Sep 18 19:39:16.926116 ip-10-0-136-185 systemd[1]: Starting Kubernetes Kubelet...
Sep 18 19:39:20.437578 ip-10-0-136-185 kubenswrapper[2585]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 18 19:39:20.437578 ip-10-0-136-185 kubenswrapper[2585]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Sep 18 19:39:20.437578 ip-10-0-136-185 kubenswrapper[2585]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 18 19:39:20.437578 ip-10-0-136-185 kubenswrapper[2585]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI.
Sep 18 19:39:20.437578 ip-10-0-136-185 kubenswrapper[2585]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 18 19:39:20.438392 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:39:20.437653    2585 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime"
Sep 18 19:39:20.440569 ip-10-0-136-185 kubenswrapper[2585]: W0918 19:39:20.440552    2585 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 18 19:39:20.440569 ip-10-0-136-185 kubenswrapper[2585]: W0918 19:39:20.440570    2585 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 18 19:39:20.440642 ip-10-0-136-185 kubenswrapper[2585]: W0918 19:39:20.440574    2585 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 18 19:39:20.440642 ip-10-0-136-185 kubenswrapper[2585]: W0918 19:39:20.440577    2585 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 18 19:39:20.440642 ip-10-0-136-185 kubenswrapper[2585]: W0918 19:39:20.440580    2585 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 18 19:39:20.440642 ip-10-0-136-185 kubenswrapper[2585]: W0918 19:39:20.440584    2585 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 18 19:39:20.440642 ip-10-0-136-185 kubenswrapper[2585]: W0918 19:39:20.440587    2585 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 18 19:39:20.440642 ip-10-0-136-185 kubenswrapper[2585]: W0918 19:39:20.440589    2585 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 18 19:39:20.440642 ip-10-0-136-185 kubenswrapper[2585]: W0918 19:39:20.440592    2585 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 18 19:39:20.440642 ip-10-0-136-185 kubenswrapper[2585]: W0918 19:39:20.440595    2585 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 18 19:39:20.440642 ip-10-0-136-185 kubenswrapper[2585]: W0918 19:39:20.440598    2585 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 18 19:39:20.440642 ip-10-0-136-185 kubenswrapper[2585]: W0918 19:39:20.440600    2585 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 18 19:39:20.440642 ip-10-0-136-185 kubenswrapper[2585]: W0918 19:39:20.440603    2585 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 18 19:39:20.440642 ip-10-0-136-185 kubenswrapper[2585]: W0918 19:39:20.440606    2585 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 18 19:39:20.440642 ip-10-0-136-185 kubenswrapper[2585]: W0918 19:39:20.440608    2585 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 18 19:39:20.440642 ip-10-0-136-185 kubenswrapper[2585]: W0918 19:39:20.440611    2585 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 18 19:39:20.440642 ip-10-0-136-185 kubenswrapper[2585]: W0918 19:39:20.440613    2585 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 18 19:39:20.440642 ip-10-0-136-185 kubenswrapper[2585]: W0918 19:39:20.440616    2585 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 18 19:39:20.440642 ip-10-0-136-185 kubenswrapper[2585]: W0918 19:39:20.440619    2585 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 18 19:39:20.440642 ip-10-0-136-185 kubenswrapper[2585]: W0918 19:39:20.440622    2585 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 18 19:39:20.440642 ip-10-0-136-185 kubenswrapper[2585]: W0918 19:39:20.440630    2585 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 18 19:39:20.440642 ip-10-0-136-185 kubenswrapper[2585]: W0918 19:39:20.440633    2585 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 18 19:39:20.441140 ip-10-0-136-185 kubenswrapper[2585]: W0918 19:39:20.440636    2585 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 18 19:39:20.441140 ip-10-0-136-185 kubenswrapper[2585]: W0918 19:39:20.440638    2585 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 18 19:39:20.441140 ip-10-0-136-185 kubenswrapper[2585]: W0918 19:39:20.440640    2585 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 18 19:39:20.441140 ip-10-0-136-185 kubenswrapper[2585]: W0918 19:39:20.440643    2585 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 18 19:39:20.441140 ip-10-0-136-185 kubenswrapper[2585]: W0918 19:39:20.440646    2585 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 18 19:39:20.441140 ip-10-0-136-185 kubenswrapper[2585]: W0918 19:39:20.440648    2585 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 18 19:39:20.441140 ip-10-0-136-185 kubenswrapper[2585]: W0918 19:39:20.440651    2585 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 18 19:39:20.441140 ip-10-0-136-185 kubenswrapper[2585]: W0918 19:39:20.440654    2585 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 18 19:39:20.441140 ip-10-0-136-185 kubenswrapper[2585]: W0918 19:39:20.440657    2585 feature_gate.go:328] unrecognized feature gate: Example2
Sep 18 19:39:20.441140 ip-10-0-136-185 kubenswrapper[2585]: W0918 19:39:20.440659    2585 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 18 19:39:20.441140 ip-10-0-136-185 kubenswrapper[2585]: W0918 19:39:20.440662    2585 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 18 19:39:20.441140 ip-10-0-136-185 kubenswrapper[2585]: W0918 19:39:20.440664    2585 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 18 19:39:20.441140 ip-10-0-136-185 kubenswrapper[2585]: W0918 19:39:20.440667    2585 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 18 19:39:20.441140 ip-10-0-136-185 kubenswrapper[2585]: W0918 19:39:20.440670    2585 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 18 19:39:20.441140 ip-10-0-136-185 kubenswrapper[2585]: W0918 19:39:20.440672    2585 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 18 19:39:20.441140 ip-10-0-136-185 kubenswrapper[2585]: W0918 19:39:20.440675    2585 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 18 19:39:20.441140 ip-10-0-136-185 kubenswrapper[2585]: W0918 19:39:20.440677    2585 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 18 19:39:20.441140 ip-10-0-136-185 kubenswrapper[2585]: W0918 19:39:20.440680    2585 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 18 19:39:20.441140 ip-10-0-136-185 kubenswrapper[2585]: W0918 19:39:20.440682    2585 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 18 19:39:20.441140 ip-10-0-136-185 kubenswrapper[2585]: W0918 19:39:20.440685    2585 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 18 19:39:20.441663 ip-10-0-136-185 kubenswrapper[2585]: W0918 19:39:20.440687    2585 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 18 19:39:20.441663 ip-10-0-136-185 kubenswrapper[2585]: W0918 19:39:20.440690    2585 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 18 19:39:20.441663 ip-10-0-136-185 kubenswrapper[2585]: W0918 19:39:20.440692    2585 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 18 19:39:20.441663 ip-10-0-136-185 kubenswrapper[2585]: W0918 19:39:20.440695    2585 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 18 19:39:20.441663 ip-10-0-136-185 kubenswrapper[2585]: W0918 19:39:20.440697    2585 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 18 19:39:20.441663 ip-10-0-136-185 kubenswrapper[2585]: W0918 19:39:20.440700    2585 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 18 19:39:20.441663 ip-10-0-136-185 kubenswrapper[2585]: W0918 19:39:20.440702    2585 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 18 19:39:20.441663 ip-10-0-136-185 kubenswrapper[2585]: W0918 19:39:20.440705    2585 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 18 19:39:20.441663 ip-10-0-136-185 kubenswrapper[2585]: W0918 19:39:20.440708    2585 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 18 19:39:20.441663 ip-10-0-136-185 kubenswrapper[2585]: W0918 19:39:20.440710    2585 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 18 19:39:20.441663 ip-10-0-136-185 kubenswrapper[2585]: W0918 19:39:20.440713    2585 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 18 19:39:20.441663 ip-10-0-136-185 kubenswrapper[2585]: W0918 19:39:20.440716    2585 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 18 19:39:20.441663 ip-10-0-136-185 kubenswrapper[2585]: W0918 19:39:20.440719    2585 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 18 19:39:20.441663 ip-10-0-136-185 kubenswrapper[2585]: W0918 19:39:20.440722    2585 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 18 19:39:20.441663 ip-10-0-136-185 kubenswrapper[2585]: W0918 19:39:20.440726    2585 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 18 19:39:20.441663 ip-10-0-136-185 kubenswrapper[2585]: W0918 19:39:20.440728    2585 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 18 19:39:20.441663 ip-10-0-136-185 kubenswrapper[2585]: W0918 19:39:20.440731    2585 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 18 19:39:20.441663 ip-10-0-136-185 kubenswrapper[2585]: W0918 19:39:20.440734    2585 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 18 19:39:20.441663 ip-10-0-136-185 kubenswrapper[2585]: W0918 19:39:20.440736    2585 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 18 19:39:20.441663 ip-10-0-136-185 kubenswrapper[2585]: W0918 19:39:20.440738    2585 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 18 19:39:20.442177 ip-10-0-136-185 kubenswrapper[2585]: W0918 19:39:20.440741    2585 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 18 19:39:20.442177 ip-10-0-136-185 kubenswrapper[2585]: W0918 19:39:20.440743    2585 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 18 19:39:20.442177 ip-10-0-136-185 kubenswrapper[2585]: W0918 19:39:20.440746    2585 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 18 19:39:20.442177 ip-10-0-136-185 kubenswrapper[2585]: W0918 19:39:20.440749    2585 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 18 19:39:20.442177 ip-10-0-136-185 kubenswrapper[2585]: W0918 19:39:20.440751    2585 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 18 19:39:20.442177 ip-10-0-136-185 kubenswrapper[2585]: W0918 19:39:20.440754    2585 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 18 19:39:20.442177 ip-10-0-136-185 kubenswrapper[2585]: W0918 19:39:20.440756    2585 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 18 19:39:20.442177 ip-10-0-136-185 kubenswrapper[2585]: W0918 19:39:20.440759    2585 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 18 19:39:20.442177 ip-10-0-136-185 kubenswrapper[2585]: W0918 19:39:20.440763    2585 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 18 19:39:20.442177 ip-10-0-136-185 kubenswrapper[2585]: W0918 19:39:20.440768    2585 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 18 19:39:20.442177 ip-10-0-136-185 kubenswrapper[2585]: W0918 19:39:20.440773    2585 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 18 19:39:20.442177 ip-10-0-136-185 kubenswrapper[2585]: W0918 19:39:20.440776    2585 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 18 19:39:20.442177 ip-10-0-136-185 kubenswrapper[2585]: W0918 19:39:20.440779    2585 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 18 19:39:20.442177 ip-10-0-136-185 kubenswrapper[2585]: W0918 19:39:20.440782    2585 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 18 19:39:20.442177 ip-10-0-136-185 kubenswrapper[2585]: W0918 19:39:20.440797    2585 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 18 19:39:20.442177 ip-10-0-136-185 kubenswrapper[2585]: W0918 19:39:20.440800    2585 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 18 19:39:20.442177 ip-10-0-136-185 kubenswrapper[2585]: W0918 19:39:20.440803    2585 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 18 19:39:20.442177 ip-10-0-136-185 kubenswrapper[2585]: W0918 19:39:20.440806    2585 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 18 19:39:20.442177 ip-10-0-136-185 kubenswrapper[2585]: W0918 19:39:20.440808    2585 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 18 19:39:20.442673 ip-10-0-136-185 kubenswrapper[2585]: W0918 19:39:20.440811    2585 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 18 19:39:20.442673 ip-10-0-136-185 kubenswrapper[2585]: W0918 19:39:20.440815    2585 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 18 19:39:20.442673 ip-10-0-136-185 kubenswrapper[2585]: W0918 19:39:20.440818    2585 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 18 19:39:20.442673 ip-10-0-136-185 kubenswrapper[2585]: W0918 19:39:20.440820    2585 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 18 19:39:20.442673 ip-10-0-136-185 kubenswrapper[2585]: W0918 19:39:20.440825    2585 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 18 19:39:20.442673 ip-10-0-136-185 kubenswrapper[2585]: W0918 19:39:20.440829    2585 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 18 19:39:20.442673 ip-10-0-136-185 kubenswrapper[2585]: W0918 19:39:20.440832    2585 feature_gate.go:328] unrecognized feature gate: Example
Sep 18 19:39:20.442673 ip-10-0-136-185 kubenswrapper[2585]: W0918 19:39:20.440836    2585 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 18 19:39:20.442673 ip-10-0-136-185 kubenswrapper[2585]: W0918 19:39:20.440838    2585 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 18 19:39:20.442673 ip-10-0-136-185 kubenswrapper[2585]: W0918 19:39:20.440841    2585 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 18 19:39:20.442673 ip-10-0-136-185 kubenswrapper[2585]: W0918 19:39:20.440844    2585 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 18 19:39:20.442673 ip-10-0-136-185 kubenswrapper[2585]: W0918 19:39:20.440847    2585 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 18 19:39:20.442673 ip-10-0-136-185 kubenswrapper[2585]: W0918 19:39:20.440849    2585 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 18 19:39:20.442673 ip-10-0-136-185 kubenswrapper[2585]: W0918 19:39:20.440852    2585 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 18 19:39:20.442673 ip-10-0-136-185 kubenswrapper[2585]: W0918 19:39:20.440855    2585 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 18 19:39:20.442673 ip-10-0-136-185 kubenswrapper[2585]: W0918 19:39:20.440857    2585 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 18 19:39:20.442673 ip-10-0-136-185 kubenswrapper[2585]: W0918 19:39:20.440860    2585 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 18 19:39:20.442673 ip-10-0-136-185 kubenswrapper[2585]: W0918 19:39:20.440862    2585 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 18 19:39:20.442673 ip-10-0-136-185 kubenswrapper[2585]: W0918 19:39:20.441625    2585 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 18 19:39:20.443149 ip-10-0-136-185 kubenswrapper[2585]: W0918 19:39:20.441632    2585 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 18 19:39:20.443149 ip-10-0-136-185 kubenswrapper[2585]: W0918 19:39:20.441635    2585 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 18 19:39:20.443149 ip-10-0-136-185 kubenswrapper[2585]: W0918 19:39:20.441639    2585 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 18 19:39:20.443149 ip-10-0-136-185 kubenswrapper[2585]: W0918 19:39:20.441642    2585 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 18 19:39:20.443149 ip-10-0-136-185 kubenswrapper[2585]: W0918 19:39:20.441645    2585 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 18 19:39:20.443149 ip-10-0-136-185 kubenswrapper[2585]: W0918 19:39:20.441648    2585 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 18 19:39:20.443149 ip-10-0-136-185 kubenswrapper[2585]: W0918 19:39:20.441650    2585 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 18 19:39:20.443149 ip-10-0-136-185 kubenswrapper[2585]: W0918 19:39:20.441653    2585 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 18 19:39:20.443149 ip-10-0-136-185 kubenswrapper[2585]: W0918 19:39:20.441656    2585 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 18 19:39:20.443149 ip-10-0-136-185 kubenswrapper[2585]: W0918 19:39:20.441658    2585 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 18 19:39:20.443149 ip-10-0-136-185 kubenswrapper[2585]: W0918 19:39:20.441661    2585 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 18 19:39:20.443149 ip-10-0-136-185 kubenswrapper[2585]: W0918 19:39:20.441664    2585 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 18 19:39:20.443149 ip-10-0-136-185 kubenswrapper[2585]: W0918 19:39:20.441667    2585 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 18 19:39:20.443149 ip-10-0-136-185 kubenswrapper[2585]: W0918 19:39:20.441670    2585 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 18 19:39:20.443149 ip-10-0-136-185 kubenswrapper[2585]: W0918 19:39:20.441672    2585 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 18 19:39:20.443149 ip-10-0-136-185 kubenswrapper[2585]: W0918 19:39:20.441675    2585 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 18 19:39:20.443149 ip-10-0-136-185 kubenswrapper[2585]: W0918 19:39:20.441678    2585 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 18 19:39:20.443149 ip-10-0-136-185 kubenswrapper[2585]: W0918 19:39:20.441681    2585 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 18 19:39:20.443149 ip-10-0-136-185 kubenswrapper[2585]: W0918 19:39:20.441684    2585 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 18 19:39:20.443149 ip-10-0-136-185 kubenswrapper[2585]: W0918 19:39:20.441688    2585 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 18 19:39:20.443673 ip-10-0-136-185 kubenswrapper[2585]: W0918 19:39:20.441691    2585 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 18 19:39:20.443673 ip-10-0-136-185 kubenswrapper[2585]: W0918 19:39:20.441694    2585 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 18 19:39:20.443673 ip-10-0-136-185 kubenswrapper[2585]: W0918 19:39:20.441697    2585 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 18 19:39:20.443673 ip-10-0-136-185 kubenswrapper[2585]: W0918 19:39:20.441699    2585 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 18 19:39:20.443673 ip-10-0-136-185 kubenswrapper[2585]: W0918 19:39:20.441702    2585 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 18 19:39:20.443673 ip-10-0-136-185 kubenswrapper[2585]: W0918 19:39:20.441704    2585 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 18 19:39:20.443673 ip-10-0-136-185 kubenswrapper[2585]: W0918 19:39:20.441707    2585 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 18 19:39:20.443673 ip-10-0-136-185 kubenswrapper[2585]: W0918 19:39:20.441709    2585 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 18 19:39:20.443673 ip-10-0-136-185 kubenswrapper[2585]: W0918 19:39:20.441712    2585 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 18 19:39:20.443673 ip-10-0-136-185 kubenswrapper[2585]: W0918 19:39:20.441715    2585 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 18 19:39:20.443673 ip-10-0-136-185 kubenswrapper[2585]: W0918 19:39:20.441717    2585 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 18 19:39:20.443673 ip-10-0-136-185 kubenswrapper[2585]: W0918 19:39:20.441720    2585 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 18 19:39:20.443673 ip-10-0-136-185 kubenswrapper[2585]: W0918 19:39:20.441722    2585 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 18 19:39:20.443673 ip-10-0-136-185 kubenswrapper[2585]: W0918 19:39:20.441725    2585 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 18 19:39:20.443673 ip-10-0-136-185 kubenswrapper[2585]: W0918 19:39:20.441727    2585 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 18 19:39:20.443673 ip-10-0-136-185 kubenswrapper[2585]: W0918 19:39:20.441730    2585 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 18 19:39:20.443673 ip-10-0-136-185 kubenswrapper[2585]: W0918 19:39:20.441733    2585 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 18 19:39:20.443673 ip-10-0-136-185 kubenswrapper[2585]: W0918 19:39:20.441735    2585 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 18 19:39:20.443673 ip-10-0-136-185 kubenswrapper[2585]: W0918 19:39:20.441739    2585 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 18 19:39:20.444151 ip-10-0-136-185 kubenswrapper[2585]: W0918 19:39:20.441743    2585 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 18 19:39:20.444151 ip-10-0-136-185 kubenswrapper[2585]: W0918 19:39:20.441747    2585 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 18 19:39:20.444151 ip-10-0-136-185 kubenswrapper[2585]: W0918 19:39:20.441750    2585 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 18 19:39:20.444151 ip-10-0-136-185 kubenswrapper[2585]: W0918 19:39:20.441753    2585 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 18 19:39:20.444151 ip-10-0-136-185 kubenswrapper[2585]: W0918 19:39:20.441756    2585 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 18 19:39:20.444151 ip-10-0-136-185 kubenswrapper[2585]: W0918 19:39:20.441758    2585 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 18 19:39:20.444151 ip-10-0-136-185 kubenswrapper[2585]: W0918 19:39:20.441761    2585 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 18 19:39:20.444151 ip-10-0-136-185 kubenswrapper[2585]: W0918 19:39:20.441764    2585 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 18 19:39:20.444151 ip-10-0-136-185 kubenswrapper[2585]: W0918 19:39:20.441766    2585 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 18 19:39:20.444151 ip-10-0-136-185 kubenswrapper[2585]: W0918 19:39:20.441769    2585 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 18 19:39:20.444151 ip-10-0-136-185 kubenswrapper[2585]: W0918 19:39:20.441771    2585 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 18 19:39:20.444151 ip-10-0-136-185 kubenswrapper[2585]: W0918 19:39:20.441774    2585 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 18 19:39:20.444151 ip-10-0-136-185 kubenswrapper[2585]: W0918 19:39:20.441779    2585 feature_gate.go:328] unrecognized feature gate: Example2
Sep 18 19:39:20.444151 ip-10-0-136-185 kubenswrapper[2585]: W0918 19:39:20.441782    2585 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 18 19:39:20.444151 ip-10-0-136-185 kubenswrapper[2585]: W0918 19:39:20.441798    2585 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 18 19:39:20.444151 ip-10-0-136-185 kubenswrapper[2585]: W0918 19:39:20.441801    2585 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 18 19:39:20.444151 ip-10-0-136-185 kubenswrapper[2585]: W0918 19:39:20.441804    2585 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 18 19:39:20.444151 ip-10-0-136-185 kubenswrapper[2585]: W0918 19:39:20.441807    2585 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 18 19:39:20.444151 ip-10-0-136-185 kubenswrapper[2585]: W0918 19:39:20.441809    2585 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 18 19:39:20.444151 ip-10-0-136-185 kubenswrapper[2585]: W0918 19:39:20.441812    2585 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 18 19:39:20.444664 ip-10-0-136-185 kubenswrapper[2585]: W0918 19:39:20.441816    2585 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 18 19:39:20.444664 ip-10-0-136-185 kubenswrapper[2585]: W0918 19:39:20.441820    2585 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 18 19:39:20.444664 ip-10-0-136-185 kubenswrapper[2585]: W0918 19:39:20.441823    2585 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 18 19:39:20.444664 ip-10-0-136-185 kubenswrapper[2585]: W0918 19:39:20.441826    2585 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 18 19:39:20.444664 ip-10-0-136-185 kubenswrapper[2585]: W0918 19:39:20.441828    2585 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 18 19:39:20.444664 ip-10-0-136-185 kubenswrapper[2585]: W0918 19:39:20.441832    2585 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 18 19:39:20.444664 ip-10-0-136-185 kubenswrapper[2585]: W0918 19:39:20.441835    2585 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 18 19:39:20.444664 ip-10-0-136-185 kubenswrapper[2585]: W0918 19:39:20.441837    2585 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 18 19:39:20.444664 ip-10-0-136-185 kubenswrapper[2585]: W0918 19:39:20.441840    2585 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 18 19:39:20.444664 ip-10-0-136-185 kubenswrapper[2585]: W0918 19:39:20.441843    2585 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 18 19:39:20.444664 ip-10-0-136-185 kubenswrapper[2585]: W0918 19:39:20.441845    2585 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 18 19:39:20.444664 ip-10-0-136-185 kubenswrapper[2585]: W0918 19:39:20.441848    2585 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 18 19:39:20.444664 ip-10-0-136-185 kubenswrapper[2585]: W0918 19:39:20.441851    2585 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 18 19:39:20.444664 ip-10-0-136-185 kubenswrapper[2585]: W0918 19:39:20.441853    2585 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 18 19:39:20.444664 ip-10-0-136-185 kubenswrapper[2585]: W0918 19:39:20.441856    2585 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 18 19:39:20.444664 ip-10-0-136-185 kubenswrapper[2585]: W0918 19:39:20.441858    2585 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 18 19:39:20.444664 ip-10-0-136-185 kubenswrapper[2585]: W0918 19:39:20.441861    2585 feature_gate.go:328] unrecognized feature gate: Example
Sep 18 19:39:20.444664 ip-10-0-136-185 kubenswrapper[2585]: W0918 19:39:20.441863    2585 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 18 19:39:20.444664 ip-10-0-136-185 kubenswrapper[2585]: W0918 19:39:20.441866    2585 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 18 19:39:20.445158 ip-10-0-136-185 kubenswrapper[2585]: W0918 19:39:20.441869    2585 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 18 19:39:20.445158 ip-10-0-136-185 kubenswrapper[2585]: W0918 19:39:20.441872    2585 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 18 19:39:20.445158 ip-10-0-136-185 kubenswrapper[2585]: W0918 19:39:20.441874    2585 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 18 19:39:20.445158 ip-10-0-136-185 kubenswrapper[2585]: W0918 19:39:20.441877    2585 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 18 19:39:20.445158 ip-10-0-136-185 kubenswrapper[2585]: W0918 19:39:20.441880    2585 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 18 19:39:20.445158 ip-10-0-136-185 kubenswrapper[2585]: W0918 19:39:20.441883    2585 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 18 19:39:20.445158 ip-10-0-136-185 kubenswrapper[2585]: W0918 19:39:20.441886    2585 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 18 19:39:20.445158 ip-10-0-136-185 kubenswrapper[2585]: W0918 19:39:20.441889    2585 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 18 19:39:20.445158 ip-10-0-136-185 kubenswrapper[2585]: W0918 19:39:20.441891    2585 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 18 19:39:20.445158 ip-10-0-136-185 kubenswrapper[2585]: W0918 19:39:20.441894    2585 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 18 19:39:20.445158 ip-10-0-136-185 kubenswrapper[2585]: W0918 19:39:20.441896    2585 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 18 19:39:20.445158 ip-10-0-136-185 kubenswrapper[2585]: W0918 19:39:20.441899    2585 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 18 19:39:20.445158 ip-10-0-136-185 kubenswrapper[2585]: W0918 19:39:20.441902    2585 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 18 19:39:20.445158 ip-10-0-136-185 kubenswrapper[2585]: W0918 19:39:20.441905    2585 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 18 19:39:20.445158 ip-10-0-136-185 kubenswrapper[2585]: W0918 19:39:20.441907    2585 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 18 19:39:20.445158 ip-10-0-136-185 kubenswrapper[2585]: W0918 19:39:20.441910    2585 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 18 19:39:20.445158 ip-10-0-136-185 kubenswrapper[2585]: W0918 19:39:20.441912    2585 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 18 19:39:20.445158 ip-10-0-136-185 kubenswrapper[2585]: W0918 19:39:20.441915    2585 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 18 19:39:20.445158 ip-10-0-136-185 kubenswrapper[2585]: W0918 19:39:20.441917    2585 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 18 19:39:20.445158 ip-10-0-136-185 kubenswrapper[2585]: W0918 19:39:20.441920    2585 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 18 19:39:20.445665 ip-10-0-136-185 kubenswrapper[2585]: W0918 19:39:20.442314    2585 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 18 19:39:20.445665 ip-10-0-136-185 kubenswrapper[2585]: W0918 19:39:20.442319    2585 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 18 19:39:20.445665 ip-10-0-136-185 kubenswrapper[2585]: W0918 19:39:20.442323    2585 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 18 19:39:20.445665 ip-10-0-136-185 kubenswrapper[2585]: W0918 19:39:20.442325    2585 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 18 19:39:20.445665 ip-10-0-136-185 kubenswrapper[2585]: W0918 19:39:20.442329    2585 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 18 19:39:20.445665 ip-10-0-136-185 kubenswrapper[2585]: W0918 19:39:20.442331    2585 feature_gate.go:328] unrecognized feature gate: Example2
Sep 18 19:39:20.445665 ip-10-0-136-185 kubenswrapper[2585]: W0918 19:39:20.442334    2585 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 18 19:39:20.445665 ip-10-0-136-185 kubenswrapper[2585]: W0918 19:39:20.442337    2585 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 18 19:39:20.445665 ip-10-0-136-185 kubenswrapper[2585]: W0918 19:39:20.442339    2585 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 18 19:39:20.445665 ip-10-0-136-185 kubenswrapper[2585]: W0918 19:39:20.442342    2585 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 18 19:39:20.445665 ip-10-0-136-185 kubenswrapper[2585]: W0918 19:39:20.442344    2585 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 18 19:39:20.445665 ip-10-0-136-185 kubenswrapper[2585]: W0918 19:39:20.442347    2585 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 18 19:39:20.445665 ip-10-0-136-185 kubenswrapper[2585]: W0918 19:39:20.442349    2585 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 18 19:39:20.445665 ip-10-0-136-185 kubenswrapper[2585]: W0918 19:39:20.442352    2585 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 18 19:39:20.445665 ip-10-0-136-185 kubenswrapper[2585]: W0918 19:39:20.442354    2585 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 18 19:39:20.445665 ip-10-0-136-185 kubenswrapper[2585]: W0918 19:39:20.442358    2585 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 18 19:39:20.445665 ip-10-0-136-185 kubenswrapper[2585]: W0918 19:39:20.442362    2585 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 18 19:39:20.445665 ip-10-0-136-185 kubenswrapper[2585]: W0918 19:39:20.442365    2585 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 18 19:39:20.445665 ip-10-0-136-185 kubenswrapper[2585]: W0918 19:39:20.442369    2585 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 18 19:39:20.446136 ip-10-0-136-185 kubenswrapper[2585]: W0918 19:39:20.442374    2585 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 18 19:39:20.446136 ip-10-0-136-185 kubenswrapper[2585]: W0918 19:39:20.442377    2585 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 18 19:39:20.446136 ip-10-0-136-185 kubenswrapper[2585]: W0918 19:39:20.442380    2585 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 18 19:39:20.446136 ip-10-0-136-185 kubenswrapper[2585]: W0918 19:39:20.442383    2585 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 18 19:39:20.446136 ip-10-0-136-185 kubenswrapper[2585]: W0918 19:39:20.442386    2585 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 18 19:39:20.446136 ip-10-0-136-185 kubenswrapper[2585]: W0918 19:39:20.442388    2585 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 18 19:39:20.446136 ip-10-0-136-185 kubenswrapper[2585]: W0918 19:39:20.442391    2585 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 18 19:39:20.446136 ip-10-0-136-185 kubenswrapper[2585]: W0918 19:39:20.442393    2585 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 18 19:39:20.446136 ip-10-0-136-185 kubenswrapper[2585]: W0918 19:39:20.442396    2585 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 18 19:39:20.446136 ip-10-0-136-185 kubenswrapper[2585]: W0918 19:39:20.442398    2585 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 18 19:39:20.446136 ip-10-0-136-185 kubenswrapper[2585]: W0918 19:39:20.442401    2585 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 18 19:39:20.446136 ip-10-0-136-185 kubenswrapper[2585]: W0918 19:39:20.442403    2585 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 18 19:39:20.446136 ip-10-0-136-185 kubenswrapper[2585]: W0918 19:39:20.442406    2585 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 18 19:39:20.446136 ip-10-0-136-185 kubenswrapper[2585]: W0918 19:39:20.442408    2585 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 18 19:39:20.446136 ip-10-0-136-185 kubenswrapper[2585]: W0918 19:39:20.442411    2585 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 18 19:39:20.446136 ip-10-0-136-185 kubenswrapper[2585]: W0918 19:39:20.442413    2585 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 18 19:39:20.446136 ip-10-0-136-185 kubenswrapper[2585]: W0918 19:39:20.442415    2585 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 18 19:39:20.446136 ip-10-0-136-185 kubenswrapper[2585]: W0918 19:39:20.442419    2585 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 18 19:39:20.446136 ip-10-0-136-185 kubenswrapper[2585]: W0918 19:39:20.442422    2585 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 18 19:39:20.446136 ip-10-0-136-185 kubenswrapper[2585]: W0918 19:39:20.442425    2585 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 18 19:39:20.446627 ip-10-0-136-185 kubenswrapper[2585]: W0918 19:39:20.442428    2585 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 18 19:39:20.446627 ip-10-0-136-185 kubenswrapper[2585]: W0918 19:39:20.442431    2585 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 18 19:39:20.446627 ip-10-0-136-185 kubenswrapper[2585]: W0918 19:39:20.442433    2585 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 18 19:39:20.446627 ip-10-0-136-185 kubenswrapper[2585]: W0918 19:39:20.442436    2585 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 18 19:39:20.446627 ip-10-0-136-185 kubenswrapper[2585]: W0918 19:39:20.442438    2585 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 18 19:39:20.446627 ip-10-0-136-185 kubenswrapper[2585]: W0918 19:39:20.442441    2585 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 18 19:39:20.446627 ip-10-0-136-185 kubenswrapper[2585]: W0918 19:39:20.442444    2585 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 18 19:39:20.446627 ip-10-0-136-185 kubenswrapper[2585]: W0918 19:39:20.442446    2585 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 18 19:39:20.446627 ip-10-0-136-185 kubenswrapper[2585]: W0918 19:39:20.442449    2585 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 18 19:39:20.446627 ip-10-0-136-185 kubenswrapper[2585]: W0918 19:39:20.442452    2585 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 18 19:39:20.446627 ip-10-0-136-185 kubenswrapper[2585]: W0918 19:39:20.442455    2585 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 18 19:39:20.446627 ip-10-0-136-185 kubenswrapper[2585]: W0918 19:39:20.442459    2585 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 18 19:39:20.446627 ip-10-0-136-185 kubenswrapper[2585]: W0918 19:39:20.442462    2585 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 18 19:39:20.446627 ip-10-0-136-185 kubenswrapper[2585]: W0918 19:39:20.442465    2585 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 18 19:39:20.446627 ip-10-0-136-185 kubenswrapper[2585]: W0918 19:39:20.442468    2585 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 18 19:39:20.446627 ip-10-0-136-185 kubenswrapper[2585]: W0918 19:39:20.442470    2585 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 18 19:39:20.446627 ip-10-0-136-185 kubenswrapper[2585]: W0918 19:39:20.442473    2585 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 18 19:39:20.446627 ip-10-0-136-185 kubenswrapper[2585]: W0918 19:39:20.442475    2585 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 18 19:39:20.446627 ip-10-0-136-185 kubenswrapper[2585]: W0918 19:39:20.442478    2585 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 18 19:39:20.446627 ip-10-0-136-185 kubenswrapper[2585]: W0918 19:39:20.442480    2585 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 18 19:39:20.447133 ip-10-0-136-185 kubenswrapper[2585]: W0918 19:39:20.442483    2585 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 18 19:39:20.447133 ip-10-0-136-185 kubenswrapper[2585]: W0918 19:39:20.442485    2585 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 18 19:39:20.447133 ip-10-0-136-185 kubenswrapper[2585]: W0918 19:39:20.442487    2585 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 18 19:39:20.447133 ip-10-0-136-185 kubenswrapper[2585]: W0918 19:39:20.442490    2585 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 18 19:39:20.447133 ip-10-0-136-185 kubenswrapper[2585]: W0918 19:39:20.442492    2585 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 18 19:39:20.447133 ip-10-0-136-185 kubenswrapper[2585]: W0918 19:39:20.442495    2585 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 18 19:39:20.447133 ip-10-0-136-185 kubenswrapper[2585]: W0918 19:39:20.442497    2585 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 18 19:39:20.447133 ip-10-0-136-185 kubenswrapper[2585]: W0918 19:39:20.442500    2585 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 18 19:39:20.447133 ip-10-0-136-185 kubenswrapper[2585]: W0918 19:39:20.442502    2585 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 18 19:39:20.447133 ip-10-0-136-185 kubenswrapper[2585]: W0918 19:39:20.442505    2585 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 18 19:39:20.447133 ip-10-0-136-185 kubenswrapper[2585]: W0918 19:39:20.442507    2585 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 18 19:39:20.447133 ip-10-0-136-185 kubenswrapper[2585]: W0918 19:39:20.442510    2585 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 18 19:39:20.447133 ip-10-0-136-185 kubenswrapper[2585]: W0918 19:39:20.442513    2585 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 18 19:39:20.447133 ip-10-0-136-185 kubenswrapper[2585]: W0918 19:39:20.442515    2585 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 18 19:39:20.447133 ip-10-0-136-185 kubenswrapper[2585]: W0918 19:39:20.442517    2585 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 18 19:39:20.447133 ip-10-0-136-185 kubenswrapper[2585]: W0918 19:39:20.442520    2585 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 18 19:39:20.447133 ip-10-0-136-185 kubenswrapper[2585]: W0918 19:39:20.442522    2585 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 18 19:39:20.447133 ip-10-0-136-185 kubenswrapper[2585]: W0918 19:39:20.442525    2585 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 18 19:39:20.447133 ip-10-0-136-185 kubenswrapper[2585]: W0918 19:39:20.442527    2585 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 18 19:39:20.447133 ip-10-0-136-185 kubenswrapper[2585]: W0918 19:39:20.442530    2585 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 18 19:39:20.447617 ip-10-0-136-185 kubenswrapper[2585]: W0918 19:39:20.442533    2585 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 18 19:39:20.447617 ip-10-0-136-185 kubenswrapper[2585]: W0918 19:39:20.442538    2585 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 18 19:39:20.447617 ip-10-0-136-185 kubenswrapper[2585]: W0918 19:39:20.442541    2585 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 18 19:39:20.447617 ip-10-0-136-185 kubenswrapper[2585]: W0918 19:39:20.442543    2585 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 18 19:39:20.447617 ip-10-0-136-185 kubenswrapper[2585]: W0918 19:39:20.442546    2585 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 18 19:39:20.447617 ip-10-0-136-185 kubenswrapper[2585]: W0918 19:39:20.442548    2585 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 18 19:39:20.447617 ip-10-0-136-185 kubenswrapper[2585]: W0918 19:39:20.442552    2585 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 18 19:39:20.447617 ip-10-0-136-185 kubenswrapper[2585]: W0918 19:39:20.442554    2585 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 18 19:39:20.447617 ip-10-0-136-185 kubenswrapper[2585]: W0918 19:39:20.442557    2585 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 18 19:39:20.447617 ip-10-0-136-185 kubenswrapper[2585]: W0918 19:39:20.442560    2585 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 18 19:39:20.447617 ip-10-0-136-185 kubenswrapper[2585]: W0918 19:39:20.442562    2585 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 18 19:39:20.447617 ip-10-0-136-185 kubenswrapper[2585]: W0918 19:39:20.442565    2585 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 18 19:39:20.447617 ip-10-0-136-185 kubenswrapper[2585]: W0918 19:39:20.442567    2585 feature_gate.go:328] unrecognized feature gate: Example
Sep 18 19:39:20.447617 ip-10-0-136-185 kubenswrapper[2585]: W0918 19:39:20.442570    2585 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 18 19:39:20.447617 ip-10-0-136-185 kubenswrapper[2585]: W0918 19:39:20.442572    2585 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 18 19:39:20.447617 ip-10-0-136-185 kubenswrapper[2585]: W0918 19:39:20.442574    2585 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 18 19:39:20.447617 ip-10-0-136-185 kubenswrapper[2585]: W0918 19:39:20.442577    2585 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 18 19:39:20.447617 ip-10-0-136-185 kubenswrapper[2585]: W0918 19:39:20.442579    2585 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 18 19:39:20.447617 ip-10-0-136-185 kubenswrapper[2585]: W0918 19:39:20.442582    2585 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 18 19:39:20.447617 ip-10-0-136-185 kubenswrapper[2585]: W0918 19:39:20.442584    2585 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 18 19:39:20.448127 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:39:20.442651    2585 flags.go:64] FLAG: --address="0.0.0.0"
Sep 18 19:39:20.448127 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:39:20.442687    2585 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Sep 18 19:39:20.448127 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:39:20.442693    2585 flags.go:64] FLAG: --anonymous-auth="true"
Sep 18 19:39:20.448127 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:39:20.442697    2585 flags.go:64] FLAG: --application-metrics-count-limit="100"
Sep 18 19:39:20.448127 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:39:20.442702    2585 flags.go:64] FLAG: --authentication-token-webhook="false"
Sep 18 19:39:20.448127 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:39:20.442705    2585 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Sep 18 19:39:20.448127 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:39:20.442709    2585 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Sep 18 19:39:20.448127 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:39:20.442713    2585 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Sep 18 19:39:20.448127 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:39:20.442717    2585 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Sep 18 19:39:20.448127 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:39:20.442720    2585 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Sep 18 19:39:20.448127 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:39:20.442723    2585 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Sep 18 19:39:20.448127 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:39:20.442727    2585 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Sep 18 19:39:20.448127 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:39:20.442730    2585 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Sep 18 19:39:20.448127 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:39:20.442734    2585 flags.go:64] FLAG: --cgroup-root=""
Sep 18 19:39:20.448127 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:39:20.442737    2585 flags.go:64] FLAG: --cgroups-per-qos="true"
Sep 18 19:39:20.448127 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:39:20.442741    2585 flags.go:64] FLAG: --client-ca-file=""
Sep 18 19:39:20.448127 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:39:20.442744    2585 flags.go:64] FLAG: --cloud-provider="external"
Sep 18 19:39:20.448127 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:39:20.442747    2585 flags.go:64] FLAG: --cluster-dns="[]"
Sep 18 19:39:20.448127 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:39:20.442751    2585 flags.go:64] FLAG: --cluster-domain=""
Sep 18 19:39:20.448127 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:39:20.442754    2585 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Sep 18 19:39:20.448127 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:39:20.442757    2585 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Sep 18 19:39:20.448127 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:39:20.442760    2585 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Sep 18 19:39:20.448127 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:39:20.442763    2585 flags.go:64] FLAG: --container-log-max-files="5"
Sep 18 19:39:20.448127 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:39:20.442767    2585 flags.go:64] FLAG: --container-log-max-size="10Mi"
Sep 18 19:39:20.448703 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:39:20.442770    2585 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Sep 18 19:39:20.448703 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:39:20.442773    2585 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Sep 18 19:39:20.448703 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:39:20.442776    2585 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Sep 18 19:39:20.448703 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:39:20.442779    2585 flags.go:64] FLAG: --contention-profiling="false"
Sep 18 19:39:20.448703 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:39:20.442782    2585 flags.go:64] FLAG: --cpu-cfs-quota="true"
Sep 18 19:39:20.448703 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:39:20.442799    2585 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Sep 18 19:39:20.448703 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:39:20.442803    2585 flags.go:64] FLAG: --cpu-manager-policy="none"
Sep 18 19:39:20.448703 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:39:20.442805    2585 flags.go:64] FLAG: --cpu-manager-policy-options=""
Sep 18 19:39:20.448703 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:39:20.442809    2585 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Sep 18 19:39:20.448703 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:39:20.442812    2585 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Sep 18 19:39:20.448703 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:39:20.442815    2585 flags.go:64] FLAG: --enable-debugging-handlers="true"
Sep 18 19:39:20.448703 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:39:20.442818    2585 flags.go:64] FLAG: --enable-load-reader="false"
Sep 18 19:39:20.448703 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:39:20.442821    2585 flags.go:64] FLAG: --enable-server="true"
Sep 18 19:39:20.448703 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:39:20.442824    2585 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Sep 18 19:39:20.448703 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:39:20.442828    2585 flags.go:64] FLAG: --event-burst="100"
Sep 18 19:39:20.448703 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:39:20.442831    2585 flags.go:64] FLAG: --event-qps="50"
Sep 18 19:39:20.448703 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:39:20.442834    2585 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Sep 18 19:39:20.448703 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:39:20.442837    2585 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Sep 18 19:39:20.448703 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:39:20.442839    2585 flags.go:64] FLAG: --eviction-hard=""
Sep 18 19:39:20.448703 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:39:20.442843    2585 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Sep 18 19:39:20.448703 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:39:20.442846    2585 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Sep 18 19:39:20.448703 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:39:20.442849    2585 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Sep 18 19:39:20.448703 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:39:20.442852    2585 flags.go:64] FLAG: --eviction-soft=""
Sep 18 19:39:20.448703 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:39:20.442855    2585 flags.go:64] FLAG: --eviction-soft-grace-period=""
Sep 18 19:39:20.448703 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:39:20.442858    2585 flags.go:64] FLAG: --exit-on-lock-contention="false"
Sep 18 19:39:20.449379 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:39:20.442862    2585 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Sep 18 19:39:20.449379 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:39:20.442865    2585 flags.go:64] FLAG: --experimental-mounter-path=""
Sep 18 19:39:20.449379 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:39:20.442868    2585 flags.go:64] FLAG: --fail-cgroupv1="false"
Sep 18 19:39:20.449379 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:39:20.442871    2585 flags.go:64] FLAG: --fail-swap-on="true"
Sep 18 19:39:20.449379 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:39:20.442874    2585 flags.go:64] FLAG: --feature-gates=""
Sep 18 19:39:20.449379 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:39:20.442878    2585 flags.go:64] FLAG: --file-check-frequency="20s"
Sep 18 19:39:20.449379 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:39:20.442881    2585 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Sep 18 19:39:20.449379 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:39:20.442884    2585 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Sep 18 19:39:20.449379 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:39:20.442887    2585 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Sep 18 19:39:20.449379 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:39:20.442890    2585 flags.go:64] FLAG: --healthz-port="10248"
Sep 18 19:39:20.449379 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:39:20.442893    2585 flags.go:64] FLAG: --help="false"
Sep 18 19:39:20.449379 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:39:20.442896    2585 flags.go:64] FLAG: --hostname-override="ip-10-0-136-185.ec2.internal"
Sep 18 19:39:20.449379 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:39:20.442899    2585 flags.go:64] FLAG: --housekeeping-interval="10s"
Sep 18 19:39:20.449379 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:39:20.442902    2585 flags.go:64] FLAG: --http-check-frequency="20s"
Sep 18 19:39:20.449379 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:39:20.442905    2585 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Sep 18 19:39:20.449379 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:39:20.442908    2585 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Sep 18 19:39:20.449379 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:39:20.442912    2585 flags.go:64] FLAG: --image-gc-high-threshold="85"
Sep 18 19:39:20.449379 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:39:20.442915    2585 flags.go:64] FLAG: --image-gc-low-threshold="80"
Sep 18 19:39:20.449379 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:39:20.442918    2585 flags.go:64] FLAG: --image-service-endpoint=""
Sep 18 19:39:20.449379 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:39:20.442921    2585 flags.go:64] FLAG: --kernel-memcg-notification="false"
Sep 18 19:39:20.449379 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:39:20.442923    2585 flags.go:64] FLAG: --kube-api-burst="100"
Sep 18 19:39:20.449379 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:39:20.442927    2585 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Sep 18 19:39:20.449379 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:39:20.442930    2585 flags.go:64] FLAG: --kube-api-qps="50"
Sep 18 19:39:20.449379 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:39:20.442933    2585 flags.go:64] FLAG: --kube-reserved=""
Sep 18 19:39:20.450030 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:39:20.442935    2585 flags.go:64] FLAG: --kube-reserved-cgroup=""
Sep 18 19:39:20.450030 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:39:20.442938    2585 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Sep 18 19:39:20.450030 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:39:20.442941    2585 flags.go:64] FLAG: --kubelet-cgroups=""
Sep 18 19:39:20.450030 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:39:20.442944    2585 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Sep 18 19:39:20.450030 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:39:20.442947    2585 flags.go:64] FLAG: --lock-file=""
Sep 18 19:39:20.450030 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:39:20.442950    2585 flags.go:64] FLAG: --log-cadvisor-usage="false"
Sep 18 19:39:20.450030 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:39:20.442953    2585 flags.go:64] FLAG: --log-flush-frequency="5s"
Sep 18 19:39:20.450030 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:39:20.442956    2585 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Sep 18 19:39:20.450030 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:39:20.442962    2585 flags.go:64] FLAG: --log-json-split-stream="false"
Sep 18 19:39:20.450030 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:39:20.442965    2585 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Sep 18 19:39:20.450030 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:39:20.442967    2585 flags.go:64] FLAG: --log-text-split-stream="false"
Sep 18 19:39:20.450030 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:39:20.442970    2585 flags.go:64] FLAG: --logging-format="text"
Sep 18 19:39:20.450030 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:39:20.442974    2585 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Sep 18 19:39:20.450030 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:39:20.442977    2585 flags.go:64] FLAG: --make-iptables-util-chains="true"
Sep 18 19:39:20.450030 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:39:20.442980    2585 flags.go:64] FLAG: --manifest-url=""
Sep 18 19:39:20.450030 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:39:20.442987    2585 flags.go:64] FLAG: --manifest-url-header=""
Sep 18 19:39:20.450030 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:39:20.442992    2585 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Sep 18 19:39:20.450030 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:39:20.442995    2585 flags.go:64] FLAG: --max-open-files="1000000"
Sep 18 19:39:20.450030 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:39:20.442999    2585 flags.go:64] FLAG: --max-pods="110"
Sep 18 19:39:20.450030 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:39:20.443002    2585 flags.go:64] FLAG: --maximum-dead-containers="-1"
Sep 18 19:39:20.450030 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:39:20.443005    2585 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Sep 18 19:39:20.450030 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:39:20.443008    2585 flags.go:64] FLAG: --memory-manager-policy="None"
Sep 18 19:39:20.450030 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:39:20.443011    2585 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Sep 18 19:39:20.450030 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:39:20.443014    2585 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Sep 18 19:39:20.450030 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:39:20.443017    2585 flags.go:64] FLAG: --node-ip="0.0.0.0"
Sep 18 19:39:20.451096 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:39:20.443020    2585 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Sep 18 19:39:20.451096 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:39:20.443027    2585 flags.go:64] FLAG: --node-status-max-images="50"
Sep 18 19:39:20.451096 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:39:20.443030    2585 flags.go:64] FLAG: --node-status-update-frequency="10s"
Sep 18 19:39:20.451096 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:39:20.443033    2585 flags.go:64] FLAG: --oom-score-adj="-999"
Sep 18 19:39:20.451096 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:39:20.443036    2585 flags.go:64] FLAG: --pod-cidr=""
Sep 18 19:39:20.451096 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:39:20.443040    2585 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:a6abe77af61c5f2e3985a83e7d43538aefc38b1b0444127da14ee0e8cf8f76fc"
Sep 18 19:39:20.451096 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:39:20.443045    2585 flags.go:64] FLAG: --pod-manifest-path=""
Sep 18 19:39:20.451096 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:39:20.443048    2585 flags.go:64] FLAG: --pod-max-pids="-1"
Sep 18 19:39:20.451096 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:39:20.443051    2585 flags.go:64] FLAG: --pods-per-core="0"
Sep 18 19:39:20.451096 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:39:20.443054    2585 flags.go:64] FLAG: --port="10250"
Sep 18 19:39:20.451096 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:39:20.443057    2585 flags.go:64] FLAG: --protect-kernel-defaults="false"
Sep 18 19:39:20.451096 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:39:20.443061    2585 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-040b4dc3a4a119949"
Sep 18 19:39:20.451096 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:39:20.443064    2585 flags.go:64] FLAG: --qos-reserved=""
Sep 18 19:39:20.451096 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:39:20.443067    2585 flags.go:64] FLAG: --read-only-port="10255"
Sep 18 19:39:20.451096 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:39:20.443070    2585 flags.go:64] FLAG: --register-node="true"
Sep 18 19:39:20.451096 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:39:20.443073    2585 flags.go:64] FLAG: --register-with-taints=""
Sep 18 19:39:20.451096 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:39:20.443077    2585 flags.go:64] FLAG: --registry-burst="10"
Sep 18 19:39:20.451096 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:39:20.443079    2585 flags.go:64] FLAG: --registry-qps="5"
Sep 18 19:39:20.451096 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:39:20.443082    2585 flags.go:64] FLAG: --reserved-cpus=""
Sep 18 19:39:20.451096 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:39:20.443085    2585 flags.go:64] FLAG: --reserved-memory=""
Sep 18 19:39:20.451096 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:39:20.443441    2585 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Sep 18 19:39:20.451096 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:39:20.443445    2585 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Sep 18 19:39:20.451096 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:39:20.443448    2585 flags.go:64] FLAG: --rotate-certificates="false"
Sep 18 19:39:20.451096 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:39:20.443451    2585 flags.go:64] FLAG: --rotate-server-certificates="false"
Sep 18 19:39:20.451096 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:39:20.443454    2585 flags.go:64] FLAG: --runonce="false"
Sep 18 19:39:20.452089 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:39:20.443457    2585 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Sep 18 19:39:20.452089 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:39:20.443460    2585 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Sep 18 19:39:20.452089 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:39:20.443463    2585 flags.go:64] FLAG: --seccomp-default="false"
Sep 18 19:39:20.452089 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:39:20.443468    2585 flags.go:64] FLAG: --serialize-image-pulls="true"
Sep 18 19:39:20.452089 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:39:20.443470    2585 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Sep 18 19:39:20.452089 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:39:20.443473    2585 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Sep 18 19:39:20.452089 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:39:20.443476    2585 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Sep 18 19:39:20.452089 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:39:20.443479    2585 flags.go:64] FLAG: --storage-driver-password="root"
Sep 18 19:39:20.452089 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:39:20.443482    2585 flags.go:64] FLAG: --storage-driver-secure="false"
Sep 18 19:39:20.452089 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:39:20.443485    2585 flags.go:64] FLAG: --storage-driver-table="stats"
Sep 18 19:39:20.452089 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:39:20.443488    2585 flags.go:64] FLAG: --storage-driver-user="root"
Sep 18 19:39:20.452089 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:39:20.443491    2585 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Sep 18 19:39:20.452089 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:39:20.443494    2585 flags.go:64] FLAG: --sync-frequency="1m0s"
Sep 18 19:39:20.452089 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:39:20.443497    2585 flags.go:64] FLAG: --system-cgroups=""
Sep 18 19:39:20.452089 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:39:20.443500    2585 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Sep 18 19:39:20.452089 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:39:20.443506    2585 flags.go:64] FLAG: --system-reserved-cgroup=""
Sep 18 19:39:20.452089 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:39:20.443508    2585 flags.go:64] FLAG: --tls-cert-file=""
Sep 18 19:39:20.452089 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:39:20.443511    2585 flags.go:64] FLAG: --tls-cipher-suites="[]"
Sep 18 19:39:20.452089 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:39:20.443515    2585 flags.go:64] FLAG: --tls-min-version=""
Sep 18 19:39:20.452089 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:39:20.443518    2585 flags.go:64] FLAG: --tls-private-key-file=""
Sep 18 19:39:20.452089 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:39:20.443522    2585 flags.go:64] FLAG: --topology-manager-policy="none"
Sep 18 19:39:20.452089 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:39:20.443525    2585 flags.go:64] FLAG: --topology-manager-policy-options=""
Sep 18 19:39:20.452089 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:39:20.443528    2585 flags.go:64] FLAG: --topology-manager-scope="container"
Sep 18 19:39:20.452089 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:39:20.443531    2585 flags.go:64] FLAG: --v="2"
Sep 18 19:39:20.452089 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:39:20.443535    2585 flags.go:64] FLAG: --version="false"
Sep 18 19:39:20.452716 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:39:20.443539    2585 flags.go:64] FLAG: --vmodule=""
Sep 18 19:39:20.452716 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:39:20.443543    2585 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Sep 18 19:39:20.452716 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:39:20.443546    2585 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Sep 18 19:39:20.452716 ip-10-0-136-185 kubenswrapper[2585]: W0918 19:39:20.443655    2585 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 18 19:39:20.452716 ip-10-0-136-185 kubenswrapper[2585]: W0918 19:39:20.443658    2585 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 18 19:39:20.452716 ip-10-0-136-185 kubenswrapper[2585]: W0918 19:39:20.443662    2585 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 18 19:39:20.452716 ip-10-0-136-185 kubenswrapper[2585]: W0918 19:39:20.443664    2585 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 18 19:39:20.452716 ip-10-0-136-185 kubenswrapper[2585]: W0918 19:39:20.443667    2585 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 18 19:39:20.452716 ip-10-0-136-185 kubenswrapper[2585]: W0918 19:39:20.443670    2585 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 18 19:39:20.452716 ip-10-0-136-185 kubenswrapper[2585]: W0918 19:39:20.443673    2585 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 18 19:39:20.452716 ip-10-0-136-185 kubenswrapper[2585]: W0918 19:39:20.443677    2585 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 18 19:39:20.452716 ip-10-0-136-185 kubenswrapper[2585]: W0918 19:39:20.443680    2585 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 18 19:39:20.452716 ip-10-0-136-185 kubenswrapper[2585]: W0918 19:39:20.443683    2585 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 18 19:39:20.452716 ip-10-0-136-185 kubenswrapper[2585]: W0918 19:39:20.443686    2585 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 18 19:39:20.452716 ip-10-0-136-185 kubenswrapper[2585]: W0918 19:39:20.443688    2585 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 18 19:39:20.452716 ip-10-0-136-185 kubenswrapper[2585]: W0918 19:39:20.443691    2585 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 18 19:39:20.452716 ip-10-0-136-185 kubenswrapper[2585]: W0918 19:39:20.443694    2585 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 18 19:39:20.452716 ip-10-0-136-185 kubenswrapper[2585]: W0918 19:39:20.443696    2585 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 18 19:39:20.452716 ip-10-0-136-185 kubenswrapper[2585]: W0918 19:39:20.443699    2585 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 18 19:39:20.452716 ip-10-0-136-185 kubenswrapper[2585]: W0918 19:39:20.443701    2585 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 18 19:39:20.453248 ip-10-0-136-185 kubenswrapper[2585]: W0918 19:39:20.443704    2585 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 18 19:39:20.453248 ip-10-0-136-185 kubenswrapper[2585]: W0918 19:39:20.443707    2585 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 18 19:39:20.453248 ip-10-0-136-185 kubenswrapper[2585]: W0918 19:39:20.443709    2585 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 18 19:39:20.453248 ip-10-0-136-185 kubenswrapper[2585]: W0918 19:39:20.443712    2585 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 18 19:39:20.453248 ip-10-0-136-185 kubenswrapper[2585]: W0918 19:39:20.443714    2585 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 18 19:39:20.453248 ip-10-0-136-185 kubenswrapper[2585]: W0918 19:39:20.443717    2585 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 18 19:39:20.453248 ip-10-0-136-185 kubenswrapper[2585]: W0918 19:39:20.443720    2585 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 18 19:39:20.453248 ip-10-0-136-185 kubenswrapper[2585]: W0918 19:39:20.443723    2585 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 18 19:39:20.453248 ip-10-0-136-185 kubenswrapper[2585]: W0918 19:39:20.443726    2585 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 18 19:39:20.453248 ip-10-0-136-185 kubenswrapper[2585]: W0918 19:39:20.443729    2585 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 18 19:39:20.453248 ip-10-0-136-185 kubenswrapper[2585]: W0918 19:39:20.443733    2585 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 18 19:39:20.453248 ip-10-0-136-185 kubenswrapper[2585]: W0918 19:39:20.443736    2585 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 18 19:39:20.453248 ip-10-0-136-185 kubenswrapper[2585]: W0918 19:39:20.443739    2585 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 18 19:39:20.453248 ip-10-0-136-185 kubenswrapper[2585]: W0918 19:39:20.443742    2585 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 18 19:39:20.453248 ip-10-0-136-185 kubenswrapper[2585]: W0918 19:39:20.443745    2585 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 18 19:39:20.453248 ip-10-0-136-185 kubenswrapper[2585]: W0918 19:39:20.443748    2585 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 18 19:39:20.453248 ip-10-0-136-185 kubenswrapper[2585]: W0918 19:39:20.443751    2585 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 18 19:39:20.453248 ip-10-0-136-185 kubenswrapper[2585]: W0918 19:39:20.443754    2585 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 18 19:39:20.453248 ip-10-0-136-185 kubenswrapper[2585]: W0918 19:39:20.443756    2585 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 18 19:39:20.453767 ip-10-0-136-185 kubenswrapper[2585]: W0918 19:39:20.443759    2585 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 18 19:39:20.453767 ip-10-0-136-185 kubenswrapper[2585]: W0918 19:39:20.443762    2585 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 18 19:39:20.453767 ip-10-0-136-185 kubenswrapper[2585]: W0918 19:39:20.443764    2585 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 18 19:39:20.453767 ip-10-0-136-185 kubenswrapper[2585]: W0918 19:39:20.443768    2585 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 18 19:39:20.453767 ip-10-0-136-185 kubenswrapper[2585]: W0918 19:39:20.443772    2585 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 18 19:39:20.453767 ip-10-0-136-185 kubenswrapper[2585]: W0918 19:39:20.443775    2585 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 18 19:39:20.453767 ip-10-0-136-185 kubenswrapper[2585]: W0918 19:39:20.443778    2585 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 18 19:39:20.453767 ip-10-0-136-185 kubenswrapper[2585]: W0918 19:39:20.443781    2585 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 18 19:39:20.453767 ip-10-0-136-185 kubenswrapper[2585]: W0918 19:39:20.443795    2585 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 18 19:39:20.453767 ip-10-0-136-185 kubenswrapper[2585]: W0918 19:39:20.443798    2585 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 18 19:39:20.453767 ip-10-0-136-185 kubenswrapper[2585]: W0918 19:39:20.443800    2585 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 18 19:39:20.453767 ip-10-0-136-185 kubenswrapper[2585]: W0918 19:39:20.443803    2585 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 18 19:39:20.453767 ip-10-0-136-185 kubenswrapper[2585]: W0918 19:39:20.443806    2585 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 18 19:39:20.453767 ip-10-0-136-185 kubenswrapper[2585]: W0918 19:39:20.443808    2585 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 18 19:39:20.453767 ip-10-0-136-185 kubenswrapper[2585]: W0918 19:39:20.443811    2585 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 18 19:39:20.453767 ip-10-0-136-185 kubenswrapper[2585]: W0918 19:39:20.443813    2585 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 18 19:39:20.453767 ip-10-0-136-185 kubenswrapper[2585]: W0918 19:39:20.443816    2585 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 18 19:39:20.453767 ip-10-0-136-185 kubenswrapper[2585]: W0918 19:39:20.443818    2585 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 18 19:39:20.453767 ip-10-0-136-185 kubenswrapper[2585]: W0918 19:39:20.443821    2585 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 18 19:39:20.454302 ip-10-0-136-185 kubenswrapper[2585]: W0918 19:39:20.443824    2585 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 18 19:39:20.454302 ip-10-0-136-185 kubenswrapper[2585]: W0918 19:39:20.443827    2585 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 18 19:39:20.454302 ip-10-0-136-185 kubenswrapper[2585]: W0918 19:39:20.443830    2585 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 18 19:39:20.454302 ip-10-0-136-185 kubenswrapper[2585]: W0918 19:39:20.443833    2585 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 18 19:39:20.454302 ip-10-0-136-185 kubenswrapper[2585]: W0918 19:39:20.443836    2585 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 18 19:39:20.454302 ip-10-0-136-185 kubenswrapper[2585]: W0918 19:39:20.443838    2585 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 18 19:39:20.454302 ip-10-0-136-185 kubenswrapper[2585]: W0918 19:39:20.443841    2585 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 18 19:39:20.454302 ip-10-0-136-185 kubenswrapper[2585]: W0918 19:39:20.443843    2585 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 18 19:39:20.454302 ip-10-0-136-185 kubenswrapper[2585]: W0918 19:39:20.443846    2585 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 18 19:39:20.454302 ip-10-0-136-185 kubenswrapper[2585]: W0918 19:39:20.443848    2585 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 18 19:39:20.454302 ip-10-0-136-185 kubenswrapper[2585]: W0918 19:39:20.443851    2585 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 18 19:39:20.454302 ip-10-0-136-185 kubenswrapper[2585]: W0918 19:39:20.443854    2585 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 18 19:39:20.454302 ip-10-0-136-185 kubenswrapper[2585]: W0918 19:39:20.443858    2585 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 18 19:39:20.454302 ip-10-0-136-185 kubenswrapper[2585]: W0918 19:39:20.443861    2585 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 18 19:39:20.454302 ip-10-0-136-185 kubenswrapper[2585]: W0918 19:39:20.443865    2585 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 18 19:39:20.454302 ip-10-0-136-185 kubenswrapper[2585]: W0918 19:39:20.443868    2585 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 18 19:39:20.454302 ip-10-0-136-185 kubenswrapper[2585]: W0918 19:39:20.443872    2585 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 18 19:39:20.454302 ip-10-0-136-185 kubenswrapper[2585]: W0918 19:39:20.443874    2585 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 18 19:39:20.454302 ip-10-0-136-185 kubenswrapper[2585]: W0918 19:39:20.443877    2585 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 18 19:39:20.454763 ip-10-0-136-185 kubenswrapper[2585]: W0918 19:39:20.443879    2585 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 18 19:39:20.454763 ip-10-0-136-185 kubenswrapper[2585]: W0918 19:39:20.443882    2585 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 18 19:39:20.454763 ip-10-0-136-185 kubenswrapper[2585]: W0918 19:39:20.443885    2585 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 18 19:39:20.454763 ip-10-0-136-185 kubenswrapper[2585]: W0918 19:39:20.443887    2585 feature_gate.go:328] unrecognized feature gate: Example2
Sep 18 19:39:20.454763 ip-10-0-136-185 kubenswrapper[2585]: W0918 19:39:20.443890    2585 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 18 19:39:20.454763 ip-10-0-136-185 kubenswrapper[2585]: W0918 19:39:20.443893    2585 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 18 19:39:20.454763 ip-10-0-136-185 kubenswrapper[2585]: W0918 19:39:20.443895    2585 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 18 19:39:20.454763 ip-10-0-136-185 kubenswrapper[2585]: W0918 19:39:20.443898    2585 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 18 19:39:20.454763 ip-10-0-136-185 kubenswrapper[2585]: W0918 19:39:20.443901    2585 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 18 19:39:20.454763 ip-10-0-136-185 kubenswrapper[2585]: W0918 19:39:20.443903    2585 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 18 19:39:20.454763 ip-10-0-136-185 kubenswrapper[2585]: W0918 19:39:20.443906    2585 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 18 19:39:20.454763 ip-10-0-136-185 kubenswrapper[2585]: W0918 19:39:20.443908    2585 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 18 19:39:20.454763 ip-10-0-136-185 kubenswrapper[2585]: W0918 19:39:20.443911    2585 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 18 19:39:20.454763 ip-10-0-136-185 kubenswrapper[2585]: W0918 19:39:20.443914    2585 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 18 19:39:20.454763 ip-10-0-136-185 kubenswrapper[2585]: W0918 19:39:20.443918    2585 feature_gate.go:328] unrecognized feature gate: Example
Sep 18 19:39:20.454763 ip-10-0-136-185 kubenswrapper[2585]: W0918 19:39:20.443921    2585 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 18 19:39:20.454763 ip-10-0-136-185 kubenswrapper[2585]: W0918 19:39:20.443923    2585 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 18 19:39:20.454763 ip-10-0-136-185 kubenswrapper[2585]: W0918 19:39:20.443926    2585 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 18 19:39:20.454763 ip-10-0-136-185 kubenswrapper[2585]: W0918 19:39:20.443929    2585 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 18 19:39:20.454763 ip-10-0-136-185 kubenswrapper[2585]: W0918 19:39:20.443931    2585 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 18 19:39:20.455278 ip-10-0-136-185 kubenswrapper[2585]: W0918 19:39:20.443934    2585 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 18 19:39:20.455278 ip-10-0-136-185 kubenswrapper[2585]: W0918 19:39:20.443937    2585 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 18 19:39:20.455278 ip-10-0-136-185 kubenswrapper[2585]: W0918 19:39:20.443939    2585 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 18 19:39:20.455278 ip-10-0-136-185 kubenswrapper[2585]: W0918 19:39:20.443942    2585 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 18 19:39:20.455278 ip-10-0-136-185 kubenswrapper[2585]: W0918 19:39:20.443944    2585 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 18 19:39:20.455278 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:39:20.444482    2585 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 18 19:39:20.455278 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:39:20.452776    2585 server.go:529] "Kubelet version" kubeletVersion="v1.34.9"
Sep 18 19:39:20.455278 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:39:20.452808    2585 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Sep 18 19:39:20.455278 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:39:20.452829    2585 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Sep 18 19:39:20.455278 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:39:20.452839    2585 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Sep 18 19:39:20.455278 ip-10-0-136-185 kubenswrapper[2585]: W0918 19:39:20.452897    2585 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 18 19:39:20.455278 ip-10-0-136-185 kubenswrapper[2585]: W0918 19:39:20.452902    2585 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 18 19:39:20.455278 ip-10-0-136-185 kubenswrapper[2585]: W0918 19:39:20.452905    2585 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 18 19:39:20.455278 ip-10-0-136-185 kubenswrapper[2585]: W0918 19:39:20.452909    2585 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 18 19:39:20.455278 ip-10-0-136-185 kubenswrapper[2585]: W0918 19:39:20.452912    2585 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 18 19:39:20.455667 ip-10-0-136-185 kubenswrapper[2585]: W0918 19:39:20.452916    2585 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 18 19:39:20.455667 ip-10-0-136-185 kubenswrapper[2585]: W0918 19:39:20.452919    2585 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 18 19:39:20.455667 ip-10-0-136-185 kubenswrapper[2585]: W0918 19:39:20.452922    2585 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 18 19:39:20.455667 ip-10-0-136-185 kubenswrapper[2585]: W0918 19:39:20.452925    2585 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 18 19:39:20.455667 ip-10-0-136-185 kubenswrapper[2585]: W0918 19:39:20.452928    2585 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 18 19:39:20.455667 ip-10-0-136-185 kubenswrapper[2585]: W0918 19:39:20.452931    2585 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 18 19:39:20.455667 ip-10-0-136-185 kubenswrapper[2585]: W0918 19:39:20.452934    2585 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 18 19:39:20.455667 ip-10-0-136-185 kubenswrapper[2585]: W0918 19:39:20.452936    2585 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 18 19:39:20.455667 ip-10-0-136-185 kubenswrapper[2585]: W0918 19:39:20.452939    2585 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 18 19:39:20.455667 ip-10-0-136-185 kubenswrapper[2585]: W0918 19:39:20.452941    2585 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 18 19:39:20.455667 ip-10-0-136-185 kubenswrapper[2585]: W0918 19:39:20.452944    2585 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 18 19:39:20.455667 ip-10-0-136-185 kubenswrapper[2585]: W0918 19:39:20.452947    2585 feature_gate.go:328] unrecognized feature gate: Example2
Sep 18 19:39:20.455667 ip-10-0-136-185 kubenswrapper[2585]: W0918 19:39:20.452949    2585 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 18 19:39:20.455667 ip-10-0-136-185 kubenswrapper[2585]: W0918 19:39:20.452952    2585 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 18 19:39:20.455667 ip-10-0-136-185 kubenswrapper[2585]: W0918 19:39:20.452955    2585 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 18 19:39:20.455667 ip-10-0-136-185 kubenswrapper[2585]: W0918 19:39:20.452958    2585 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 18 19:39:20.455667 ip-10-0-136-185 kubenswrapper[2585]: W0918 19:39:20.452961    2585 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 18 19:39:20.455667 ip-10-0-136-185 kubenswrapper[2585]: W0918 19:39:20.452964    2585 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 18 19:39:20.455667 ip-10-0-136-185 kubenswrapper[2585]: W0918 19:39:20.452967    2585 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 18 19:39:20.455667 ip-10-0-136-185 kubenswrapper[2585]: W0918 19:39:20.452969    2585 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 18 19:39:20.456235 ip-10-0-136-185 kubenswrapper[2585]: W0918 19:39:20.452972    2585 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 18 19:39:20.456235 ip-10-0-136-185 kubenswrapper[2585]: W0918 19:39:20.452975    2585 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 18 19:39:20.456235 ip-10-0-136-185 kubenswrapper[2585]: W0918 19:39:20.452977    2585 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 18 19:39:20.456235 ip-10-0-136-185 kubenswrapper[2585]: W0918 19:39:20.452980    2585 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 18 19:39:20.456235 ip-10-0-136-185 kubenswrapper[2585]: W0918 19:39:20.452983    2585 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 18 19:39:20.456235 ip-10-0-136-185 kubenswrapper[2585]: W0918 19:39:20.452986    2585 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 18 19:39:20.456235 ip-10-0-136-185 kubenswrapper[2585]: W0918 19:39:20.452988    2585 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 18 19:39:20.456235 ip-10-0-136-185 kubenswrapper[2585]: W0918 19:39:20.452991    2585 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 18 19:39:20.456235 ip-10-0-136-185 kubenswrapper[2585]: W0918 19:39:20.452994    2585 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 18 19:39:20.456235 ip-10-0-136-185 kubenswrapper[2585]: W0918 19:39:20.452996    2585 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 18 19:39:20.456235 ip-10-0-136-185 kubenswrapper[2585]: W0918 19:39:20.453001    2585 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 18 19:39:20.456235 ip-10-0-136-185 kubenswrapper[2585]: W0918 19:39:20.453004    2585 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 18 19:39:20.456235 ip-10-0-136-185 kubenswrapper[2585]: W0918 19:39:20.453007    2585 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 18 19:39:20.456235 ip-10-0-136-185 kubenswrapper[2585]: W0918 19:39:20.453009    2585 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 18 19:39:20.456235 ip-10-0-136-185 kubenswrapper[2585]: W0918 19:39:20.453012    2585 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 18 19:39:20.456235 ip-10-0-136-185 kubenswrapper[2585]: W0918 19:39:20.453015    2585 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 18 19:39:20.456235 ip-10-0-136-185 kubenswrapper[2585]: W0918 19:39:20.453017    2585 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 18 19:39:20.456235 ip-10-0-136-185 kubenswrapper[2585]: W0918 19:39:20.453020    2585 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 18 19:39:20.456235 ip-10-0-136-185 kubenswrapper[2585]: W0918 19:39:20.453023    2585 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 18 19:39:20.456700 ip-10-0-136-185 kubenswrapper[2585]: W0918 19:39:20.453025    2585 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 18 19:39:20.456700 ip-10-0-136-185 kubenswrapper[2585]: W0918 19:39:20.453028    2585 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 18 19:39:20.456700 ip-10-0-136-185 kubenswrapper[2585]: W0918 19:39:20.453031    2585 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 18 19:39:20.456700 ip-10-0-136-185 kubenswrapper[2585]: W0918 19:39:20.453034    2585 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 18 19:39:20.456700 ip-10-0-136-185 kubenswrapper[2585]: W0918 19:39:20.453037    2585 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 18 19:39:20.456700 ip-10-0-136-185 kubenswrapper[2585]: W0918 19:39:20.453040    2585 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 18 19:39:20.456700 ip-10-0-136-185 kubenswrapper[2585]: W0918 19:39:20.453043    2585 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 18 19:39:20.456700 ip-10-0-136-185 kubenswrapper[2585]: W0918 19:39:20.453045    2585 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 18 19:39:20.456700 ip-10-0-136-185 kubenswrapper[2585]: W0918 19:39:20.453048    2585 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 18 19:39:20.456700 ip-10-0-136-185 kubenswrapper[2585]: W0918 19:39:20.453050    2585 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 18 19:39:20.456700 ip-10-0-136-185 kubenswrapper[2585]: W0918 19:39:20.453053    2585 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 18 19:39:20.456700 ip-10-0-136-185 kubenswrapper[2585]: W0918 19:39:20.453056    2585 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 18 19:39:20.456700 ip-10-0-136-185 kubenswrapper[2585]: W0918 19:39:20.453059    2585 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 18 19:39:20.456700 ip-10-0-136-185 kubenswrapper[2585]: W0918 19:39:20.453061    2585 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 18 19:39:20.456700 ip-10-0-136-185 kubenswrapper[2585]: W0918 19:39:20.453064    2585 feature_gate.go:328] unrecognized feature gate: Example
Sep 18 19:39:20.456700 ip-10-0-136-185 kubenswrapper[2585]: W0918 19:39:20.453068    2585 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 18 19:39:20.456700 ip-10-0-136-185 kubenswrapper[2585]: W0918 19:39:20.453071    2585 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 18 19:39:20.456700 ip-10-0-136-185 kubenswrapper[2585]: W0918 19:39:20.453075    2585 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 18 19:39:20.456700 ip-10-0-136-185 kubenswrapper[2585]: W0918 19:39:20.453078    2585 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 18 19:39:20.456700 ip-10-0-136-185 kubenswrapper[2585]: W0918 19:39:20.453081    2585 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 18 19:39:20.457347 ip-10-0-136-185 kubenswrapper[2585]: W0918 19:39:20.453083    2585 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 18 19:39:20.457347 ip-10-0-136-185 kubenswrapper[2585]: W0918 19:39:20.453086    2585 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 18 19:39:20.457347 ip-10-0-136-185 kubenswrapper[2585]: W0918 19:39:20.453089    2585 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 18 19:39:20.457347 ip-10-0-136-185 kubenswrapper[2585]: W0918 19:39:20.453091    2585 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 18 19:39:20.457347 ip-10-0-136-185 kubenswrapper[2585]: W0918 19:39:20.453094    2585 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 18 19:39:20.457347 ip-10-0-136-185 kubenswrapper[2585]: W0918 19:39:20.453097    2585 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 18 19:39:20.457347 ip-10-0-136-185 kubenswrapper[2585]: W0918 19:39:20.453100    2585 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 18 19:39:20.457347 ip-10-0-136-185 kubenswrapper[2585]: W0918 19:39:20.453102    2585 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 18 19:39:20.457347 ip-10-0-136-185 kubenswrapper[2585]: W0918 19:39:20.453105    2585 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 18 19:39:20.457347 ip-10-0-136-185 kubenswrapper[2585]: W0918 19:39:20.453107    2585 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 18 19:39:20.457347 ip-10-0-136-185 kubenswrapper[2585]: W0918 19:39:20.453110    2585 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 18 19:39:20.457347 ip-10-0-136-185 kubenswrapper[2585]: W0918 19:39:20.453113    2585 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 18 19:39:20.457347 ip-10-0-136-185 kubenswrapper[2585]: W0918 19:39:20.453115    2585 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 18 19:39:20.457347 ip-10-0-136-185 kubenswrapper[2585]: W0918 19:39:20.453118    2585 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 18 19:39:20.457347 ip-10-0-136-185 kubenswrapper[2585]: W0918 19:39:20.453121    2585 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 18 19:39:20.457347 ip-10-0-136-185 kubenswrapper[2585]: W0918 19:39:20.453124    2585 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 18 19:39:20.457347 ip-10-0-136-185 kubenswrapper[2585]: W0918 19:39:20.453127    2585 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 18 19:39:20.457347 ip-10-0-136-185 kubenswrapper[2585]: W0918 19:39:20.453129    2585 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 18 19:39:20.457347 ip-10-0-136-185 kubenswrapper[2585]: W0918 19:39:20.453132    2585 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 18 19:39:20.457347 ip-10-0-136-185 kubenswrapper[2585]: W0918 19:39:20.453135    2585 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 18 19:39:20.457858 ip-10-0-136-185 kubenswrapper[2585]: W0918 19:39:20.453139    2585 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 18 19:39:20.457858 ip-10-0-136-185 kubenswrapper[2585]: W0918 19:39:20.453142    2585 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 18 19:39:20.457858 ip-10-0-136-185 kubenswrapper[2585]: W0918 19:39:20.453145    2585 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 18 19:39:20.457858 ip-10-0-136-185 kubenswrapper[2585]: W0918 19:39:20.453148    2585 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 18 19:39:20.457858 ip-10-0-136-185 kubenswrapper[2585]: W0918 19:39:20.453151    2585 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 18 19:39:20.457858 ip-10-0-136-185 kubenswrapper[2585]: W0918 19:39:20.453153    2585 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 18 19:39:20.457858 ip-10-0-136-185 kubenswrapper[2585]: W0918 19:39:20.453156    2585 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 18 19:39:20.457858 ip-10-0-136-185 kubenswrapper[2585]: W0918 19:39:20.453158    2585 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 18 19:39:20.457858 ip-10-0-136-185 kubenswrapper[2585]: W0918 19:39:20.453161    2585 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 18 19:39:20.457858 ip-10-0-136-185 kubenswrapper[2585]: W0918 19:39:20.453164    2585 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 18 19:39:20.457858 ip-10-0-136-185 kubenswrapper[2585]: W0918 19:39:20.453167    2585 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 18 19:39:20.457858 ip-10-0-136-185 kubenswrapper[2585]: W0918 19:39:20.453169    2585 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 18 19:39:20.457858 ip-10-0-136-185 kubenswrapper[2585]: W0918 19:39:20.453172    2585 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 18 19:39:20.457858 ip-10-0-136-185 kubenswrapper[2585]: W0918 19:39:20.453175    2585 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 18 19:39:20.457858 ip-10-0-136-185 kubenswrapper[2585]: W0918 19:39:20.453178    2585 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 18 19:39:20.458221 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:39:20.453182    2585 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 18 19:39:20.458221 ip-10-0-136-185 kubenswrapper[2585]: W0918 19:39:20.453314    2585 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 18 19:39:20.458221 ip-10-0-136-185 kubenswrapper[2585]: W0918 19:39:20.453320    2585 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 18 19:39:20.458221 ip-10-0-136-185 kubenswrapper[2585]: W0918 19:39:20.453323    2585 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 18 19:39:20.458221 ip-10-0-136-185 kubenswrapper[2585]: W0918 19:39:20.453325    2585 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 18 19:39:20.458221 ip-10-0-136-185 kubenswrapper[2585]: W0918 19:39:20.453328    2585 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 18 19:39:20.458221 ip-10-0-136-185 kubenswrapper[2585]: W0918 19:39:20.453331    2585 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 18 19:39:20.458221 ip-10-0-136-185 kubenswrapper[2585]: W0918 19:39:20.453334    2585 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 18 19:39:20.458221 ip-10-0-136-185 kubenswrapper[2585]: W0918 19:39:20.453336    2585 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 18 19:39:20.458221 ip-10-0-136-185 kubenswrapper[2585]: W0918 19:39:20.453339    2585 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 18 19:39:20.458221 ip-10-0-136-185 kubenswrapper[2585]: W0918 19:39:20.453341    2585 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 18 19:39:20.458221 ip-10-0-136-185 kubenswrapper[2585]: W0918 19:39:20.453344    2585 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 18 19:39:20.458221 ip-10-0-136-185 kubenswrapper[2585]: W0918 19:39:20.453347    2585 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 18 19:39:20.458221 ip-10-0-136-185 kubenswrapper[2585]: W0918 19:39:20.453350    2585 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 18 19:39:20.458221 ip-10-0-136-185 kubenswrapper[2585]: W0918 19:39:20.453353    2585 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 18 19:39:20.458591 ip-10-0-136-185 kubenswrapper[2585]: W0918 19:39:20.453356    2585 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 18 19:39:20.458591 ip-10-0-136-185 kubenswrapper[2585]: W0918 19:39:20.453359    2585 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 18 19:39:20.458591 ip-10-0-136-185 kubenswrapper[2585]: W0918 19:39:20.453361    2585 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 18 19:39:20.458591 ip-10-0-136-185 kubenswrapper[2585]: W0918 19:39:20.453364    2585 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 18 19:39:20.458591 ip-10-0-136-185 kubenswrapper[2585]: W0918 19:39:20.453367    2585 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 18 19:39:20.458591 ip-10-0-136-185 kubenswrapper[2585]: W0918 19:39:20.453370    2585 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 18 19:39:20.458591 ip-10-0-136-185 kubenswrapper[2585]: W0918 19:39:20.453372    2585 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 18 19:39:20.458591 ip-10-0-136-185 kubenswrapper[2585]: W0918 19:39:20.453375    2585 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 18 19:39:20.458591 ip-10-0-136-185 kubenswrapper[2585]: W0918 19:39:20.453377    2585 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 18 19:39:20.458591 ip-10-0-136-185 kubenswrapper[2585]: W0918 19:39:20.453380    2585 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 18 19:39:20.458591 ip-10-0-136-185 kubenswrapper[2585]: W0918 19:39:20.453383    2585 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 18 19:39:20.458591 ip-10-0-136-185 kubenswrapper[2585]: W0918 19:39:20.453385    2585 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 18 19:39:20.458591 ip-10-0-136-185 kubenswrapper[2585]: W0918 19:39:20.453388    2585 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 18 19:39:20.458591 ip-10-0-136-185 kubenswrapper[2585]: W0918 19:39:20.453390    2585 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 18 19:39:20.458591 ip-10-0-136-185 kubenswrapper[2585]: W0918 19:39:20.453393    2585 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 18 19:39:20.458591 ip-10-0-136-185 kubenswrapper[2585]: W0918 19:39:20.453396    2585 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 18 19:39:20.458591 ip-10-0-136-185 kubenswrapper[2585]: W0918 19:39:20.453398    2585 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 18 19:39:20.458591 ip-10-0-136-185 kubenswrapper[2585]: W0918 19:39:20.453400    2585 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 18 19:39:20.458591 ip-10-0-136-185 kubenswrapper[2585]: W0918 19:39:20.453403    2585 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 18 19:39:20.458591 ip-10-0-136-185 kubenswrapper[2585]: W0918 19:39:20.453406    2585 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 18 19:39:20.459097 ip-10-0-136-185 kubenswrapper[2585]: W0918 19:39:20.453408    2585 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 18 19:39:20.459097 ip-10-0-136-185 kubenswrapper[2585]: W0918 19:39:20.453411    2585 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 18 19:39:20.459097 ip-10-0-136-185 kubenswrapper[2585]: W0918 19:39:20.453413    2585 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 18 19:39:20.459097 ip-10-0-136-185 kubenswrapper[2585]: W0918 19:39:20.453416    2585 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 18 19:39:20.459097 ip-10-0-136-185 kubenswrapper[2585]: W0918 19:39:20.453418    2585 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 18 19:39:20.459097 ip-10-0-136-185 kubenswrapper[2585]: W0918 19:39:20.453421    2585 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 18 19:39:20.459097 ip-10-0-136-185 kubenswrapper[2585]: W0918 19:39:20.453423    2585 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 18 19:39:20.459097 ip-10-0-136-185 kubenswrapper[2585]: W0918 19:39:20.453426    2585 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 18 19:39:20.459097 ip-10-0-136-185 kubenswrapper[2585]: W0918 19:39:20.453429    2585 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 18 19:39:20.459097 ip-10-0-136-185 kubenswrapper[2585]: W0918 19:39:20.453431    2585 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 18 19:39:20.459097 ip-10-0-136-185 kubenswrapper[2585]: W0918 19:39:20.453434    2585 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 18 19:39:20.459097 ip-10-0-136-185 kubenswrapper[2585]: W0918 19:39:20.453437    2585 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 18 19:39:20.459097 ip-10-0-136-185 kubenswrapper[2585]: W0918 19:39:20.453439    2585 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 18 19:39:20.459097 ip-10-0-136-185 kubenswrapper[2585]: W0918 19:39:20.453442    2585 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 18 19:39:20.459097 ip-10-0-136-185 kubenswrapper[2585]: W0918 19:39:20.453444    2585 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 18 19:39:20.459097 ip-10-0-136-185 kubenswrapper[2585]: W0918 19:39:20.453447    2585 feature_gate.go:328] unrecognized feature gate: Example
Sep 18 19:39:20.459097 ip-10-0-136-185 kubenswrapper[2585]: W0918 19:39:20.453450    2585 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 18 19:39:20.459097 ip-10-0-136-185 kubenswrapper[2585]: W0918 19:39:20.453452    2585 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 18 19:39:20.459097 ip-10-0-136-185 kubenswrapper[2585]: W0918 19:39:20.453455    2585 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 18 19:39:20.459097 ip-10-0-136-185 kubenswrapper[2585]: W0918 19:39:20.453457    2585 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 18 19:39:20.459589 ip-10-0-136-185 kubenswrapper[2585]: W0918 19:39:20.453462    2585 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 18 19:39:20.459589 ip-10-0-136-185 kubenswrapper[2585]: W0918 19:39:20.453465    2585 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 18 19:39:20.459589 ip-10-0-136-185 kubenswrapper[2585]: W0918 19:39:20.453468    2585 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 18 19:39:20.459589 ip-10-0-136-185 kubenswrapper[2585]: W0918 19:39:20.453472    2585 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 18 19:39:20.459589 ip-10-0-136-185 kubenswrapper[2585]: W0918 19:39:20.453475    2585 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 18 19:39:20.459589 ip-10-0-136-185 kubenswrapper[2585]: W0918 19:39:20.453477    2585 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 18 19:39:20.459589 ip-10-0-136-185 kubenswrapper[2585]: W0918 19:39:20.453480    2585 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 18 19:39:20.459589 ip-10-0-136-185 kubenswrapper[2585]: W0918 19:39:20.453483    2585 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 18 19:39:20.459589 ip-10-0-136-185 kubenswrapper[2585]: W0918 19:39:20.453486    2585 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 18 19:39:20.459589 ip-10-0-136-185 kubenswrapper[2585]: W0918 19:39:20.453489    2585 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 18 19:39:20.459589 ip-10-0-136-185 kubenswrapper[2585]: W0918 19:39:20.453492    2585 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 18 19:39:20.459589 ip-10-0-136-185 kubenswrapper[2585]: W0918 19:39:20.453495    2585 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 18 19:39:20.459589 ip-10-0-136-185 kubenswrapper[2585]: W0918 19:39:20.453497    2585 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 18 19:39:20.459589 ip-10-0-136-185 kubenswrapper[2585]: W0918 19:39:20.453500    2585 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 18 19:39:20.459589 ip-10-0-136-185 kubenswrapper[2585]: W0918 19:39:20.453502    2585 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 18 19:39:20.459589 ip-10-0-136-185 kubenswrapper[2585]: W0918 19:39:20.453505    2585 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 18 19:39:20.459589 ip-10-0-136-185 kubenswrapper[2585]: W0918 19:39:20.453508    2585 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 18 19:39:20.459589 ip-10-0-136-185 kubenswrapper[2585]: W0918 19:39:20.453510    2585 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 18 19:39:20.459589 ip-10-0-136-185 kubenswrapper[2585]: W0918 19:39:20.453512    2585 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 18 19:39:20.460069 ip-10-0-136-185 kubenswrapper[2585]: W0918 19:39:20.453515    2585 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 18 19:39:20.460069 ip-10-0-136-185 kubenswrapper[2585]: W0918 19:39:20.453518    2585 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 18 19:39:20.460069 ip-10-0-136-185 kubenswrapper[2585]: W0918 19:39:20.453521    2585 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 18 19:39:20.460069 ip-10-0-136-185 kubenswrapper[2585]: W0918 19:39:20.453525    2585 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 18 19:39:20.460069 ip-10-0-136-185 kubenswrapper[2585]: W0918 19:39:20.453528    2585 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 18 19:39:20.460069 ip-10-0-136-185 kubenswrapper[2585]: W0918 19:39:20.453533    2585 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 18 19:39:20.460069 ip-10-0-136-185 kubenswrapper[2585]: W0918 19:39:20.453536    2585 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 18 19:39:20.460069 ip-10-0-136-185 kubenswrapper[2585]: W0918 19:39:20.453539    2585 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 18 19:39:20.460069 ip-10-0-136-185 kubenswrapper[2585]: W0918 19:39:20.453542    2585 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 18 19:39:20.460069 ip-10-0-136-185 kubenswrapper[2585]: W0918 19:39:20.453546    2585 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 18 19:39:20.460069 ip-10-0-136-185 kubenswrapper[2585]: W0918 19:39:20.453549    2585 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 18 19:39:20.460069 ip-10-0-136-185 kubenswrapper[2585]: W0918 19:39:20.453552    2585 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 18 19:39:20.460069 ip-10-0-136-185 kubenswrapper[2585]: W0918 19:39:20.453555    2585 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 18 19:39:20.460069 ip-10-0-136-185 kubenswrapper[2585]: W0918 19:39:20.453557    2585 feature_gate.go:328] unrecognized feature gate: Example2
Sep 18 19:39:20.460069 ip-10-0-136-185 kubenswrapper[2585]: W0918 19:39:20.453560    2585 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 18 19:39:20.460069 ip-10-0-136-185 kubenswrapper[2585]: W0918 19:39:20.453562    2585 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 18 19:39:20.460069 ip-10-0-136-185 kubenswrapper[2585]: W0918 19:39:20.453565    2585 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 18 19:39:20.460069 ip-10-0-136-185 kubenswrapper[2585]: W0918 19:39:20.453567    2585 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 18 19:39:20.460069 ip-10-0-136-185 kubenswrapper[2585]: W0918 19:39:20.453570    2585 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 18 19:39:20.460069 ip-10-0-136-185 kubenswrapper[2585]: W0918 19:39:20.453573    2585 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 18 19:39:20.460563 ip-10-0-136-185 kubenswrapper[2585]: W0918 19:39:20.453575    2585 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 18 19:39:20.460563 ip-10-0-136-185 kubenswrapper[2585]: W0918 19:39:20.453577    2585 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 18 19:39:20.460563 ip-10-0-136-185 kubenswrapper[2585]: W0918 19:39:20.453580    2585 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 18 19:39:20.460563 ip-10-0-136-185 kubenswrapper[2585]: W0918 19:39:20.453583    2585 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 18 19:39:20.460563 ip-10-0-136-185 kubenswrapper[2585]: W0918 19:39:20.453585    2585 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 18 19:39:20.460563 ip-10-0-136-185 kubenswrapper[2585]: W0918 19:39:20.453588    2585 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 18 19:39:20.460563 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:39:20.453593    2585 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 18 19:39:20.460563 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:39:20.453715    2585 server.go:962] "Client rotation is on, will bootstrap in background"
Sep 18 19:39:20.460563 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:39:20.456295    2585 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Sep 18 19:39:20.460563 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:39:20.457167    2585 server.go:1019] "Starting client certificate rotation"
Sep 18 19:39:20.460563 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:39:20.457270    2585 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 18 19:39:20.460563 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:39:20.457314    2585 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 18 19:39:20.460563 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:39:20.459537    2585 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Sep 18 19:39:20.460944 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:39:20.460721    2585 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Sep 18 19:39:20.467001 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:39:20.466982    2585 log.go:25] "Validated CRI v1 runtime API"
Sep 18 19:39:20.468590 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:39:20.468577    2585 log.go:25] "Validated CRI v1 image API"
Sep 18 19:39:20.469421 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:39:20.469405    2585 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Sep 18 19:39:20.473015 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:39:20.472994    2585 fs.go:135] Filesystem UUIDs: map[7B77-95E7:/dev/nvme0n1p2 af1c7d62-dbe3-4908-9df8-4f6788fd1f90:/dev/nvme0n1p3 b62eb347-a64e-4e5f-8001-5accb863a8e4:/dev/nvme0n1p4]
Sep 18 19:39:20.473089 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:39:20.473013    2585 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}]
Sep 18 19:39:20.478611 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:39:20.478485    2585 manager.go:216] Machine: {Timestamp:2026-09-18 19:39:20.477958596 +0000 UTC m=+2.002564010 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3103739 MemoryCapacity:33164472320 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2ed6768a12ba2a53de4e61f50ac0a3 SystemUUID:ec2ed676-8a12-ba2a-53de-4e61f50ac0a3 BootID:4ed6ca92-3dd1-4544-827d-fb3908da89e6 Filesystems:[{Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632894464 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582238208 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582234112 Type:vfs Inodes:4048397 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:ff:d8:3a:6b:1d Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:ff:d8:3a:6b:1d Speed:0 Mtu:9001} {Name:ovs-system MacAddress:e6:0d:1f:95:8b:ca Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164472320 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Sep 18 19:39:20.478611 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:39:20.478602    2585 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Sep 18 19:39:20.478756 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:39:20.478683    2585 manager.go:232] Version: {KernelVersion:5.14.0-570.141.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260914-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Sep 18 19:39:20.480132 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:39:20.480110    2585 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Sep 18 19:39:20.480329 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:39:20.480133    2585 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-136-185.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Sep 18 19:39:20.481387 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:39:20.481376    2585 topology_manager.go:138] "Creating topology manager with none policy"
Sep 18 19:39:20.481421 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:39:20.481390    2585 container_manager_linux.go:309] "Creating device plugin manager"
Sep 18 19:39:20.481421 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:39:20.481402    2585 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 18 19:39:20.481421 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:39:20.481417    2585 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 18 19:39:20.481502 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:39:20.481485    2585 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager"
Sep 18 19:39:20.484460 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:39:20.484447    2585 state_mem.go:36] "Initialized new in-memory state store"
Sep 18 19:39:20.484566 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:39:20.484557    2585 server.go:1240] "Using root directory" path="/var/lib/kubelet"
Sep 18 19:39:20.484617 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:39:20.484589    2585 kubelet.go:486] "Attempting to sync node with API server"
Sep 18 19:39:20.484617 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:39:20.484601    2585 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests"
Sep 18 19:39:20.484617 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:39:20.484612    2585 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Sep 18 19:39:20.484750 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:39:20.484654    2585 kubelet.go:398] "Adding apiserver pod source"
Sep 18 19:39:20.484750 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:39:20.484663    2585 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Sep 18 19:39:20.486671 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:39:20.486599    2585 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Sep 18 19:39:20.486883 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:39:20.486870    2585 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Sep 18 19:39:20.488966 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:39:20.488939    2585 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Sep 18 19:39:20.490329 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:39:20.490313    2585 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.13-6.rhaos4.21.git6253162.el9" apiVersion="v1"
Sep 18 19:39:20.492427 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:39:20.492414    2585 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Sep 18 19:39:20.492476 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:39:20.492443    2585 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Sep 18 19:39:20.493500 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:39:20.493489    2585 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Sep 18 19:39:20.493561 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:39:20.493505    2585 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Sep 18 19:39:20.493561 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:39:20.493511    2585 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Sep 18 19:39:20.493561 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:39:20.493517    2585 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Sep 18 19:39:20.493561 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:39:20.493526    2585 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Sep 18 19:39:20.493561 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:39:20.493532    2585 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Sep 18 19:39:20.493561 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:39:20.493545    2585 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Sep 18 19:39:20.493561 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:39:20.493551    2585 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Sep 18 19:39:20.493561 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:39:20.493557    2585 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Sep 18 19:39:20.493561 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:39:20.493563    2585 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Sep 18 19:39:20.493811 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:39:20.493572    2585 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Sep 18 19:39:20.493811 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:39:20.493577    2585 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Sep 18 19:39:20.493811 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:39:20.493602    2585 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Sep 18 19:39:20.493811 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:39:20.493608    2585 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Sep 18 19:39:20.495653 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:39:20.495633    2585 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-136-185.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 18 19:39:20.495742 ip-10-0-136-185 kubenswrapper[2585]: E0918 19:39:20.495653    2585 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
Sep 18 19:39:20.495742 ip-10-0-136-185 kubenswrapper[2585]: E0918 19:39:20.495671    2585 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-136-185.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
Sep 18 19:39:20.497971 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:39:20.497959    2585 server.go:1268] "Started kubelet"
Sep 18 19:39:20.498041 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:39:20.498016    2585 server.go:180] "Starting to listen" address="0.0.0.0" port=10250
Sep 18 19:39:20.498113 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:39:20.498067    2585 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Sep 18 19:39:20.498159 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:39:20.498140    2585 server_v1.go:49] "podresources" method="list" useActivePods=true
Sep 18 19:39:20.499018 ip-10-0-136-185 systemd[1]: Started Kubernetes Kubelet.
Sep 18 19:39:20.499570 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:39:20.499414    2585 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Sep 18 19:39:20.499801 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:39:20.499772    2585 server.go:310] "Adding debug handlers to kubelet server"
Sep 18 19:39:20.503836 ip-10-0-136-185 kubenswrapper[2585]: E0918 19:39:20.503818    2585 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Sep 18 19:39:20.505284 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:39:20.505263    2585 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
Sep 18 19:39:20.505344 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:39:20.505282    2585 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Sep 18 19:39:20.505392 ip-10-0-136-185 kubenswrapper[2585]: E0918 19:39:20.505355    2585 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-185.ec2.internal\" not found"
Sep 18 19:39:20.505868 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:39:20.505826    2585 volume_manager.go:311] "The desired_state_of_world populator starts"
Sep 18 19:39:20.505868 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:39:20.505868    2585 volume_manager.go:313] "Starting Kubelet Volume Manager"
Sep 18 19:39:20.506010 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:39:20.505893    2585 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Sep 18 19:39:20.506060 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:39:20.506036    2585 reconstruct.go:97] "Volume reconstruction finished"
Sep 18 19:39:20.506060 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:39:20.506044    2585 reconciler.go:29] "Reconciler: start to sync state"
Sep 18 19:39:20.506423 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:39:20.506405    2585 factory.go:153] Registering CRI-O factory
Sep 18 19:39:20.506495 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:39:20.506426    2585 factory.go:223] Registration of the crio container factory successfully
Sep 18 19:39:20.506495 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:39:20.506477    2585 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Sep 18 19:39:20.506495 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:39:20.506487    2585 factory.go:55] Registering systemd factory
Sep 18 19:39:20.506495 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:39:20.506495    2585 factory.go:223] Registration of the systemd container factory successfully
Sep 18 19:39:20.506619 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:39:20.506512    2585 factory.go:103] Registering Raw factory
Sep 18 19:39:20.506619 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:39:20.506526    2585 manager.go:1196] Started watching for new ooms in manager
Sep 18 19:39:20.506992 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:39:20.506980    2585 manager.go:349] Starting recovery of all containers
Sep 18 19:39:20.508552 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:39:20.508529    2585 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-6rscm"
Sep 18 19:39:20.530691 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:39:20.530483    2585 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-6rscm"
Sep 18 19:39:20.533428 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:39:20.533407    2585 manager.go:354] Recovery completed
Sep 18 19:39:20.533634 ip-10-0-136-185 kubenswrapper[2585]: E0918 19:39:20.533607    2585 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver"
Sep 18 19:39:20.533766 ip-10-0-136-185 kubenswrapper[2585]: E0918 19:39:20.533733    2585 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-136-185.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
Sep 18 19:39:20.538566 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:39:20.538547    2585 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 18 19:39:20.542404 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:39:20.542389    2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-185.ec2.internal" event="NodeHasSufficientMemory"
Sep 18 19:39:20.542464 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:39:20.542417    2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-185.ec2.internal" event="NodeHasNoDiskPressure"
Sep 18 19:39:20.542464 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:39:20.542427    2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-185.ec2.internal" event="NodeHasSufficientPID"
Sep 18 19:39:20.542985 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:39:20.542968    2585 cpu_manager.go:222] "Starting CPU manager" policy="none"
Sep 18 19:39:20.542985 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:39:20.542981    2585 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s"
Sep 18 19:39:20.543102 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:39:20.542995    2585 state_mem.go:36] "Initialized new in-memory state store"
Sep 18 19:39:20.543102 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:39:20.543006    2585 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 18 19:39:20.543738 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:39:20.543722    2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-185.ec2.internal" event="NodeHasSufficientMemory"
Sep 18 19:39:20.543813 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:39:20.543754    2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-185.ec2.internal" event="NodeHasNoDiskPressure"
Sep 18 19:39:20.543813 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:39:20.543769    2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-185.ec2.internal" event="NodeHasSufficientPID"
Sep 18 19:39:20.544139 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:39:20.544125    2585 policy_none.go:49] "None policy: Start"
Sep 18 19:39:20.544192 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:39:20.544141    2585 memory_manager.go:187] "Starting memorymanager" policy="None"
Sep 18 19:39:20.544192 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:39:20.544151    2585 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Sep 18 19:39:20.545155 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:39:20.545144    2585 policy_none.go:47] "Start"
Sep 18 19:39:20.579381 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:39:20.579364    2585 manager.go:337] "Starting Device Plugin manager"
Sep 18 19:39:20.595310 ip-10-0-136-185 kubenswrapper[2585]: E0918 19:39:20.579449    2585 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Sep 18 19:39:20.595310 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:39:20.579462    2585 server.go:87] "Starting device plugin registration server"
Sep 18 19:39:20.595310 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:39:20.579729    2585 eviction_manager.go:189] "Eviction manager: starting control loop"
Sep 18 19:39:20.595310 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:39:20.579749    2585 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Sep 18 19:39:20.595310 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:39:20.579843    2585 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Sep 18 19:39:20.595310 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:39:20.579917    2585 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts"
Sep 18 19:39:20.595310 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:39:20.579925    2585 plugin_manager.go:118] "Starting Kubelet Plugin Manager"
Sep 18 19:39:20.595310 ip-10-0-136-185 kubenswrapper[2585]: E0918 19:39:20.580492    2585 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Sep 18 19:39:20.595310 ip-10-0-136-185 kubenswrapper[2585]: E0918 19:39:20.580566    2585 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-136-185.ec2.internal\" not found"
Sep 18 19:39:20.653942 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:39:20.653902    2585 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Sep 18 19:39:20.655139 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:39:20.655127    2585 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Sep 18 19:39:20.655193 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:39:20.655177    2585 status_manager.go:244] "Starting to sync pod status with apiserver"
Sep 18 19:39:20.655235 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:39:20.655203    2585 kubelet.go:2448] "Starting kubelet main sync loop"
Sep 18 19:39:20.655279 ip-10-0-136-185 kubenswrapper[2585]: E0918 19:39:20.655241    2585 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Sep 18 19:39:20.657425 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:39:20.657404    2585 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 18 19:39:20.680308 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:39:20.680277    2585 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 18 19:39:20.681533 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:39:20.681514    2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-185.ec2.internal" event="NodeHasSufficientMemory"
Sep 18 19:39:20.681634 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:39:20.681544    2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-185.ec2.internal" event="NodeHasNoDiskPressure"
Sep 18 19:39:20.681634 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:39:20.681560    2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-185.ec2.internal" event="NodeHasSufficientPID"
Sep 18 19:39:20.681634 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:39:20.681587    2585 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-136-185.ec2.internal"
Sep 18 19:39:20.692179 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:39:20.692118    2585 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-136-185.ec2.internal"
Sep 18 19:39:20.692179 ip-10-0-136-185 kubenswrapper[2585]: E0918 19:39:20.692147    2585 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-136-185.ec2.internal\": node \"ip-10-0-136-185.ec2.internal\" not found"
Sep 18 19:39:20.710190 ip-10-0-136-185 kubenswrapper[2585]: E0918 19:39:20.710166    2585 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-185.ec2.internal\" not found"
Sep 18 19:39:20.755657 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:39:20.755629    2585 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-185.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-136-185.ec2.internal"]
Sep 18 19:39:20.755762 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:39:20.755714    2585 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 18 19:39:20.757267 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:39:20.757251    2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-185.ec2.internal" event="NodeHasSufficientMemory"
Sep 18 19:39:20.757370 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:39:20.757293    2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-185.ec2.internal" event="NodeHasNoDiskPressure"
Sep 18 19:39:20.757370 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:39:20.757307    2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-185.ec2.internal" event="NodeHasSufficientPID"
Sep 18 19:39:20.759210 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:39:20.759195    2585 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 18 19:39:20.759342 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:39:20.759326    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-185.ec2.internal"
Sep 18 19:39:20.759392 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:39:20.759358    2585 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 18 19:39:20.760009 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:39:20.759991    2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-185.ec2.internal" event="NodeHasSufficientMemory"
Sep 18 19:39:20.760093 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:39:20.760012    2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-185.ec2.internal" event="NodeHasSufficientMemory"
Sep 18 19:39:20.760093 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:39:20.760021    2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-185.ec2.internal" event="NodeHasNoDiskPressure"
Sep 18 19:39:20.760093 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:39:20.760031    2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-185.ec2.internal" event="NodeHasSufficientPID"
Sep 18 19:39:20.760093 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:39:20.760033    2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-185.ec2.internal" event="NodeHasNoDiskPressure"
Sep 18 19:39:20.760093 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:39:20.760043    2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-185.ec2.internal" event="NodeHasSufficientPID"
Sep 18 19:39:20.761888 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:39:20.761874    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-136-185.ec2.internal"
Sep 18 19:39:20.761964 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:39:20.761897    2585 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 18 19:39:20.762628 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:39:20.762610    2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-185.ec2.internal" event="NodeHasSufficientMemory"
Sep 18 19:39:20.762712 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:39:20.762638    2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-185.ec2.internal" event="NodeHasNoDiskPressure"
Sep 18 19:39:20.762712 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:39:20.762648    2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-185.ec2.internal" event="NodeHasSufficientPID"
Sep 18 19:39:20.797567 ip-10-0-136-185 kubenswrapper[2585]: E0918 19:39:20.797543    2585 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-136-185.ec2.internal\" not found" node="ip-10-0-136-185.ec2.internal"
Sep 18 19:39:20.802018 ip-10-0-136-185 kubenswrapper[2585]: E0918 19:39:20.802000    2585 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-136-185.ec2.internal\" not found" node="ip-10-0-136-185.ec2.internal"
Sep 18 19:39:20.807201 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:39:20.807182    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/5989c800ad96c7a3440a3290994469f4-config\") pod \"kube-apiserver-proxy-ip-10-0-136-185.ec2.internal\" (UID: \"5989c800ad96c7a3440a3290994469f4\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-136-185.ec2.internal"
Sep 18 19:39:20.807294 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:39:20.807214    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/acc0abc3eedde3c5f89486311f9adfed-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-136-185.ec2.internal\" (UID: \"acc0abc3eedde3c5f89486311f9adfed\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-185.ec2.internal"
Sep 18 19:39:20.807294 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:39:20.807239    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/acc0abc3eedde3c5f89486311f9adfed-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-136-185.ec2.internal\" (UID: \"acc0abc3eedde3c5f89486311f9adfed\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-185.ec2.internal"
Sep 18 19:39:20.811260 ip-10-0-136-185 kubenswrapper[2585]: E0918 19:39:20.811242    2585 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-185.ec2.internal\" not found"
Sep 18 19:39:20.907703 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:39:20.907668    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/acc0abc3eedde3c5f89486311f9adfed-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-136-185.ec2.internal\" (UID: \"acc0abc3eedde3c5f89486311f9adfed\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-185.ec2.internal"
Sep 18 19:39:20.907703 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:39:20.907710    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/acc0abc3eedde3c5f89486311f9adfed-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-136-185.ec2.internal\" (UID: \"acc0abc3eedde3c5f89486311f9adfed\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-185.ec2.internal"
Sep 18 19:39:20.907954 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:39:20.907728    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/5989c800ad96c7a3440a3290994469f4-config\") pod \"kube-apiserver-proxy-ip-10-0-136-185.ec2.internal\" (UID: \"5989c800ad96c7a3440a3290994469f4\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-136-185.ec2.internal"
Sep 18 19:39:20.907954 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:39:20.907754    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/5989c800ad96c7a3440a3290994469f4-config\") pod \"kube-apiserver-proxy-ip-10-0-136-185.ec2.internal\" (UID: \"5989c800ad96c7a3440a3290994469f4\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-136-185.ec2.internal"
Sep 18 19:39:20.907954 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:39:20.907749    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/acc0abc3eedde3c5f89486311f9adfed-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-136-185.ec2.internal\" (UID: \"acc0abc3eedde3c5f89486311f9adfed\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-185.ec2.internal"
Sep 18 19:39:20.907954 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:39:20.907777    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/acc0abc3eedde3c5f89486311f9adfed-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-136-185.ec2.internal\" (UID: \"acc0abc3eedde3c5f89486311f9adfed\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-185.ec2.internal"
Sep 18 19:39:20.911799 ip-10-0-136-185 kubenswrapper[2585]: E0918 19:39:20.911762    2585 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-185.ec2.internal\" not found"
Sep 18 19:39:21.012607 ip-10-0-136-185 kubenswrapper[2585]: E0918 19:39:21.012527    2585 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-185.ec2.internal\" not found"
Sep 18 19:39:21.100742 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:39:21.100716    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-185.ec2.internal"
Sep 18 19:39:21.105967 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:39:21.105697    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-136-185.ec2.internal"
Sep 18 19:39:21.113362 ip-10-0-136-185 kubenswrapper[2585]: E0918 19:39:21.113338    2585 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-185.ec2.internal\" not found"
Sep 18 19:39:21.213912 ip-10-0-136-185 kubenswrapper[2585]: E0918 19:39:21.213862    2585 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-185.ec2.internal\" not found"
Sep 18 19:39:21.314413 ip-10-0-136-185 kubenswrapper[2585]: E0918 19:39:21.314340    2585 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-185.ec2.internal\" not found"
Sep 18 19:39:21.414862 ip-10-0-136-185 kubenswrapper[2585]: E0918 19:39:21.414833    2585 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-185.ec2.internal\" not found"
Sep 18 19:39:21.457026 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:39:21.456987    2585 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Sep 18 19:39:21.457661 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:39:21.457150    2585 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Sep 18 19:39:21.493174 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:39:21.493147    2585 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 18 19:39:21.515275 ip-10-0-136-185 kubenswrapper[2585]: E0918 19:39:21.515248    2585 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-185.ec2.internal\" not found"
Sep 18 19:39:21.535443 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:39:21.535405    2585 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-09-17 19:34:20 +0000 UTC" deadline="2028-02-15 20:04:27.580460906 +0000 UTC"
Sep 18 19:39:21.535443 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:39:21.535437    2585 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="12360h25m6.045027689s"
Sep 18 19:39:21.616311 ip-10-0-136-185 kubenswrapper[2585]: E0918 19:39:21.616045    2585 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-185.ec2.internal\" not found"
Sep 18 19:39:21.709577 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:39:21.709554    2585 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 18 19:39:21.716334 ip-10-0-136-185 kubenswrapper[2585]: E0918 19:39:21.716311    2585 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-185.ec2.internal\" not found"
Sep 18 19:39:21.727148 ip-10-0-136-185 kubenswrapper[2585]: W0918 19:39:21.727117    2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5989c800ad96c7a3440a3290994469f4.slice/crio-9fb563f2f03d9a1fbea6693832f235c93ac8c76742e6a5db82967c4b21641854 WatchSource:0}: Error finding container 9fb563f2f03d9a1fbea6693832f235c93ac8c76742e6a5db82967c4b21641854: Status 404 returned error can't find the container with id 9fb563f2f03d9a1fbea6693832f235c93ac8c76742e6a5db82967c4b21641854
Sep 18 19:39:21.727435 ip-10-0-136-185 kubenswrapper[2585]: W0918 19:39:21.727423    2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podacc0abc3eedde3c5f89486311f9adfed.slice/crio-27eae363456f8d8e55fcb30d27a16c429bda3f5115091e35516129123f28fd38 WatchSource:0}: Error finding container 27eae363456f8d8e55fcb30d27a16c429bda3f5115091e35516129123f28fd38: Status 404 returned error can't find the container with id 27eae363456f8d8e55fcb30d27a16c429bda3f5115091e35516129123f28fd38
Sep 18 19:39:21.733872 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:39:21.733856    2585 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 18 19:39:21.816824 ip-10-0-136-185 kubenswrapper[2585]: E0918 19:39:21.816765    2585 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-185.ec2.internal\" not found"
Sep 18 19:39:21.917245 ip-10-0-136-185 kubenswrapper[2585]: E0918 19:39:21.917224    2585 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-185.ec2.internal\" not found"
Sep 18 19:39:22.018051 ip-10-0-136-185 kubenswrapper[2585]: E0918 19:39:22.018028    2585 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-185.ec2.internal\" not found"
Sep 18 19:39:22.078141 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:39:22.078108    2585 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 18 19:39:22.106089 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:39:22.106045    2585 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-185.ec2.internal"
Sep 18 19:39:22.116783 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:39:22.116759    2585 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Sep 18 19:39:22.117732 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:39:22.117711    2585 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-136-185.ec2.internal"
Sep 18 19:39:22.123720 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:39:22.123698    2585 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Sep 18 19:39:22.485613 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:39:22.485577    2585 apiserver.go:52] "Watching apiserver"
Sep 18 19:39:22.492733 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:39:22.492708    2585 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Sep 18 19:39:22.493197 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:39:22.493169    2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ovn-kubernetes/ovnkube-node-rctld","kube-system/global-pull-secret-syncer-npsrh","kube-system/konnectivity-agent-mbhzp","openshift-cluster-node-tuning-operator/tuned-cmv8r","openshift-image-registry/node-ca-llt9z","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-185.ec2.internal","openshift-multus/multus-w27m9","openshift-multus/network-metrics-daemon-4njfl","openshift-network-operator/iptables-alerter-fpnxg","kube-system/kube-apiserver-proxy-ip-10-0-136-185.ec2.internal","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-m4b5p","openshift-dns/node-resolver-ltq6c","openshift-multus/multus-additional-cni-plugins-x7kdg","openshift-network-diagnostics/network-check-target-zxh4h"]
Sep 18 19:39:22.495697 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:39:22.495659    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-w27m9"
Sep 18 19:39:22.498204 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:39:22.498129    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Sep 18 19:39:22.498204 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:39:22.498191    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Sep 18 19:39:22.498375 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:39:22.498302    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-gmtbw\""
Sep 18 19:39:22.498375 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:39:22.498324    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Sep 18 19:39:22.498375 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:39:22.498345    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Sep 18 19:39:22.498495 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:39:22.498450    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-x7kdg"
Sep 18 19:39:22.500197 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:39:22.500176    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-mbhzp"
Sep 18 19:39:22.500777 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:39:22.500558    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Sep 18 19:39:22.500777 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:39:22.500566    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-m7nqj\""
Sep 18 19:39:22.500777 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:39:22.500562    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Sep 18 19:39:22.502013 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:39:22.501993    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-nh8kh\""
Sep 18 19:39:22.502102 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:39:22.502059    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Sep 18 19:39:22.502324 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:39:22.502306    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Sep 18 19:39:22.502505 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:39:22.502449    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-cmv8r"
Sep 18 19:39:22.504770 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:39:22.504316    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Sep 18 19:39:22.504770 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:39:22.504456    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-hblpp\""
Sep 18 19:39:22.504770 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:39:22.504514    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Sep 18 19:39:22.504770 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:39:22.504685    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-llt9z"
Sep 18 19:39:22.506062 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:39:22.506040    2585 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Sep 18 19:39:22.506397 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:39:22.506379    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Sep 18 19:39:22.507165 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:39:22.506700    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Sep 18 19:39:22.507165 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:39:22.506776    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Sep 18 19:39:22.507165 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:39:22.506972    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zxh4h"
Sep 18 19:39:22.507165 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:39:22.507025    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-cvmz8\""
Sep 18 19:39:22.507165 ip-10-0-136-185 kubenswrapper[2585]: E0918 19:39:22.507074    2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zxh4h" podUID="13f14c60-b4bd-4146-b4ee-0dd90be01349"
Sep 18 19:39:22.510195 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:39:22.509442    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-fpnxg"
Sep 18 19:39:22.511998 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:39:22.511473    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Sep 18 19:39:22.511998 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:39:22.511803    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-5l25g\""
Sep 18 19:39:22.511998 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:39:22.511894    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Sep 18 19:39:22.512312 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:39:22.512243    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-rctld"
Sep 18 19:39:22.514145 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:39:22.514126    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Sep 18 19:39:22.515059 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:39:22.514875    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Sep 18 19:39:22.515693 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:39:22.515673    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-npsrh"
Sep 18 19:39:22.515843 ip-10-0-136-185 kubenswrapper[2585]: E0918 19:39:22.515741    2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-npsrh" podUID="7e47b713-e1a2-48ff-b961-ad41e706a225"
Sep 18 19:39:22.516228 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:39:22.516137    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Sep 18 19:39:22.516228 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:39:22.516147    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Sep 18 19:39:22.516371 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:39:22.516294    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Sep 18 19:39:22.516961 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:39:22.516934    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-qkpgr\""
Sep 18 19:39:22.517085 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:39:22.517023    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/fe7306b4-9adf-4628-804d-a0ff961d5135-run\") pod \"tuned-cmv8r\" (UID: \"fe7306b4-9adf-4628-804d-a0ff961d5135\") " pod="openshift-cluster-node-tuning-operator/tuned-cmv8r"
Sep 18 19:39:22.517085 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:39:22.517054    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/fe7306b4-9adf-4628-804d-a0ff961d5135-var-lib-kubelet\") pod \"tuned-cmv8r\" (UID: \"fe7306b4-9adf-4628-804d-a0ff961d5135\") " pod="openshift-cluster-node-tuning-operator/tuned-cmv8r"
Sep 18 19:39:22.517085 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:39:22.517079    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/3a884214-2df1-4958-a6b9-e3f2c903ae22-host\") pod \"node-ca-llt9z\" (UID: \"3a884214-2df1-4958-a6b9-e3f2c903ae22\") " pod="openshift-image-registry/node-ca-llt9z"
Sep 18 19:39:22.517238 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:39:22.517102    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/e34af56e-8a55-46f2-96b1-a81fbf4effb4-host-var-lib-cni-multus\") pod \"multus-w27m9\" (UID: \"e34af56e-8a55-46f2-96b1-a81fbf4effb4\") " pod="openshift-multus/multus-w27m9"
Sep 18 19:39:22.517238 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:39:22.517143    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/17b9ee59-c638-43de-8ec6-9544968d07f5-system-cni-dir\") pod \"multus-additional-cni-plugins-x7kdg\" (UID: \"17b9ee59-c638-43de-8ec6-9544968d07f5\") " pod="openshift-multus/multus-additional-cni-plugins-x7kdg"
Sep 18 19:39:22.517238 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:39:22.517175    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/17b9ee59-c638-43de-8ec6-9544968d07f5-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-x7kdg\" (UID: \"17b9ee59-c638-43de-8ec6-9544968d07f5\") " pod="openshift-multus/multus-additional-cni-plugins-x7kdg"
Sep 18 19:39:22.517238 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:39:22.517207    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/d2fa3583-1483-4960-a44b-694496d13f45-konnectivity-ca\") pod \"konnectivity-agent-mbhzp\" (UID: \"d2fa3583-1483-4960-a44b-694496d13f45\") " pod="kube-system/konnectivity-agent-mbhzp"
Sep 18 19:39:22.517238 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:39:22.517234    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/fe7306b4-9adf-4628-804d-a0ff961d5135-etc-sysctl-conf\") pod \"tuned-cmv8r\" (UID: \"fe7306b4-9adf-4628-804d-a0ff961d5135\") " pod="openshift-cluster-node-tuning-operator/tuned-cmv8r"
Sep 18 19:39:22.517501 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:39:22.517291    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xc47f\" (UniqueName: \"kubernetes.io/projected/3a884214-2df1-4958-a6b9-e3f2c903ae22-kube-api-access-xc47f\") pod \"node-ca-llt9z\" (UID: \"3a884214-2df1-4958-a6b9-e3f2c903ae22\") " pod="openshift-image-registry/node-ca-llt9z"
Sep 18 19:39:22.517501 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:39:22.517333    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/e34af56e-8a55-46f2-96b1-a81fbf4effb4-multus-cni-dir\") pod \"multus-w27m9\" (UID: \"e34af56e-8a55-46f2-96b1-a81fbf4effb4\") " pod="openshift-multus/multus-w27m9"
Sep 18 19:39:22.517501 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:39:22.517363    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/e34af56e-8a55-46f2-96b1-a81fbf4effb4-host-var-lib-cni-bin\") pod \"multus-w27m9\" (UID: \"e34af56e-8a55-46f2-96b1-a81fbf4effb4\") " pod="openshift-multus/multus-w27m9"
Sep 18 19:39:22.517501 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:39:22.517388    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/17b9ee59-c638-43de-8ec6-9544968d07f5-cnibin\") pod \"multus-additional-cni-plugins-x7kdg\" (UID: \"17b9ee59-c638-43de-8ec6-9544968d07f5\") " pod="openshift-multus/multus-additional-cni-plugins-x7kdg"
Sep 18 19:39:22.517501 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:39:22.517411    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Sep 18 19:39:22.517501 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:39:22.517412    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/fe7306b4-9adf-4628-804d-a0ff961d5135-etc-modprobe-d\") pod \"tuned-cmv8r\" (UID: \"fe7306b4-9adf-4628-804d-a0ff961d5135\") " pod="openshift-cluster-node-tuning-operator/tuned-cmv8r"
Sep 18 19:39:22.517501 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:39:22.517475    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/fe7306b4-9adf-4628-804d-a0ff961d5135-tmp\") pod \"tuned-cmv8r\" (UID: \"fe7306b4-9adf-4628-804d-a0ff961d5135\") " pod="openshift-cluster-node-tuning-operator/tuned-cmv8r"
Sep 18 19:39:22.517501 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:39:22.517500    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6bfnn\" (UniqueName: \"kubernetes.io/projected/13f14c60-b4bd-4146-b4ee-0dd90be01349-kube-api-access-6bfnn\") pod \"network-check-target-zxh4h\" (UID: \"13f14c60-b4bd-4146-b4ee-0dd90be01349\") " pod="openshift-network-diagnostics/network-check-target-zxh4h"
Sep 18 19:39:22.517904 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:39:22.517521    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/e34af56e-8a55-46f2-96b1-a81fbf4effb4-cnibin\") pod \"multus-w27m9\" (UID: \"e34af56e-8a55-46f2-96b1-a81fbf4effb4\") " pod="openshift-multus/multus-w27m9"
Sep 18 19:39:22.517904 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:39:22.517541    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/e34af56e-8a55-46f2-96b1-a81fbf4effb4-cni-binary-copy\") pod \"multus-w27m9\" (UID: \"e34af56e-8a55-46f2-96b1-a81fbf4effb4\") " pod="openshift-multus/multus-w27m9"
Sep 18 19:39:22.517904 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:39:22.517565    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/e34af56e-8a55-46f2-96b1-a81fbf4effb4-host-var-lib-kubelet\") pod \"multus-w27m9\" (UID: \"e34af56e-8a55-46f2-96b1-a81fbf4effb4\") " pod="openshift-multus/multus-w27m9"
Sep 18 19:39:22.517904 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:39:22.517588    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/17b9ee59-c638-43de-8ec6-9544968d07f5-tuning-conf-dir\") pod \"multus-additional-cni-plugins-x7kdg\" (UID: \"17b9ee59-c638-43de-8ec6-9544968d07f5\") " pod="openshift-multus/multus-additional-cni-plugins-x7kdg"
Sep 18 19:39:22.517904 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:39:22.517626    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/fe7306b4-9adf-4628-804d-a0ff961d5135-etc-sysctl-d\") pod \"tuned-cmv8r\" (UID: \"fe7306b4-9adf-4628-804d-a0ff961d5135\") " pod="openshift-cluster-node-tuning-operator/tuned-cmv8r"
Sep 18 19:39:22.517904 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:39:22.517650    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/fe7306b4-9adf-4628-804d-a0ff961d5135-etc-systemd\") pod \"tuned-cmv8r\" (UID: \"fe7306b4-9adf-4628-804d-a0ff961d5135\") " pod="openshift-cluster-node-tuning-operator/tuned-cmv8r"
Sep 18 19:39:22.517904 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:39:22.517674    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/e34af56e-8a55-46f2-96b1-a81fbf4effb4-multus-socket-dir-parent\") pod \"multus-w27m9\" (UID: \"e34af56e-8a55-46f2-96b1-a81fbf4effb4\") " pod="openshift-multus/multus-w27m9"
Sep 18 19:39:22.517904 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:39:22.517707    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/e34af56e-8a55-46f2-96b1-a81fbf4effb4-host-run-k8s-cni-cncf-io\") pod \"multus-w27m9\" (UID: \"e34af56e-8a55-46f2-96b1-a81fbf4effb4\") " pod="openshift-multus/multus-w27m9"
Sep 18 19:39:22.517904 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:39:22.517735    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e34af56e-8a55-46f2-96b1-a81fbf4effb4-etc-kubernetes\") pod \"multus-w27m9\" (UID: \"e34af56e-8a55-46f2-96b1-a81fbf4effb4\") " pod="openshift-multus/multus-w27m9"
Sep 18 19:39:22.517904 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:39:22.517749    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Sep 18 19:39:22.517904 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:39:22.517759    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/17b9ee59-c638-43de-8ec6-9544968d07f5-cni-binary-copy\") pod \"multus-additional-cni-plugins-x7kdg\" (UID: \"17b9ee59-c638-43de-8ec6-9544968d07f5\") " pod="openshift-multus/multus-additional-cni-plugins-x7kdg"
Sep 18 19:39:22.517904 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:39:22.517827    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/fe7306b4-9adf-4628-804d-a0ff961d5135-etc-tuned\") pod \"tuned-cmv8r\" (UID: \"fe7306b4-9adf-4628-804d-a0ff961d5135\") " pod="openshift-cluster-node-tuning-operator/tuned-cmv8r"
Sep 18 19:39:22.517904 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:39:22.517859    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/17b9ee59-c638-43de-8ec6-9544968d07f5-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-x7kdg\" (UID: \"17b9ee59-c638-43de-8ec6-9544968d07f5\") " pod="openshift-multus/multus-additional-cni-plugins-x7kdg"
Sep 18 19:39:22.517904 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:39:22.517883    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/fe7306b4-9adf-4628-804d-a0ff961d5135-lib-modules\") pod \"tuned-cmv8r\" (UID: \"fe7306b4-9adf-4628-804d-a0ff961d5135\") " pod="openshift-cluster-node-tuning-operator/tuned-cmv8r"
Sep 18 19:39:22.517904 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:39:22.517906    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/3a884214-2df1-4958-a6b9-e3f2c903ae22-serviceca\") pod \"node-ca-llt9z\" (UID: \"3a884214-2df1-4958-a6b9-e3f2c903ae22\") " pod="openshift-image-registry/node-ca-llt9z"
Sep 18 19:39:22.518653 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:39:22.517928    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/e34af56e-8a55-46f2-96b1-a81fbf4effb4-system-cni-dir\") pod \"multus-w27m9\" (UID: \"e34af56e-8a55-46f2-96b1-a81fbf4effb4\") " pod="openshift-multus/multus-w27m9"
Sep 18 19:39:22.518653 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:39:22.517951    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/e34af56e-8a55-46f2-96b1-a81fbf4effb4-host-run-netns\") pod \"multus-w27m9\" (UID: \"e34af56e-8a55-46f2-96b1-a81fbf4effb4\") " pod="openshift-multus/multus-w27m9"
Sep 18 19:39:22.518653 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:39:22.517974    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/e34af56e-8a55-46f2-96b1-a81fbf4effb4-host-run-multus-certs\") pod \"multus-w27m9\" (UID: \"e34af56e-8a55-46f2-96b1-a81fbf4effb4\") " pod="openshift-multus/multus-w27m9"
Sep 18 19:39:22.518653 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:39:22.518006    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/fe7306b4-9adf-4628-804d-a0ff961d5135-etc-sysconfig\") pod \"tuned-cmv8r\" (UID: \"fe7306b4-9adf-4628-804d-a0ff961d5135\") " pod="openshift-cluster-node-tuning-operator/tuned-cmv8r"
Sep 18 19:39:22.518653 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:39:22.518038    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/fe7306b4-9adf-4628-804d-a0ff961d5135-etc-kubernetes\") pod \"tuned-cmv8r\" (UID: \"fe7306b4-9adf-4628-804d-a0ff961d5135\") " pod="openshift-cluster-node-tuning-operator/tuned-cmv8r"
Sep 18 19:39:22.518653 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:39:22.518081    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/fe7306b4-9adf-4628-804d-a0ff961d5135-sys\") pod \"tuned-cmv8r\" (UID: \"fe7306b4-9adf-4628-804d-a0ff961d5135\") " pod="openshift-cluster-node-tuning-operator/tuned-cmv8r"
Sep 18 19:39:22.518653 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:39:22.518109    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/fe7306b4-9adf-4628-804d-a0ff961d5135-host\") pod \"tuned-cmv8r\" (UID: \"fe7306b4-9adf-4628-804d-a0ff961d5135\") " pod="openshift-cluster-node-tuning-operator/tuned-cmv8r"
Sep 18 19:39:22.518653 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:39:22.518133    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/e34af56e-8a55-46f2-96b1-a81fbf4effb4-hostroot\") pod \"multus-w27m9\" (UID: \"e34af56e-8a55-46f2-96b1-a81fbf4effb4\") " pod="openshift-multus/multus-w27m9"
Sep 18 19:39:22.518653 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:39:22.518156    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h849s\" (UniqueName: \"kubernetes.io/projected/e34af56e-8a55-46f2-96b1-a81fbf4effb4-kube-api-access-h849s\") pod \"multus-w27m9\" (UID: \"e34af56e-8a55-46f2-96b1-a81fbf4effb4\") " pod="openshift-multus/multus-w27m9"
Sep 18 19:39:22.518653 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:39:22.518180    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vxt9v\" (UniqueName: \"kubernetes.io/projected/fe7306b4-9adf-4628-804d-a0ff961d5135-kube-api-access-vxt9v\") pod \"tuned-cmv8r\" (UID: \"fe7306b4-9adf-4628-804d-a0ff961d5135\") " pod="openshift-cluster-node-tuning-operator/tuned-cmv8r"
Sep 18 19:39:22.518653 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:39:22.518203    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/e34af56e-8a55-46f2-96b1-a81fbf4effb4-multus-conf-dir\") pod \"multus-w27m9\" (UID: \"e34af56e-8a55-46f2-96b1-a81fbf4effb4\") " pod="openshift-multus/multus-w27m9"
Sep 18 19:39:22.518653 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:39:22.518234    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/e34af56e-8a55-46f2-96b1-a81fbf4effb4-multus-daemon-config\") pod \"multus-w27m9\" (UID: \"e34af56e-8a55-46f2-96b1-a81fbf4effb4\") " pod="openshift-multus/multus-w27m9"
Sep 18 19:39:22.518653 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:39:22.518259    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sp49h\" (UniqueName: \"kubernetes.io/projected/17b9ee59-c638-43de-8ec6-9544968d07f5-kube-api-access-sp49h\") pod \"multus-additional-cni-plugins-x7kdg\" (UID: \"17b9ee59-c638-43de-8ec6-9544968d07f5\") " pod="openshift-multus/multus-additional-cni-plugins-x7kdg"
Sep 18 19:39:22.518653 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:39:22.518283    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/d2fa3583-1483-4960-a44b-694496d13f45-agent-certs\") pod \"konnectivity-agent-mbhzp\" (UID: \"d2fa3583-1483-4960-a44b-694496d13f45\") " pod="kube-system/konnectivity-agent-mbhzp"
Sep 18 19:39:22.519415 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:39:22.519189    2585 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Sep 18 19:39:22.519488 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:39:22.519470    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4njfl"
Sep 18 19:39:22.519555 ip-10-0-136-185 kubenswrapper[2585]: E0918 19:39:22.519534    2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4njfl" podUID="3eb3cd73-4ea1-4950-8bfd-82b5fa60fd09"
Sep 18 19:39:22.521684 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:39:22.521665    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-m4b5p"
Sep 18 19:39:22.523469 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:39:22.523450    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Sep 18 19:39:22.523758 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:39:22.523739    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-92kf7\""
Sep 18 19:39:22.523969 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:39:22.523953    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Sep 18 19:39:22.524060 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:39:22.524010    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-ltq6c"
Sep 18 19:39:22.524114 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:39:22.524076    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Sep 18 19:39:22.526083 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:39:22.525887    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Sep 18 19:39:22.526083 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:39:22.525933    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-dlrjv\""
Sep 18 19:39:22.526083 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:39:22.525957    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Sep 18 19:39:22.544292 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:39:22.544268    2585 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-ggtvn"
Sep 18 19:39:22.552007 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:39:22.551984    2585 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-ggtvn"
Sep 18 19:39:22.609581 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:39:22.608146    2585 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Sep 18 19:39:22.619514 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:39:22.619478    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k8r8m\" (UniqueName: \"kubernetes.io/projected/cc0e0de2-75d5-43a1-b64b-b071427cd8ce-kube-api-access-k8r8m\") pod \"ovnkube-node-rctld\" (UID: \"cc0e0de2-75d5-43a1-b64b-b071427cd8ce\") " pod="openshift-ovn-kubernetes/ovnkube-node-rctld"
Sep 18 19:39:22.619685 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:39:22.619525    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/718b7f76-1cfb-4499-a480-9d5b1567f113-kubelet-dir\") pod \"aws-ebs-csi-driver-node-m4b5p\" (UID: \"718b7f76-1cfb-4499-a480-9d5b1567f113\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-m4b5p"
Sep 18 19:39:22.619685 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:39:22.619556    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/fe7306b4-9adf-4628-804d-a0ff961d5135-etc-tuned\") pod \"tuned-cmv8r\" (UID: \"fe7306b4-9adf-4628-804d-a0ff961d5135\") " pod="openshift-cluster-node-tuning-operator/tuned-cmv8r"
Sep 18 19:39:22.619685 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:39:22.619602    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3eb3cd73-4ea1-4950-8bfd-82b5fa60fd09-metrics-certs\") pod \"network-metrics-daemon-4njfl\" (UID: \"3eb3cd73-4ea1-4950-8bfd-82b5fa60fd09\") " pod="openshift-multus/network-metrics-daemon-4njfl"
Sep 18 19:39:22.619685 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:39:22.619661    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/fe7306b4-9adf-4628-804d-a0ff961d5135-lib-modules\") pod \"tuned-cmv8r\" (UID: \"fe7306b4-9adf-4628-804d-a0ff961d5135\") " pod="openshift-cluster-node-tuning-operator/tuned-cmv8r"
Sep 18 19:39:22.619915 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:39:22.619716    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/e34af56e-8a55-46f2-96b1-a81fbf4effb4-system-cni-dir\") pod \"multus-w27m9\" (UID: \"e34af56e-8a55-46f2-96b1-a81fbf4effb4\") " pod="openshift-multus/multus-w27m9"
Sep 18 19:39:22.619915 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:39:22.619748    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/e34af56e-8a55-46f2-96b1-a81fbf4effb4-host-run-netns\") pod \"multus-w27m9\" (UID: \"e34af56e-8a55-46f2-96b1-a81fbf4effb4\") " pod="openshift-multus/multus-w27m9"
Sep 18 19:39:22.619915 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:39:22.619773    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/eb928dda-649a-48de-841d-06bcb524d0d4-host-slash\") pod \"iptables-alerter-fpnxg\" (UID: \"eb928dda-649a-48de-841d-06bcb524d0d4\") " pod="openshift-network-operator/iptables-alerter-fpnxg"
Sep 18 19:39:22.619915 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:39:22.619807    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/fe7306b4-9adf-4628-804d-a0ff961d5135-lib-modules\") pod \"tuned-cmv8r\" (UID: \"fe7306b4-9adf-4628-804d-a0ff961d5135\") " pod="openshift-cluster-node-tuning-operator/tuned-cmv8r"
Sep 18 19:39:22.619915 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:39:22.619815    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/cc0e0de2-75d5-43a1-b64b-b071427cd8ce-run-ovn\") pod \"ovnkube-node-rctld\" (UID: \"cc0e0de2-75d5-43a1-b64b-b071427cd8ce\") " pod="openshift-ovn-kubernetes/ovnkube-node-rctld"
Sep 18 19:39:22.619915 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:39:22.619838    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/718b7f76-1cfb-4499-a480-9d5b1567f113-registration-dir\") pod \"aws-ebs-csi-driver-node-m4b5p\" (UID: \"718b7f76-1cfb-4499-a480-9d5b1567f113\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-m4b5p"
Sep 18 19:39:22.619915 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:39:22.619862    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/e34af56e-8a55-46f2-96b1-a81fbf4effb4-host-run-netns\") pod \"multus-w27m9\" (UID: \"e34af56e-8a55-46f2-96b1-a81fbf4effb4\") " pod="openshift-multus/multus-w27m9"
Sep 18 19:39:22.619915 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:39:22.619864    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9fq7b\" (UniqueName: \"kubernetes.io/projected/718b7f76-1cfb-4499-a480-9d5b1567f113-kube-api-access-9fq7b\") pod \"aws-ebs-csi-driver-node-m4b5p\" (UID: \"718b7f76-1cfb-4499-a480-9d5b1567f113\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-m4b5p"
Sep 18 19:39:22.620287 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:39:22.619916    2585 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Sep 18 19:39:22.620287 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:39:22.619927    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/e34af56e-8a55-46f2-96b1-a81fbf4effb4-system-cni-dir\") pod \"multus-w27m9\" (UID: \"e34af56e-8a55-46f2-96b1-a81fbf4effb4\") " pod="openshift-multus/multus-w27m9"
Sep 18 19:39:22.620287 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:39:22.619937    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/fe7306b4-9adf-4628-804d-a0ff961d5135-etc-sysconfig\") pod \"tuned-cmv8r\" (UID: \"fe7306b4-9adf-4628-804d-a0ff961d5135\") " pod="openshift-cluster-node-tuning-operator/tuned-cmv8r"
Sep 18 19:39:22.620287 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:39:22.619990    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/fe7306b4-9adf-4628-804d-a0ff961d5135-etc-kubernetes\") pod \"tuned-cmv8r\" (UID: \"fe7306b4-9adf-4628-804d-a0ff961d5135\") " pod="openshift-cluster-node-tuning-operator/tuned-cmv8r"
Sep 18 19:39:22.620287 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:39:22.620013    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/fe7306b4-9adf-4628-804d-a0ff961d5135-sys\") pod \"tuned-cmv8r\" (UID: \"fe7306b4-9adf-4628-804d-a0ff961d5135\") " pod="openshift-cluster-node-tuning-operator/tuned-cmv8r"
Sep 18 19:39:22.620287 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:39:22.620017    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/fe7306b4-9adf-4628-804d-a0ff961d5135-etc-sysconfig\") pod \"tuned-cmv8r\" (UID: \"fe7306b4-9adf-4628-804d-a0ff961d5135\") " pod="openshift-cluster-node-tuning-operator/tuned-cmv8r"
Sep 18 19:39:22.620287 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:39:22.620030    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/fe7306b4-9adf-4628-804d-a0ff961d5135-host\") pod \"tuned-cmv8r\" (UID: \"fe7306b4-9adf-4628-804d-a0ff961d5135\") " pod="openshift-cluster-node-tuning-operator/tuned-cmv8r"
Sep 18 19:39:22.620287 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:39:22.620071    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/fe7306b4-9adf-4628-804d-a0ff961d5135-etc-kubernetes\") pod \"tuned-cmv8r\" (UID: \"fe7306b4-9adf-4628-804d-a0ff961d5135\") " pod="openshift-cluster-node-tuning-operator/tuned-cmv8r"
Sep 18 19:39:22.620287 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:39:22.620073    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/fe7306b4-9adf-4628-804d-a0ff961d5135-host\") pod \"tuned-cmv8r\" (UID: \"fe7306b4-9adf-4628-804d-a0ff961d5135\") " pod="openshift-cluster-node-tuning-operator/tuned-cmv8r"
Sep 18 19:39:22.620287 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:39:22.620072    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/fe7306b4-9adf-4628-804d-a0ff961d5135-sys\") pod \"tuned-cmv8r\" (UID: \"fe7306b4-9adf-4628-804d-a0ff961d5135\") " pod="openshift-cluster-node-tuning-operator/tuned-cmv8r"
Sep 18 19:39:22.620287 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:39:22.620074    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/e34af56e-8a55-46f2-96b1-a81fbf4effb4-hostroot\") pod \"multus-w27m9\" (UID: \"e34af56e-8a55-46f2-96b1-a81fbf4effb4\") " pod="openshift-multus/multus-w27m9"
Sep 18 19:39:22.620287 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:39:22.620114    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/cc0e0de2-75d5-43a1-b64b-b071427cd8ce-etc-openvswitch\") pod \"ovnkube-node-rctld\" (UID: \"cc0e0de2-75d5-43a1-b64b-b071427cd8ce\") " pod="openshift-ovn-kubernetes/ovnkube-node-rctld"
Sep 18 19:39:22.620287 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:39:22.620140    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/cc0e0de2-75d5-43a1-b64b-b071427cd8ce-node-log\") pod \"ovnkube-node-rctld\" (UID: \"cc0e0de2-75d5-43a1-b64b-b071427cd8ce\") " pod="openshift-ovn-kubernetes/ovnkube-node-rctld"
Sep 18 19:39:22.620287 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:39:22.620117    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/e34af56e-8a55-46f2-96b1-a81fbf4effb4-hostroot\") pod \"multus-w27m9\" (UID: \"e34af56e-8a55-46f2-96b1-a81fbf4effb4\") " pod="openshift-multus/multus-w27m9"
Sep 18 19:39:22.620287 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:39:22.620167    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/e34af56e-8a55-46f2-96b1-a81fbf4effb4-multus-conf-dir\") pod \"multus-w27m9\" (UID: \"e34af56e-8a55-46f2-96b1-a81fbf4effb4\") " pod="openshift-multus/multus-w27m9"
Sep 18 19:39:22.620287 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:39:22.620193    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/e34af56e-8a55-46f2-96b1-a81fbf4effb4-multus-daemon-config\") pod \"multus-w27m9\" (UID: \"e34af56e-8a55-46f2-96b1-a81fbf4effb4\") " pod="openshift-multus/multus-w27m9"
Sep 18 19:39:22.620287 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:39:22.620216    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/d2fa3583-1483-4960-a44b-694496d13f45-agent-certs\") pod \"konnectivity-agent-mbhzp\" (UID: \"d2fa3583-1483-4960-a44b-694496d13f45\") " pod="kube-system/konnectivity-agent-mbhzp"
Sep 18 19:39:22.620287 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:39:22.620225    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/e34af56e-8a55-46f2-96b1-a81fbf4effb4-multus-conf-dir\") pod \"multus-w27m9\" (UID: \"e34af56e-8a55-46f2-96b1-a81fbf4effb4\") " pod="openshift-multus/multus-w27m9"
Sep 18 19:39:22.621263 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:39:22.620243    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/cc0e0de2-75d5-43a1-b64b-b071427cd8ce-host-run-ovn-kubernetes\") pod \"ovnkube-node-rctld\" (UID: \"cc0e0de2-75d5-43a1-b64b-b071427cd8ce\") " pod="openshift-ovn-kubernetes/ovnkube-node-rctld"
Sep 18 19:39:22.621263 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:39:22.620266    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/cc0e0de2-75d5-43a1-b64b-b071427cd8ce-ovnkube-config\") pod \"ovnkube-node-rctld\" (UID: \"cc0e0de2-75d5-43a1-b64b-b071427cd8ce\") " pod="openshift-ovn-kubernetes/ovnkube-node-rctld"
Sep 18 19:39:22.621263 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:39:22.620300    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/fe7306b4-9adf-4628-804d-a0ff961d5135-var-lib-kubelet\") pod \"tuned-cmv8r\" (UID: \"fe7306b4-9adf-4628-804d-a0ff961d5135\") " pod="openshift-cluster-node-tuning-operator/tuned-cmv8r"
Sep 18 19:39:22.621263 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:39:22.620321    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/3a884214-2df1-4958-a6b9-e3f2c903ae22-host\") pod \"node-ca-llt9z\" (UID: \"3a884214-2df1-4958-a6b9-e3f2c903ae22\") " pod="openshift-image-registry/node-ca-llt9z"
Sep 18 19:39:22.621263 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:39:22.620345    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/e34af56e-8a55-46f2-96b1-a81fbf4effb4-host-var-lib-cni-multus\") pod \"multus-w27m9\" (UID: \"e34af56e-8a55-46f2-96b1-a81fbf4effb4\") " pod="openshift-multus/multus-w27m9"
Sep 18 19:39:22.621263 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:39:22.620370    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/17b9ee59-c638-43de-8ec6-9544968d07f5-system-cni-dir\") pod \"multus-additional-cni-plugins-x7kdg\" (UID: \"17b9ee59-c638-43de-8ec6-9544968d07f5\") " pod="openshift-multus/multus-additional-cni-plugins-x7kdg"
Sep 18 19:39:22.621263 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:39:22.620389    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/fe7306b4-9adf-4628-804d-a0ff961d5135-var-lib-kubelet\") pod \"tuned-cmv8r\" (UID: \"fe7306b4-9adf-4628-804d-a0ff961d5135\") " pod="openshift-cluster-node-tuning-operator/tuned-cmv8r"
Sep 18 19:39:22.621263 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:39:22.620395    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/17b9ee59-c638-43de-8ec6-9544968d07f5-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-x7kdg\" (UID: \"17b9ee59-c638-43de-8ec6-9544968d07f5\") " pod="openshift-multus/multus-additional-cni-plugins-x7kdg"
Sep 18 19:39:22.621263 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:39:22.620443    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/cc0e0de2-75d5-43a1-b64b-b071427cd8ce-host-cni-netd\") pod \"ovnkube-node-rctld\" (UID: \"cc0e0de2-75d5-43a1-b64b-b071427cd8ce\") " pod="openshift-ovn-kubernetes/ovnkube-node-rctld"
Sep 18 19:39:22.621263 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:39:22.620470    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/cc0e0de2-75d5-43a1-b64b-b071427cd8ce-ovnkube-script-lib\") pod \"ovnkube-node-rctld\" (UID: \"cc0e0de2-75d5-43a1-b64b-b071427cd8ce\") " pod="openshift-ovn-kubernetes/ovnkube-node-rctld"
Sep 18 19:39:22.621263 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:39:22.620498    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jnwc7\" (UniqueName: \"kubernetes.io/projected/bdda9ef7-2f71-4b51-84a9-56da43dd64d9-kube-api-access-jnwc7\") pod \"node-resolver-ltq6c\" (UID: \"bdda9ef7-2f71-4b51-84a9-56da43dd64d9\") " pod="openshift-dns/node-resolver-ltq6c"
Sep 18 19:39:22.621263 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:39:22.620526    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/fe7306b4-9adf-4628-804d-a0ff961d5135-etc-sysctl-conf\") pod \"tuned-cmv8r\" (UID: \"fe7306b4-9adf-4628-804d-a0ff961d5135\") " pod="openshift-cluster-node-tuning-operator/tuned-cmv8r"
Sep 18 19:39:22.621263 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:39:22.620549    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xc47f\" (UniqueName: \"kubernetes.io/projected/3a884214-2df1-4958-a6b9-e3f2c903ae22-kube-api-access-xc47f\") pod \"node-ca-llt9z\" (UID: \"3a884214-2df1-4958-a6b9-e3f2c903ae22\") " pod="openshift-image-registry/node-ca-llt9z"
Sep 18 19:39:22.621263 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:39:22.620572    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/e34af56e-8a55-46f2-96b1-a81fbf4effb4-host-var-lib-cni-bin\") pod \"multus-w27m9\" (UID: \"e34af56e-8a55-46f2-96b1-a81fbf4effb4\") " pod="openshift-multus/multus-w27m9"
Sep 18 19:39:22.621263 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:39:22.620599    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/cc0e0de2-75d5-43a1-b64b-b071427cd8ce-log-socket\") pod \"ovnkube-node-rctld\" (UID: \"cc0e0de2-75d5-43a1-b64b-b071427cd8ce\") " pod="openshift-ovn-kubernetes/ovnkube-node-rctld"
Sep 18 19:39:22.621263 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:39:22.620620    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/cc0e0de2-75d5-43a1-b64b-b071427cd8ce-env-overrides\") pod \"ovnkube-node-rctld\" (UID: \"cc0e0de2-75d5-43a1-b64b-b071427cd8ce\") " pod="openshift-ovn-kubernetes/ovnkube-node-rctld"
Sep 18 19:39:22.621263 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:39:22.620645    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/fe7306b4-9adf-4628-804d-a0ff961d5135-tmp\") pod \"tuned-cmv8r\" (UID: \"fe7306b4-9adf-4628-804d-a0ff961d5135\") " pod="openshift-cluster-node-tuning-operator/tuned-cmv8r"
Sep 18 19:39:22.622105 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:39:22.620671    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6bfnn\" (UniqueName: \"kubernetes.io/projected/13f14c60-b4bd-4146-b4ee-0dd90be01349-kube-api-access-6bfnn\") pod \"network-check-target-zxh4h\" (UID: \"13f14c60-b4bd-4146-b4ee-0dd90be01349\") " pod="openshift-network-diagnostics/network-check-target-zxh4h"
Sep 18 19:39:22.622105 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:39:22.620700    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/e34af56e-8a55-46f2-96b1-a81fbf4effb4-host-var-lib-kubelet\") pod \"multus-w27m9\" (UID: \"e34af56e-8a55-46f2-96b1-a81fbf4effb4\") " pod="openshift-multus/multus-w27m9"
Sep 18 19:39:22.622105 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:39:22.620739    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/17b9ee59-c638-43de-8ec6-9544968d07f5-tuning-conf-dir\") pod \"multus-additional-cni-plugins-x7kdg\" (UID: \"17b9ee59-c638-43de-8ec6-9544968d07f5\") " pod="openshift-multus/multus-additional-cni-plugins-x7kdg"
Sep 18 19:39:22.622105 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:39:22.620763    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/cc0e0de2-75d5-43a1-b64b-b071427cd8ce-ovn-node-metrics-cert\") pod \"ovnkube-node-rctld\" (UID: \"cc0e0de2-75d5-43a1-b64b-b071427cd8ce\") " pod="openshift-ovn-kubernetes/ovnkube-node-rctld"
Sep 18 19:39:22.622105 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:39:22.620802    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/bdda9ef7-2f71-4b51-84a9-56da43dd64d9-hosts-file\") pod \"node-resolver-ltq6c\" (UID: \"bdda9ef7-2f71-4b51-84a9-56da43dd64d9\") " pod="openshift-dns/node-resolver-ltq6c"
Sep 18 19:39:22.622105 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:39:22.620828    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/fe7306b4-9adf-4628-804d-a0ff961d5135-etc-sysctl-d\") pod \"tuned-cmv8r\" (UID: \"fe7306b4-9adf-4628-804d-a0ff961d5135\") " pod="openshift-cluster-node-tuning-operator/tuned-cmv8r"
Sep 18 19:39:22.622105 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:39:22.620828    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/e34af56e-8a55-46f2-96b1-a81fbf4effb4-multus-daemon-config\") pod \"multus-w27m9\" (UID: \"e34af56e-8a55-46f2-96b1-a81fbf4effb4\") " pod="openshift-multus/multus-w27m9"
Sep 18 19:39:22.622105 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:39:22.620850    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/e34af56e-8a55-46f2-96b1-a81fbf4effb4-host-run-k8s-cni-cncf-io\") pod \"multus-w27m9\" (UID: \"e34af56e-8a55-46f2-96b1-a81fbf4effb4\") " pod="openshift-multus/multus-w27m9"
Sep 18 19:39:22.622105 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:39:22.620874    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e34af56e-8a55-46f2-96b1-a81fbf4effb4-etc-kubernetes\") pod \"multus-w27m9\" (UID: \"e34af56e-8a55-46f2-96b1-a81fbf4effb4\") " pod="openshift-multus/multus-w27m9"
Sep 18 19:39:22.622105 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:39:22.620897    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/7e47b713-e1a2-48ff-b961-ad41e706a225-kubelet-config\") pod \"global-pull-secret-syncer-npsrh\" (UID: \"7e47b713-e1a2-48ff-b961-ad41e706a225\") " pod="kube-system/global-pull-secret-syncer-npsrh"
Sep 18 19:39:22.622105 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:39:22.620904    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/e34af56e-8a55-46f2-96b1-a81fbf4effb4-host-var-lib-cni-multus\") pod \"multus-w27m9\" (UID: \"e34af56e-8a55-46f2-96b1-a81fbf4effb4\") " pod="openshift-multus/multus-w27m9"
Sep 18 19:39:22.622105 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:39:22.620925    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/17b9ee59-c638-43de-8ec6-9544968d07f5-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-x7kdg\" (UID: \"17b9ee59-c638-43de-8ec6-9544968d07f5\") " pod="openshift-multus/multus-additional-cni-plugins-x7kdg"
Sep 18 19:39:22.622105 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:39:22.620943    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/3a884214-2df1-4958-a6b9-e3f2c903ae22-host\") pod \"node-ca-llt9z\" (UID: \"3a884214-2df1-4958-a6b9-e3f2c903ae22\") " pod="openshift-image-registry/node-ca-llt9z"
Sep 18 19:39:22.622105 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:39:22.620952    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b5dph\" (UniqueName: \"kubernetes.io/projected/eb928dda-649a-48de-841d-06bcb524d0d4-kube-api-access-b5dph\") pod \"iptables-alerter-fpnxg\" (UID: \"eb928dda-649a-48de-841d-06bcb524d0d4\") " pod="openshift-network-operator/iptables-alerter-fpnxg"
Sep 18 19:39:22.622105 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:39:22.620964    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/17b9ee59-c638-43de-8ec6-9544968d07f5-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-x7kdg\" (UID: \"17b9ee59-c638-43de-8ec6-9544968d07f5\") " pod="openshift-multus/multus-additional-cni-plugins-x7kdg"
Sep 18 19:39:22.622105 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:39:22.620988    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/17b9ee59-c638-43de-8ec6-9544968d07f5-system-cni-dir\") pod \"multus-additional-cni-plugins-x7kdg\" (UID: \"17b9ee59-c638-43de-8ec6-9544968d07f5\") " pod="openshift-multus/multus-additional-cni-plugins-x7kdg"
Sep 18 19:39:22.622105 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:39:22.621016    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/718b7f76-1cfb-4499-a480-9d5b1567f113-device-dir\") pod \"aws-ebs-csi-driver-node-m4b5p\" (UID: \"718b7f76-1cfb-4499-a480-9d5b1567f113\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-m4b5p"
Sep 18 19:39:22.622900 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:39:22.621037    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/3a884214-2df1-4958-a6b9-e3f2c903ae22-serviceca\") pod \"node-ca-llt9z\" (UID: \"3a884214-2df1-4958-a6b9-e3f2c903ae22\") " pod="openshift-image-registry/node-ca-llt9z"
Sep 18 19:39:22.622900 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:39:22.621056    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/e34af56e-8a55-46f2-96b1-a81fbf4effb4-host-run-multus-certs\") pod \"multus-w27m9\" (UID: \"e34af56e-8a55-46f2-96b1-a81fbf4effb4\") " pod="openshift-multus/multus-w27m9"
Sep 18 19:39:22.622900 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:39:22.621080    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/cc0e0de2-75d5-43a1-b64b-b071427cd8ce-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-rctld\" (UID: \"cc0e0de2-75d5-43a1-b64b-b071427cd8ce\") " pod="openshift-ovn-kubernetes/ovnkube-node-rctld"
Sep 18 19:39:22.622900 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:39:22.621101    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bdda9ef7-2f71-4b51-84a9-56da43dd64d9-tmp-dir\") pod \"node-resolver-ltq6c\" (UID: \"bdda9ef7-2f71-4b51-84a9-56da43dd64d9\") " pod="openshift-dns/node-resolver-ltq6c"
Sep 18 19:39:22.622900 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:39:22.621121    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h849s\" (UniqueName: \"kubernetes.io/projected/e34af56e-8a55-46f2-96b1-a81fbf4effb4-kube-api-access-h849s\") pod \"multus-w27m9\" (UID: \"e34af56e-8a55-46f2-96b1-a81fbf4effb4\") " pod="openshift-multus/multus-w27m9"
Sep 18 19:39:22.622900 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:39:22.621143    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/718b7f76-1cfb-4499-a480-9d5b1567f113-socket-dir\") pod \"aws-ebs-csi-driver-node-m4b5p\" (UID: \"718b7f76-1cfb-4499-a480-9d5b1567f113\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-m4b5p"
Sep 18 19:39:22.622900 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:39:22.621153    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/fe7306b4-9adf-4628-804d-a0ff961d5135-etc-sysctl-conf\") pod \"tuned-cmv8r\" (UID: \"fe7306b4-9adf-4628-804d-a0ff961d5135\") " pod="openshift-cluster-node-tuning-operator/tuned-cmv8r"
Sep 18 19:39:22.622900 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:39:22.621166    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/718b7f76-1cfb-4499-a480-9d5b1567f113-etc-selinux\") pod \"aws-ebs-csi-driver-node-m4b5p\" (UID: \"718b7f76-1cfb-4499-a480-9d5b1567f113\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-m4b5p"
Sep 18 19:39:22.622900 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:39:22.621184    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/17b9ee59-c638-43de-8ec6-9544968d07f5-tuning-conf-dir\") pod \"multus-additional-cni-plugins-x7kdg\" (UID: \"17b9ee59-c638-43de-8ec6-9544968d07f5\") " pod="openshift-multus/multus-additional-cni-plugins-x7kdg"
Sep 18 19:39:22.622900 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:39:22.621257    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/fe7306b4-9adf-4628-804d-a0ff961d5135-etc-sysctl-d\") pod \"tuned-cmv8r\" (UID: \"fe7306b4-9adf-4628-804d-a0ff961d5135\") " pod="openshift-cluster-node-tuning-operator/tuned-cmv8r"
Sep 18 19:39:22.622900 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:39:22.621346    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/e34af56e-8a55-46f2-96b1-a81fbf4effb4-host-var-lib-cni-bin\") pod \"multus-w27m9\" (UID: \"e34af56e-8a55-46f2-96b1-a81fbf4effb4\") " pod="openshift-multus/multus-w27m9"
Sep 18 19:39:22.622900 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:39:22.621350    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e34af56e-8a55-46f2-96b1-a81fbf4effb4-etc-kubernetes\") pod \"multus-w27m9\" (UID: \"e34af56e-8a55-46f2-96b1-a81fbf4effb4\") " pod="openshift-multus/multus-w27m9"
Sep 18 19:39:22.622900 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:39:22.621391    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/e34af56e-8a55-46f2-96b1-a81fbf4effb4-host-run-multus-certs\") pod \"multus-w27m9\" (UID: \"e34af56e-8a55-46f2-96b1-a81fbf4effb4\") " pod="openshift-multus/multus-w27m9"
Sep 18 19:39:22.622900 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:39:22.621403    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/718b7f76-1cfb-4499-a480-9d5b1567f113-sys-fs\") pod \"aws-ebs-csi-driver-node-m4b5p\" (UID: \"718b7f76-1cfb-4499-a480-9d5b1567f113\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-m4b5p"
Sep 18 19:39:22.622900 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:39:22.621434    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vxt9v\" (UniqueName: \"kubernetes.io/projected/fe7306b4-9adf-4628-804d-a0ff961d5135-kube-api-access-vxt9v\") pod \"tuned-cmv8r\" (UID: \"fe7306b4-9adf-4628-804d-a0ff961d5135\") " pod="openshift-cluster-node-tuning-operator/tuned-cmv8r"
Sep 18 19:39:22.622900 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:39:22.621475    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sp49h\" (UniqueName: \"kubernetes.io/projected/17b9ee59-c638-43de-8ec6-9544968d07f5-kube-api-access-sp49h\") pod \"multus-additional-cni-plugins-x7kdg\" (UID: \"17b9ee59-c638-43de-8ec6-9544968d07f5\") " pod="openshift-multus/multus-additional-cni-plugins-x7kdg"
Sep 18 19:39:22.622900 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:39:22.621504    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/cc0e0de2-75d5-43a1-b64b-b071427cd8ce-systemd-units\") pod \"ovnkube-node-rctld\" (UID: \"cc0e0de2-75d5-43a1-b64b-b071427cd8ce\") " pod="openshift-ovn-kubernetes/ovnkube-node-rctld"
Sep 18 19:39:22.623685 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:39:22.621532    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/fe7306b4-9adf-4628-804d-a0ff961d5135-run\") pod \"tuned-cmv8r\" (UID: \"fe7306b4-9adf-4628-804d-a0ff961d5135\") " pod="openshift-cluster-node-tuning-operator/tuned-cmv8r"
Sep 18 19:39:22.623685 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:39:22.621573    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/d2fa3583-1483-4960-a44b-694496d13f45-konnectivity-ca\") pod \"konnectivity-agent-mbhzp\" (UID: \"d2fa3583-1483-4960-a44b-694496d13f45\") " pod="kube-system/konnectivity-agent-mbhzp"
Sep 18 19:39:22.623685 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:39:22.621600    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/eb928dda-649a-48de-841d-06bcb524d0d4-iptables-alerter-script\") pod \"iptables-alerter-fpnxg\" (UID: \"eb928dda-649a-48de-841d-06bcb524d0d4\") " pod="openshift-network-operator/iptables-alerter-fpnxg"
Sep 18 19:39:22.623685 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:39:22.621642    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/cc0e0de2-75d5-43a1-b64b-b071427cd8ce-host-slash\") pod \"ovnkube-node-rctld\" (UID: \"cc0e0de2-75d5-43a1-b64b-b071427cd8ce\") " pod="openshift-ovn-kubernetes/ovnkube-node-rctld"
Sep 18 19:39:22.623685 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:39:22.621667    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/cc0e0de2-75d5-43a1-b64b-b071427cd8ce-var-lib-openvswitch\") pod \"ovnkube-node-rctld\" (UID: \"cc0e0de2-75d5-43a1-b64b-b071427cd8ce\") " pod="openshift-ovn-kubernetes/ovnkube-node-rctld"
Sep 18 19:39:22.623685 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:39:22.621679    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/e34af56e-8a55-46f2-96b1-a81fbf4effb4-host-run-k8s-cni-cncf-io\") pod \"multus-w27m9\" (UID: \"e34af56e-8a55-46f2-96b1-a81fbf4effb4\") " pod="openshift-multus/multus-w27m9"
Sep 18 19:39:22.623685 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:39:22.621691    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/7e47b713-e1a2-48ff-b961-ad41e706a225-original-pull-secret\") pod \"global-pull-secret-syncer-npsrh\" (UID: \"7e47b713-e1a2-48ff-b961-ad41e706a225\") " pod="kube-system/global-pull-secret-syncer-npsrh"
Sep 18 19:39:22.623685 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:39:22.621718    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/e34af56e-8a55-46f2-96b1-a81fbf4effb4-host-var-lib-kubelet\") pod \"multus-w27m9\" (UID: \"e34af56e-8a55-46f2-96b1-a81fbf4effb4\") " pod="openshift-multus/multus-w27m9"
Sep 18 19:39:22.623685 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:39:22.621643    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/3a884214-2df1-4958-a6b9-e3f2c903ae22-serviceca\") pod \"node-ca-llt9z\" (UID: \"3a884214-2df1-4958-a6b9-e3f2c903ae22\") " pod="openshift-image-registry/node-ca-llt9z"
Sep 18 19:39:22.623685 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:39:22.622005    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/e34af56e-8a55-46f2-96b1-a81fbf4effb4-multus-cni-dir\") pod \"multus-w27m9\" (UID: \"e34af56e-8a55-46f2-96b1-a81fbf4effb4\") " pod="openshift-multus/multus-w27m9"
Sep 18 19:39:22.623685 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:39:22.622041    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/17b9ee59-c638-43de-8ec6-9544968d07f5-cnibin\") pod \"multus-additional-cni-plugins-x7kdg\" (UID: \"17b9ee59-c638-43de-8ec6-9544968d07f5\") " pod="openshift-multus/multus-additional-cni-plugins-x7kdg"
Sep 18 19:39:22.623685 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:39:22.622067    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/cc0e0de2-75d5-43a1-b64b-b071427cd8ce-host-kubelet\") pod \"ovnkube-node-rctld\" (UID: \"cc0e0de2-75d5-43a1-b64b-b071427cd8ce\") " pod="openshift-ovn-kubernetes/ovnkube-node-rctld"
Sep 18 19:39:22.623685 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:39:22.622091    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/cc0e0de2-75d5-43a1-b64b-b071427cd8ce-host-run-netns\") pod \"ovnkube-node-rctld\" (UID: \"cc0e0de2-75d5-43a1-b64b-b071427cd8ce\") " pod="openshift-ovn-kubernetes/ovnkube-node-rctld"
Sep 18 19:39:22.623685 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:39:22.622118    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/cc0e0de2-75d5-43a1-b64b-b071427cd8ce-host-cni-bin\") pod \"ovnkube-node-rctld\" (UID: \"cc0e0de2-75d5-43a1-b64b-b071427cd8ce\") " pod="openshift-ovn-kubernetes/ovnkube-node-rctld"
Sep 18 19:39:22.623685 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:39:22.622178    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/fe7306b4-9adf-4628-804d-a0ff961d5135-etc-modprobe-d\") pod \"tuned-cmv8r\" (UID: \"fe7306b4-9adf-4628-804d-a0ff961d5135\") " pod="openshift-cluster-node-tuning-operator/tuned-cmv8r"
Sep 18 19:39:22.623685 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:39:22.622206    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/e34af56e-8a55-46f2-96b1-a81fbf4effb4-cnibin\") pod \"multus-w27m9\" (UID: \"e34af56e-8a55-46f2-96b1-a81fbf4effb4\") " pod="openshift-multus/multus-w27m9"
Sep 18 19:39:22.623685 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:39:22.622232    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/e34af56e-8a55-46f2-96b1-a81fbf4effb4-cni-binary-copy\") pod \"multus-w27m9\" (UID: \"e34af56e-8a55-46f2-96b1-a81fbf4effb4\") " pod="openshift-multus/multus-w27m9"
Sep 18 19:39:22.624515 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:39:22.622257    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/cc0e0de2-75d5-43a1-b64b-b071427cd8ce-run-systemd\") pod \"ovnkube-node-rctld\" (UID: \"cc0e0de2-75d5-43a1-b64b-b071427cd8ce\") " pod="openshift-ovn-kubernetes/ovnkube-node-rctld"
Sep 18 19:39:22.624515 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:39:22.622282    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/cc0e0de2-75d5-43a1-b64b-b071427cd8ce-run-openvswitch\") pod \"ovnkube-node-rctld\" (UID: \"cc0e0de2-75d5-43a1-b64b-b071427cd8ce\") " pod="openshift-ovn-kubernetes/ovnkube-node-rctld"
Sep 18 19:39:22.624515 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:39:22.622313    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/7e47b713-e1a2-48ff-b961-ad41e706a225-dbus\") pod \"global-pull-secret-syncer-npsrh\" (UID: \"7e47b713-e1a2-48ff-b961-ad41e706a225\") " pod="kube-system/global-pull-secret-syncer-npsrh"
Sep 18 19:39:22.624515 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:39:22.622339    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dl9pt\" (UniqueName: \"kubernetes.io/projected/3eb3cd73-4ea1-4950-8bfd-82b5fa60fd09-kube-api-access-dl9pt\") pod \"network-metrics-daemon-4njfl\" (UID: \"3eb3cd73-4ea1-4950-8bfd-82b5fa60fd09\") " pod="openshift-multus/network-metrics-daemon-4njfl"
Sep 18 19:39:22.624515 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:39:22.622368    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/fe7306b4-9adf-4628-804d-a0ff961d5135-etc-systemd\") pod \"tuned-cmv8r\" (UID: \"fe7306b4-9adf-4628-804d-a0ff961d5135\") " pod="openshift-cluster-node-tuning-operator/tuned-cmv8r"
Sep 18 19:39:22.624515 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:39:22.622394    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/e34af56e-8a55-46f2-96b1-a81fbf4effb4-multus-socket-dir-parent\") pod \"multus-w27m9\" (UID: \"e34af56e-8a55-46f2-96b1-a81fbf4effb4\") " pod="openshift-multus/multus-w27m9"
Sep 18 19:39:22.624515 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:39:22.622423    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/17b9ee59-c638-43de-8ec6-9544968d07f5-cni-binary-copy\") pod \"multus-additional-cni-plugins-x7kdg\" (UID: \"17b9ee59-c638-43de-8ec6-9544968d07f5\") " pod="openshift-multus/multus-additional-cni-plugins-x7kdg"
Sep 18 19:39:22.624515 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:39:22.622460    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/d2fa3583-1483-4960-a44b-694496d13f45-konnectivity-ca\") pod \"konnectivity-agent-mbhzp\" (UID: \"d2fa3583-1483-4960-a44b-694496d13f45\") " pod="kube-system/konnectivity-agent-mbhzp"
Sep 18 19:39:22.624515 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:39:22.622503    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/17b9ee59-c638-43de-8ec6-9544968d07f5-cnibin\") pod \"multus-additional-cni-plugins-x7kdg\" (UID: \"17b9ee59-c638-43de-8ec6-9544968d07f5\") " pod="openshift-multus/multus-additional-cni-plugins-x7kdg"
Sep 18 19:39:22.624515 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:39:22.622509    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/e34af56e-8a55-46f2-96b1-a81fbf4effb4-cnibin\") pod \"multus-w27m9\" (UID: \"e34af56e-8a55-46f2-96b1-a81fbf4effb4\") " pod="openshift-multus/multus-w27m9"
Sep 18 19:39:22.624515 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:39:22.622521    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/e34af56e-8a55-46f2-96b1-a81fbf4effb4-multus-cni-dir\") pod \"multus-w27m9\" (UID: \"e34af56e-8a55-46f2-96b1-a81fbf4effb4\") " pod="openshift-multus/multus-w27m9"
Sep 18 19:39:22.624515 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:39:22.622582    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/fe7306b4-9adf-4628-804d-a0ff961d5135-etc-modprobe-d\") pod \"tuned-cmv8r\" (UID: \"fe7306b4-9adf-4628-804d-a0ff961d5135\") " pod="openshift-cluster-node-tuning-operator/tuned-cmv8r"
Sep 18 19:39:22.624515 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:39:22.622709    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/e34af56e-8a55-46f2-96b1-a81fbf4effb4-multus-socket-dir-parent\") pod \"multus-w27m9\" (UID: \"e34af56e-8a55-46f2-96b1-a81fbf4effb4\") " pod="openshift-multus/multus-w27m9"
Sep 18 19:39:22.624515 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:39:22.622742    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/fe7306b4-9adf-4628-804d-a0ff961d5135-etc-systemd\") pod \"tuned-cmv8r\" (UID: \"fe7306b4-9adf-4628-804d-a0ff961d5135\") " pod="openshift-cluster-node-tuning-operator/tuned-cmv8r"
Sep 18 19:39:22.624515 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:39:22.622818    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/fe7306b4-9adf-4628-804d-a0ff961d5135-run\") pod \"tuned-cmv8r\" (UID: \"fe7306b4-9adf-4628-804d-a0ff961d5135\") " pod="openshift-cluster-node-tuning-operator/tuned-cmv8r"
Sep 18 19:39:22.624515 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:39:22.622964    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/17b9ee59-c638-43de-8ec6-9544968d07f5-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-x7kdg\" (UID: \"17b9ee59-c638-43de-8ec6-9544968d07f5\") " pod="openshift-multus/multus-additional-cni-plugins-x7kdg"
Sep 18 19:39:22.624515 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:39:22.623024    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/e34af56e-8a55-46f2-96b1-a81fbf4effb4-cni-binary-copy\") pod \"multus-w27m9\" (UID: \"e34af56e-8a55-46f2-96b1-a81fbf4effb4\") " pod="openshift-multus/multus-w27m9"
Sep 18 19:39:22.625391 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:39:22.623291    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/17b9ee59-c638-43de-8ec6-9544968d07f5-cni-binary-copy\") pod \"multus-additional-cni-plugins-x7kdg\" (UID: \"17b9ee59-c638-43de-8ec6-9544968d07f5\") " pod="openshift-multus/multus-additional-cni-plugins-x7kdg"
Sep 18 19:39:22.625391 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:39:22.624965    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/fe7306b4-9adf-4628-804d-a0ff961d5135-tmp\") pod \"tuned-cmv8r\" (UID: \"fe7306b4-9adf-4628-804d-a0ff961d5135\") " pod="openshift-cluster-node-tuning-operator/tuned-cmv8r"
Sep 18 19:39:22.625391 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:39:22.624987    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/fe7306b4-9adf-4628-804d-a0ff961d5135-etc-tuned\") pod \"tuned-cmv8r\" (UID: \"fe7306b4-9adf-4628-804d-a0ff961d5135\") " pod="openshift-cluster-node-tuning-operator/tuned-cmv8r"
Sep 18 19:39:22.625724 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:39:22.625703    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/d2fa3583-1483-4960-a44b-694496d13f45-agent-certs\") pod \"konnectivity-agent-mbhzp\" (UID: \"d2fa3583-1483-4960-a44b-694496d13f45\") " pod="kube-system/konnectivity-agent-mbhzp"
Sep 18 19:39:22.632291 ip-10-0-136-185 kubenswrapper[2585]: E0918 19:39:22.632268    2585 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 18 19:39:22.632432 ip-10-0-136-185 kubenswrapper[2585]: E0918 19:39:22.632296    2585 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 18 19:39:22.632432 ip-10-0-136-185 kubenswrapper[2585]: E0918 19:39:22.632311    2585 projected.go:196] Error preparing data for projected volume kube-api-access-6bfnn for pod openshift-network-diagnostics/network-check-target-zxh4h: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 18 19:39:22.632432 ip-10-0-136-185 kubenswrapper[2585]: E0918 19:39:22.632420    2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/13f14c60-b4bd-4146-b4ee-0dd90be01349-kube-api-access-6bfnn podName:13f14c60-b4bd-4146-b4ee-0dd90be01349 nodeName:}" failed. No retries permitted until 2026-09-18 19:39:23.13236611 +0000 UTC m=+4.656971534 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-6bfnn" (UniqueName: "kubernetes.io/projected/13f14c60-b4bd-4146-b4ee-0dd90be01349-kube-api-access-6bfnn") pod "network-check-target-zxh4h" (UID: "13f14c60-b4bd-4146-b4ee-0dd90be01349") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 18 19:39:22.634674 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:39:22.634644    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xc47f\" (UniqueName: \"kubernetes.io/projected/3a884214-2df1-4958-a6b9-e3f2c903ae22-kube-api-access-xc47f\") pod \"node-ca-llt9z\" (UID: \"3a884214-2df1-4958-a6b9-e3f2c903ae22\") " pod="openshift-image-registry/node-ca-llt9z"
Sep 18 19:39:22.635138 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:39:22.635117    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sp49h\" (UniqueName: \"kubernetes.io/projected/17b9ee59-c638-43de-8ec6-9544968d07f5-kube-api-access-sp49h\") pod \"multus-additional-cni-plugins-x7kdg\" (UID: \"17b9ee59-c638-43de-8ec6-9544968d07f5\") " pod="openshift-multus/multus-additional-cni-plugins-x7kdg"
Sep 18 19:39:22.635271 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:39:22.635145    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-h849s\" (UniqueName: \"kubernetes.io/projected/e34af56e-8a55-46f2-96b1-a81fbf4effb4-kube-api-access-h849s\") pod \"multus-w27m9\" (UID: \"e34af56e-8a55-46f2-96b1-a81fbf4effb4\") " pod="openshift-multus/multus-w27m9"
Sep 18 19:39:22.635271 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:39:22.635187    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vxt9v\" (UniqueName: \"kubernetes.io/projected/fe7306b4-9adf-4628-804d-a0ff961d5135-kube-api-access-vxt9v\") pod \"tuned-cmv8r\" (UID: \"fe7306b4-9adf-4628-804d-a0ff961d5135\") " pod="openshift-cluster-node-tuning-operator/tuned-cmv8r"
Sep 18 19:39:22.659967 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:39:22.659916    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-185.ec2.internal" event={"ID":"acc0abc3eedde3c5f89486311f9adfed","Type":"ContainerStarted","Data":"27eae363456f8d8e55fcb30d27a16c429bda3f5115091e35516129123f28fd38"}
Sep 18 19:39:22.661154 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:39:22.661122    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-136-185.ec2.internal" event={"ID":"5989c800ad96c7a3440a3290994469f4","Type":"ContainerStarted","Data":"9fb563f2f03d9a1fbea6693832f235c93ac8c76742e6a5db82967c4b21641854"}
Sep 18 19:39:22.723062 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:39:22.723029    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/7e47b713-e1a2-48ff-b961-ad41e706a225-kubelet-config\") pod \"global-pull-secret-syncer-npsrh\" (UID: \"7e47b713-e1a2-48ff-b961-ad41e706a225\") " pod="kube-system/global-pull-secret-syncer-npsrh"
Sep 18 19:39:22.723062 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:39:22.723070    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b5dph\" (UniqueName: \"kubernetes.io/projected/eb928dda-649a-48de-841d-06bcb524d0d4-kube-api-access-b5dph\") pod \"iptables-alerter-fpnxg\" (UID: \"eb928dda-649a-48de-841d-06bcb524d0d4\") " pod="openshift-network-operator/iptables-alerter-fpnxg"
Sep 18 19:39:22.723306 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:39:22.723093    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/718b7f76-1cfb-4499-a480-9d5b1567f113-device-dir\") pod \"aws-ebs-csi-driver-node-m4b5p\" (UID: \"718b7f76-1cfb-4499-a480-9d5b1567f113\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-m4b5p"
Sep 18 19:39:22.723306 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:39:22.723117    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/cc0e0de2-75d5-43a1-b64b-b071427cd8ce-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-rctld\" (UID: \"cc0e0de2-75d5-43a1-b64b-b071427cd8ce\") " pod="openshift-ovn-kubernetes/ovnkube-node-rctld"
Sep 18 19:39:22.723306 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:39:22.723157    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bdda9ef7-2f71-4b51-84a9-56da43dd64d9-tmp-dir\") pod \"node-resolver-ltq6c\" (UID: \"bdda9ef7-2f71-4b51-84a9-56da43dd64d9\") " pod="openshift-dns/node-resolver-ltq6c"
Sep 18 19:39:22.723306 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:39:22.723174    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/718b7f76-1cfb-4499-a480-9d5b1567f113-device-dir\") pod \"aws-ebs-csi-driver-node-m4b5p\" (UID: \"718b7f76-1cfb-4499-a480-9d5b1567f113\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-m4b5p"
Sep 18 19:39:22.723306 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:39:22.723179    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/718b7f76-1cfb-4499-a480-9d5b1567f113-socket-dir\") pod \"aws-ebs-csi-driver-node-m4b5p\" (UID: \"718b7f76-1cfb-4499-a480-9d5b1567f113\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-m4b5p"
Sep 18 19:39:22.723306 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:39:22.723220    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/cc0e0de2-75d5-43a1-b64b-b071427cd8ce-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-rctld\" (UID: \"cc0e0de2-75d5-43a1-b64b-b071427cd8ce\") " pod="openshift-ovn-kubernetes/ovnkube-node-rctld"
Sep 18 19:39:22.723306 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:39:22.723240    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/718b7f76-1cfb-4499-a480-9d5b1567f113-etc-selinux\") pod \"aws-ebs-csi-driver-node-m4b5p\" (UID: \"718b7f76-1cfb-4499-a480-9d5b1567f113\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-m4b5p"
Sep 18 19:39:22.723306 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:39:22.723175    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/7e47b713-e1a2-48ff-b961-ad41e706a225-kubelet-config\") pod \"global-pull-secret-syncer-npsrh\" (UID: \"7e47b713-e1a2-48ff-b961-ad41e706a225\") " pod="kube-system/global-pull-secret-syncer-npsrh"
Sep 18 19:39:22.723306 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:39:22.723264    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/718b7f76-1cfb-4499-a480-9d5b1567f113-sys-fs\") pod \"aws-ebs-csi-driver-node-m4b5p\" (UID: \"718b7f76-1cfb-4499-a480-9d5b1567f113\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-m4b5p"
Sep 18 19:39:22.723306 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:39:22.723302    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/cc0e0de2-75d5-43a1-b64b-b071427cd8ce-systemd-units\") pod \"ovnkube-node-rctld\" (UID: \"cc0e0de2-75d5-43a1-b64b-b071427cd8ce\") " pod="openshift-ovn-kubernetes/ovnkube-node-rctld"
Sep 18 19:39:22.723767 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:39:22.723316    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/718b7f76-1cfb-4499-a480-9d5b1567f113-sys-fs\") pod \"aws-ebs-csi-driver-node-m4b5p\" (UID: \"718b7f76-1cfb-4499-a480-9d5b1567f113\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-m4b5p"
Sep 18 19:39:22.723767 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:39:22.723341    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/eb928dda-649a-48de-841d-06bcb524d0d4-iptables-alerter-script\") pod \"iptables-alerter-fpnxg\" (UID: \"eb928dda-649a-48de-841d-06bcb524d0d4\") " pod="openshift-network-operator/iptables-alerter-fpnxg"
Sep 18 19:39:22.723767 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:39:22.723345    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/718b7f76-1cfb-4499-a480-9d5b1567f113-socket-dir\") pod \"aws-ebs-csi-driver-node-m4b5p\" (UID: \"718b7f76-1cfb-4499-a480-9d5b1567f113\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-m4b5p"
Sep 18 19:39:22.723767 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:39:22.723346    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/718b7f76-1cfb-4499-a480-9d5b1567f113-etc-selinux\") pod \"aws-ebs-csi-driver-node-m4b5p\" (UID: \"718b7f76-1cfb-4499-a480-9d5b1567f113\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-m4b5p"
Sep 18 19:39:22.723767 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:39:22.723360    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/cc0e0de2-75d5-43a1-b64b-b071427cd8ce-host-slash\") pod \"ovnkube-node-rctld\" (UID: \"cc0e0de2-75d5-43a1-b64b-b071427cd8ce\") " pod="openshift-ovn-kubernetes/ovnkube-node-rctld"
Sep 18 19:39:22.723767 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:39:22.723375    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/cc0e0de2-75d5-43a1-b64b-b071427cd8ce-systemd-units\") pod \"ovnkube-node-rctld\" (UID: \"cc0e0de2-75d5-43a1-b64b-b071427cd8ce\") " pod="openshift-ovn-kubernetes/ovnkube-node-rctld"
Sep 18 19:39:22.723767 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:39:22.723393    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/cc0e0de2-75d5-43a1-b64b-b071427cd8ce-var-lib-openvswitch\") pod \"ovnkube-node-rctld\" (UID: \"cc0e0de2-75d5-43a1-b64b-b071427cd8ce\") " pod="openshift-ovn-kubernetes/ovnkube-node-rctld"
Sep 18 19:39:22.723767 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:39:22.723421    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/7e47b713-e1a2-48ff-b961-ad41e706a225-original-pull-secret\") pod \"global-pull-secret-syncer-npsrh\" (UID: \"7e47b713-e1a2-48ff-b961-ad41e706a225\") " pod="kube-system/global-pull-secret-syncer-npsrh"
Sep 18 19:39:22.723767 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:39:22.723424    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/cc0e0de2-75d5-43a1-b64b-b071427cd8ce-host-slash\") pod \"ovnkube-node-rctld\" (UID: \"cc0e0de2-75d5-43a1-b64b-b071427cd8ce\") " pod="openshift-ovn-kubernetes/ovnkube-node-rctld"
Sep 18 19:39:22.723767 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:39:22.723460    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/cc0e0de2-75d5-43a1-b64b-b071427cd8ce-host-kubelet\") pod \"ovnkube-node-rctld\" (UID: \"cc0e0de2-75d5-43a1-b64b-b071427cd8ce\") " pod="openshift-ovn-kubernetes/ovnkube-node-rctld"
Sep 18 19:39:22.723767 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:39:22.723486    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bdda9ef7-2f71-4b51-84a9-56da43dd64d9-tmp-dir\") pod \"node-resolver-ltq6c\" (UID: \"bdda9ef7-2f71-4b51-84a9-56da43dd64d9\") " pod="openshift-dns/node-resolver-ltq6c"
Sep 18 19:39:22.723767 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:39:22.723498    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/cc0e0de2-75d5-43a1-b64b-b071427cd8ce-host-kubelet\") pod \"ovnkube-node-rctld\" (UID: \"cc0e0de2-75d5-43a1-b64b-b071427cd8ce\") " pod="openshift-ovn-kubernetes/ovnkube-node-rctld"
Sep 18 19:39:22.723767 ip-10-0-136-185 kubenswrapper[2585]: E0918 19:39:22.723506    2585 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 18 19:39:22.723767 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:39:22.723533    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/cc0e0de2-75d5-43a1-b64b-b071427cd8ce-var-lib-openvswitch\") pod \"ovnkube-node-rctld\" (UID: \"cc0e0de2-75d5-43a1-b64b-b071427cd8ce\") " pod="openshift-ovn-kubernetes/ovnkube-node-rctld"
Sep 18 19:39:22.723767 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:39:22.723556    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/cc0e0de2-75d5-43a1-b64b-b071427cd8ce-host-run-netns\") pod \"ovnkube-node-rctld\" (UID: \"cc0e0de2-75d5-43a1-b64b-b071427cd8ce\") " pod="openshift-ovn-kubernetes/ovnkube-node-rctld"
Sep 18 19:39:22.723767 ip-10-0-136-185 kubenswrapper[2585]: E0918 19:39:22.723575    2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7e47b713-e1a2-48ff-b961-ad41e706a225-original-pull-secret podName:7e47b713-e1a2-48ff-b961-ad41e706a225 nodeName:}" failed. No retries permitted until 2026-09-18 19:39:23.223555741 +0000 UTC m=+4.748161156 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/7e47b713-e1a2-48ff-b961-ad41e706a225-original-pull-secret") pod "global-pull-secret-syncer-npsrh" (UID: "7e47b713-e1a2-48ff-b961-ad41e706a225") : object "kube-system"/"original-pull-secret" not registered
Sep 18 19:39:22.723767 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:39:22.723586    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/cc0e0de2-75d5-43a1-b64b-b071427cd8ce-host-run-netns\") pod \"ovnkube-node-rctld\" (UID: \"cc0e0de2-75d5-43a1-b64b-b071427cd8ce\") " pod="openshift-ovn-kubernetes/ovnkube-node-rctld"
Sep 18 19:39:22.724592 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:39:22.723601    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/cc0e0de2-75d5-43a1-b64b-b071427cd8ce-host-cni-bin\") pod \"ovnkube-node-rctld\" (UID: \"cc0e0de2-75d5-43a1-b64b-b071427cd8ce\") " pod="openshift-ovn-kubernetes/ovnkube-node-rctld"
Sep 18 19:39:22.724592 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:39:22.723628    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/cc0e0de2-75d5-43a1-b64b-b071427cd8ce-run-systemd\") pod \"ovnkube-node-rctld\" (UID: \"cc0e0de2-75d5-43a1-b64b-b071427cd8ce\") " pod="openshift-ovn-kubernetes/ovnkube-node-rctld"
Sep 18 19:39:22.724592 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:39:22.723652    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/cc0e0de2-75d5-43a1-b64b-b071427cd8ce-run-openvswitch\") pod \"ovnkube-node-rctld\" (UID: \"cc0e0de2-75d5-43a1-b64b-b071427cd8ce\") " pod="openshift-ovn-kubernetes/ovnkube-node-rctld"
Sep 18 19:39:22.724592 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:39:22.723675    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/7e47b713-e1a2-48ff-b961-ad41e706a225-dbus\") pod \"global-pull-secret-syncer-npsrh\" (UID: \"7e47b713-e1a2-48ff-b961-ad41e706a225\") " pod="kube-system/global-pull-secret-syncer-npsrh"
Sep 18 19:39:22.724592 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:39:22.723697    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/cc0e0de2-75d5-43a1-b64b-b071427cd8ce-run-systemd\") pod \"ovnkube-node-rctld\" (UID: \"cc0e0de2-75d5-43a1-b64b-b071427cd8ce\") " pod="openshift-ovn-kubernetes/ovnkube-node-rctld"
Sep 18 19:39:22.724592 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:39:22.723699    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/cc0e0de2-75d5-43a1-b64b-b071427cd8ce-host-cni-bin\") pod \"ovnkube-node-rctld\" (UID: \"cc0e0de2-75d5-43a1-b64b-b071427cd8ce\") " pod="openshift-ovn-kubernetes/ovnkube-node-rctld"
Sep 18 19:39:22.724592 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:39:22.723702    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dl9pt\" (UniqueName: \"kubernetes.io/projected/3eb3cd73-4ea1-4950-8bfd-82b5fa60fd09-kube-api-access-dl9pt\") pod \"network-metrics-daemon-4njfl\" (UID: \"3eb3cd73-4ea1-4950-8bfd-82b5fa60fd09\") " pod="openshift-multus/network-metrics-daemon-4njfl"
Sep 18 19:39:22.724592 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:39:22.723737    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/cc0e0de2-75d5-43a1-b64b-b071427cd8ce-run-openvswitch\") pod \"ovnkube-node-rctld\" (UID: \"cc0e0de2-75d5-43a1-b64b-b071427cd8ce\") " pod="openshift-ovn-kubernetes/ovnkube-node-rctld"
Sep 18 19:39:22.724592 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:39:22.723766    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k8r8m\" (UniqueName: \"kubernetes.io/projected/cc0e0de2-75d5-43a1-b64b-b071427cd8ce-kube-api-access-k8r8m\") pod \"ovnkube-node-rctld\" (UID: \"cc0e0de2-75d5-43a1-b64b-b071427cd8ce\") " pod="openshift-ovn-kubernetes/ovnkube-node-rctld"
Sep 18 19:39:22.724592 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:39:22.723810    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/718b7f76-1cfb-4499-a480-9d5b1567f113-kubelet-dir\") pod \"aws-ebs-csi-driver-node-m4b5p\" (UID: \"718b7f76-1cfb-4499-a480-9d5b1567f113\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-m4b5p"
Sep 18 19:39:22.724592 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:39:22.723836    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3eb3cd73-4ea1-4950-8bfd-82b5fa60fd09-metrics-certs\") pod \"network-metrics-daemon-4njfl\" (UID: \"3eb3cd73-4ea1-4950-8bfd-82b5fa60fd09\") " pod="openshift-multus/network-metrics-daemon-4njfl"
Sep 18 19:39:22.724592 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:39:22.723862    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/eb928dda-649a-48de-841d-06bcb524d0d4-host-slash\") pod \"iptables-alerter-fpnxg\" (UID: \"eb928dda-649a-48de-841d-06bcb524d0d4\") " pod="openshift-network-operator/iptables-alerter-fpnxg"
Sep 18 19:39:22.724592 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:39:22.723864    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/7e47b713-e1a2-48ff-b961-ad41e706a225-dbus\") pod \"global-pull-secret-syncer-npsrh\" (UID: \"7e47b713-e1a2-48ff-b961-ad41e706a225\") " pod="kube-system/global-pull-secret-syncer-npsrh"
Sep 18 19:39:22.724592 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:39:22.723904    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/718b7f76-1cfb-4499-a480-9d5b1567f113-kubelet-dir\") pod \"aws-ebs-csi-driver-node-m4b5p\" (UID: \"718b7f76-1cfb-4499-a480-9d5b1567f113\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-m4b5p"
Sep 18 19:39:22.724592 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:39:22.723889    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/eb928dda-649a-48de-841d-06bcb524d0d4-iptables-alerter-script\") pod \"iptables-alerter-fpnxg\" (UID: \"eb928dda-649a-48de-841d-06bcb524d0d4\") " pod="openshift-network-operator/iptables-alerter-fpnxg"
Sep 18 19:39:22.724592 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:39:22.723950    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/eb928dda-649a-48de-841d-06bcb524d0d4-host-slash\") pod \"iptables-alerter-fpnxg\" (UID: \"eb928dda-649a-48de-841d-06bcb524d0d4\") " pod="openshift-network-operator/iptables-alerter-fpnxg"
Sep 18 19:39:22.724592 ip-10-0-136-185 kubenswrapper[2585]: E0918 19:39:22.723963    2585 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 18 19:39:22.724592 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:39:22.723968    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/cc0e0de2-75d5-43a1-b64b-b071427cd8ce-run-ovn\") pod \"ovnkube-node-rctld\" (UID: \"cc0e0de2-75d5-43a1-b64b-b071427cd8ce\") " pod="openshift-ovn-kubernetes/ovnkube-node-rctld"
Sep 18 19:39:22.725360 ip-10-0-136-185 kubenswrapper[2585]: E0918 19:39:22.724010    2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3eb3cd73-4ea1-4950-8bfd-82b5fa60fd09-metrics-certs podName:3eb3cd73-4ea1-4950-8bfd-82b5fa60fd09 nodeName:}" failed. No retries permitted until 2026-09-18 19:39:23.223993997 +0000 UTC m=+4.748599413 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/3eb3cd73-4ea1-4950-8bfd-82b5fa60fd09-metrics-certs") pod "network-metrics-daemon-4njfl" (UID: "3eb3cd73-4ea1-4950-8bfd-82b5fa60fd09") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 18 19:39:22.725360 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:39:22.724016    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/cc0e0de2-75d5-43a1-b64b-b071427cd8ce-run-ovn\") pod \"ovnkube-node-rctld\" (UID: \"cc0e0de2-75d5-43a1-b64b-b071427cd8ce\") " pod="openshift-ovn-kubernetes/ovnkube-node-rctld"
Sep 18 19:39:22.725360 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:39:22.724028    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/718b7f76-1cfb-4499-a480-9d5b1567f113-registration-dir\") pod \"aws-ebs-csi-driver-node-m4b5p\" (UID: \"718b7f76-1cfb-4499-a480-9d5b1567f113\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-m4b5p"
Sep 18 19:39:22.725360 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:39:22.724055    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9fq7b\" (UniqueName: \"kubernetes.io/projected/718b7f76-1cfb-4499-a480-9d5b1567f113-kube-api-access-9fq7b\") pod \"aws-ebs-csi-driver-node-m4b5p\" (UID: \"718b7f76-1cfb-4499-a480-9d5b1567f113\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-m4b5p"
Sep 18 19:39:22.725360 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:39:22.724091    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/cc0e0de2-75d5-43a1-b64b-b071427cd8ce-etc-openvswitch\") pod \"ovnkube-node-rctld\" (UID: \"cc0e0de2-75d5-43a1-b64b-b071427cd8ce\") " pod="openshift-ovn-kubernetes/ovnkube-node-rctld"
Sep 18 19:39:22.725360 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:39:22.724059    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/718b7f76-1cfb-4499-a480-9d5b1567f113-registration-dir\") pod \"aws-ebs-csi-driver-node-m4b5p\" (UID: \"718b7f76-1cfb-4499-a480-9d5b1567f113\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-m4b5p"
Sep 18 19:39:22.725360 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:39:22.724114    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/cc0e0de2-75d5-43a1-b64b-b071427cd8ce-node-log\") pod \"ovnkube-node-rctld\" (UID: \"cc0e0de2-75d5-43a1-b64b-b071427cd8ce\") " pod="openshift-ovn-kubernetes/ovnkube-node-rctld"
Sep 18 19:39:22.725360 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:39:22.724155    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/cc0e0de2-75d5-43a1-b64b-b071427cd8ce-host-run-ovn-kubernetes\") pod \"ovnkube-node-rctld\" (UID: \"cc0e0de2-75d5-43a1-b64b-b071427cd8ce\") " pod="openshift-ovn-kubernetes/ovnkube-node-rctld"
Sep 18 19:39:22.725360 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:39:22.724179    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/cc0e0de2-75d5-43a1-b64b-b071427cd8ce-ovnkube-config\") pod \"ovnkube-node-rctld\" (UID: \"cc0e0de2-75d5-43a1-b64b-b071427cd8ce\") " pod="openshift-ovn-kubernetes/ovnkube-node-rctld"
Sep 18 19:39:22.725360 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:39:22.724193    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/cc0e0de2-75d5-43a1-b64b-b071427cd8ce-etc-openvswitch\") pod \"ovnkube-node-rctld\" (UID: \"cc0e0de2-75d5-43a1-b64b-b071427cd8ce\") " pod="openshift-ovn-kubernetes/ovnkube-node-rctld"
Sep 18 19:39:22.725360 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:39:22.724208    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/cc0e0de2-75d5-43a1-b64b-b071427cd8ce-host-cni-netd\") pod \"ovnkube-node-rctld\" (UID: \"cc0e0de2-75d5-43a1-b64b-b071427cd8ce\") " pod="openshift-ovn-kubernetes/ovnkube-node-rctld"
Sep 18 19:39:22.725360 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:39:22.724243    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/cc0e0de2-75d5-43a1-b64b-b071427cd8ce-host-run-ovn-kubernetes\") pod \"ovnkube-node-rctld\" (UID: \"cc0e0de2-75d5-43a1-b64b-b071427cd8ce\") " pod="openshift-ovn-kubernetes/ovnkube-node-rctld"
Sep 18 19:39:22.725360 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:39:22.724241    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/cc0e0de2-75d5-43a1-b64b-b071427cd8ce-node-log\") pod \"ovnkube-node-rctld\" (UID: \"cc0e0de2-75d5-43a1-b64b-b071427cd8ce\") " pod="openshift-ovn-kubernetes/ovnkube-node-rctld"
Sep 18 19:39:22.725360 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:39:22.724258    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/cc0e0de2-75d5-43a1-b64b-b071427cd8ce-host-cni-netd\") pod \"ovnkube-node-rctld\" (UID: \"cc0e0de2-75d5-43a1-b64b-b071427cd8ce\") " pod="openshift-ovn-kubernetes/ovnkube-node-rctld"
Sep 18 19:39:22.725360 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:39:22.724270    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/cc0e0de2-75d5-43a1-b64b-b071427cd8ce-ovnkube-script-lib\") pod \"ovnkube-node-rctld\" (UID: \"cc0e0de2-75d5-43a1-b64b-b071427cd8ce\") " pod="openshift-ovn-kubernetes/ovnkube-node-rctld"
Sep 18 19:39:22.725360 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:39:22.724293    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jnwc7\" (UniqueName: \"kubernetes.io/projected/bdda9ef7-2f71-4b51-84a9-56da43dd64d9-kube-api-access-jnwc7\") pod \"node-resolver-ltq6c\" (UID: \"bdda9ef7-2f71-4b51-84a9-56da43dd64d9\") " pod="openshift-dns/node-resolver-ltq6c"
Sep 18 19:39:22.725910 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:39:22.724311    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/cc0e0de2-75d5-43a1-b64b-b071427cd8ce-log-socket\") pod \"ovnkube-node-rctld\" (UID: \"cc0e0de2-75d5-43a1-b64b-b071427cd8ce\") " pod="openshift-ovn-kubernetes/ovnkube-node-rctld"
Sep 18 19:39:22.725910 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:39:22.724327    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/cc0e0de2-75d5-43a1-b64b-b071427cd8ce-env-overrides\") pod \"ovnkube-node-rctld\" (UID: \"cc0e0de2-75d5-43a1-b64b-b071427cd8ce\") " pod="openshift-ovn-kubernetes/ovnkube-node-rctld"
Sep 18 19:39:22.725910 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:39:22.724351    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/cc0e0de2-75d5-43a1-b64b-b071427cd8ce-ovn-node-metrics-cert\") pod \"ovnkube-node-rctld\" (UID: \"cc0e0de2-75d5-43a1-b64b-b071427cd8ce\") " pod="openshift-ovn-kubernetes/ovnkube-node-rctld"
Sep 18 19:39:22.725910 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:39:22.724369    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/bdda9ef7-2f71-4b51-84a9-56da43dd64d9-hosts-file\") pod \"node-resolver-ltq6c\" (UID: \"bdda9ef7-2f71-4b51-84a9-56da43dd64d9\") " pod="openshift-dns/node-resolver-ltq6c"
Sep 18 19:39:22.725910 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:39:22.724464    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/bdda9ef7-2f71-4b51-84a9-56da43dd64d9-hosts-file\") pod \"node-resolver-ltq6c\" (UID: \"bdda9ef7-2f71-4b51-84a9-56da43dd64d9\") " pod="openshift-dns/node-resolver-ltq6c"
Sep 18 19:39:22.725910 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:39:22.724856    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/cc0e0de2-75d5-43a1-b64b-b071427cd8ce-ovnkube-config\") pod \"ovnkube-node-rctld\" (UID: \"cc0e0de2-75d5-43a1-b64b-b071427cd8ce\") " pod="openshift-ovn-kubernetes/ovnkube-node-rctld"
Sep 18 19:39:22.725910 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:39:22.724880    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/cc0e0de2-75d5-43a1-b64b-b071427cd8ce-ovnkube-script-lib\") pod \"ovnkube-node-rctld\" (UID: \"cc0e0de2-75d5-43a1-b64b-b071427cd8ce\") " pod="openshift-ovn-kubernetes/ovnkube-node-rctld"
Sep 18 19:39:22.725910 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:39:22.724922    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/cc0e0de2-75d5-43a1-b64b-b071427cd8ce-log-socket\") pod \"ovnkube-node-rctld\" (UID: \"cc0e0de2-75d5-43a1-b64b-b071427cd8ce\") " pod="openshift-ovn-kubernetes/ovnkube-node-rctld"
Sep 18 19:39:22.725910 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:39:22.724991    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/cc0e0de2-75d5-43a1-b64b-b071427cd8ce-env-overrides\") pod \"ovnkube-node-rctld\" (UID: \"cc0e0de2-75d5-43a1-b64b-b071427cd8ce\") " pod="openshift-ovn-kubernetes/ovnkube-node-rctld"
Sep 18 19:39:22.727079 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:39:22.727058    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/cc0e0de2-75d5-43a1-b64b-b071427cd8ce-ovn-node-metrics-cert\") pod \"ovnkube-node-rctld\" (UID: \"cc0e0de2-75d5-43a1-b64b-b071427cd8ce\") " pod="openshift-ovn-kubernetes/ovnkube-node-rctld"
Sep 18 19:39:22.738939 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:39:22.738915    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jnwc7\" (UniqueName: \"kubernetes.io/projected/bdda9ef7-2f71-4b51-84a9-56da43dd64d9-kube-api-access-jnwc7\") pod \"node-resolver-ltq6c\" (UID: \"bdda9ef7-2f71-4b51-84a9-56da43dd64d9\") " pod="openshift-dns/node-resolver-ltq6c"
Sep 18 19:39:22.739177 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:39:22.739156    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-b5dph\" (UniqueName: \"kubernetes.io/projected/eb928dda-649a-48de-841d-06bcb524d0d4-kube-api-access-b5dph\") pod \"iptables-alerter-fpnxg\" (UID: \"eb928dda-649a-48de-841d-06bcb524d0d4\") " pod="openshift-network-operator/iptables-alerter-fpnxg"
Sep 18 19:39:22.739266 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:39:22.739235    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9fq7b\" (UniqueName: \"kubernetes.io/projected/718b7f76-1cfb-4499-a480-9d5b1567f113-kube-api-access-9fq7b\") pod \"aws-ebs-csi-driver-node-m4b5p\" (UID: \"718b7f76-1cfb-4499-a480-9d5b1567f113\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-m4b5p"
Sep 18 19:39:22.739537 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:39:22.739515    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dl9pt\" (UniqueName: \"kubernetes.io/projected/3eb3cd73-4ea1-4950-8bfd-82b5fa60fd09-kube-api-access-dl9pt\") pod \"network-metrics-daemon-4njfl\" (UID: \"3eb3cd73-4ea1-4950-8bfd-82b5fa60fd09\") " pod="openshift-multus/network-metrics-daemon-4njfl"
Sep 18 19:39:22.740372 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:39:22.740352    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-k8r8m\" (UniqueName: \"kubernetes.io/projected/cc0e0de2-75d5-43a1-b64b-b071427cd8ce-kube-api-access-k8r8m\") pod \"ovnkube-node-rctld\" (UID: \"cc0e0de2-75d5-43a1-b64b-b071427cd8ce\") " pod="openshift-ovn-kubernetes/ovnkube-node-rctld"
Sep 18 19:39:22.809358 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:39:22.809286    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-w27m9"
Sep 18 19:39:22.817881 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:39:22.817859    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-mbhzp"
Sep 18 19:39:22.826658 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:39:22.826632    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-x7kdg"
Sep 18 19:39:22.833398 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:39:22.833379    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-cmv8r"
Sep 18 19:39:22.852161 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:39:22.852136    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-llt9z"
Sep 18 19:39:22.860708 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:39:22.860691    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-fpnxg"
Sep 18 19:39:22.868263 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:39:22.868245    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-rctld"
Sep 18 19:39:22.875866 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:39:22.875849    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-m4b5p"
Sep 18 19:39:22.876859 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:39:22.876843    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-ltq6c"
Sep 18 19:39:22.906838 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:39:22.906811    2585 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 18 19:39:23.226541 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:39:23.226509    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/7e47b713-e1a2-48ff-b961-ad41e706a225-original-pull-secret\") pod \"global-pull-secret-syncer-npsrh\" (UID: \"7e47b713-e1a2-48ff-b961-ad41e706a225\") " pod="kube-system/global-pull-secret-syncer-npsrh"
Sep 18 19:39:23.226683 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:39:23.226564    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3eb3cd73-4ea1-4950-8bfd-82b5fa60fd09-metrics-certs\") pod \"network-metrics-daemon-4njfl\" (UID: \"3eb3cd73-4ea1-4950-8bfd-82b5fa60fd09\") " pod="openshift-multus/network-metrics-daemon-4njfl"
Sep 18 19:39:23.226683 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:39:23.226596    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6bfnn\" (UniqueName: \"kubernetes.io/projected/13f14c60-b4bd-4146-b4ee-0dd90be01349-kube-api-access-6bfnn\") pod \"network-check-target-zxh4h\" (UID: \"13f14c60-b4bd-4146-b4ee-0dd90be01349\") " pod="openshift-network-diagnostics/network-check-target-zxh4h"
Sep 18 19:39:23.226683 ip-10-0-136-185 kubenswrapper[2585]: E0918 19:39:23.226618    2585 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 18 19:39:23.226683 ip-10-0-136-185 kubenswrapper[2585]: E0918 19:39:23.226684    2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7e47b713-e1a2-48ff-b961-ad41e706a225-original-pull-secret podName:7e47b713-e1a2-48ff-b961-ad41e706a225 nodeName:}" failed. No retries permitted until 2026-09-18 19:39:24.226669382 +0000 UTC m=+5.751274782 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/7e47b713-e1a2-48ff-b961-ad41e706a225-original-pull-secret") pod "global-pull-secret-syncer-npsrh" (UID: "7e47b713-e1a2-48ff-b961-ad41e706a225") : object "kube-system"/"original-pull-secret" not registered
Sep 18 19:39:23.226840 ip-10-0-136-185 kubenswrapper[2585]: E0918 19:39:23.226694    2585 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 18 19:39:23.226840 ip-10-0-136-185 kubenswrapper[2585]: E0918 19:39:23.226710    2585 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 18 19:39:23.226840 ip-10-0-136-185 kubenswrapper[2585]: E0918 19:39:23.226727    2585 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 18 19:39:23.226840 ip-10-0-136-185 kubenswrapper[2585]: E0918 19:39:23.226738    2585 projected.go:196] Error preparing data for projected volume kube-api-access-6bfnn for pod openshift-network-diagnostics/network-check-target-zxh4h: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 18 19:39:23.226840 ip-10-0-136-185 kubenswrapper[2585]: E0918 19:39:23.226753    2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3eb3cd73-4ea1-4950-8bfd-82b5fa60fd09-metrics-certs podName:3eb3cd73-4ea1-4950-8bfd-82b5fa60fd09 nodeName:}" failed. No retries permitted until 2026-09-18 19:39:24.226736922 +0000 UTC m=+5.751342338 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/3eb3cd73-4ea1-4950-8bfd-82b5fa60fd09-metrics-certs") pod "network-metrics-daemon-4njfl" (UID: "3eb3cd73-4ea1-4950-8bfd-82b5fa60fd09") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 18 19:39:23.226840 ip-10-0-136-185 kubenswrapper[2585]: E0918 19:39:23.226773    2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/13f14c60-b4bd-4146-b4ee-0dd90be01349-kube-api-access-6bfnn podName:13f14c60-b4bd-4146-b4ee-0dd90be01349 nodeName:}" failed. No retries permitted until 2026-09-18 19:39:24.226762168 +0000 UTC m=+5.751367570 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-6bfnn" (UniqueName: "kubernetes.io/projected/13f14c60-b4bd-4146-b4ee-0dd90be01349-kube-api-access-6bfnn") pod "network-check-target-zxh4h" (UID: "13f14c60-b4bd-4146-b4ee-0dd90be01349") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 18 19:39:23.555383 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:39:23.555296    2585 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-17 19:34:22 +0000 UTC" deadline="2028-06-16 17:30:15.519855416 +0000 UTC"
Sep 18 19:39:23.555383 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:39:23.555344    2585 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15285h50m51.964513897s"
Sep 18 19:39:23.596398 ip-10-0-136-185 kubenswrapper[2585]: W0918 19:39:23.596164    2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbdda9ef7_2f71_4b51_84a9_56da43dd64d9.slice/crio-e6d15ac508679a8700c7577af1d88bcf99d00df328a070b138ea41e4db35d855 WatchSource:0}: Error finding container e6d15ac508679a8700c7577af1d88bcf99d00df328a070b138ea41e4db35d855: Status 404 returned error can't find the container with id e6d15ac508679a8700c7577af1d88bcf99d00df328a070b138ea41e4db35d855
Sep 18 19:39:23.598938 ip-10-0-136-185 kubenswrapper[2585]: W0918 19:39:23.598860    2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfe7306b4_9adf_4628_804d_a0ff961d5135.slice/crio-ada47b6c74a800ebfeb9ee37c3cf57a4c926a9fef7e68ccaa4e9f6b8f1043c15 WatchSource:0}: Error finding container ada47b6c74a800ebfeb9ee37c3cf57a4c926a9fef7e68ccaa4e9f6b8f1043c15: Status 404 returned error can't find the container with id ada47b6c74a800ebfeb9ee37c3cf57a4c926a9fef7e68ccaa4e9f6b8f1043c15
Sep 18 19:39:23.607495 ip-10-0-136-185 kubenswrapper[2585]: W0918 19:39:23.607472    2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3a884214_2df1_4958_a6b9_e3f2c903ae22.slice/crio-60185531b5d5bb0fdd1b8a7c513e98684d54e1f84c717bb374c4fe54b838f93f WatchSource:0}: Error finding container 60185531b5d5bb0fdd1b8a7c513e98684d54e1f84c717bb374c4fe54b838f93f: Status 404 returned error can't find the container with id 60185531b5d5bb0fdd1b8a7c513e98684d54e1f84c717bb374c4fe54b838f93f
Sep 18 19:39:23.608013 ip-10-0-136-185 kubenswrapper[2585]: W0918 19:39:23.607990    2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod17b9ee59_c638_43de_8ec6_9544968d07f5.slice/crio-ecc741dee9cea560698f1ba65662549947c147eb11c9b145a095412116aa92a4 WatchSource:0}: Error finding container ecc741dee9cea560698f1ba65662549947c147eb11c9b145a095412116aa92a4: Status 404 returned error can't find the container with id ecc741dee9cea560698f1ba65662549947c147eb11c9b145a095412116aa92a4
Sep 18 19:39:23.609586 ip-10-0-136-185 kubenswrapper[2585]: W0918 19:39:23.609487    2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcc0e0de2_75d5_43a1_b64b_b071427cd8ce.slice/crio-cbfc026cdaa4707a0e26fd84c0c00006fb2e1b7f53387589022546c32100308e WatchSource:0}: Error finding container cbfc026cdaa4707a0e26fd84c0c00006fb2e1b7f53387589022546c32100308e: Status 404 returned error can't find the container with id cbfc026cdaa4707a0e26fd84c0c00006fb2e1b7f53387589022546c32100308e
Sep 18 19:39:23.610534 ip-10-0-136-185 kubenswrapper[2585]: W0918 19:39:23.610431    2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeb928dda_649a_48de_841d_06bcb524d0d4.slice/crio-778e27aa99328c94634ab0fd7cc16b3efbbadaf7f7c0a706e1744269b6b19878 WatchSource:0}: Error finding container 778e27aa99328c94634ab0fd7cc16b3efbbadaf7f7c0a706e1744269b6b19878: Status 404 returned error can't find the container with id 778e27aa99328c94634ab0fd7cc16b3efbbadaf7f7c0a706e1744269b6b19878
Sep 18 19:39:23.612297 ip-10-0-136-185 kubenswrapper[2585]: W0918 19:39:23.612102    2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod718b7f76_1cfb_4499_a480_9d5b1567f113.slice/crio-2fe4e82060d137363a2b34467dddaf7118cca5d3379d9a4ae45e9675e2bef99a WatchSource:0}: Error finding container 2fe4e82060d137363a2b34467dddaf7118cca5d3379d9a4ae45e9675e2bef99a: Status 404 returned error can't find the container with id 2fe4e82060d137363a2b34467dddaf7118cca5d3379d9a4ae45e9675e2bef99a
Sep 18 19:39:23.612475 ip-10-0-136-185 kubenswrapper[2585]: W0918 19:39:23.612455    2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode34af56e_8a55_46f2_96b1_a81fbf4effb4.slice/crio-3dde6fb4ab57722e148c526b9f6264c727b1e05c79d7587b69404101b2b8ca9e WatchSource:0}: Error finding container 3dde6fb4ab57722e148c526b9f6264c727b1e05c79d7587b69404101b2b8ca9e: Status 404 returned error can't find the container with id 3dde6fb4ab57722e148c526b9f6264c727b1e05c79d7587b69404101b2b8ca9e
Sep 18 19:39:23.614208 ip-10-0-136-185 kubenswrapper[2585]: W0918 19:39:23.613748    2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd2fa3583_1483_4960_a44b_694496d13f45.slice/crio-570a712b489710d5bff721ebfe5d6a167b429a4dbc852e82cd54d1d750d7947f WatchSource:0}: Error finding container 570a712b489710d5bff721ebfe5d6a167b429a4dbc852e82cd54d1d750d7947f: Status 404 returned error can't find the container with id 570a712b489710d5bff721ebfe5d6a167b429a4dbc852e82cd54d1d750d7947f
Sep 18 19:39:23.664347 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:39:23.664313    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-fpnxg" event={"ID":"eb928dda-649a-48de-841d-06bcb524d0d4","Type":"ContainerStarted","Data":"778e27aa99328c94634ab0fd7cc16b3efbbadaf7f7c0a706e1744269b6b19878"}
Sep 18 19:39:23.665468 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:39:23.665443    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-m4b5p" event={"ID":"718b7f76-1cfb-4499-a480-9d5b1567f113","Type":"ContainerStarted","Data":"2fe4e82060d137363a2b34467dddaf7118cca5d3379d9a4ae45e9675e2bef99a"}
Sep 18 19:39:23.666991 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:39:23.666961    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-rctld" event={"ID":"cc0e0de2-75d5-43a1-b64b-b071427cd8ce","Type":"ContainerStarted","Data":"cbfc026cdaa4707a0e26fd84c0c00006fb2e1b7f53387589022546c32100308e"}
Sep 18 19:39:23.668137 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:39:23.668115    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-cmv8r" event={"ID":"fe7306b4-9adf-4628-804d-a0ff961d5135","Type":"ContainerStarted","Data":"ada47b6c74a800ebfeb9ee37c3cf57a4c926a9fef7e68ccaa4e9f6b8f1043c15"}
Sep 18 19:39:23.669264 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:39:23.669239    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-ltq6c" event={"ID":"bdda9ef7-2f71-4b51-84a9-56da43dd64d9","Type":"ContainerStarted","Data":"e6d15ac508679a8700c7577af1d88bcf99d00df328a070b138ea41e4db35d855"}
Sep 18 19:39:23.670393 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:39:23.670373    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-mbhzp" event={"ID":"d2fa3583-1483-4960-a44b-694496d13f45","Type":"ContainerStarted","Data":"570a712b489710d5bff721ebfe5d6a167b429a4dbc852e82cd54d1d750d7947f"}
Sep 18 19:39:23.671531 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:39:23.671509    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-w27m9" event={"ID":"e34af56e-8a55-46f2-96b1-a81fbf4effb4","Type":"ContainerStarted","Data":"3dde6fb4ab57722e148c526b9f6264c727b1e05c79d7587b69404101b2b8ca9e"}
Sep 18 19:39:23.672574 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:39:23.672548    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-x7kdg" event={"ID":"17b9ee59-c638-43de-8ec6-9544968d07f5","Type":"ContainerStarted","Data":"ecc741dee9cea560698f1ba65662549947c147eb11c9b145a095412116aa92a4"}
Sep 18 19:39:23.673779 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:39:23.673757    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-llt9z" event={"ID":"3a884214-2df1-4958-a6b9-e3f2c903ae22","Type":"ContainerStarted","Data":"60185531b5d5bb0fdd1b8a7c513e98684d54e1f84c717bb374c4fe54b838f93f"}
Sep 18 19:39:24.235006 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:39:24.234074    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3eb3cd73-4ea1-4950-8bfd-82b5fa60fd09-metrics-certs\") pod \"network-metrics-daemon-4njfl\" (UID: \"3eb3cd73-4ea1-4950-8bfd-82b5fa60fd09\") " pod="openshift-multus/network-metrics-daemon-4njfl"
Sep 18 19:39:24.235006 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:39:24.234137    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6bfnn\" (UniqueName: \"kubernetes.io/projected/13f14c60-b4bd-4146-b4ee-0dd90be01349-kube-api-access-6bfnn\") pod \"network-check-target-zxh4h\" (UID: \"13f14c60-b4bd-4146-b4ee-0dd90be01349\") " pod="openshift-network-diagnostics/network-check-target-zxh4h"
Sep 18 19:39:24.235006 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:39:24.234179    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/7e47b713-e1a2-48ff-b961-ad41e706a225-original-pull-secret\") pod \"global-pull-secret-syncer-npsrh\" (UID: \"7e47b713-e1a2-48ff-b961-ad41e706a225\") " pod="kube-system/global-pull-secret-syncer-npsrh"
Sep 18 19:39:24.235006 ip-10-0-136-185 kubenswrapper[2585]: E0918 19:39:24.234295    2585 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 18 19:39:24.235006 ip-10-0-136-185 kubenswrapper[2585]: E0918 19:39:24.234355    2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7e47b713-e1a2-48ff-b961-ad41e706a225-original-pull-secret podName:7e47b713-e1a2-48ff-b961-ad41e706a225 nodeName:}" failed. No retries permitted until 2026-09-18 19:39:26.234337577 +0000 UTC m=+7.758942984 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/7e47b713-e1a2-48ff-b961-ad41e706a225-original-pull-secret") pod "global-pull-secret-syncer-npsrh" (UID: "7e47b713-e1a2-48ff-b961-ad41e706a225") : object "kube-system"/"original-pull-secret" not registered
Sep 18 19:39:24.235006 ip-10-0-136-185 kubenswrapper[2585]: E0918 19:39:24.234744    2585 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 18 19:39:24.235006 ip-10-0-136-185 kubenswrapper[2585]: E0918 19:39:24.234816    2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3eb3cd73-4ea1-4950-8bfd-82b5fa60fd09-metrics-certs podName:3eb3cd73-4ea1-4950-8bfd-82b5fa60fd09 nodeName:}" failed. No retries permitted until 2026-09-18 19:39:26.234781252 +0000 UTC m=+7.759386659 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/3eb3cd73-4ea1-4950-8bfd-82b5fa60fd09-metrics-certs") pod "network-metrics-daemon-4njfl" (UID: "3eb3cd73-4ea1-4950-8bfd-82b5fa60fd09") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 18 19:39:24.235006 ip-10-0-136-185 kubenswrapper[2585]: E0918 19:39:24.234904    2585 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 18 19:39:24.235006 ip-10-0-136-185 kubenswrapper[2585]: E0918 19:39:24.234921    2585 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 18 19:39:24.235006 ip-10-0-136-185 kubenswrapper[2585]: E0918 19:39:24.234934    2585 projected.go:196] Error preparing data for projected volume kube-api-access-6bfnn for pod openshift-network-diagnostics/network-check-target-zxh4h: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 18 19:39:24.235006 ip-10-0-136-185 kubenswrapper[2585]: E0918 19:39:24.234968    2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/13f14c60-b4bd-4146-b4ee-0dd90be01349-kube-api-access-6bfnn podName:13f14c60-b4bd-4146-b4ee-0dd90be01349 nodeName:}" failed. No retries permitted until 2026-09-18 19:39:26.234957706 +0000 UTC m=+7.759563111 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-6bfnn" (UniqueName: "kubernetes.io/projected/13f14c60-b4bd-4146-b4ee-0dd90be01349-kube-api-access-6bfnn") pod "network-check-target-zxh4h" (UID: "13f14c60-b4bd-4146-b4ee-0dd90be01349") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 18 19:39:24.557106 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:39:24.556985    2585 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-17 19:34:22 +0000 UTC" deadline="2028-02-23 01:44:12.960743271 +0000 UTC"
Sep 18 19:39:24.557106 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:39:24.557030    2585 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12534h4m48.403719009s"
Sep 18 19:39:24.656934 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:39:24.656903    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zxh4h"
Sep 18 19:39:24.657115 ip-10-0-136-185 kubenswrapper[2585]: E0918 19:39:24.657046    2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zxh4h" podUID="13f14c60-b4bd-4146-b4ee-0dd90be01349"
Sep 18 19:39:24.657927 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:39:24.657479    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-npsrh"
Sep 18 19:39:24.657927 ip-10-0-136-185 kubenswrapper[2585]: E0918 19:39:24.657583    2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-npsrh" podUID="7e47b713-e1a2-48ff-b961-ad41e706a225"
Sep 18 19:39:24.657927 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:39:24.657748    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4njfl"
Sep 18 19:39:24.657927 ip-10-0-136-185 kubenswrapper[2585]: E0918 19:39:24.657858    2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4njfl" podUID="3eb3cd73-4ea1-4950-8bfd-82b5fa60fd09"
Sep 18 19:39:24.688024 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:39:24.687984    2585 generic.go:356] "Generic (PLEG): container finished" podID="acc0abc3eedde3c5f89486311f9adfed" containerID="622c8d4f3f1bb8917757741fb228844c6c35474d2c2961dfe0d9d5e8f6838a35" exitCode=0
Sep 18 19:39:24.688595 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:39:24.688571    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-185.ec2.internal" event={"ID":"acc0abc3eedde3c5f89486311f9adfed","Type":"ContainerDied","Data":"622c8d4f3f1bb8917757741fb228844c6c35474d2c2961dfe0d9d5e8f6838a35"}
Sep 18 19:39:24.719840 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:39:24.719274    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-136-185.ec2.internal" event={"ID":"5989c800ad96c7a3440a3290994469f4","Type":"ContainerStarted","Data":"e624ebc3fe36e47b8796f7a5e8e4f1766ad3d0a803d586d4949d8869cc6bed4d"}
Sep 18 19:39:25.729303 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:39:25.729257    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-185.ec2.internal" event={"ID":"acc0abc3eedde3c5f89486311f9adfed","Type":"ContainerStarted","Data":"576731a42069f604e38a53ae25a38472dfa50e7b8d2d6a08a4d71ab14ea9551e"}
Sep 18 19:39:25.747861 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:39:25.746644    2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-185.ec2.internal" podStartSLOduration=3.746622721 podStartE2EDuration="3.746622721s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-18 19:39:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 19:39:25.74645921 +0000 UTC m=+7.271064633" watchObservedRunningTime="2026-09-18 19:39:25.746622721 +0000 UTC m=+7.271228141"
Sep 18 19:39:25.747861 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:39:25.747019    2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-136-185.ec2.internal" podStartSLOduration=3.747010982 podStartE2EDuration="3.747010982s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-18 19:39:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 19:39:24.745075818 +0000 UTC m=+6.269681242" watchObservedRunningTime="2026-09-18 19:39:25.747010982 +0000 UTC m=+7.271616407"
Sep 18 19:39:26.251178 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:39:26.251139    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/7e47b713-e1a2-48ff-b961-ad41e706a225-original-pull-secret\") pod \"global-pull-secret-syncer-npsrh\" (UID: \"7e47b713-e1a2-48ff-b961-ad41e706a225\") " pod="kube-system/global-pull-secret-syncer-npsrh"
Sep 18 19:39:26.251481 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:39:26.251206    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3eb3cd73-4ea1-4950-8bfd-82b5fa60fd09-metrics-certs\") pod \"network-metrics-daemon-4njfl\" (UID: \"3eb3cd73-4ea1-4950-8bfd-82b5fa60fd09\") " pod="openshift-multus/network-metrics-daemon-4njfl"
Sep 18 19:39:26.251481 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:39:26.251253    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6bfnn\" (UniqueName: \"kubernetes.io/projected/13f14c60-b4bd-4146-b4ee-0dd90be01349-kube-api-access-6bfnn\") pod \"network-check-target-zxh4h\" (UID: \"13f14c60-b4bd-4146-b4ee-0dd90be01349\") " pod="openshift-network-diagnostics/network-check-target-zxh4h"
Sep 18 19:39:26.251481 ip-10-0-136-185 kubenswrapper[2585]: E0918 19:39:26.251399    2585 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 18 19:39:26.251481 ip-10-0-136-185 kubenswrapper[2585]: E0918 19:39:26.251430    2585 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 18 19:39:26.251481 ip-10-0-136-185 kubenswrapper[2585]: E0918 19:39:26.251443    2585 projected.go:196] Error preparing data for projected volume kube-api-access-6bfnn for pod openshift-network-diagnostics/network-check-target-zxh4h: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 18 19:39:26.251761 ip-10-0-136-185 kubenswrapper[2585]: E0918 19:39:26.251503    2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/13f14c60-b4bd-4146-b4ee-0dd90be01349-kube-api-access-6bfnn podName:13f14c60-b4bd-4146-b4ee-0dd90be01349 nodeName:}" failed. No retries permitted until 2026-09-18 19:39:30.251484967 +0000 UTC m=+11.776090373 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-6bfnn" (UniqueName: "kubernetes.io/projected/13f14c60-b4bd-4146-b4ee-0dd90be01349-kube-api-access-6bfnn") pod "network-check-target-zxh4h" (UID: "13f14c60-b4bd-4146-b4ee-0dd90be01349") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 18 19:39:26.251982 ip-10-0-136-185 kubenswrapper[2585]: E0918 19:39:26.251927    2585 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 18 19:39:26.251982 ip-10-0-136-185 kubenswrapper[2585]: E0918 19:39:26.251978    2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7e47b713-e1a2-48ff-b961-ad41e706a225-original-pull-secret podName:7e47b713-e1a2-48ff-b961-ad41e706a225 nodeName:}" failed. No retries permitted until 2026-09-18 19:39:30.251963118 +0000 UTC m=+11.776568539 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/7e47b713-e1a2-48ff-b961-ad41e706a225-original-pull-secret") pod "global-pull-secret-syncer-npsrh" (UID: "7e47b713-e1a2-48ff-b961-ad41e706a225") : object "kube-system"/"original-pull-secret" not registered
Sep 18 19:39:26.252148 ip-10-0-136-185 kubenswrapper[2585]: E0918 19:39:26.252034    2585 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 18 19:39:26.252148 ip-10-0-136-185 kubenswrapper[2585]: E0918 19:39:26.252065    2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3eb3cd73-4ea1-4950-8bfd-82b5fa60fd09-metrics-certs podName:3eb3cd73-4ea1-4950-8bfd-82b5fa60fd09 nodeName:}" failed. No retries permitted until 2026-09-18 19:39:30.252054129 +0000 UTC m=+11.776659533 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/3eb3cd73-4ea1-4950-8bfd-82b5fa60fd09-metrics-certs") pod "network-metrics-daemon-4njfl" (UID: "3eb3cd73-4ea1-4950-8bfd-82b5fa60fd09") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 18 19:39:26.655887 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:39:26.655834    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zxh4h"
Sep 18 19:39:26.656065 ip-10-0-136-185 kubenswrapper[2585]: E0918 19:39:26.655965    2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zxh4h" podUID="13f14c60-b4bd-4146-b4ee-0dd90be01349"
Sep 18 19:39:26.656421 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:39:26.656386    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-npsrh"
Sep 18 19:39:26.656528 ip-10-0-136-185 kubenswrapper[2585]: E0918 19:39:26.656481    2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-npsrh" podUID="7e47b713-e1a2-48ff-b961-ad41e706a225"
Sep 18 19:39:26.656588 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:39:26.656552    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4njfl"
Sep 18 19:39:26.656677 ip-10-0-136-185 kubenswrapper[2585]: E0918 19:39:26.656635    2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4njfl" podUID="3eb3cd73-4ea1-4950-8bfd-82b5fa60fd09"
Sep 18 19:39:28.655996 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:39:28.655960    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zxh4h"
Sep 18 19:39:28.656457 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:39:28.655960    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-npsrh"
Sep 18 19:39:28.656457 ip-10-0-136-185 kubenswrapper[2585]: E0918 19:39:28.656092    2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zxh4h" podUID="13f14c60-b4bd-4146-b4ee-0dd90be01349"
Sep 18 19:39:28.656457 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:39:28.656100    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4njfl"
Sep 18 19:39:28.656457 ip-10-0-136-185 kubenswrapper[2585]: E0918 19:39:28.656174    2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-npsrh" podUID="7e47b713-e1a2-48ff-b961-ad41e706a225"
Sep 18 19:39:28.656457 ip-10-0-136-185 kubenswrapper[2585]: E0918 19:39:28.656283    2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4njfl" podUID="3eb3cd73-4ea1-4950-8bfd-82b5fa60fd09"
Sep 18 19:39:30.286259 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:39:30.286216    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3eb3cd73-4ea1-4950-8bfd-82b5fa60fd09-metrics-certs\") pod \"network-metrics-daemon-4njfl\" (UID: \"3eb3cd73-4ea1-4950-8bfd-82b5fa60fd09\") " pod="openshift-multus/network-metrics-daemon-4njfl"
Sep 18 19:39:30.286729 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:39:30.286299    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6bfnn\" (UniqueName: \"kubernetes.io/projected/13f14c60-b4bd-4146-b4ee-0dd90be01349-kube-api-access-6bfnn\") pod \"network-check-target-zxh4h\" (UID: \"13f14c60-b4bd-4146-b4ee-0dd90be01349\") " pod="openshift-network-diagnostics/network-check-target-zxh4h"
Sep 18 19:39:30.286729 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:39:30.286345    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/7e47b713-e1a2-48ff-b961-ad41e706a225-original-pull-secret\") pod \"global-pull-secret-syncer-npsrh\" (UID: \"7e47b713-e1a2-48ff-b961-ad41e706a225\") " pod="kube-system/global-pull-secret-syncer-npsrh"
Sep 18 19:39:30.286729 ip-10-0-136-185 kubenswrapper[2585]: E0918 19:39:30.286428    2585 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 18 19:39:30.286729 ip-10-0-136-185 kubenswrapper[2585]: E0918 19:39:30.286488    2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7e47b713-e1a2-48ff-b961-ad41e706a225-original-pull-secret podName:7e47b713-e1a2-48ff-b961-ad41e706a225 nodeName:}" failed. No retries permitted until 2026-09-18 19:39:38.286470176 +0000 UTC m=+19.811075581 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/7e47b713-e1a2-48ff-b961-ad41e706a225-original-pull-secret") pod "global-pull-secret-syncer-npsrh" (UID: "7e47b713-e1a2-48ff-b961-ad41e706a225") : object "kube-system"/"original-pull-secret" not registered
Sep 18 19:39:30.286729 ip-10-0-136-185 kubenswrapper[2585]: E0918 19:39:30.286426    2585 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 18 19:39:30.286729 ip-10-0-136-185 kubenswrapper[2585]: E0918 19:39:30.286534    2585 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 18 19:39:30.286729 ip-10-0-136-185 kubenswrapper[2585]: E0918 19:39:30.286552    2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3eb3cd73-4ea1-4950-8bfd-82b5fa60fd09-metrics-certs podName:3eb3cd73-4ea1-4950-8bfd-82b5fa60fd09 nodeName:}" failed. No retries permitted until 2026-09-18 19:39:38.286540308 +0000 UTC m=+19.811145728 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/3eb3cd73-4ea1-4950-8bfd-82b5fa60fd09-metrics-certs") pod "network-metrics-daemon-4njfl" (UID: "3eb3cd73-4ea1-4950-8bfd-82b5fa60fd09") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 18 19:39:30.286729 ip-10-0-136-185 kubenswrapper[2585]: E0918 19:39:30.286558    2585 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 18 19:39:30.286729 ip-10-0-136-185 kubenswrapper[2585]: E0918 19:39:30.286567    2585 projected.go:196] Error preparing data for projected volume kube-api-access-6bfnn for pod openshift-network-diagnostics/network-check-target-zxh4h: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 18 19:39:30.286729 ip-10-0-136-185 kubenswrapper[2585]: E0918 19:39:30.286615    2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/13f14c60-b4bd-4146-b4ee-0dd90be01349-kube-api-access-6bfnn podName:13f14c60-b4bd-4146-b4ee-0dd90be01349 nodeName:}" failed. No retries permitted until 2026-09-18 19:39:38.286602201 +0000 UTC m=+19.811207605 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-6bfnn" (UniqueName: "kubernetes.io/projected/13f14c60-b4bd-4146-b4ee-0dd90be01349-kube-api-access-6bfnn") pod "network-check-target-zxh4h" (UID: "13f14c60-b4bd-4146-b4ee-0dd90be01349") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 18 19:39:30.659689 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:39:30.659659    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zxh4h"
Sep 18 19:39:30.659689 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:39:30.659692    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-npsrh"
Sep 18 19:39:30.659954 ip-10-0-136-185 kubenswrapper[2585]: E0918 19:39:30.659804    2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zxh4h" podUID="13f14c60-b4bd-4146-b4ee-0dd90be01349"
Sep 18 19:39:30.659954 ip-10-0-136-185 kubenswrapper[2585]: E0918 19:39:30.659835    2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-npsrh" podUID="7e47b713-e1a2-48ff-b961-ad41e706a225"
Sep 18 19:39:30.659954 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:39:30.659875    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4njfl"
Sep 18 19:39:30.660096 ip-10-0-136-185 kubenswrapper[2585]: E0918 19:39:30.659952    2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4njfl" podUID="3eb3cd73-4ea1-4950-8bfd-82b5fa60fd09"
Sep 18 19:39:32.659739 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:39:32.659397    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zxh4h"
Sep 18 19:39:32.659739 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:39:32.659504    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-npsrh"
Sep 18 19:39:32.659739 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:39:32.659530    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4njfl"
Sep 18 19:39:32.659739 ip-10-0-136-185 kubenswrapper[2585]: E0918 19:39:32.659524    2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zxh4h" podUID="13f14c60-b4bd-4146-b4ee-0dd90be01349"
Sep 18 19:39:32.659739 ip-10-0-136-185 kubenswrapper[2585]: E0918 19:39:32.659604    2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-npsrh" podUID="7e47b713-e1a2-48ff-b961-ad41e706a225"
Sep 18 19:39:32.659739 ip-10-0-136-185 kubenswrapper[2585]: E0918 19:39:32.659689    2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4njfl" podUID="3eb3cd73-4ea1-4950-8bfd-82b5fa60fd09"
Sep 18 19:39:34.656533 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:39:34.656480    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zxh4h"
Sep 18 19:39:34.656533 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:39:34.656517    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4njfl"
Sep 18 19:39:34.656533 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:39:34.656480    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-npsrh"
Sep 18 19:39:34.657127 ip-10-0-136-185 kubenswrapper[2585]: E0918 19:39:34.656641    2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zxh4h" podUID="13f14c60-b4bd-4146-b4ee-0dd90be01349"
Sep 18 19:39:34.657127 ip-10-0-136-185 kubenswrapper[2585]: E0918 19:39:34.656711    2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-npsrh" podUID="7e47b713-e1a2-48ff-b961-ad41e706a225"
Sep 18 19:39:34.657127 ip-10-0-136-185 kubenswrapper[2585]: E0918 19:39:34.656799    2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4njfl" podUID="3eb3cd73-4ea1-4950-8bfd-82b5fa60fd09"
Sep 18 19:39:36.659062 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:39:36.659029    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-npsrh"
Sep 18 19:39:36.659531 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:39:36.659074    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4njfl"
Sep 18 19:39:36.659531 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:39:36.659029    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zxh4h"
Sep 18 19:39:36.659531 ip-10-0-136-185 kubenswrapper[2585]: E0918 19:39:36.659155    2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-npsrh" podUID="7e47b713-e1a2-48ff-b961-ad41e706a225"
Sep 18 19:39:36.659531 ip-10-0-136-185 kubenswrapper[2585]: E0918 19:39:36.659222    2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zxh4h" podUID="13f14c60-b4bd-4146-b4ee-0dd90be01349"
Sep 18 19:39:36.659531 ip-10-0-136-185 kubenswrapper[2585]: E0918 19:39:36.659343    2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4njfl" podUID="3eb3cd73-4ea1-4950-8bfd-82b5fa60fd09"
Sep 18 19:39:38.345361 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:39:38.345325    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6bfnn\" (UniqueName: \"kubernetes.io/projected/13f14c60-b4bd-4146-b4ee-0dd90be01349-kube-api-access-6bfnn\") pod \"network-check-target-zxh4h\" (UID: \"13f14c60-b4bd-4146-b4ee-0dd90be01349\") " pod="openshift-network-diagnostics/network-check-target-zxh4h"
Sep 18 19:39:38.345913 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:39:38.345378    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/7e47b713-e1a2-48ff-b961-ad41e706a225-original-pull-secret\") pod \"global-pull-secret-syncer-npsrh\" (UID: \"7e47b713-e1a2-48ff-b961-ad41e706a225\") " pod="kube-system/global-pull-secret-syncer-npsrh"
Sep 18 19:39:38.345913 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:39:38.345414    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3eb3cd73-4ea1-4950-8bfd-82b5fa60fd09-metrics-certs\") pod \"network-metrics-daemon-4njfl\" (UID: \"3eb3cd73-4ea1-4950-8bfd-82b5fa60fd09\") " pod="openshift-multus/network-metrics-daemon-4njfl"
Sep 18 19:39:38.345913 ip-10-0-136-185 kubenswrapper[2585]: E0918 19:39:38.345507    2585 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 18 19:39:38.345913 ip-10-0-136-185 kubenswrapper[2585]: E0918 19:39:38.345562    2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3eb3cd73-4ea1-4950-8bfd-82b5fa60fd09-metrics-certs podName:3eb3cd73-4ea1-4950-8bfd-82b5fa60fd09 nodeName:}" failed. No retries permitted until 2026-09-18 19:39:54.345547454 +0000 UTC m=+35.870152856 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/3eb3cd73-4ea1-4950-8bfd-82b5fa60fd09-metrics-certs") pod "network-metrics-daemon-4njfl" (UID: "3eb3cd73-4ea1-4950-8bfd-82b5fa60fd09") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 18 19:39:38.345913 ip-10-0-136-185 kubenswrapper[2585]: E0918 19:39:38.345559    2585 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 18 19:39:38.345913 ip-10-0-136-185 kubenswrapper[2585]: E0918 19:39:38.345509    2585 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 18 19:39:38.345913 ip-10-0-136-185 kubenswrapper[2585]: E0918 19:39:38.345600    2585 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 18 19:39:38.345913 ip-10-0-136-185 kubenswrapper[2585]: E0918 19:39:38.345614    2585 projected.go:196] Error preparing data for projected volume kube-api-access-6bfnn for pod openshift-network-diagnostics/network-check-target-zxh4h: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 18 19:39:38.345913 ip-10-0-136-185 kubenswrapper[2585]: E0918 19:39:38.345620    2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7e47b713-e1a2-48ff-b961-ad41e706a225-original-pull-secret podName:7e47b713-e1a2-48ff-b961-ad41e706a225 nodeName:}" failed. No retries permitted until 2026-09-18 19:39:54.345605373 +0000 UTC m=+35.870210776 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/7e47b713-e1a2-48ff-b961-ad41e706a225-original-pull-secret") pod "global-pull-secret-syncer-npsrh" (UID: "7e47b713-e1a2-48ff-b961-ad41e706a225") : object "kube-system"/"original-pull-secret" not registered
Sep 18 19:39:38.345913 ip-10-0-136-185 kubenswrapper[2585]: E0918 19:39:38.345665    2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/13f14c60-b4bd-4146-b4ee-0dd90be01349-kube-api-access-6bfnn podName:13f14c60-b4bd-4146-b4ee-0dd90be01349 nodeName:}" failed. No retries permitted until 2026-09-18 19:39:54.345653696 +0000 UTC m=+35.870259108 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-6bfnn" (UniqueName: "kubernetes.io/projected/13f14c60-b4bd-4146-b4ee-0dd90be01349-kube-api-access-6bfnn") pod "network-check-target-zxh4h" (UID: "13f14c60-b4bd-4146-b4ee-0dd90be01349") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 18 19:39:38.656210 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:39:38.655775    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zxh4h"
Sep 18 19:39:38.656210 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:39:38.655783    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-npsrh"
Sep 18 19:39:38.656210 ip-10-0-136-185 kubenswrapper[2585]: E0918 19:39:38.655891    2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zxh4h" podUID="13f14c60-b4bd-4146-b4ee-0dd90be01349"
Sep 18 19:39:38.656210 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:39:38.655911    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4njfl"
Sep 18 19:39:38.656210 ip-10-0-136-185 kubenswrapper[2585]: E0918 19:39:38.656006    2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-npsrh" podUID="7e47b713-e1a2-48ff-b961-ad41e706a225"
Sep 18 19:39:38.656210 ip-10-0-136-185 kubenswrapper[2585]: E0918 19:39:38.656087    2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4njfl" podUID="3eb3cd73-4ea1-4950-8bfd-82b5fa60fd09"
Sep 18 19:39:40.656525 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:39:40.656492    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-npsrh"
Sep 18 19:39:40.656909 ip-10-0-136-185 kubenswrapper[2585]: E0918 19:39:40.656588    2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-npsrh" podUID="7e47b713-e1a2-48ff-b961-ad41e706a225"
Sep 18 19:39:40.656909 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:39:40.656672    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zxh4h"
Sep 18 19:39:40.656909 ip-10-0-136-185 kubenswrapper[2585]: E0918 19:39:40.656802    2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zxh4h" podUID="13f14c60-b4bd-4146-b4ee-0dd90be01349"
Sep 18 19:39:40.656909 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:39:40.656860    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4njfl"
Sep 18 19:39:40.657059 ip-10-0-136-185 kubenswrapper[2585]: E0918 19:39:40.656948    2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4njfl" podUID="3eb3cd73-4ea1-4950-8bfd-82b5fa60fd09"
Sep 18 19:39:41.762182 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:39:41.761939    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-m4b5p" event={"ID":"718b7f76-1cfb-4499-a480-9d5b1567f113","Type":"ContainerStarted","Data":"49c0315d45b75cbb47861ac4c85c045eec0205b7789dc1bb0ed7d56ef74017c5"}
Sep 18 19:39:41.764713 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:39:41.764691    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-rctld_cc0e0de2-75d5-43a1-b64b-b071427cd8ce/ovn-acl-logging/0.log"
Sep 18 19:39:41.765106 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:39:41.765085    2585 generic.go:356] "Generic (PLEG): container finished" podID="cc0e0de2-75d5-43a1-b64b-b071427cd8ce" containerID="4298c3f50069a32b4301c8fdacbece0e4402ce50b6385af92ddfd25a649edc6b" exitCode=1
Sep 18 19:39:41.765189 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:39:41.765158    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-rctld" event={"ID":"cc0e0de2-75d5-43a1-b64b-b071427cd8ce","Type":"ContainerStarted","Data":"a3a468907d286751f0a853386db79372c670b92c72f686a2cae4b91217db8d3f"}
Sep 18 19:39:41.765268 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:39:41.765250    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-rctld" event={"ID":"cc0e0de2-75d5-43a1-b64b-b071427cd8ce","Type":"ContainerStarted","Data":"af469ec22b9b585aa391f60766fe3d3577f28ec1bae2ede2dd6ec3622be9db15"}
Sep 18 19:39:41.765323 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:39:41.765310    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-rctld" event={"ID":"cc0e0de2-75d5-43a1-b64b-b071427cd8ce","Type":"ContainerStarted","Data":"7a92ea3db2bc319603eb36bbb7b5c353f314862b5a372c3ff12bf52e524734be"}
Sep 18 19:39:41.765372 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:39:41.765360    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-rctld" event={"ID":"cc0e0de2-75d5-43a1-b64b-b071427cd8ce","Type":"ContainerStarted","Data":"008dcef630367946993cd9a99c8615d24c4cd7b2c5413585c2a6a4ebfeb8fc08"}
Sep 18 19:39:41.765416 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:39:41.765401    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-rctld" event={"ID":"cc0e0de2-75d5-43a1-b64b-b071427cd8ce","Type":"ContainerDied","Data":"4298c3f50069a32b4301c8fdacbece0e4402ce50b6385af92ddfd25a649edc6b"}
Sep 18 19:39:41.765466 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:39:41.765448    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-rctld" event={"ID":"cc0e0de2-75d5-43a1-b64b-b071427cd8ce","Type":"ContainerStarted","Data":"b8a217a0e3cefd1b3b889f4cb30968bf9894a42913f25b39c52b270372e25ac7"}
Sep 18 19:39:41.766939 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:39:41.766913    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-cmv8r" event={"ID":"fe7306b4-9adf-4628-804d-a0ff961d5135","Type":"ContainerStarted","Data":"856d7da35ac1499c375d81ff48b2a9e455ca84bf3bece4e8f4b16499c7a77d9a"}
Sep 18 19:39:41.768843 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:39:41.768819    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-ltq6c" event={"ID":"bdda9ef7-2f71-4b51-84a9-56da43dd64d9","Type":"ContainerStarted","Data":"056f571c9c9a6325766bd7be5ee174bf7411bb2a7e51d46de120501bae758509"}
Sep 18 19:39:41.770648 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:39:41.770621    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-mbhzp" event={"ID":"d2fa3583-1483-4960-a44b-694496d13f45","Type":"ContainerStarted","Data":"2ea0dcc83fd9765d8e2c1fe3331b73c462fd6c8f562e42b92c2fb6305888fe20"}
Sep 18 19:39:41.772265 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:39:41.772239    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-w27m9" event={"ID":"e34af56e-8a55-46f2-96b1-a81fbf4effb4","Type":"ContainerStarted","Data":"3870f218bf5938a2634e703c38d64685a39b451b7e4d4675f251762bf01a94fb"}
Sep 18 19:39:41.773804 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:39:41.773765    2585 generic.go:356] "Generic (PLEG): container finished" podID="17b9ee59-c638-43de-8ec6-9544968d07f5" containerID="af77311d612560821bb57eb7c7d9d3372ba7c779f18e1f19af87781ba4059dea" exitCode=0
Sep 18 19:39:41.773905 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:39:41.773823    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-x7kdg" event={"ID":"17b9ee59-c638-43de-8ec6-9544968d07f5","Type":"ContainerDied","Data":"af77311d612560821bb57eb7c7d9d3372ba7c779f18e1f19af87781ba4059dea"}
Sep 18 19:39:41.779736 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:39:41.779561    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-llt9z" event={"ID":"3a884214-2df1-4958-a6b9-e3f2c903ae22","Type":"ContainerStarted","Data":"7da2fe665847d20c23915854b598dcd4e0e0ee6edfbf9b047a5ec16a54356d93"}
Sep 18 19:39:41.797413 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:39:41.797370    2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-cmv8r" podStartSLOduration=4.528629066 podStartE2EDuration="21.79735597s" totalImagesPullingTime="17.268726904s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-18 19:39:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 19:39:41.784335405 +0000 UTC m=+23.308940829" watchObservedRunningTime="2026-09-18 19:39:41.79735597 +0000 UTC m=+23.321961396"
Sep 18 19:39:41.797526 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:39:41.797474    2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-llt9z" podStartSLOduration=4.538460033 podStartE2EDuration="21.797467431s" totalImagesPullingTime="17.259007398s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-18 19:39:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 19:39:41.797176367 +0000 UTC m=+23.321781794" watchObservedRunningTime="2026-09-18 19:39:41.797467431 +0000 UTC m=+23.322072853"
Sep 18 19:39:41.810324 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:39:41.810276    2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-ltq6c" podStartSLOduration=4.539386311 podStartE2EDuration="21.810260392s" totalImagesPullingTime="17.270874081s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-18 19:39:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 19:39:41.809996767 +0000 UTC m=+23.334602195" watchObservedRunningTime="2026-09-18 19:39:41.810260392 +0000 UTC m=+23.334865816"
Sep 18 19:39:41.829772 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:39:41.829722    2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-w27m9" podStartSLOduration=4.552522863 podStartE2EDuration="21.829703891s" totalImagesPullingTime="17.277181028s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-18 19:39:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 19:39:41.828950853 +0000 UTC m=+23.353556278" watchObservedRunningTime="2026-09-18 19:39:41.829703891 +0000 UTC m=+23.354309315"
Sep 18 19:39:41.865889 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:39:41.865839    2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-mbhzp" podStartSLOduration=11.998175965 podStartE2EDuration="21.865821943s" totalImagesPullingTime="9.867645978s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 19:39:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 19:39:41.86565989 +0000 UTC m=+23.390265314" watchObservedRunningTime="2026-09-18 19:39:41.865821943 +0000 UTC m=+23.390427366"
Sep 18 19:39:42.571833 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:39:42.571802    2585 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Sep 18 19:39:42.592612 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:39:42.592507    2585 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-09-18T19:39:42.571829238Z","UUID":"cf0c9a1a-4327-4728-b747-0ef531d11137","Handler":null,"Name":"","Endpoint":""}
Sep 18 19:39:42.594407 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:39:42.594388    2585 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Sep 18 19:39:42.594507 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:39:42.594421    2585 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Sep 18 19:39:42.656145 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:39:42.656112    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zxh4h"
Sep 18 19:39:42.656306 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:39:42.656118    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-npsrh"
Sep 18 19:39:42.656306 ip-10-0-136-185 kubenswrapper[2585]: E0918 19:39:42.656242    2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zxh4h" podUID="13f14c60-b4bd-4146-b4ee-0dd90be01349"
Sep 18 19:39:42.656306 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:39:42.656118    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4njfl"
Sep 18 19:39:42.656469 ip-10-0-136-185 kubenswrapper[2585]: E0918 19:39:42.656312    2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-npsrh" podUID="7e47b713-e1a2-48ff-b961-ad41e706a225"
Sep 18 19:39:42.656469 ip-10-0-136-185 kubenswrapper[2585]: E0918 19:39:42.656405    2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4njfl" podUID="3eb3cd73-4ea1-4950-8bfd-82b5fa60fd09"
Sep 18 19:39:42.784198 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:39:42.784123    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-fpnxg" event={"ID":"eb928dda-649a-48de-841d-06bcb524d0d4","Type":"ContainerStarted","Data":"e6f4b12eed01898a4f0e4113026b0ee7453af9cd5c8d489cc9a3576610edc3a8"}
Sep 18 19:39:42.786342 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:39:42.786306    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-m4b5p" event={"ID":"718b7f76-1cfb-4499-a480-9d5b1567f113","Type":"ContainerStarted","Data":"1975937f464919a8868c96743f5d9162f3a04725c9b177a11c90ef3cda63e67b"}
Sep 18 19:39:42.798943 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:39:42.798896    2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-fpnxg" podStartSLOduration=5.542242801 podStartE2EDuration="22.798855724s" totalImagesPullingTime="17.256612923s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-18 19:39:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 19:39:42.798337314 +0000 UTC m=+24.322942774" watchObservedRunningTime="2026-09-18 19:39:42.798855724 +0000 UTC m=+24.323461146"
Sep 18 19:39:44.655988 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:39:44.655746    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4njfl"
Sep 18 19:39:44.656602 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:39:44.655746    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-npsrh"
Sep 18 19:39:44.656602 ip-10-0-136-185 kubenswrapper[2585]: E0918 19:39:44.656082    2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4njfl" podUID="3eb3cd73-4ea1-4950-8bfd-82b5fa60fd09"
Sep 18 19:39:44.656602 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:39:44.655802    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zxh4h"
Sep 18 19:39:44.656602 ip-10-0-136-185 kubenswrapper[2585]: E0918 19:39:44.656186    2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-npsrh" podUID="7e47b713-e1a2-48ff-b961-ad41e706a225"
Sep 18 19:39:44.656602 ip-10-0-136-185 kubenswrapper[2585]: E0918 19:39:44.656265    2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zxh4h" podUID="13f14c60-b4bd-4146-b4ee-0dd90be01349"
Sep 18 19:39:44.794126 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:39:44.794088    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-m4b5p" event={"ID":"718b7f76-1cfb-4499-a480-9d5b1567f113","Type":"ContainerStarted","Data":"a782d3761fcb685ec08ccadccee39d3bd05632786d6f2d599df7a0ffcd99e2db"}
Sep 18 19:39:44.797600 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:39:44.797547    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-rctld_cc0e0de2-75d5-43a1-b64b-b071427cd8ce/ovn-acl-logging/0.log"
Sep 18 19:39:44.798076 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:39:44.798002    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-rctld" event={"ID":"cc0e0de2-75d5-43a1-b64b-b071427cd8ce","Type":"ContainerStarted","Data":"ceef617a0d5f48ae6bfdce81f6e71e73d758ea0101f9eed63f5973bba2c4c2f3"}
Sep 18 19:39:44.813942 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:39:44.813885    2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-m4b5p" podStartSLOduration=5.014907723 podStartE2EDuration="24.813869087s" totalImagesPullingTime="19.798961364s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-18 19:39:20 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 19:39:44.813594916 +0000 UTC m=+26.338200336" watchObservedRunningTime="2026-09-18 19:39:44.813869087 +0000 UTC m=+26.338474525"
Sep 18 19:39:46.655722 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:39:46.655469    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-npsrh"
Sep 18 19:39:46.656392 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:39:46.655517    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zxh4h"
Sep 18 19:39:46.656392 ip-10-0-136-185 kubenswrapper[2585]: E0918 19:39:46.655856    2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-npsrh" podUID="7e47b713-e1a2-48ff-b961-ad41e706a225"
Sep 18 19:39:46.656392 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:39:46.655535    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4njfl"
Sep 18 19:39:46.656392 ip-10-0-136-185 kubenswrapper[2585]: E0918 19:39:46.655890    2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zxh4h" podUID="13f14c60-b4bd-4146-b4ee-0dd90be01349"
Sep 18 19:39:46.656392 ip-10-0-136-185 kubenswrapper[2585]: E0918 19:39:46.655970    2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4njfl" podUID="3eb3cd73-4ea1-4950-8bfd-82b5fa60fd09"
Sep 18 19:39:46.721962 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:39:46.721927    2585 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-mbhzp"
Sep 18 19:39:46.722558 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:39:46.722542    2585 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-mbhzp"
Sep 18 19:39:46.804247 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:39:46.804211    2585 generic.go:356] "Generic (PLEG): container finished" podID="17b9ee59-c638-43de-8ec6-9544968d07f5" containerID="e8aad139dd564cf5ffc8ac07ca8092c9545decd6fe8480eb1163f11c74ffa0b5" exitCode=0
Sep 18 19:39:46.804386 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:39:46.804288    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-x7kdg" event={"ID":"17b9ee59-c638-43de-8ec6-9544968d07f5","Type":"ContainerDied","Data":"e8aad139dd564cf5ffc8ac07ca8092c9545decd6fe8480eb1163f11c74ffa0b5"}
Sep 18 19:39:46.807988 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:39:46.807966    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-rctld_cc0e0de2-75d5-43a1-b64b-b071427cd8ce/ovn-acl-logging/0.log"
Sep 18 19:39:46.808414 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:39:46.808394    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-rctld" event={"ID":"cc0e0de2-75d5-43a1-b64b-b071427cd8ce","Type":"ContainerStarted","Data":"150dd513341d155c5d9732beb569a76a71809398af04271663ce735435a68b44"}
Sep 18 19:39:46.809222 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:39:46.808773    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-mbhzp"
Sep 18 19:39:46.809222 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:39:46.808823    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-rctld"
Sep 18 19:39:46.809222 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:39:46.808864    2585 scope.go:117] "RemoveContainer" containerID="4298c3f50069a32b4301c8fdacbece0e4402ce50b6385af92ddfd25a649edc6b"
Sep 18 19:39:46.809222 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:39:46.808873    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-rctld"
Sep 18 19:39:46.809222 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:39:46.808921    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-rctld"
Sep 18 19:39:46.809222 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:39:46.809228    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-mbhzp"
Sep 18 19:39:46.824436 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:39:46.824346    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-rctld"
Sep 18 19:39:46.825054 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:39:46.825040    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-rctld"
Sep 18 19:39:47.814969 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:39:47.814939    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-rctld_cc0e0de2-75d5-43a1-b64b-b071427cd8ce/ovn-acl-logging/0.log"
Sep 18 19:39:47.815443 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:39:47.815378    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-rctld" event={"ID":"cc0e0de2-75d5-43a1-b64b-b071427cd8ce","Type":"ContainerStarted","Data":"ef2f7de7e3da954f0ed4e08e4d635d91bee54c24396e281e3034c92c891f2a67"}
Sep 18 19:39:47.847603 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:39:47.847550    2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-rctld" podStartSLOduration=10.542487836 podStartE2EDuration="27.847533495s" totalImagesPullingTime="17.305045659s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-18 19:39:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 19:39:47.846242732 +0000 UTC m=+29.370848156" watchObservedRunningTime="2026-09-18 19:39:47.847533495 +0000 UTC m=+29.372138917"
Sep 18 19:39:48.114473 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:39:48.114383    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-npsrh"]
Sep 18 19:39:48.114607 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:39:48.114539    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-npsrh"
Sep 18 19:39:48.114648 ip-10-0-136-185 kubenswrapper[2585]: E0918 19:39:48.114621    2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-npsrh" podUID="7e47b713-e1a2-48ff-b961-ad41e706a225"
Sep 18 19:39:48.118279 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:39:48.118251    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-zxh4h"]
Sep 18 19:39:48.118442 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:39:48.118373    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zxh4h"
Sep 18 19:39:48.118498 ip-10-0-136-185 kubenswrapper[2585]: E0918 19:39:48.118465    2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zxh4h" podUID="13f14c60-b4bd-4146-b4ee-0dd90be01349"
Sep 18 19:39:48.118954 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:39:48.118934    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-4njfl"]
Sep 18 19:39:48.119058 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:39:48.119047    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4njfl"
Sep 18 19:39:48.119180 ip-10-0-136-185 kubenswrapper[2585]: E0918 19:39:48.119160    2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4njfl" podUID="3eb3cd73-4ea1-4950-8bfd-82b5fa60fd09"
Sep 18 19:39:48.820064 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:39:48.820034    2585 generic.go:356] "Generic (PLEG): container finished" podID="17b9ee59-c638-43de-8ec6-9544968d07f5" containerID="1fb5a39c9695a0a557bc6a63acf696d6cadc7fb44beeb577dc0fdefed13c2edd" exitCode=0
Sep 18 19:39:48.820537 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:39:48.820115    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-x7kdg" event={"ID":"17b9ee59-c638-43de-8ec6-9544968d07f5","Type":"ContainerDied","Data":"1fb5a39c9695a0a557bc6a63acf696d6cadc7fb44beeb577dc0fdefed13c2edd"}
Sep 18 19:39:49.655485 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:39:49.655454    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zxh4h"
Sep 18 19:39:49.655696 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:39:49.655544    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-npsrh"
Sep 18 19:39:49.655696 ip-10-0-136-185 kubenswrapper[2585]: E0918 19:39:49.655552    2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zxh4h" podUID="13f14c60-b4bd-4146-b4ee-0dd90be01349"
Sep 18 19:39:49.655696 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:39:49.655572    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4njfl"
Sep 18 19:39:49.655696 ip-10-0-136-185 kubenswrapper[2585]: E0918 19:39:49.655648    2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-npsrh" podUID="7e47b713-e1a2-48ff-b961-ad41e706a225"
Sep 18 19:39:49.655855 ip-10-0-136-185 kubenswrapper[2585]: E0918 19:39:49.655709    2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4njfl" podUID="3eb3cd73-4ea1-4950-8bfd-82b5fa60fd09"
Sep 18 19:39:50.826112 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:39:50.826079    2585 generic.go:356] "Generic (PLEG): container finished" podID="17b9ee59-c638-43de-8ec6-9544968d07f5" containerID="95d49709f6f4a8b9bc7769a4b54c5c839a839d0f2e713a79ffe1bde69f257e24" exitCode=0
Sep 18 19:39:50.826625 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:39:50.826131    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-x7kdg" event={"ID":"17b9ee59-c638-43de-8ec6-9544968d07f5","Type":"ContainerDied","Data":"95d49709f6f4a8b9bc7769a4b54c5c839a839d0f2e713a79ffe1bde69f257e24"}
Sep 18 19:39:51.656161 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:39:51.656130    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4njfl"
Sep 18 19:39:51.656353 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:39:51.656131    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-npsrh"
Sep 18 19:39:51.656353 ip-10-0-136-185 kubenswrapper[2585]: E0918 19:39:51.656261    2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4njfl" podUID="3eb3cd73-4ea1-4950-8bfd-82b5fa60fd09"
Sep 18 19:39:51.656353 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:39:51.656131    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zxh4h"
Sep 18 19:39:51.656353 ip-10-0-136-185 kubenswrapper[2585]: E0918 19:39:51.656344    2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-npsrh" podUID="7e47b713-e1a2-48ff-b961-ad41e706a225"
Sep 18 19:39:51.656547 ip-10-0-136-185 kubenswrapper[2585]: E0918 19:39:51.656426    2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zxh4h" podUID="13f14c60-b4bd-4146-b4ee-0dd90be01349"
Sep 18 19:39:53.655956 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:39:53.655914    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zxh4h"
Sep 18 19:39:53.655956 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:39:53.655952    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-npsrh"
Sep 18 19:39:53.656563 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:39:53.655956    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4njfl"
Sep 18 19:39:53.656563 ip-10-0-136-185 kubenswrapper[2585]: E0918 19:39:53.656042    2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zxh4h" podUID="13f14c60-b4bd-4146-b4ee-0dd90be01349"
Sep 18 19:39:53.656563 ip-10-0-136-185 kubenswrapper[2585]: E0918 19:39:53.656155    2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-npsrh" podUID="7e47b713-e1a2-48ff-b961-ad41e706a225"
Sep 18 19:39:53.656563 ip-10-0-136-185 kubenswrapper[2585]: E0918 19:39:53.656265    2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4njfl" podUID="3eb3cd73-4ea1-4950-8bfd-82b5fa60fd09"
Sep 18 19:39:54.360414 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:39:54.360331    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/7e47b713-e1a2-48ff-b961-ad41e706a225-original-pull-secret\") pod \"global-pull-secret-syncer-npsrh\" (UID: \"7e47b713-e1a2-48ff-b961-ad41e706a225\") " pod="kube-system/global-pull-secret-syncer-npsrh"
Sep 18 19:39:54.360414 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:39:54.360382    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3eb3cd73-4ea1-4950-8bfd-82b5fa60fd09-metrics-certs\") pod \"network-metrics-daemon-4njfl\" (UID: \"3eb3cd73-4ea1-4950-8bfd-82b5fa60fd09\") " pod="openshift-multus/network-metrics-daemon-4njfl"
Sep 18 19:39:54.360660 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:39:54.360421    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6bfnn\" (UniqueName: \"kubernetes.io/projected/13f14c60-b4bd-4146-b4ee-0dd90be01349-kube-api-access-6bfnn\") pod \"network-check-target-zxh4h\" (UID: \"13f14c60-b4bd-4146-b4ee-0dd90be01349\") " pod="openshift-network-diagnostics/network-check-target-zxh4h"
Sep 18 19:39:54.360660 ip-10-0-136-185 kubenswrapper[2585]: E0918 19:39:54.360501    2585 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 18 19:39:54.360660 ip-10-0-136-185 kubenswrapper[2585]: E0918 19:39:54.360526    2585 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 18 19:39:54.360660 ip-10-0-136-185 kubenswrapper[2585]: E0918 19:39:54.360531    2585 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 18 19:39:54.360660 ip-10-0-136-185 kubenswrapper[2585]: E0918 19:39:54.360550    2585 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 18 19:39:54.360660 ip-10-0-136-185 kubenswrapper[2585]: E0918 19:39:54.360560    2585 projected.go:196] Error preparing data for projected volume kube-api-access-6bfnn for pod openshift-network-diagnostics/network-check-target-zxh4h: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 18 19:39:54.360660 ip-10-0-136-185 kubenswrapper[2585]: E0918 19:39:54.360578    2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7e47b713-e1a2-48ff-b961-ad41e706a225-original-pull-secret podName:7e47b713-e1a2-48ff-b961-ad41e706a225 nodeName:}" failed. No retries permitted until 2026-09-18 19:40:26.360558057 +0000 UTC m=+67.885163472 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/7e47b713-e1a2-48ff-b961-ad41e706a225-original-pull-secret") pod "global-pull-secret-syncer-npsrh" (UID: "7e47b713-e1a2-48ff-b961-ad41e706a225") : object "kube-system"/"original-pull-secret" not registered
Sep 18 19:39:54.360660 ip-10-0-136-185 kubenswrapper[2585]: E0918 19:39:54.360592    2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3eb3cd73-4ea1-4950-8bfd-82b5fa60fd09-metrics-certs podName:3eb3cd73-4ea1-4950-8bfd-82b5fa60fd09 nodeName:}" failed. No retries permitted until 2026-09-18 19:40:26.360585935 +0000 UTC m=+67.885191335 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/3eb3cd73-4ea1-4950-8bfd-82b5fa60fd09-metrics-certs") pod "network-metrics-daemon-4njfl" (UID: "3eb3cd73-4ea1-4950-8bfd-82b5fa60fd09") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 18 19:39:54.360660 ip-10-0-136-185 kubenswrapper[2585]: E0918 19:39:54.360607    2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/13f14c60-b4bd-4146-b4ee-0dd90be01349-kube-api-access-6bfnn podName:13f14c60-b4bd-4146-b4ee-0dd90be01349 nodeName:}" failed. No retries permitted until 2026-09-18 19:40:26.360598934 +0000 UTC m=+67.885204343 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-6bfnn" (UniqueName: "kubernetes.io/projected/13f14c60-b4bd-4146-b4ee-0dd90be01349-kube-api-access-6bfnn") pod "network-check-target-zxh4h" (UID: "13f14c60-b4bd-4146-b4ee-0dd90be01349") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 18 19:39:54.797655 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:39:54.797629    2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-185.ec2.internal" event="NodeReady"
Sep 18 19:39:54.798141 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:39:54.797755    2585 kubelet_node_status.go:489] "Fast updating node status as it just became ready"
Sep 18 19:39:54.850695 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:39:54.850663    2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-grfwt"]
Sep 18 19:39:54.865328 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:39:54.865297    2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-kxzjt"]
Sep 18 19:39:54.865499 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:39:54.865416    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-grfwt"
Sep 18 19:39:54.869348 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:39:54.868841    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-v8jjp\""
Sep 18 19:39:54.869348 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:39:54.869134    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Sep 18 19:39:54.869348 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:39:54.869202    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Sep 18 19:39:54.886929 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:39:54.886895    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-kxzjt"]
Sep 18 19:39:54.886929 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:39:54.886931    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-grfwt"]
Sep 18 19:39:54.887148 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:39:54.887031    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-kxzjt"
Sep 18 19:39:54.889538 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:39:54.889516    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Sep 18 19:39:54.889538 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:39:54.889543    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Sep 18 19:39:54.889746 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:39:54.889544    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-wfwhm\""
Sep 18 19:39:54.889746 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:39:54.889546    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Sep 18 19:39:54.965363 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:39:54.965326    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/13a57b4c-73f9-4960-89d4-e9c25d392279-metrics-tls\") pod \"dns-default-grfwt\" (UID: \"13a57b4c-73f9-4960-89d4-e9c25d392279\") " pod="openshift-dns/dns-default-grfwt"
Sep 18 19:39:54.965363 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:39:54.965361    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/13a57b4c-73f9-4960-89d4-e9c25d392279-tmp-dir\") pod \"dns-default-grfwt\" (UID: \"13a57b4c-73f9-4960-89d4-e9c25d392279\") " pod="openshift-dns/dns-default-grfwt"
Sep 18 19:39:54.965567 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:39:54.965386    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ccbt8\" (UniqueName: \"kubernetes.io/projected/13a57b4c-73f9-4960-89d4-e9c25d392279-kube-api-access-ccbt8\") pod \"dns-default-grfwt\" (UID: \"13a57b4c-73f9-4960-89d4-e9c25d392279\") " pod="openshift-dns/dns-default-grfwt"
Sep 18 19:39:54.965567 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:39:54.965411    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/a797d295-00fd-4085-b1cc-bdfffc6af82f-cert\") pod \"ingress-canary-kxzjt\" (UID: \"a797d295-00fd-4085-b1cc-bdfffc6af82f\") " pod="openshift-ingress-canary/ingress-canary-kxzjt"
Sep 18 19:39:54.965567 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:39:54.965477    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/13a57b4c-73f9-4960-89d4-e9c25d392279-config-volume\") pod \"dns-default-grfwt\" (UID: \"13a57b4c-73f9-4960-89d4-e9c25d392279\") " pod="openshift-dns/dns-default-grfwt"
Sep 18 19:39:54.965567 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:39:54.965518    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ppcff\" (UniqueName: \"kubernetes.io/projected/a797d295-00fd-4085-b1cc-bdfffc6af82f-kube-api-access-ppcff\") pod \"ingress-canary-kxzjt\" (UID: \"a797d295-00fd-4085-b1cc-bdfffc6af82f\") " pod="openshift-ingress-canary/ingress-canary-kxzjt"
Sep 18 19:39:55.066857 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:39:55.066819    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/13a57b4c-73f9-4960-89d4-e9c25d392279-metrics-tls\") pod \"dns-default-grfwt\" (UID: \"13a57b4c-73f9-4960-89d4-e9c25d392279\") " pod="openshift-dns/dns-default-grfwt"
Sep 18 19:39:55.067039 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:39:55.066865    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/13a57b4c-73f9-4960-89d4-e9c25d392279-tmp-dir\") pod \"dns-default-grfwt\" (UID: \"13a57b4c-73f9-4960-89d4-e9c25d392279\") " pod="openshift-dns/dns-default-grfwt"
Sep 18 19:39:55.067039 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:39:55.066910    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ccbt8\" (UniqueName: \"kubernetes.io/projected/13a57b4c-73f9-4960-89d4-e9c25d392279-kube-api-access-ccbt8\") pod \"dns-default-grfwt\" (UID: \"13a57b4c-73f9-4960-89d4-e9c25d392279\") " pod="openshift-dns/dns-default-grfwt"
Sep 18 19:39:55.067039 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:39:55.066958    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/a797d295-00fd-4085-b1cc-bdfffc6af82f-cert\") pod \"ingress-canary-kxzjt\" (UID: \"a797d295-00fd-4085-b1cc-bdfffc6af82f\") " pod="openshift-ingress-canary/ingress-canary-kxzjt"
Sep 18 19:39:55.067039 ip-10-0-136-185 kubenswrapper[2585]: E0918 19:39:55.066969    2585 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 18 19:39:55.067039 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:39:55.067009    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/13a57b4c-73f9-4960-89d4-e9c25d392279-config-volume\") pod \"dns-default-grfwt\" (UID: \"13a57b4c-73f9-4960-89d4-e9c25d392279\") " pod="openshift-dns/dns-default-grfwt"
Sep 18 19:39:55.067039 ip-10-0-136-185 kubenswrapper[2585]: E0918 19:39:55.067043    2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/13a57b4c-73f9-4960-89d4-e9c25d392279-metrics-tls podName:13a57b4c-73f9-4960-89d4-e9c25d392279 nodeName:}" failed. No retries permitted until 2026-09-18 19:39:55.567021332 +0000 UTC m=+37.091626748 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/13a57b4c-73f9-4960-89d4-e9c25d392279-metrics-tls") pod "dns-default-grfwt" (UID: "13a57b4c-73f9-4960-89d4-e9c25d392279") : secret "dns-default-metrics-tls" not found
Sep 18 19:39:55.067340 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:39:55.067081    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ppcff\" (UniqueName: \"kubernetes.io/projected/a797d295-00fd-4085-b1cc-bdfffc6af82f-kube-api-access-ppcff\") pod \"ingress-canary-kxzjt\" (UID: \"a797d295-00fd-4085-b1cc-bdfffc6af82f\") " pod="openshift-ingress-canary/ingress-canary-kxzjt"
Sep 18 19:39:55.067340 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:39:55.067194    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/13a57b4c-73f9-4960-89d4-e9c25d392279-tmp-dir\") pod \"dns-default-grfwt\" (UID: \"13a57b4c-73f9-4960-89d4-e9c25d392279\") " pod="openshift-dns/dns-default-grfwt"
Sep 18 19:39:55.067573 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:39:55.067550    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/13a57b4c-73f9-4960-89d4-e9c25d392279-config-volume\") pod \"dns-default-grfwt\" (UID: \"13a57b4c-73f9-4960-89d4-e9c25d392279\") " pod="openshift-dns/dns-default-grfwt"
Sep 18 19:39:55.071128 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:39:55.071108    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/a797d295-00fd-4085-b1cc-bdfffc6af82f-cert\") pod \"ingress-canary-kxzjt\" (UID: \"a797d295-00fd-4085-b1cc-bdfffc6af82f\") " pod="openshift-ingress-canary/ingress-canary-kxzjt"
Sep 18 19:39:55.079040 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:39:55.079012    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ppcff\" (UniqueName: \"kubernetes.io/projected/a797d295-00fd-4085-b1cc-bdfffc6af82f-kube-api-access-ppcff\") pod \"ingress-canary-kxzjt\" (UID: \"a797d295-00fd-4085-b1cc-bdfffc6af82f\") " pod="openshift-ingress-canary/ingress-canary-kxzjt"
Sep 18 19:39:55.079156 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:39:55.079095    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ccbt8\" (UniqueName: \"kubernetes.io/projected/13a57b4c-73f9-4960-89d4-e9c25d392279-kube-api-access-ccbt8\") pod \"dns-default-grfwt\" (UID: \"13a57b4c-73f9-4960-89d4-e9c25d392279\") " pod="openshift-dns/dns-default-grfwt"
Sep 18 19:39:55.195777 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:39:55.195591    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-kxzjt"
Sep 18 19:39:55.528858 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:39:55.528725    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-kxzjt"]
Sep 18 19:39:55.533986 ip-10-0-136-185 kubenswrapper[2585]: W0918 19:39:55.533954    2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda797d295_00fd_4085_b1cc_bdfffc6af82f.slice/crio-4c89950b39233bcbcb8d09186aa732aceb0abab2aab40ba2dce4966a34fda1f6 WatchSource:0}: Error finding container 4c89950b39233bcbcb8d09186aa732aceb0abab2aab40ba2dce4966a34fda1f6: Status 404 returned error can't find the container with id 4c89950b39233bcbcb8d09186aa732aceb0abab2aab40ba2dce4966a34fda1f6
Sep 18 19:39:55.571027 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:39:55.571006    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/13a57b4c-73f9-4960-89d4-e9c25d392279-metrics-tls\") pod \"dns-default-grfwt\" (UID: \"13a57b4c-73f9-4960-89d4-e9c25d392279\") " pod="openshift-dns/dns-default-grfwt"
Sep 18 19:39:55.571170 ip-10-0-136-185 kubenswrapper[2585]: E0918 19:39:55.571151    2585 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 18 19:39:55.571227 ip-10-0-136-185 kubenswrapper[2585]: E0918 19:39:55.571216    2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/13a57b4c-73f9-4960-89d4-e9c25d392279-metrics-tls podName:13a57b4c-73f9-4960-89d4-e9c25d392279 nodeName:}" failed. No retries permitted until 2026-09-18 19:39:56.571200688 +0000 UTC m=+38.095806109 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/13a57b4c-73f9-4960-89d4-e9c25d392279-metrics-tls") pod "dns-default-grfwt" (UID: "13a57b4c-73f9-4960-89d4-e9c25d392279") : secret "dns-default-metrics-tls" not found
Sep 18 19:39:55.655686 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:39:55.655647    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4njfl"
Sep 18 19:39:55.655686 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:39:55.655672    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-npsrh"
Sep 18 19:39:55.655686 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:39:55.655680    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zxh4h"
Sep 18 19:39:55.658429 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:39:55.658408    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-t5jrl\""
Sep 18 19:39:55.659203 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:39:55.659188    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-m9tpx\""
Sep 18 19:39:55.659276 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:39:55.659246    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Sep 18 19:39:55.659339 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:39:55.659296    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Sep 18 19:39:55.659339 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:39:55.659309    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Sep 18 19:39:55.659432 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:39:55.659294    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Sep 18 19:39:55.840336 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:39:55.840255    2585 generic.go:356] "Generic (PLEG): container finished" podID="17b9ee59-c638-43de-8ec6-9544968d07f5" containerID="6f1e0e8eb6f714a89fe764e2d11c84fdf925cd8eb977db20844c42f729668f76" exitCode=0
Sep 18 19:39:55.840336 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:39:55.840320    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-x7kdg" event={"ID":"17b9ee59-c638-43de-8ec6-9544968d07f5","Type":"ContainerDied","Data":"6f1e0e8eb6f714a89fe764e2d11c84fdf925cd8eb977db20844c42f729668f76"}
Sep 18 19:39:55.841531 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:39:55.841457    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-kxzjt" event={"ID":"a797d295-00fd-4085-b1cc-bdfffc6af82f","Type":"ContainerStarted","Data":"4c89950b39233bcbcb8d09186aa732aceb0abab2aab40ba2dce4966a34fda1f6"}
Sep 18 19:39:56.580632 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:39:56.580598    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/13a57b4c-73f9-4960-89d4-e9c25d392279-metrics-tls\") pod \"dns-default-grfwt\" (UID: \"13a57b4c-73f9-4960-89d4-e9c25d392279\") " pod="openshift-dns/dns-default-grfwt"
Sep 18 19:39:56.580816 ip-10-0-136-185 kubenswrapper[2585]: E0918 19:39:56.580742    2585 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 18 19:39:56.580888 ip-10-0-136-185 kubenswrapper[2585]: E0918 19:39:56.580819    2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/13a57b4c-73f9-4960-89d4-e9c25d392279-metrics-tls podName:13a57b4c-73f9-4960-89d4-e9c25d392279 nodeName:}" failed. No retries permitted until 2026-09-18 19:39:58.5807835 +0000 UTC m=+40.105389026 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/13a57b4c-73f9-4960-89d4-e9c25d392279-metrics-tls") pod "dns-default-grfwt" (UID: "13a57b4c-73f9-4960-89d4-e9c25d392279") : secret "dns-default-metrics-tls" not found
Sep 18 19:39:56.848375 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:39:56.848272    2585 generic.go:356] "Generic (PLEG): container finished" podID="17b9ee59-c638-43de-8ec6-9544968d07f5" containerID="4e8c895cb9a2d7d4a1c51cdd5d76eed7d35c17f6d909f71f704614d73afb1f62" exitCode=0
Sep 18 19:39:56.848375 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:39:56.848353    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-x7kdg" event={"ID":"17b9ee59-c638-43de-8ec6-9544968d07f5","Type":"ContainerDied","Data":"4e8c895cb9a2d7d4a1c51cdd5d76eed7d35c17f6d909f71f704614d73afb1f62"}
Sep 18 19:39:57.854904 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:39:57.854624    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-x7kdg" event={"ID":"17b9ee59-c638-43de-8ec6-9544968d07f5","Type":"ContainerStarted","Data":"4589e447bff0a7685f1c9afda0dcd7a12d30727c886273a2a64c6eace1b35af0"}
Sep 18 19:39:57.878834 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:39:57.878761    2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-x7kdg" podStartSLOduration=10.049458444 podStartE2EDuration="37.87874413s" totalImagesPullingTime="27.829285686s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-18 19:39:20 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 19:39:57.878346643 +0000 UTC m=+39.402952065" watchObservedRunningTime="2026-09-18 19:39:57.87874413 +0000 UTC m=+39.403349553"
Sep 18 19:39:58.596770 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:39:58.596736    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/13a57b4c-73f9-4960-89d4-e9c25d392279-metrics-tls\") pod \"dns-default-grfwt\" (UID: \"13a57b4c-73f9-4960-89d4-e9c25d392279\") " pod="openshift-dns/dns-default-grfwt"
Sep 18 19:39:58.596926 ip-10-0-136-185 kubenswrapper[2585]: E0918 19:39:58.596868    2585 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 18 19:39:58.596961 ip-10-0-136-185 kubenswrapper[2585]: E0918 19:39:58.596928    2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/13a57b4c-73f9-4960-89d4-e9c25d392279-metrics-tls podName:13a57b4c-73f9-4960-89d4-e9c25d392279 nodeName:}" failed. No retries permitted until 2026-09-18 19:40:02.596911953 +0000 UTC m=+44.121517358 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/13a57b4c-73f9-4960-89d4-e9c25d392279-metrics-tls") pod "dns-default-grfwt" (UID: "13a57b4c-73f9-4960-89d4-e9c25d392279") : secret "dns-default-metrics-tls" not found
Sep 18 19:39:58.858288 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:39:58.858204    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-kxzjt" event={"ID":"a797d295-00fd-4085-b1cc-bdfffc6af82f","Type":"ContainerStarted","Data":"4b783c3f0df093fb470ff07079e49b617603310d8f70e35e02e3289dc18daf91"}
Sep 18 19:39:58.876878 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:39:58.876827    2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-kxzjt" podStartSLOduration=2.392519137 podStartE2EDuration="4.876810709s" totalImagesPullingTime="2.484291572s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 19:39:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 19:39:58.875970125 +0000 UTC m=+40.400575538" watchObservedRunningTime="2026-09-18 19:39:58.876810709 +0000 UTC m=+40.401416123"
Sep 18 19:40:02.623253 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:40:02.623212    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/13a57b4c-73f9-4960-89d4-e9c25d392279-metrics-tls\") pod \"dns-default-grfwt\" (UID: \"13a57b4c-73f9-4960-89d4-e9c25d392279\") " pod="openshift-dns/dns-default-grfwt"
Sep 18 19:40:02.623639 ip-10-0-136-185 kubenswrapper[2585]: E0918 19:40:02.623371    2585 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 18 19:40:02.623639 ip-10-0-136-185 kubenswrapper[2585]: E0918 19:40:02.623433    2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/13a57b4c-73f9-4960-89d4-e9c25d392279-metrics-tls podName:13a57b4c-73f9-4960-89d4-e9c25d392279 nodeName:}" failed. No retries permitted until 2026-09-18 19:40:10.623418275 +0000 UTC m=+52.148023676 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/13a57b4c-73f9-4960-89d4-e9c25d392279-metrics-tls") pod "dns-default-grfwt" (UID: "13a57b4c-73f9-4960-89d4-e9c25d392279") : secret "dns-default-metrics-tls" not found
Sep 18 19:40:10.677534 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:40:10.677499    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/13a57b4c-73f9-4960-89d4-e9c25d392279-metrics-tls\") pod \"dns-default-grfwt\" (UID: \"13a57b4c-73f9-4960-89d4-e9c25d392279\") " pod="openshift-dns/dns-default-grfwt"
Sep 18 19:40:10.677949 ip-10-0-136-185 kubenswrapper[2585]: E0918 19:40:10.677643    2585 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 18 19:40:10.677949 ip-10-0-136-185 kubenswrapper[2585]: E0918 19:40:10.677702    2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/13a57b4c-73f9-4960-89d4-e9c25d392279-metrics-tls podName:13a57b4c-73f9-4960-89d4-e9c25d392279 nodeName:}" failed. No retries permitted until 2026-09-18 19:40:26.677685833 +0000 UTC m=+68.202291234 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/13a57b4c-73f9-4960-89d4-e9c25d392279-metrics-tls") pod "dns-default-grfwt" (UID: "13a57b4c-73f9-4960-89d4-e9c25d392279") : secret "dns-default-metrics-tls" not found
Sep 18 19:40:18.835171 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:40:18.835146    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-rctld"
Sep 18 19:40:26.391727 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:40:26.391681    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/7e47b713-e1a2-48ff-b961-ad41e706a225-original-pull-secret\") pod \"global-pull-secret-syncer-npsrh\" (UID: \"7e47b713-e1a2-48ff-b961-ad41e706a225\") " pod="kube-system/global-pull-secret-syncer-npsrh"
Sep 18 19:40:26.391727 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:40:26.391731    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3eb3cd73-4ea1-4950-8bfd-82b5fa60fd09-metrics-certs\") pod \"network-metrics-daemon-4njfl\" (UID: \"3eb3cd73-4ea1-4950-8bfd-82b5fa60fd09\") " pod="openshift-multus/network-metrics-daemon-4njfl"
Sep 18 19:40:26.392231 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:40:26.391769    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6bfnn\" (UniqueName: \"kubernetes.io/projected/13f14c60-b4bd-4146-b4ee-0dd90be01349-kube-api-access-6bfnn\") pod \"network-check-target-zxh4h\" (UID: \"13f14c60-b4bd-4146-b4ee-0dd90be01349\") " pod="openshift-network-diagnostics/network-check-target-zxh4h"
Sep 18 19:40:26.394667 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:40:26.394645    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Sep 18 19:40:26.394738 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:40:26.394714    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Sep 18 19:40:26.394738 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:40:26.394722    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Sep 18 19:40:26.402347 ip-10-0-136-185 kubenswrapper[2585]: E0918 19:40:26.402325    2585 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Sep 18 19:40:26.402469 ip-10-0-136-185 kubenswrapper[2585]: E0918 19:40:26.402396    2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3eb3cd73-4ea1-4950-8bfd-82b5fa60fd09-metrics-certs podName:3eb3cd73-4ea1-4950-8bfd-82b5fa60fd09 nodeName:}" failed. No retries permitted until 2026-09-18 19:41:30.40238263 +0000 UTC m=+131.926988045 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/3eb3cd73-4ea1-4950-8bfd-82b5fa60fd09-metrics-certs") pod "network-metrics-daemon-4njfl" (UID: "3eb3cd73-4ea1-4950-8bfd-82b5fa60fd09") : secret "metrics-daemon-secret" not found
Sep 18 19:40:26.404087 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:40:26.404070    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Sep 18 19:40:26.406042 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:40:26.406024    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/7e47b713-e1a2-48ff-b961-ad41e706a225-original-pull-secret\") pod \"global-pull-secret-syncer-npsrh\" (UID: \"7e47b713-e1a2-48ff-b961-ad41e706a225\") " pod="kube-system/global-pull-secret-syncer-npsrh"
Sep 18 19:40:26.414882 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:40:26.414861    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6bfnn\" (UniqueName: \"kubernetes.io/projected/13f14c60-b4bd-4146-b4ee-0dd90be01349-kube-api-access-6bfnn\") pod \"network-check-target-zxh4h\" (UID: \"13f14c60-b4bd-4146-b4ee-0dd90be01349\") " pod="openshift-network-diagnostics/network-check-target-zxh4h"
Sep 18 19:40:26.565482 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:40:26.565428    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-npsrh"
Sep 18 19:40:26.572771 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:40:26.572748    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-t5jrl\""
Sep 18 19:40:26.580805 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:40:26.580770    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zxh4h"
Sep 18 19:40:26.693892 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:40:26.693852    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/13a57b4c-73f9-4960-89d4-e9c25d392279-metrics-tls\") pod \"dns-default-grfwt\" (UID: \"13a57b4c-73f9-4960-89d4-e9c25d392279\") " pod="openshift-dns/dns-default-grfwt"
Sep 18 19:40:26.694083 ip-10-0-136-185 kubenswrapper[2585]: E0918 19:40:26.694006    2585 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 18 19:40:26.694083 ip-10-0-136-185 kubenswrapper[2585]: E0918 19:40:26.694077    2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/13a57b4c-73f9-4960-89d4-e9c25d392279-metrics-tls podName:13a57b4c-73f9-4960-89d4-e9c25d392279 nodeName:}" failed. No retries permitted until 2026-09-18 19:40:58.694057919 +0000 UTC m=+100.218663320 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/13a57b4c-73f9-4960-89d4-e9c25d392279-metrics-tls") pod "dns-default-grfwt" (UID: "13a57b4c-73f9-4960-89d4-e9c25d392279") : secret "dns-default-metrics-tls" not found
Sep 18 19:40:26.839523 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:40:26.839488    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-npsrh"]
Sep 18 19:40:26.842255 ip-10-0-136-185 kubenswrapper[2585]: W0918 19:40:26.842221    2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7e47b713_e1a2_48ff_b961_ad41e706a225.slice/crio-40b760386bcfe1ef233d8a10229d5f57815c4811a6349c55f892452541b9d9a3 WatchSource:0}: Error finding container 40b760386bcfe1ef233d8a10229d5f57815c4811a6349c55f892452541b9d9a3: Status 404 returned error can't find the container with id 40b760386bcfe1ef233d8a10229d5f57815c4811a6349c55f892452541b9d9a3
Sep 18 19:40:26.858202 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:40:26.858174    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-zxh4h"]
Sep 18 19:40:26.861253 ip-10-0-136-185 kubenswrapper[2585]: W0918 19:40:26.861228    2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod13f14c60_b4bd_4146_b4ee_0dd90be01349.slice/crio-f3fb56de178af48f792d64b887bb683b00f7fcd8ace12c3333a78395b9dfe8c4 WatchSource:0}: Error finding container f3fb56de178af48f792d64b887bb683b00f7fcd8ace12c3333a78395b9dfe8c4: Status 404 returned error can't find the container with id f3fb56de178af48f792d64b887bb683b00f7fcd8ace12c3333a78395b9dfe8c4
Sep 18 19:40:26.923862 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:40:26.923820    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-npsrh" event={"ID":"7e47b713-e1a2-48ff-b961-ad41e706a225","Type":"ContainerStarted","Data":"40b760386bcfe1ef233d8a10229d5f57815c4811a6349c55f892452541b9d9a3"}
Sep 18 19:40:26.924926 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:40:26.924906    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-zxh4h" event={"ID":"13f14c60-b4bd-4146-b4ee-0dd90be01349","Type":"ContainerStarted","Data":"f3fb56de178af48f792d64b887bb683b00f7fcd8ace12c3333a78395b9dfe8c4"}
Sep 18 19:40:31.939835 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:40:31.939780    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-npsrh" event={"ID":"7e47b713-e1a2-48ff-b961-ad41e706a225","Type":"ContainerStarted","Data":"5d9a04ab33e09fdf47ea924339888ac44db369de01979a18f46ed70658f5c0b6"}
Sep 18 19:40:31.941258 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:40:31.941234    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-zxh4h" event={"ID":"13f14c60-b4bd-4146-b4ee-0dd90be01349","Type":"ContainerStarted","Data":"1fc5a774d78b3d268ad95d6ce50dcfc3d2ee5d5ee09f15d1f80cd625f51713d1"}
Sep 18 19:40:31.941386 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:40:31.941366    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-zxh4h"
Sep 18 19:40:31.954669 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:40:31.954618    2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-npsrh" podStartSLOduration=67.582969167 podStartE2EDuration="1m11.954601787s" totalImagesPullingTime="4.37163262s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-18 19:39:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 19:40:31.954205572 +0000 UTC m=+73.478810987" watchObservedRunningTime="2026-09-18 19:40:31.954601787 +0000 UTC m=+73.479207209"
Sep 18 19:40:31.969680 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:40:31.969641    2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-zxh4h" podStartSLOduration=67.620413548 podStartE2EDuration="1m11.969632764s" totalImagesPullingTime="4.349219216s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 19:39:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 19:40:31.969188848 +0000 UTC m=+73.493794281" watchObservedRunningTime="2026-09-18 19:40:31.969632764 +0000 UTC m=+73.494238187"
Sep 18 19:40:41.950754 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:40:41.950723    2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7786448cdb-pqk46"]
Sep 18 19:40:41.953218 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:40:41.953191    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7786448cdb-pqk46"
Sep 18 19:40:41.955650 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:40:41.955629    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\""
Sep 18 19:40:41.955751 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:40:41.955650    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\""
Sep 18 19:40:41.956580 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:40:41.956558    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\""
Sep 18 19:40:41.956818 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:40:41.956804    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-dockercfg-xjh8s\""
Sep 18 19:40:41.956871 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:40:41.956860    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-hub-kubeconfig\""
Sep 18 19:40:41.961370 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:40:41.961350    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7786448cdb-pqk46"]
Sep 18 19:40:41.987014 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:40:41.986988    2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-854d4bb9f9-d7fxb"]
Sep 18 19:40:41.989077 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:40:41.989062    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-854d4bb9f9-d7fxb"
Sep 18 19:40:41.991271 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:40:41.991249    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"work-manager-hub-kubeconfig\""
Sep 18 19:40:41.998101 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:40:41.998080    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rn4mh\" (UniqueName: \"kubernetes.io/projected/050e1951-5ccc-4c96-80f4-8a7f52621f7b-kube-api-access-rn4mh\") pod \"managed-serviceaccount-addon-agent-7786448cdb-pqk46\" (UID: \"050e1951-5ccc-4c96-80f4-8a7f52621f7b\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7786448cdb-pqk46"
Sep 18 19:40:41.998190 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:40:41.998120    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/050e1951-5ccc-4c96-80f4-8a7f52621f7b-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-7786448cdb-pqk46\" (UID: \"050e1951-5ccc-4c96-80f4-8a7f52621f7b\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7786448cdb-pqk46"
Sep 18 19:40:42.000346 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:40:42.000322    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-854d4bb9f9-d7fxb"]
Sep 18 19:40:42.099355 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:40:42.099319    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rn4mh\" (UniqueName: \"kubernetes.io/projected/050e1951-5ccc-4c96-80f4-8a7f52621f7b-kube-api-access-rn4mh\") pod \"managed-serviceaccount-addon-agent-7786448cdb-pqk46\" (UID: \"050e1951-5ccc-4c96-80f4-8a7f52621f7b\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7786448cdb-pqk46"
Sep 18 19:40:42.099355 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:40:42.099365    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/050e1951-5ccc-4c96-80f4-8a7f52621f7b-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-7786448cdb-pqk46\" (UID: \"050e1951-5ccc-4c96-80f4-8a7f52621f7b\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7786448cdb-pqk46"
Sep 18 19:40:42.099602 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:40:42.099395    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/7f1db892-607a-4a03-b019-b33e96d89944-tmp\") pod \"klusterlet-addon-workmgr-854d4bb9f9-d7fxb\" (UID: \"7f1db892-607a-4a03-b019-b33e96d89944\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-854d4bb9f9-d7fxb"
Sep 18 19:40:42.099602 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:40:42.099413    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/7f1db892-607a-4a03-b019-b33e96d89944-klusterlet-config\") pod \"klusterlet-addon-workmgr-854d4bb9f9-d7fxb\" (UID: \"7f1db892-607a-4a03-b019-b33e96d89944\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-854d4bb9f9-d7fxb"
Sep 18 19:40:42.099602 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:40:42.099447    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-np9pj\" (UniqueName: \"kubernetes.io/projected/7f1db892-607a-4a03-b019-b33e96d89944-kube-api-access-np9pj\") pod \"klusterlet-addon-workmgr-854d4bb9f9-d7fxb\" (UID: \"7f1db892-607a-4a03-b019-b33e96d89944\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-854d4bb9f9-d7fxb"
Sep 18 19:40:42.101821 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:40:42.101782    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/050e1951-5ccc-4c96-80f4-8a7f52621f7b-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-7786448cdb-pqk46\" (UID: \"050e1951-5ccc-4c96-80f4-8a7f52621f7b\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7786448cdb-pqk46"
Sep 18 19:40:42.107593 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:40:42.107568    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rn4mh\" (UniqueName: \"kubernetes.io/projected/050e1951-5ccc-4c96-80f4-8a7f52621f7b-kube-api-access-rn4mh\") pod \"managed-serviceaccount-addon-agent-7786448cdb-pqk46\" (UID: \"050e1951-5ccc-4c96-80f4-8a7f52621f7b\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7786448cdb-pqk46"
Sep 18 19:40:42.199875 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:40:42.199843    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/7f1db892-607a-4a03-b019-b33e96d89944-tmp\") pod \"klusterlet-addon-workmgr-854d4bb9f9-d7fxb\" (UID: \"7f1db892-607a-4a03-b019-b33e96d89944\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-854d4bb9f9-d7fxb"
Sep 18 19:40:42.199875 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:40:42.199877    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/7f1db892-607a-4a03-b019-b33e96d89944-klusterlet-config\") pod \"klusterlet-addon-workmgr-854d4bb9f9-d7fxb\" (UID: \"7f1db892-607a-4a03-b019-b33e96d89944\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-854d4bb9f9-d7fxb"
Sep 18 19:40:42.200057 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:40:42.199921    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-np9pj\" (UniqueName: \"kubernetes.io/projected/7f1db892-607a-4a03-b019-b33e96d89944-kube-api-access-np9pj\") pod \"klusterlet-addon-workmgr-854d4bb9f9-d7fxb\" (UID: \"7f1db892-607a-4a03-b019-b33e96d89944\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-854d4bb9f9-d7fxb"
Sep 18 19:40:42.200339 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:40:42.200322    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/7f1db892-607a-4a03-b019-b33e96d89944-tmp\") pod \"klusterlet-addon-workmgr-854d4bb9f9-d7fxb\" (UID: \"7f1db892-607a-4a03-b019-b33e96d89944\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-854d4bb9f9-d7fxb"
Sep 18 19:40:42.202383 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:40:42.202323    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/7f1db892-607a-4a03-b019-b33e96d89944-klusterlet-config\") pod \"klusterlet-addon-workmgr-854d4bb9f9-d7fxb\" (UID: \"7f1db892-607a-4a03-b019-b33e96d89944\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-854d4bb9f9-d7fxb"
Sep 18 19:40:42.211278 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:40:42.211250    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-np9pj\" (UniqueName: \"kubernetes.io/projected/7f1db892-607a-4a03-b019-b33e96d89944-kube-api-access-np9pj\") pod \"klusterlet-addon-workmgr-854d4bb9f9-d7fxb\" (UID: \"7f1db892-607a-4a03-b019-b33e96d89944\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-854d4bb9f9-d7fxb"
Sep 18 19:40:42.274244 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:40:42.274212    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7786448cdb-pqk46"
Sep 18 19:40:42.297683 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:40:42.297656    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-854d4bb9f9-d7fxb"
Sep 18 19:40:42.544487 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:40:42.544403    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7786448cdb-pqk46"]
Sep 18 19:40:42.548618 ip-10-0-136-185 kubenswrapper[2585]: W0918 19:40:42.548595    2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod050e1951_5ccc_4c96_80f4_8a7f52621f7b.slice/crio-f342d81a66fb92863f3d3845bee154f6df2fdf1e9132d0ce424f38d01a698c5f WatchSource:0}: Error finding container f342d81a66fb92863f3d3845bee154f6df2fdf1e9132d0ce424f38d01a698c5f: Status 404 returned error can't find the container with id f342d81a66fb92863f3d3845bee154f6df2fdf1e9132d0ce424f38d01a698c5f
Sep 18 19:40:42.570438 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:40:42.570410    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-854d4bb9f9-d7fxb"]
Sep 18 19:40:42.573369 ip-10-0-136-185 kubenswrapper[2585]: W0918 19:40:42.573344    2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7f1db892_607a_4a03_b019_b33e96d89944.slice/crio-589309848a7d32acf292d2cb376d1b4cf397c685b4329978b0019dbe69ed5897 WatchSource:0}: Error finding container 589309848a7d32acf292d2cb376d1b4cf397c685b4329978b0019dbe69ed5897: Status 404 returned error can't find the container with id 589309848a7d32acf292d2cb376d1b4cf397c685b4329978b0019dbe69ed5897
Sep 18 19:40:42.969347 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:40:42.969306    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-854d4bb9f9-d7fxb" event={"ID":"7f1db892-607a-4a03-b019-b33e96d89944","Type":"ContainerStarted","Data":"589309848a7d32acf292d2cb376d1b4cf397c685b4329978b0019dbe69ed5897"}
Sep 18 19:40:42.970297 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:40:42.970276    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7786448cdb-pqk46" event={"ID":"050e1951-5ccc-4c96-80f4-8a7f52621f7b","Type":"ContainerStarted","Data":"f342d81a66fb92863f3d3845bee154f6df2fdf1e9132d0ce424f38d01a698c5f"}
Sep 18 19:40:47.988257 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:40:47.988216    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-854d4bb9f9-d7fxb" event={"ID":"7f1db892-607a-4a03-b019-b33e96d89944","Type":"ContainerStarted","Data":"f7b50a4d6fe31fd1301b7acee0dfdaa3c9a9bd8f3d03bd741fc944c1003db062"}
Sep 18 19:40:47.988724 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:40:47.988394    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-854d4bb9f9-d7fxb"
Sep 18 19:40:47.989756 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:40:47.989715    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7786448cdb-pqk46" event={"ID":"050e1951-5ccc-4c96-80f4-8a7f52621f7b","Type":"ContainerStarted","Data":"43e1740eeb8cb15895748f228638e892c9cc9a83d04b0891f2d303211b475a62"}
Sep 18 19:40:47.990079 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:40:47.990057    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-854d4bb9f9-d7fxb"
Sep 18 19:40:48.005145 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:40:48.005103    2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-854d4bb9f9-d7fxb" podStartSLOduration=2.469530959 podStartE2EDuration="7.005089758s" totalImagesPullingTime="4.535558799s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 19:40:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 19:40:48.004534197 +0000 UTC m=+89.529139633" watchObservedRunningTime="2026-09-18 19:40:48.005089758 +0000 UTC m=+89.529695178"
Sep 18 19:40:48.019677 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:40:48.019482    2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7786448cdb-pqk46" podStartSLOduration=2.476453297 podStartE2EDuration="7.019464742s" totalImagesPullingTime="4.543011445s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 19:40:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 19:40:48.019065029 +0000 UTC m=+89.543670451" watchObservedRunningTime="2026-09-18 19:40:48.019464742 +0000 UTC m=+89.544070167"
Sep 18 19:40:58.716530 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:40:58.716489    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/13a57b4c-73f9-4960-89d4-e9c25d392279-metrics-tls\") pod \"dns-default-grfwt\" (UID: \"13a57b4c-73f9-4960-89d4-e9c25d392279\") " pod="openshift-dns/dns-default-grfwt"
Sep 18 19:40:58.716992 ip-10-0-136-185 kubenswrapper[2585]: E0918 19:40:58.716615    2585 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 18 19:40:58.716992 ip-10-0-136-185 kubenswrapper[2585]: E0918 19:40:58.716667    2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/13a57b4c-73f9-4960-89d4-e9c25d392279-metrics-tls podName:13a57b4c-73f9-4960-89d4-e9c25d392279 nodeName:}" failed. No retries permitted until 2026-09-18 19:42:02.716654353 +0000 UTC m=+164.241259754 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/13a57b4c-73f9-4960-89d4-e9c25d392279-metrics-tls") pod "dns-default-grfwt" (UID: "13a57b4c-73f9-4960-89d4-e9c25d392279") : secret "dns-default-metrics-tls" not found
Sep 18 19:41:02.946487 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:41:02.946456    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-zxh4h"
Sep 18 19:41:30.437445 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:41:30.437392    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3eb3cd73-4ea1-4950-8bfd-82b5fa60fd09-metrics-certs\") pod \"network-metrics-daemon-4njfl\" (UID: \"3eb3cd73-4ea1-4950-8bfd-82b5fa60fd09\") " pod="openshift-multus/network-metrics-daemon-4njfl"
Sep 18 19:41:30.438060 ip-10-0-136-185 kubenswrapper[2585]: E0918 19:41:30.437547    2585 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Sep 18 19:41:30.438060 ip-10-0-136-185 kubenswrapper[2585]: E0918 19:41:30.437628    2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3eb3cd73-4ea1-4950-8bfd-82b5fa60fd09-metrics-certs podName:3eb3cd73-4ea1-4950-8bfd-82b5fa60fd09 nodeName:}" failed. No retries permitted until 2026-09-18 19:43:32.437611934 +0000 UTC m=+253.962217348 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/3eb3cd73-4ea1-4950-8bfd-82b5fa60fd09-metrics-certs") pod "network-metrics-daemon-4njfl" (UID: "3eb3cd73-4ea1-4950-8bfd-82b5fa60fd09") : secret "metrics-daemon-secret" not found
Sep 18 19:41:57.876168 ip-10-0-136-185 kubenswrapper[2585]: E0918 19:41:57.876121    2585 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-dns/dns-default-grfwt" podUID="13a57b4c-73f9-4960-89d4-e9c25d392279"
Sep 18 19:41:58.188911 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:41:58.188885    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-grfwt"
Sep 18 19:41:58.675914 ip-10-0-136-185 kubenswrapper[2585]: E0918 19:41:58.675864    2585 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-certs], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-multus/network-metrics-daemon-4njfl" podUID="3eb3cd73-4ea1-4950-8bfd-82b5fa60fd09"
Sep 18 19:42:02.757808 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:42:02.757749    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/13a57b4c-73f9-4960-89d4-e9c25d392279-metrics-tls\") pod \"dns-default-grfwt\" (UID: \"13a57b4c-73f9-4960-89d4-e9c25d392279\") " pod="openshift-dns/dns-default-grfwt"
Sep 18 19:42:02.758293 ip-10-0-136-185 kubenswrapper[2585]: E0918 19:42:02.757893    2585 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 18 19:42:02.758293 ip-10-0-136-185 kubenswrapper[2585]: E0918 19:42:02.757965    2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/13a57b4c-73f9-4960-89d4-e9c25d392279-metrics-tls podName:13a57b4c-73f9-4960-89d4-e9c25d392279 nodeName:}" failed. No retries permitted until 2026-09-18 19:44:04.757945585 +0000 UTC m=+286.282550987 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/13a57b4c-73f9-4960-89d4-e9c25d392279-metrics-tls") pod "dns-default-grfwt" (UID: "13a57b4c-73f9-4960-89d4-e9c25d392279") : secret "dns-default-metrics-tls" not found
Sep 18 19:42:10.656755 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:42:10.656719    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4njfl"
Sep 18 19:42:47.328287 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:42:47.328254    2585 generic.go:356] "Generic (PLEG): container finished" podID="7f1db892-607a-4a03-b019-b33e96d89944" containerID="f7b50a4d6fe31fd1301b7acee0dfdaa3c9a9bd8f3d03bd741fc944c1003db062" exitCode=1
Sep 18 19:42:47.328810 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:42:47.328334    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-854d4bb9f9-d7fxb" event={"ID":"7f1db892-607a-4a03-b019-b33e96d89944","Type":"ContainerDied","Data":"f7b50a4d6fe31fd1301b7acee0dfdaa3c9a9bd8f3d03bd741fc944c1003db062"}
Sep 18 19:42:47.328810 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:42:47.328731    2585 scope.go:117] "RemoveContainer" containerID="f7b50a4d6fe31fd1301b7acee0dfdaa3c9a9bd8f3d03bd741fc944c1003db062"
Sep 18 19:42:47.329891 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:42:47.329868    2585 generic.go:356] "Generic (PLEG): container finished" podID="050e1951-5ccc-4c96-80f4-8a7f52621f7b" containerID="43e1740eeb8cb15895748f228638e892c9cc9a83d04b0891f2d303211b475a62" exitCode=255
Sep 18 19:42:47.330006 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:42:47.329890    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7786448cdb-pqk46" event={"ID":"050e1951-5ccc-4c96-80f4-8a7f52621f7b","Type":"ContainerDied","Data":"43e1740eeb8cb15895748f228638e892c9cc9a83d04b0891f2d303211b475a62"}
Sep 18 19:42:47.335880 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:42:47.335862    2585 scope.go:117] "RemoveContainer" containerID="43e1740eeb8cb15895748f228638e892c9cc9a83d04b0891f2d303211b475a62"
Sep 18 19:42:47.988561 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:42:47.988524    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-854d4bb9f9-d7fxb"
Sep 18 19:42:48.334361 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:42:48.334267    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7786448cdb-pqk46" event={"ID":"050e1951-5ccc-4c96-80f4-8a7f52621f7b","Type":"ContainerStarted","Data":"bba64f06c25114948acf77234617f3d8a7218b577683902568f65dfa11ff45bc"}
Sep 18 19:42:48.335893 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:42:48.335870    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-854d4bb9f9-d7fxb" event={"ID":"7f1db892-607a-4a03-b019-b33e96d89944","Type":"ContainerStarted","Data":"28d6e16ba307d335231baacc01f325418f003451272fc73281f0336ecf9caa1e"}
Sep 18 19:42:48.336066 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:42:48.336052    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-854d4bb9f9-d7fxb"
Sep 18 19:42:48.337096 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:42:48.337051    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-854d4bb9f9-d7fxb"
Sep 18 19:43:32.459371 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:43:32.459282    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3eb3cd73-4ea1-4950-8bfd-82b5fa60fd09-metrics-certs\") pod \"network-metrics-daemon-4njfl\" (UID: \"3eb3cd73-4ea1-4950-8bfd-82b5fa60fd09\") " pod="openshift-multus/network-metrics-daemon-4njfl"
Sep 18 19:43:32.459870 ip-10-0-136-185 kubenswrapper[2585]: E0918 19:43:32.459425    2585 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Sep 18 19:43:32.459870 ip-10-0-136-185 kubenswrapper[2585]: E0918 19:43:32.459478    2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3eb3cd73-4ea1-4950-8bfd-82b5fa60fd09-metrics-certs podName:3eb3cd73-4ea1-4950-8bfd-82b5fa60fd09 nodeName:}" failed. No retries permitted until 2026-09-18 19:45:34.459464561 +0000 UTC m=+375.984069962 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/3eb3cd73-4ea1-4950-8bfd-82b5fa60fd09-metrics-certs") pod "network-metrics-daemon-4njfl" (UID: "3eb3cd73-4ea1-4950-8bfd-82b5fa60fd09") : secret "metrics-daemon-secret" not found
Sep 18 19:43:47.447865 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:43:47.447834    2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-service-ca-operator/service-ca-operator-57cdc4b768-xpd8n"]
Sep 18 19:43:47.449770 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:43:47.449753    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-57cdc4b768-xpd8n"
Sep 18 19:43:47.452381 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:43:47.452355    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"serving-cert\""
Sep 18 19:43:47.452489 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:43:47.452357    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"openshift-service-ca.crt\""
Sep 18 19:43:47.452489 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:43:47.452358    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-config\""
Sep 18 19:43:47.452489 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:43:47.452452    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-dockercfg-z6rz4\""
Sep 18 19:43:47.453110 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:43:47.453094    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"kube-root-ca.crt\""
Sep 18 19:43:47.455206 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:43:47.455183    2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-5c4db8bdf9-gpcjl"]
Sep 18 19:43:47.457219 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:43:47.457084    2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-operator-6f9c644dd9-6bc55"]
Sep 18 19:43:47.457660 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:43:47.457642    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-5c4db8bdf9-gpcjl"
Sep 18 19:43:47.458869 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:43:47.458849    2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-75d9fb895b-85gqr"]
Sep 18 19:43:47.458986 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:43:47.458970    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-6f9c644dd9-6bc55"
Sep 18 19:43:47.460591 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:43:47.460573    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-75d9fb895b-85gqr"
Sep 18 19:43:47.463848 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:43:47.463824    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Sep 18 19:43:47.464070 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:43:47.463860    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-storage-operator\"/\"volume-data-source-validator-dockercfg-f86hc\""
Sep 18 19:43:47.464168 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:43:47.463939    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\""
Sep 18 19:43:47.464228 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:43:47.464074    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"kube-root-ca.crt\""
Sep 18 19:43:47.464647 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:43:47.464584    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-57cdc4b768-xpd8n"]
Sep 18 19:43:47.464647 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:43:47.464634    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"openshift-insights-serving-cert\""
Sep 18 19:43:47.464808 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:43:47.464634    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Sep 18 19:43:47.468128 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:43:47.468105    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\""
Sep 18 19:43:47.468242 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:43:47.468144    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\""
Sep 18 19:43:47.468485 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:43:47.468393    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\""
Sep 18 19:43:47.468485 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:43:47.468401    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-b9rnh\""
Sep 18 19:43:47.468485 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:43:47.468437    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"openshift-service-ca.crt\""
Sep 18 19:43:47.468485 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:43:47.468448    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-9x7s9\""
Sep 18 19:43:47.473154 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:43:47.473041    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-5c4db8bdf9-gpcjl"]
Sep 18 19:43:47.474395 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:43:47.474082    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\""
Sep 18 19:43:47.481587 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:43:47.477609    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-6f9c644dd9-6bc55"]
Sep 18 19:43:47.481587 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:43:47.480840    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-75d9fb895b-85gqr"]
Sep 18 19:43:47.482038 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:43:47.482021    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\""
Sep 18 19:43:47.562990 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:43:47.562955    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-59cdq\" (UniqueName: \"kubernetes.io/projected/79747afc-194b-4ded-bdd1-d6332f8412b7-kube-api-access-59cdq\") pod \"image-registry-75d9fb895b-85gqr\" (UID: \"79747afc-194b-4ded-bdd1-d6332f8412b7\") " pod="openshift-image-registry/image-registry-75d9fb895b-85gqr"
Sep 18 19:43:47.563171 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:43:47.563007    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tt25v\" (UniqueName: \"kubernetes.io/projected/d6560c03-8e80-4d69-b52c-73e63032f2ec-kube-api-access-tt25v\") pod \"insights-operator-6f9c644dd9-6bc55\" (UID: \"d6560c03-8e80-4d69-b52c-73e63032f2ec\") " pod="openshift-insights/insights-operator-6f9c644dd9-6bc55"
Sep 18 19:43:47.563171 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:43:47.563052    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/79747afc-194b-4ded-bdd1-d6332f8412b7-trusted-ca\") pod \"image-registry-75d9fb895b-85gqr\" (UID: \"79747afc-194b-4ded-bdd1-d6332f8412b7\") " pod="openshift-image-registry/image-registry-75d9fb895b-85gqr"
Sep 18 19:43:47.563171 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:43:47.563097    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/79747afc-194b-4ded-bdd1-d6332f8412b7-bound-sa-token\") pod \"image-registry-75d9fb895b-85gqr\" (UID: \"79747afc-194b-4ded-bdd1-d6332f8412b7\") " pod="openshift-image-registry/image-registry-75d9fb895b-85gqr"
Sep 18 19:43:47.563171 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:43:47.563118    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/79747afc-194b-4ded-bdd1-d6332f8412b7-registry-tls\") pod \"image-registry-75d9fb895b-85gqr\" (UID: \"79747afc-194b-4ded-bdd1-d6332f8412b7\") " pod="openshift-image-registry/image-registry-75d9fb895b-85gqr"
Sep 18 19:43:47.563171 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:43:47.563138    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d6560c03-8e80-4d69-b52c-73e63032f2ec-service-ca-bundle\") pod \"insights-operator-6f9c644dd9-6bc55\" (UID: \"d6560c03-8e80-4d69-b52c-73e63032f2ec\") " pod="openshift-insights/insights-operator-6f9c644dd9-6bc55"
Sep 18 19:43:47.563171 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:43:47.563158    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/79747afc-194b-4ded-bdd1-d6332f8412b7-installation-pull-secrets\") pod \"image-registry-75d9fb895b-85gqr\" (UID: \"79747afc-194b-4ded-bdd1-d6332f8412b7\") " pod="openshift-image-registry/image-registry-75d9fb895b-85gqr"
Sep 18 19:43:47.563386 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:43:47.563182    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d6560c03-8e80-4d69-b52c-73e63032f2ec-tmp\") pod \"insights-operator-6f9c644dd9-6bc55\" (UID: \"d6560c03-8e80-4d69-b52c-73e63032f2ec\") " pod="openshift-insights/insights-operator-6f9c644dd9-6bc55"
Sep 18 19:43:47.563386 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:43:47.563201    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d6560c03-8e80-4d69-b52c-73e63032f2ec-trusted-ca-bundle\") pod \"insights-operator-6f9c644dd9-6bc55\" (UID: \"d6560c03-8e80-4d69-b52c-73e63032f2ec\") " pod="openshift-insights/insights-operator-6f9c644dd9-6bc55"
Sep 18 19:43:47.563386 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:43:47.563217    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/d6560c03-8e80-4d69-b52c-73e63032f2ec-snapshots\") pod \"insights-operator-6f9c644dd9-6bc55\" (UID: \"d6560c03-8e80-4d69-b52c-73e63032f2ec\") " pod="openshift-insights/insights-operator-6f9c644dd9-6bc55"
Sep 18 19:43:47.563386 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:43:47.563249    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/79747afc-194b-4ded-bdd1-d6332f8412b7-ca-trust-extracted\") pod \"image-registry-75d9fb895b-85gqr\" (UID: \"79747afc-194b-4ded-bdd1-d6332f8412b7\") " pod="openshift-image-registry/image-registry-75d9fb895b-85gqr"
Sep 18 19:43:47.563386 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:43:47.563275    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/a9344bdb-53d7-4066-a8ef-7d28173a0617-serving-cert\") pod \"service-ca-operator-57cdc4b768-xpd8n\" (UID: \"a9344bdb-53d7-4066-a8ef-7d28173a0617\") " pod="openshift-service-ca-operator/service-ca-operator-57cdc4b768-xpd8n"
Sep 18 19:43:47.563386 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:43:47.563294    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/79747afc-194b-4ded-bdd1-d6332f8412b7-registry-certificates\") pod \"image-registry-75d9fb895b-85gqr\" (UID: \"79747afc-194b-4ded-bdd1-d6332f8412b7\") " pod="openshift-image-registry/image-registry-75d9fb895b-85gqr"
Sep 18 19:43:47.563386 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:43:47.563319    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/d6560c03-8e80-4d69-b52c-73e63032f2ec-serving-cert\") pod \"insights-operator-6f9c644dd9-6bc55\" (UID: \"d6560c03-8e80-4d69-b52c-73e63032f2ec\") " pod="openshift-insights/insights-operator-6f9c644dd9-6bc55"
Sep 18 19:43:47.563386 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:43:47.563341    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g52kt\" (UniqueName: \"kubernetes.io/projected/a9344bdb-53d7-4066-a8ef-7d28173a0617-kube-api-access-g52kt\") pod \"service-ca-operator-57cdc4b768-xpd8n\" (UID: \"a9344bdb-53d7-4066-a8ef-7d28173a0617\") " pod="openshift-service-ca-operator/service-ca-operator-57cdc4b768-xpd8n"
Sep 18 19:43:47.563644 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:43:47.563388    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/a9344bdb-53d7-4066-a8ef-7d28173a0617-config\") pod \"service-ca-operator-57cdc4b768-xpd8n\" (UID: \"a9344bdb-53d7-4066-a8ef-7d28173a0617\") " pod="openshift-service-ca-operator/service-ca-operator-57cdc4b768-xpd8n"
Sep 18 19:43:47.563644 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:43:47.563408    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/79747afc-194b-4ded-bdd1-d6332f8412b7-image-registry-private-configuration\") pod \"image-registry-75d9fb895b-85gqr\" (UID: \"79747afc-194b-4ded-bdd1-d6332f8412b7\") " pod="openshift-image-registry/image-registry-75d9fb895b-85gqr"
Sep 18 19:43:47.563644 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:43:47.563427    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qn5t8\" (UniqueName: \"kubernetes.io/projected/16c53ed7-e3cd-4eb5-8afa-3ee430dbe58f-kube-api-access-qn5t8\") pod \"volume-data-source-validator-5c4db8bdf9-gpcjl\" (UID: \"16c53ed7-e3cd-4eb5-8afa-3ee430dbe58f\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-5c4db8bdf9-gpcjl"
Sep 18 19:43:47.664432 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:43:47.664391    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tt25v\" (UniqueName: \"kubernetes.io/projected/d6560c03-8e80-4d69-b52c-73e63032f2ec-kube-api-access-tt25v\") pod \"insights-operator-6f9c644dd9-6bc55\" (UID: \"d6560c03-8e80-4d69-b52c-73e63032f2ec\") " pod="openshift-insights/insights-operator-6f9c644dd9-6bc55"
Sep 18 19:43:47.664432 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:43:47.664431    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/79747afc-194b-4ded-bdd1-d6332f8412b7-trusted-ca\") pod \"image-registry-75d9fb895b-85gqr\" (UID: \"79747afc-194b-4ded-bdd1-d6332f8412b7\") " pod="openshift-image-registry/image-registry-75d9fb895b-85gqr"
Sep 18 19:43:47.664668 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:43:47.664464    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/79747afc-194b-4ded-bdd1-d6332f8412b7-bound-sa-token\") pod \"image-registry-75d9fb895b-85gqr\" (UID: \"79747afc-194b-4ded-bdd1-d6332f8412b7\") " pod="openshift-image-registry/image-registry-75d9fb895b-85gqr"
Sep 18 19:43:47.664668 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:43:47.664492    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/79747afc-194b-4ded-bdd1-d6332f8412b7-registry-tls\") pod \"image-registry-75d9fb895b-85gqr\" (UID: \"79747afc-194b-4ded-bdd1-d6332f8412b7\") " pod="openshift-image-registry/image-registry-75d9fb895b-85gqr"
Sep 18 19:43:47.664668 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:43:47.664518    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d6560c03-8e80-4d69-b52c-73e63032f2ec-service-ca-bundle\") pod \"insights-operator-6f9c644dd9-6bc55\" (UID: \"d6560c03-8e80-4d69-b52c-73e63032f2ec\") " pod="openshift-insights/insights-operator-6f9c644dd9-6bc55"
Sep 18 19:43:47.664668 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:43:47.664547    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/79747afc-194b-4ded-bdd1-d6332f8412b7-installation-pull-secrets\") pod \"image-registry-75d9fb895b-85gqr\" (UID: \"79747afc-194b-4ded-bdd1-d6332f8412b7\") " pod="openshift-image-registry/image-registry-75d9fb895b-85gqr"
Sep 18 19:43:47.664668 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:43:47.664568    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d6560c03-8e80-4d69-b52c-73e63032f2ec-tmp\") pod \"insights-operator-6f9c644dd9-6bc55\" (UID: \"d6560c03-8e80-4d69-b52c-73e63032f2ec\") " pod="openshift-insights/insights-operator-6f9c644dd9-6bc55"
Sep 18 19:43:47.664668 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:43:47.664592    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d6560c03-8e80-4d69-b52c-73e63032f2ec-trusted-ca-bundle\") pod \"insights-operator-6f9c644dd9-6bc55\" (UID: \"d6560c03-8e80-4d69-b52c-73e63032f2ec\") " pod="openshift-insights/insights-operator-6f9c644dd9-6bc55"
Sep 18 19:43:47.664668 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:43:47.664617    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/d6560c03-8e80-4d69-b52c-73e63032f2ec-snapshots\") pod \"insights-operator-6f9c644dd9-6bc55\" (UID: \"d6560c03-8e80-4d69-b52c-73e63032f2ec\") " pod="openshift-insights/insights-operator-6f9c644dd9-6bc55"
Sep 18 19:43:47.664668 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:43:47.664637    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/79747afc-194b-4ded-bdd1-d6332f8412b7-ca-trust-extracted\") pod \"image-registry-75d9fb895b-85gqr\" (UID: \"79747afc-194b-4ded-bdd1-d6332f8412b7\") " pod="openshift-image-registry/image-registry-75d9fb895b-85gqr"
Sep 18 19:43:47.664668 ip-10-0-136-185 kubenswrapper[2585]: E0918 19:43:47.664644    2585 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 18 19:43:47.664668 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:43:47.664661    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/a9344bdb-53d7-4066-a8ef-7d28173a0617-serving-cert\") pod \"service-ca-operator-57cdc4b768-xpd8n\" (UID: \"a9344bdb-53d7-4066-a8ef-7d28173a0617\") " pod="openshift-service-ca-operator/service-ca-operator-57cdc4b768-xpd8n"
Sep 18 19:43:47.665127 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:43:47.664688    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/79747afc-194b-4ded-bdd1-d6332f8412b7-registry-certificates\") pod \"image-registry-75d9fb895b-85gqr\" (UID: \"79747afc-194b-4ded-bdd1-d6332f8412b7\") " pod="openshift-image-registry/image-registry-75d9fb895b-85gqr"
Sep 18 19:43:47.665127 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:43:47.664709    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/d6560c03-8e80-4d69-b52c-73e63032f2ec-serving-cert\") pod \"insights-operator-6f9c644dd9-6bc55\" (UID: \"d6560c03-8e80-4d69-b52c-73e63032f2ec\") " pod="openshift-insights/insights-operator-6f9c644dd9-6bc55"
Sep 18 19:43:47.665127 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:43:47.664731    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g52kt\" (UniqueName: \"kubernetes.io/projected/a9344bdb-53d7-4066-a8ef-7d28173a0617-kube-api-access-g52kt\") pod \"service-ca-operator-57cdc4b768-xpd8n\" (UID: \"a9344bdb-53d7-4066-a8ef-7d28173a0617\") " pod="openshift-service-ca-operator/service-ca-operator-57cdc4b768-xpd8n"
Sep 18 19:43:47.665127 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:43:47.664758    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/a9344bdb-53d7-4066-a8ef-7d28173a0617-config\") pod \"service-ca-operator-57cdc4b768-xpd8n\" (UID: \"a9344bdb-53d7-4066-a8ef-7d28173a0617\") " pod="openshift-service-ca-operator/service-ca-operator-57cdc4b768-xpd8n"
Sep 18 19:43:47.665127 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:43:47.664817    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/79747afc-194b-4ded-bdd1-d6332f8412b7-image-registry-private-configuration\") pod \"image-registry-75d9fb895b-85gqr\" (UID: \"79747afc-194b-4ded-bdd1-d6332f8412b7\") " pod="openshift-image-registry/image-registry-75d9fb895b-85gqr"
Sep 18 19:43:47.665127 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:43:47.664848    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qn5t8\" (UniqueName: \"kubernetes.io/projected/16c53ed7-e3cd-4eb5-8afa-3ee430dbe58f-kube-api-access-qn5t8\") pod \"volume-data-source-validator-5c4db8bdf9-gpcjl\" (UID: \"16c53ed7-e3cd-4eb5-8afa-3ee430dbe58f\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-5c4db8bdf9-gpcjl"
Sep 18 19:43:47.665127 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:43:47.664875    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-59cdq\" (UniqueName: \"kubernetes.io/projected/79747afc-194b-4ded-bdd1-d6332f8412b7-kube-api-access-59cdq\") pod \"image-registry-75d9fb895b-85gqr\" (UID: \"79747afc-194b-4ded-bdd1-d6332f8412b7\") " pod="openshift-image-registry/image-registry-75d9fb895b-85gqr"
Sep 18 19:43:47.665127 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:43:47.665069    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d6560c03-8e80-4d69-b52c-73e63032f2ec-tmp\") pod \"insights-operator-6f9c644dd9-6bc55\" (UID: \"d6560c03-8e80-4d69-b52c-73e63032f2ec\") " pod="openshift-insights/insights-operator-6f9c644dd9-6bc55"
Sep 18 19:43:47.665504 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:43:47.665346    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/79747afc-194b-4ded-bdd1-d6332f8412b7-ca-trust-extracted\") pod \"image-registry-75d9fb895b-85gqr\" (UID: \"79747afc-194b-4ded-bdd1-d6332f8412b7\") " pod="openshift-image-registry/image-registry-75d9fb895b-85gqr"
Sep 18 19:43:47.665504 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:43:47.665389    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/d6560c03-8e80-4d69-b52c-73e63032f2ec-snapshots\") pod \"insights-operator-6f9c644dd9-6bc55\" (UID: \"d6560c03-8e80-4d69-b52c-73e63032f2ec\") " pod="openshift-insights/insights-operator-6f9c644dd9-6bc55"
Sep 18 19:43:47.665504 ip-10-0-136-185 kubenswrapper[2585]: E0918 19:43:47.664666    2585 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-75d9fb895b-85gqr: secret "image-registry-tls" not found
Sep 18 19:43:47.665683 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:43:47.665656    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/79747afc-194b-4ded-bdd1-d6332f8412b7-trusted-ca\") pod \"image-registry-75d9fb895b-85gqr\" (UID: \"79747afc-194b-4ded-bdd1-d6332f8412b7\") " pod="openshift-image-registry/image-registry-75d9fb895b-85gqr"
Sep 18 19:43:47.665819 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:43:47.665775    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d6560c03-8e80-4d69-b52c-73e63032f2ec-service-ca-bundle\") pod \"insights-operator-6f9c644dd9-6bc55\" (UID: \"d6560c03-8e80-4d69-b52c-73e63032f2ec\") " pod="openshift-insights/insights-operator-6f9c644dd9-6bc55"
Sep 18 19:43:47.665941 ip-10-0-136-185 kubenswrapper[2585]: E0918 19:43:47.665809    2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/79747afc-194b-4ded-bdd1-d6332f8412b7-registry-tls podName:79747afc-194b-4ded-bdd1-d6332f8412b7 nodeName:}" failed. No retries permitted until 2026-09-18 19:43:48.165770932 +0000 UTC m=+269.690376348 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/79747afc-194b-4ded-bdd1-d6332f8412b7-registry-tls") pod "image-registry-75d9fb895b-85gqr" (UID: "79747afc-194b-4ded-bdd1-d6332f8412b7") : secret "image-registry-tls" not found
Sep 18 19:43:47.665941 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:43:47.665883    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/a9344bdb-53d7-4066-a8ef-7d28173a0617-config\") pod \"service-ca-operator-57cdc4b768-xpd8n\" (UID: \"a9344bdb-53d7-4066-a8ef-7d28173a0617\") " pod="openshift-service-ca-operator/service-ca-operator-57cdc4b768-xpd8n"
Sep 18 19:43:47.666171 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:43:47.666151    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/79747afc-194b-4ded-bdd1-d6332f8412b7-registry-certificates\") pod \"image-registry-75d9fb895b-85gqr\" (UID: \"79747afc-194b-4ded-bdd1-d6332f8412b7\") " pod="openshift-image-registry/image-registry-75d9fb895b-85gqr"
Sep 18 19:43:47.666600 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:43:47.666574    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d6560c03-8e80-4d69-b52c-73e63032f2ec-trusted-ca-bundle\") pod \"insights-operator-6f9c644dd9-6bc55\" (UID: \"d6560c03-8e80-4d69-b52c-73e63032f2ec\") " pod="openshift-insights/insights-operator-6f9c644dd9-6bc55"
Sep 18 19:43:47.667419 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:43:47.667394    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/d6560c03-8e80-4d69-b52c-73e63032f2ec-serving-cert\") pod \"insights-operator-6f9c644dd9-6bc55\" (UID: \"d6560c03-8e80-4d69-b52c-73e63032f2ec\") " pod="openshift-insights/insights-operator-6f9c644dd9-6bc55"
Sep 18 19:43:47.667640 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:43:47.667623    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/79747afc-194b-4ded-bdd1-d6332f8412b7-installation-pull-secrets\") pod \"image-registry-75d9fb895b-85gqr\" (UID: \"79747afc-194b-4ded-bdd1-d6332f8412b7\") " pod="openshift-image-registry/image-registry-75d9fb895b-85gqr"
Sep 18 19:43:47.667857 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:43:47.667841    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/a9344bdb-53d7-4066-a8ef-7d28173a0617-serving-cert\") pod \"service-ca-operator-57cdc4b768-xpd8n\" (UID: \"a9344bdb-53d7-4066-a8ef-7d28173a0617\") " pod="openshift-service-ca-operator/service-ca-operator-57cdc4b768-xpd8n"
Sep 18 19:43:47.667943 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:43:47.667926    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/79747afc-194b-4ded-bdd1-d6332f8412b7-image-registry-private-configuration\") pod \"image-registry-75d9fb895b-85gqr\" (UID: \"79747afc-194b-4ded-bdd1-d6332f8412b7\") " pod="openshift-image-registry/image-registry-75d9fb895b-85gqr"
Sep 18 19:43:47.673853 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:43:47.673829    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tt25v\" (UniqueName: \"kubernetes.io/projected/d6560c03-8e80-4d69-b52c-73e63032f2ec-kube-api-access-tt25v\") pod \"insights-operator-6f9c644dd9-6bc55\" (UID: \"d6560c03-8e80-4d69-b52c-73e63032f2ec\") " pod="openshift-insights/insights-operator-6f9c644dd9-6bc55"
Sep 18 19:43:47.674047 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:43:47.674022    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/79747afc-194b-4ded-bdd1-d6332f8412b7-bound-sa-token\") pod \"image-registry-75d9fb895b-85gqr\" (UID: \"79747afc-194b-4ded-bdd1-d6332f8412b7\") " pod="openshift-image-registry/image-registry-75d9fb895b-85gqr"
Sep 18 19:43:47.674389 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:43:47.674371    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-59cdq\" (UniqueName: \"kubernetes.io/projected/79747afc-194b-4ded-bdd1-d6332f8412b7-kube-api-access-59cdq\") pod \"image-registry-75d9fb895b-85gqr\" (UID: \"79747afc-194b-4ded-bdd1-d6332f8412b7\") " pod="openshift-image-registry/image-registry-75d9fb895b-85gqr"
Sep 18 19:43:47.674512 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:43:47.674496    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-g52kt\" (UniqueName: \"kubernetes.io/projected/a9344bdb-53d7-4066-a8ef-7d28173a0617-kube-api-access-g52kt\") pod \"service-ca-operator-57cdc4b768-xpd8n\" (UID: \"a9344bdb-53d7-4066-a8ef-7d28173a0617\") " pod="openshift-service-ca-operator/service-ca-operator-57cdc4b768-xpd8n"
Sep 18 19:43:47.674719 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:43:47.674695    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qn5t8\" (UniqueName: \"kubernetes.io/projected/16c53ed7-e3cd-4eb5-8afa-3ee430dbe58f-kube-api-access-qn5t8\") pod \"volume-data-source-validator-5c4db8bdf9-gpcjl\" (UID: \"16c53ed7-e3cd-4eb5-8afa-3ee430dbe58f\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-5c4db8bdf9-gpcjl"
Sep 18 19:43:47.760600 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:43:47.760510    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-57cdc4b768-xpd8n"
Sep 18 19:43:47.770514 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:43:47.770416    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-5c4db8bdf9-gpcjl"
Sep 18 19:43:47.779582 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:43:47.779411    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-6f9c644dd9-6bc55"
Sep 18 19:43:48.041238 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:43:48.041169    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-57cdc4b768-xpd8n"]
Sep 18 19:43:48.044054 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:43:48.044030    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-5c4db8bdf9-gpcjl"]
Sep 18 19:43:48.044506 ip-10-0-136-185 kubenswrapper[2585]: W0918 19:43:48.044483    2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda9344bdb_53d7_4066_a8ef_7d28173a0617.slice/crio-a7f5c1f8497a722aa09076b039f435ad2a8015cca55fc14ff7318f7276adc6b0 WatchSource:0}: Error finding container a7f5c1f8497a722aa09076b039f435ad2a8015cca55fc14ff7318f7276adc6b0: Status 404 returned error can't find the container with id a7f5c1f8497a722aa09076b039f435ad2a8015cca55fc14ff7318f7276adc6b0
Sep 18 19:43:48.046815 ip-10-0-136-185 kubenswrapper[2585]: W0918 19:43:48.046775    2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod16c53ed7_e3cd_4eb5_8afa_3ee430dbe58f.slice/crio-721c493925cdbc75a5f11b5660299ceda84b558b75366ebcb49cd457e393119f WatchSource:0}: Error finding container 721c493925cdbc75a5f11b5660299ceda84b558b75366ebcb49cd457e393119f: Status 404 returned error can't find the container with id 721c493925cdbc75a5f11b5660299ceda84b558b75366ebcb49cd457e393119f
Sep 18 19:43:48.067672 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:43:48.067641    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-6f9c644dd9-6bc55"]
Sep 18 19:43:48.070334 ip-10-0-136-185 kubenswrapper[2585]: W0918 19:43:48.070306    2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd6560c03_8e80_4d69_b52c_73e63032f2ec.slice/crio-aa71ccd142ee13d85b129159f4c5f92ba491d0db94e5ccee6851e7625df96470 WatchSource:0}: Error finding container aa71ccd142ee13d85b129159f4c5f92ba491d0db94e5ccee6851e7625df96470: Status 404 returned error can't find the container with id aa71ccd142ee13d85b129159f4c5f92ba491d0db94e5ccee6851e7625df96470
Sep 18 19:43:48.168783 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:43:48.168748    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/79747afc-194b-4ded-bdd1-d6332f8412b7-registry-tls\") pod \"image-registry-75d9fb895b-85gqr\" (UID: \"79747afc-194b-4ded-bdd1-d6332f8412b7\") " pod="openshift-image-registry/image-registry-75d9fb895b-85gqr"
Sep 18 19:43:48.168962 ip-10-0-136-185 kubenswrapper[2585]: E0918 19:43:48.168898    2585 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 18 19:43:48.168962 ip-10-0-136-185 kubenswrapper[2585]: E0918 19:43:48.168920    2585 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-75d9fb895b-85gqr: secret "image-registry-tls" not found
Sep 18 19:43:48.169030 ip-10-0-136-185 kubenswrapper[2585]: E0918 19:43:48.168972    2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/79747afc-194b-4ded-bdd1-d6332f8412b7-registry-tls podName:79747afc-194b-4ded-bdd1-d6332f8412b7 nodeName:}" failed. No retries permitted until 2026-09-18 19:43:49.168957665 +0000 UTC m=+270.693563071 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/79747afc-194b-4ded-bdd1-d6332f8412b7-registry-tls") pod "image-registry-75d9fb895b-85gqr" (UID: "79747afc-194b-4ded-bdd1-d6332f8412b7") : secret "image-registry-tls" not found
Sep 18 19:43:48.511059 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:43:48.510966    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-57cdc4b768-xpd8n" event={"ID":"a9344bdb-53d7-4066-a8ef-7d28173a0617","Type":"ContainerStarted","Data":"a7f5c1f8497a722aa09076b039f435ad2a8015cca55fc14ff7318f7276adc6b0"}
Sep 18 19:43:48.512654 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:43:48.512618    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-6f9c644dd9-6bc55" event={"ID":"d6560c03-8e80-4d69-b52c-73e63032f2ec","Type":"ContainerStarted","Data":"aa71ccd142ee13d85b129159f4c5f92ba491d0db94e5ccee6851e7625df96470"}
Sep 18 19:43:48.515585 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:43:48.515532    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-5c4db8bdf9-gpcjl" event={"ID":"16c53ed7-e3cd-4eb5-8afa-3ee430dbe58f","Type":"ContainerStarted","Data":"721c493925cdbc75a5f11b5660299ceda84b558b75366ebcb49cd457e393119f"}
Sep 18 19:43:49.177488 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:43:49.177446    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/79747afc-194b-4ded-bdd1-d6332f8412b7-registry-tls\") pod \"image-registry-75d9fb895b-85gqr\" (UID: \"79747afc-194b-4ded-bdd1-d6332f8412b7\") " pod="openshift-image-registry/image-registry-75d9fb895b-85gqr"
Sep 18 19:43:49.177702 ip-10-0-136-185 kubenswrapper[2585]: E0918 19:43:49.177617    2585 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 18 19:43:49.177702 ip-10-0-136-185 kubenswrapper[2585]: E0918 19:43:49.177640    2585 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-75d9fb895b-85gqr: secret "image-registry-tls" not found
Sep 18 19:43:49.177702 ip-10-0-136-185 kubenswrapper[2585]: E0918 19:43:49.177701    2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/79747afc-194b-4ded-bdd1-d6332f8412b7-registry-tls podName:79747afc-194b-4ded-bdd1-d6332f8412b7 nodeName:}" failed. No retries permitted until 2026-09-18 19:43:51.177683035 +0000 UTC m=+272.702288438 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/79747afc-194b-4ded-bdd1-d6332f8412b7-registry-tls") pod "image-registry-75d9fb895b-85gqr" (UID: "79747afc-194b-4ded-bdd1-d6332f8412b7") : secret "image-registry-tls" not found
Sep 18 19:43:49.521226 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:43:49.521185    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-5c4db8bdf9-gpcjl" event={"ID":"16c53ed7-e3cd-4eb5-8afa-3ee430dbe58f","Type":"ContainerStarted","Data":"77cf89bf4ee60031f198e19ede41a886ee85d0a5f5c42a91236c0f0049e508da"}
Sep 18 19:43:50.525731 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:43:50.525696    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-6f9c644dd9-6bc55" event={"ID":"d6560c03-8e80-4d69-b52c-73e63032f2ec","Type":"ContainerStarted","Data":"e294a4c944c9a9de9266c3a9fcc6264209ec84b375ba66a42eedec9d325d6611"}
Sep 18 19:43:50.527231 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:43:50.527209    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-57cdc4b768-xpd8n" event={"ID":"a9344bdb-53d7-4066-a8ef-7d28173a0617","Type":"ContainerStarted","Data":"5f3e7d9d557dcc0dbe73ab04ba3b748892424d1ef374ede43a167d0ed4951e1f"}
Sep 18 19:43:50.546042 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:43:50.545938    2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-operator-6f9c644dd9-6bc55" podStartSLOduration=1.338374192 podStartE2EDuration="3.545916712s" totalImagesPullingTime="2.20754252s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 19:43:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 19:43:50.544598902 +0000 UTC m=+272.069204325" watchObservedRunningTime="2026-09-18 19:43:50.545916712 +0000 UTC m=+272.070522133"
Sep 18 19:43:50.546337 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:43:50.546308    2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-storage-operator/volume-data-source-validator-5c4db8bdf9-gpcjl" podStartSLOduration=2.606569901 podStartE2EDuration="3.546301056s" totalImagesPullingTime="939.731155ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 19:43:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 19:43:49.537213487 +0000 UTC m=+271.061818911" watchObservedRunningTime="2026-09-18 19:43:50.546301056 +0000 UTC m=+272.070906480"
Sep 18 19:43:50.562018 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:43:50.561958    2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-service-ca-operator/service-ca-operator-57cdc4b768-xpd8n" podStartSLOduration=1.326276815 podStartE2EDuration="3.561938629s" totalImagesPullingTime="2.235661814s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 19:43:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 19:43:50.561477519 +0000 UTC m=+272.086082944" watchObservedRunningTime="2026-09-18 19:43:50.561938629 +0000 UTC m=+272.086544054"
Sep 18 19:43:51.194775 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:43:51.194739    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/79747afc-194b-4ded-bdd1-d6332f8412b7-registry-tls\") pod \"image-registry-75d9fb895b-85gqr\" (UID: \"79747afc-194b-4ded-bdd1-d6332f8412b7\") " pod="openshift-image-registry/image-registry-75d9fb895b-85gqr"
Sep 18 19:43:51.194960 ip-10-0-136-185 kubenswrapper[2585]: E0918 19:43:51.194907    2585 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 18 19:43:51.194960 ip-10-0-136-185 kubenswrapper[2585]: E0918 19:43:51.194928    2585 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-75d9fb895b-85gqr: secret "image-registry-tls" not found
Sep 18 19:43:51.195097 ip-10-0-136-185 kubenswrapper[2585]: E0918 19:43:51.194993    2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/79747afc-194b-4ded-bdd1-d6332f8412b7-registry-tls podName:79747afc-194b-4ded-bdd1-d6332f8412b7 nodeName:}" failed. No retries permitted until 2026-09-18 19:43:55.194973743 +0000 UTC m=+276.719579162 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/79747afc-194b-4ded-bdd1-d6332f8412b7-registry-tls") pod "image-registry-75d9fb895b-85gqr" (UID: "79747afc-194b-4ded-bdd1-d6332f8412b7") : secret "image-registry-tls" not found
Sep 18 19:43:51.818835 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:43:51.818801    2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-7czbj"]
Sep 18 19:43:51.821051 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:43:51.821030    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-7czbj"
Sep 18 19:43:51.823519 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:43:51.823494    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Sep 18 19:43:51.824011 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:43:51.823988    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Sep 18 19:43:51.824352 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:43:51.824330    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-ldqld\""
Sep 18 19:43:51.832853 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:43:51.832828    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-7czbj"]
Sep 18 19:43:52.000779 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:43:52.000744    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n4tjf\" (UniqueName: \"kubernetes.io/projected/f3c00964-2160-44d6-8699-354dae4ac4a5-kube-api-access-n4tjf\") pod \"insights-runtime-extractor-7czbj\" (UID: \"f3c00964-2160-44d6-8699-354dae4ac4a5\") " pod="openshift-insights/insights-runtime-extractor-7czbj"
Sep 18 19:43:52.000961 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:43:52.000782    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/f3c00964-2160-44d6-8699-354dae4ac4a5-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-7czbj\" (UID: \"f3c00964-2160-44d6-8699-354dae4ac4a5\") " pod="openshift-insights/insights-runtime-extractor-7czbj"
Sep 18 19:43:52.000961 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:43:52.000883    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/f3c00964-2160-44d6-8699-354dae4ac4a5-data-volume\") pod \"insights-runtime-extractor-7czbj\" (UID: \"f3c00964-2160-44d6-8699-354dae4ac4a5\") " pod="openshift-insights/insights-runtime-extractor-7czbj"
Sep 18 19:43:52.000961 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:43:52.000928    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/f3c00964-2160-44d6-8699-354dae4ac4a5-crio-socket\") pod \"insights-runtime-extractor-7czbj\" (UID: \"f3c00964-2160-44d6-8699-354dae4ac4a5\") " pod="openshift-insights/insights-runtime-extractor-7czbj"
Sep 18 19:43:52.000961 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:43:52.000947    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/f3c00964-2160-44d6-8699-354dae4ac4a5-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-7czbj\" (UID: \"f3c00964-2160-44d6-8699-354dae4ac4a5\") " pod="openshift-insights/insights-runtime-extractor-7czbj"
Sep 18 19:43:52.102318 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:43:52.102222    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/f3c00964-2160-44d6-8699-354dae4ac4a5-data-volume\") pod \"insights-runtime-extractor-7czbj\" (UID: \"f3c00964-2160-44d6-8699-354dae4ac4a5\") " pod="openshift-insights/insights-runtime-extractor-7czbj"
Sep 18 19:43:52.102318 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:43:52.102288    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/f3c00964-2160-44d6-8699-354dae4ac4a5-crio-socket\") pod \"insights-runtime-extractor-7czbj\" (UID: \"f3c00964-2160-44d6-8699-354dae4ac4a5\") " pod="openshift-insights/insights-runtime-extractor-7czbj"
Sep 18 19:43:52.102318 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:43:52.102317    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/f3c00964-2160-44d6-8699-354dae4ac4a5-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-7czbj\" (UID: \"f3c00964-2160-44d6-8699-354dae4ac4a5\") " pod="openshift-insights/insights-runtime-extractor-7czbj"
Sep 18 19:43:52.102585 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:43:52.102364    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n4tjf\" (UniqueName: \"kubernetes.io/projected/f3c00964-2160-44d6-8699-354dae4ac4a5-kube-api-access-n4tjf\") pod \"insights-runtime-extractor-7czbj\" (UID: \"f3c00964-2160-44d6-8699-354dae4ac4a5\") " pod="openshift-insights/insights-runtime-extractor-7czbj"
Sep 18 19:43:52.102585 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:43:52.102392    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/f3c00964-2160-44d6-8699-354dae4ac4a5-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-7czbj\" (UID: \"f3c00964-2160-44d6-8699-354dae4ac4a5\") " pod="openshift-insights/insights-runtime-extractor-7czbj"
Sep 18 19:43:52.102585 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:43:52.102427    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/f3c00964-2160-44d6-8699-354dae4ac4a5-crio-socket\") pod \"insights-runtime-extractor-7czbj\" (UID: \"f3c00964-2160-44d6-8699-354dae4ac4a5\") " pod="openshift-insights/insights-runtime-extractor-7czbj"
Sep 18 19:43:52.102585 ip-10-0-136-185 kubenswrapper[2585]: E0918 19:43:52.102446    2585 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 18 19:43:52.102585 ip-10-0-136-185 kubenswrapper[2585]: E0918 19:43:52.102529    2585 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 18 19:43:52.102585 ip-10-0-136-185 kubenswrapper[2585]: E0918 19:43:52.102545    2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/f3c00964-2160-44d6-8699-354dae4ac4a5-kube-rbac-proxy-cm podName:f3c00964-2160-44d6-8699-354dae4ac4a5 nodeName:}" failed. No retries permitted until 2026-09-18 19:43:52.602525694 +0000 UTC m=+274.127131101 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/f3c00964-2160-44d6-8699-354dae4ac4a5-kube-rbac-proxy-cm") pod "insights-runtime-extractor-7czbj" (UID: "f3c00964-2160-44d6-8699-354dae4ac4a5") : configmap "kube-rbac-proxy" not found
Sep 18 19:43:52.102585 ip-10-0-136-185 kubenswrapper[2585]: E0918 19:43:52.102590    2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f3c00964-2160-44d6-8699-354dae4ac4a5-insights-runtime-extractor-tls podName:f3c00964-2160-44d6-8699-354dae4ac4a5 nodeName:}" failed. No retries permitted until 2026-09-18 19:43:52.602571681 +0000 UTC m=+274.127177098 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/f3c00964-2160-44d6-8699-354dae4ac4a5-insights-runtime-extractor-tls") pod "insights-runtime-extractor-7czbj" (UID: "f3c00964-2160-44d6-8699-354dae4ac4a5") : secret "insights-runtime-extractor-tls" not found
Sep 18 19:43:52.102894 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:43:52.102627    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/f3c00964-2160-44d6-8699-354dae4ac4a5-data-volume\") pod \"insights-runtime-extractor-7czbj\" (UID: \"f3c00964-2160-44d6-8699-354dae4ac4a5\") " pod="openshift-insights/insights-runtime-extractor-7czbj"
Sep 18 19:43:52.111147 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:43:52.111130    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-n4tjf\" (UniqueName: \"kubernetes.io/projected/f3c00964-2160-44d6-8699-354dae4ac4a5-kube-api-access-n4tjf\") pod \"insights-runtime-extractor-7czbj\" (UID: \"f3c00964-2160-44d6-8699-354dae4ac4a5\") " pod="openshift-insights/insights-runtime-extractor-7czbj"
Sep 18 19:43:52.605070 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:43:52.605029    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/f3c00964-2160-44d6-8699-354dae4ac4a5-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-7czbj\" (UID: \"f3c00964-2160-44d6-8699-354dae4ac4a5\") " pod="openshift-insights/insights-runtime-extractor-7czbj"
Sep 18 19:43:52.605259 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:43:52.605088    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/f3c00964-2160-44d6-8699-354dae4ac4a5-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-7czbj\" (UID: \"f3c00964-2160-44d6-8699-354dae4ac4a5\") " pod="openshift-insights/insights-runtime-extractor-7czbj"
Sep 18 19:43:52.605259 ip-10-0-136-185 kubenswrapper[2585]: E0918 19:43:52.605176    2585 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 18 19:43:52.605259 ip-10-0-136-185 kubenswrapper[2585]: E0918 19:43:52.605203    2585 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 18 19:43:52.605259 ip-10-0-136-185 kubenswrapper[2585]: E0918 19:43:52.605249    2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f3c00964-2160-44d6-8699-354dae4ac4a5-insights-runtime-extractor-tls podName:f3c00964-2160-44d6-8699-354dae4ac4a5 nodeName:}" failed. No retries permitted until 2026-09-18 19:43:53.60523734 +0000 UTC m=+275.129842741 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/f3c00964-2160-44d6-8699-354dae4ac4a5-insights-runtime-extractor-tls") pod "insights-runtime-extractor-7czbj" (UID: "f3c00964-2160-44d6-8699-354dae4ac4a5") : secret "insights-runtime-extractor-tls" not found
Sep 18 19:43:52.605259 ip-10-0-136-185 kubenswrapper[2585]: E0918 19:43:52.605263    2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/f3c00964-2160-44d6-8699-354dae4ac4a5-kube-rbac-proxy-cm podName:f3c00964-2160-44d6-8699-354dae4ac4a5 nodeName:}" failed. No retries permitted until 2026-09-18 19:43:53.605256136 +0000 UTC m=+275.129861537 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/f3c00964-2160-44d6-8699-354dae4ac4a5-kube-rbac-proxy-cm") pod "insights-runtime-extractor-7czbj" (UID: "f3c00964-2160-44d6-8699-354dae4ac4a5") : configmap "kube-rbac-proxy" not found
Sep 18 19:43:53.611931 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:43:53.611896    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/f3c00964-2160-44d6-8699-354dae4ac4a5-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-7czbj\" (UID: \"f3c00964-2160-44d6-8699-354dae4ac4a5\") " pod="openshift-insights/insights-runtime-extractor-7czbj"
Sep 18 19:43:53.612308 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:43:53.611954    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/f3c00964-2160-44d6-8699-354dae4ac4a5-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-7czbj\" (UID: \"f3c00964-2160-44d6-8699-354dae4ac4a5\") " pod="openshift-insights/insights-runtime-extractor-7czbj"
Sep 18 19:43:53.612308 ip-10-0-136-185 kubenswrapper[2585]: E0918 19:43:53.612012    2585 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 18 19:43:53.612308 ip-10-0-136-185 kubenswrapper[2585]: E0918 19:43:53.612053    2585 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 18 19:43:53.612308 ip-10-0-136-185 kubenswrapper[2585]: E0918 19:43:53.612075    2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/f3c00964-2160-44d6-8699-354dae4ac4a5-kube-rbac-proxy-cm podName:f3c00964-2160-44d6-8699-354dae4ac4a5 nodeName:}" failed. No retries permitted until 2026-09-18 19:43:55.612059925 +0000 UTC m=+277.136665342 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/f3c00964-2160-44d6-8699-354dae4ac4a5-kube-rbac-proxy-cm") pod "insights-runtime-extractor-7czbj" (UID: "f3c00964-2160-44d6-8699-354dae4ac4a5") : configmap "kube-rbac-proxy" not found
Sep 18 19:43:53.612308 ip-10-0-136-185 kubenswrapper[2585]: E0918 19:43:53.612107    2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f3c00964-2160-44d6-8699-354dae4ac4a5-insights-runtime-extractor-tls podName:f3c00964-2160-44d6-8699-354dae4ac4a5 nodeName:}" failed. No retries permitted until 2026-09-18 19:43:55.612092517 +0000 UTC m=+277.136697918 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/f3c00964-2160-44d6-8699-354dae4ac4a5-insights-runtime-extractor-tls") pod "insights-runtime-extractor-7czbj" (UID: "f3c00964-2160-44d6-8699-354dae4ac4a5") : secret "insights-runtime-extractor-tls" not found
Sep 18 19:43:53.900726 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:43:53.900690    2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-service-ca/service-ca-55f6fc7b86-rqk6f"]
Sep 18 19:43:53.903750 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:43:53.903728    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-55f6fc7b86-rqk6f"
Sep 18 19:43:53.906549 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:43:53.906528    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"service-ca-dockercfg-42l6c\""
Sep 18 19:43:53.907108 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:43:53.907085    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"signing-cabundle\""
Sep 18 19:43:53.907222 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:43:53.907129    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"openshift-service-ca.crt\""
Sep 18 19:43:53.907413 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:43:53.907334    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"signing-key\""
Sep 18 19:43:53.907413 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:43:53.907354    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"kube-root-ca.crt\""
Sep 18 19:43:53.916675 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:43:53.916654    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-55f6fc7b86-rqk6f"]
Sep 18 19:43:54.016404 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:43:54.016352    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fcqxj\" (UniqueName: \"kubernetes.io/projected/07203fa1-5f7a-4ac3-964c-2401d7850aac-kube-api-access-fcqxj\") pod \"service-ca-55f6fc7b86-rqk6f\" (UID: \"07203fa1-5f7a-4ac3-964c-2401d7850aac\") " pod="openshift-service-ca/service-ca-55f6fc7b86-rqk6f"
Sep 18 19:43:54.016581 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:43:54.016424    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/07203fa1-5f7a-4ac3-964c-2401d7850aac-signing-key\") pod \"service-ca-55f6fc7b86-rqk6f\" (UID: \"07203fa1-5f7a-4ac3-964c-2401d7850aac\") " pod="openshift-service-ca/service-ca-55f6fc7b86-rqk6f"
Sep 18 19:43:54.016581 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:43:54.016489    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/07203fa1-5f7a-4ac3-964c-2401d7850aac-signing-cabundle\") pod \"service-ca-55f6fc7b86-rqk6f\" (UID: \"07203fa1-5f7a-4ac3-964c-2401d7850aac\") " pod="openshift-service-ca/service-ca-55f6fc7b86-rqk6f"
Sep 18 19:43:54.117454 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:43:54.117397    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fcqxj\" (UniqueName: \"kubernetes.io/projected/07203fa1-5f7a-4ac3-964c-2401d7850aac-kube-api-access-fcqxj\") pod \"service-ca-55f6fc7b86-rqk6f\" (UID: \"07203fa1-5f7a-4ac3-964c-2401d7850aac\") " pod="openshift-service-ca/service-ca-55f6fc7b86-rqk6f"
Sep 18 19:43:54.117454 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:43:54.117460    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/07203fa1-5f7a-4ac3-964c-2401d7850aac-signing-key\") pod \"service-ca-55f6fc7b86-rqk6f\" (UID: \"07203fa1-5f7a-4ac3-964c-2401d7850aac\") " pod="openshift-service-ca/service-ca-55f6fc7b86-rqk6f"
Sep 18 19:43:54.117669 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:43:54.117506    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/07203fa1-5f7a-4ac3-964c-2401d7850aac-signing-cabundle\") pod \"service-ca-55f6fc7b86-rqk6f\" (UID: \"07203fa1-5f7a-4ac3-964c-2401d7850aac\") " pod="openshift-service-ca/service-ca-55f6fc7b86-rqk6f"
Sep 18 19:43:54.118244 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:43:54.118218    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/07203fa1-5f7a-4ac3-964c-2401d7850aac-signing-cabundle\") pod \"service-ca-55f6fc7b86-rqk6f\" (UID: \"07203fa1-5f7a-4ac3-964c-2401d7850aac\") " pod="openshift-service-ca/service-ca-55f6fc7b86-rqk6f"
Sep 18 19:43:54.119921 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:43:54.119902    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/07203fa1-5f7a-4ac3-964c-2401d7850aac-signing-key\") pod \"service-ca-55f6fc7b86-rqk6f\" (UID: \"07203fa1-5f7a-4ac3-964c-2401d7850aac\") " pod="openshift-service-ca/service-ca-55f6fc7b86-rqk6f"
Sep 18 19:43:54.125903 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:43:54.125883    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fcqxj\" (UniqueName: \"kubernetes.io/projected/07203fa1-5f7a-4ac3-964c-2401d7850aac-kube-api-access-fcqxj\") pod \"service-ca-55f6fc7b86-rqk6f\" (UID: \"07203fa1-5f7a-4ac3-964c-2401d7850aac\") " pod="openshift-service-ca/service-ca-55f6fc7b86-rqk6f"
Sep 18 19:43:54.213956 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:43:54.213870    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-55f6fc7b86-rqk6f"
Sep 18 19:43:54.484022 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:43:54.483955    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-55f6fc7b86-rqk6f"]
Sep 18 19:43:54.486210 ip-10-0-136-185 kubenswrapper[2585]: W0918 19:43:54.486180    2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod07203fa1_5f7a_4ac3_964c_2401d7850aac.slice/crio-3611c64ba0bc77ee83fedefe54439d52304a34dc09dbd9ef39646b22f6675a69 WatchSource:0}: Error finding container 3611c64ba0bc77ee83fedefe54439d52304a34dc09dbd9ef39646b22f6675a69: Status 404 returned error can't find the container with id 3611c64ba0bc77ee83fedefe54439d52304a34dc09dbd9ef39646b22f6675a69
Sep 18 19:43:54.541675 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:43:54.541649    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-55f6fc7b86-rqk6f" event={"ID":"07203fa1-5f7a-4ac3-964c-2401d7850aac","Type":"ContainerStarted","Data":"daa9fe1dbe3322ba0ee005a01b26e05116ba71d8316e531da07f6d685fe57a64"}
Sep 18 19:43:54.541783 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:43:54.541700    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-55f6fc7b86-rqk6f" event={"ID":"07203fa1-5f7a-4ac3-964c-2401d7850aac","Type":"ContainerStarted","Data":"3611c64ba0bc77ee83fedefe54439d52304a34dc09dbd9ef39646b22f6675a69"}
Sep 18 19:43:54.566133 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:43:54.566077    2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-service-ca/service-ca-55f6fc7b86-rqk6f" podStartSLOduration=1.566056058 podStartE2EDuration="1.566056058s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 19:43:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 19:43:54.565260319 +0000 UTC m=+276.089865742" watchObservedRunningTime="2026-09-18 19:43:54.566056058 +0000 UTC m=+276.090661483"
Sep 18 19:43:54.756666 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:43:54.756590    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-ltq6c_bdda9ef7-2f71-4b51-84a9-56da43dd64d9/dns-node-resolver/0.log"
Sep 18 19:43:55.224083 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:43:55.224047    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/79747afc-194b-4ded-bdd1-d6332f8412b7-registry-tls\") pod \"image-registry-75d9fb895b-85gqr\" (UID: \"79747afc-194b-4ded-bdd1-d6332f8412b7\") " pod="openshift-image-registry/image-registry-75d9fb895b-85gqr"
Sep 18 19:43:55.224300 ip-10-0-136-185 kubenswrapper[2585]: E0918 19:43:55.224181    2585 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 18 19:43:55.224300 ip-10-0-136-185 kubenswrapper[2585]: E0918 19:43:55.224194    2585 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-75d9fb895b-85gqr: secret "image-registry-tls" not found
Sep 18 19:43:55.224300 ip-10-0-136-185 kubenswrapper[2585]: E0918 19:43:55.224241    2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/79747afc-194b-4ded-bdd1-d6332f8412b7-registry-tls podName:79747afc-194b-4ded-bdd1-d6332f8412b7 nodeName:}" failed. No retries permitted until 2026-09-18 19:44:03.224228803 +0000 UTC m=+284.748834205 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/79747afc-194b-4ded-bdd1-d6332f8412b7-registry-tls") pod "image-registry-75d9fb895b-85gqr" (UID: "79747afc-194b-4ded-bdd1-d6332f8412b7") : secret "image-registry-tls" not found
Sep 18 19:43:55.358604 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:43:55.358363    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-llt9z_3a884214-2df1-4958-a6b9-e3f2c903ae22/node-ca/0.log"
Sep 18 19:43:55.628639 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:43:55.628515    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/f3c00964-2160-44d6-8699-354dae4ac4a5-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-7czbj\" (UID: \"f3c00964-2160-44d6-8699-354dae4ac4a5\") " pod="openshift-insights/insights-runtime-extractor-7czbj"
Sep 18 19:43:55.628817 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:43:55.628690    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/f3c00964-2160-44d6-8699-354dae4ac4a5-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-7czbj\" (UID: \"f3c00964-2160-44d6-8699-354dae4ac4a5\") " pod="openshift-insights/insights-runtime-extractor-7czbj"
Sep 18 19:43:55.629266 ip-10-0-136-185 kubenswrapper[2585]: E0918 19:43:55.628971    2585 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 18 19:43:55.629266 ip-10-0-136-185 kubenswrapper[2585]: E0918 19:43:55.629010    2585 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 18 19:43:55.629266 ip-10-0-136-185 kubenswrapper[2585]: E0918 19:43:55.629037    2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/f3c00964-2160-44d6-8699-354dae4ac4a5-kube-rbac-proxy-cm podName:f3c00964-2160-44d6-8699-354dae4ac4a5 nodeName:}" failed. No retries permitted until 2026-09-18 19:43:59.629018185 +0000 UTC m=+281.153623607 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/f3c00964-2160-44d6-8699-354dae4ac4a5-kube-rbac-proxy-cm") pod "insights-runtime-extractor-7czbj" (UID: "f3c00964-2160-44d6-8699-354dae4ac4a5") : configmap "kube-rbac-proxy" not found
Sep 18 19:43:55.629266 ip-10-0-136-185 kubenswrapper[2585]: E0918 19:43:55.629062    2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f3c00964-2160-44d6-8699-354dae4ac4a5-insights-runtime-extractor-tls podName:f3c00964-2160-44d6-8699-354dae4ac4a5 nodeName:}" failed. No retries permitted until 2026-09-18 19:43:59.629045777 +0000 UTC m=+281.153651182 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/f3c00964-2160-44d6-8699-354dae4ac4a5-insights-runtime-extractor-tls") pod "insights-runtime-extractor-7czbj" (UID: "f3c00964-2160-44d6-8699-354dae4ac4a5") : secret "insights-runtime-extractor-tls" not found
Sep 18 19:43:56.158651 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:43:56.158620    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-kxzjt_a797d295-00fd-4085-b1cc-bdfffc6af82f/serve-healthcheck-canary/0.log"
Sep 18 19:43:59.661644 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:43:59.661609    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/f3c00964-2160-44d6-8699-354dae4ac4a5-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-7czbj\" (UID: \"f3c00964-2160-44d6-8699-354dae4ac4a5\") " pod="openshift-insights/insights-runtime-extractor-7czbj"
Sep 18 19:43:59.662074 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:43:59.661694    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/f3c00964-2160-44d6-8699-354dae4ac4a5-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-7czbj\" (UID: \"f3c00964-2160-44d6-8699-354dae4ac4a5\") " pod="openshift-insights/insights-runtime-extractor-7czbj"
Sep 18 19:43:59.662074 ip-10-0-136-185 kubenswrapper[2585]: E0918 19:43:59.661758    2585 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 18 19:43:59.662074 ip-10-0-136-185 kubenswrapper[2585]: E0918 19:43:59.661776    2585 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 18 19:43:59.662074 ip-10-0-136-185 kubenswrapper[2585]: E0918 19:43:59.661832    2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/f3c00964-2160-44d6-8699-354dae4ac4a5-kube-rbac-proxy-cm podName:f3c00964-2160-44d6-8699-354dae4ac4a5 nodeName:}" failed. No retries permitted until 2026-09-18 19:44:07.661817653 +0000 UTC m=+289.186423064 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/f3c00964-2160-44d6-8699-354dae4ac4a5-kube-rbac-proxy-cm") pod "insights-runtime-extractor-7czbj" (UID: "f3c00964-2160-44d6-8699-354dae4ac4a5") : configmap "kube-rbac-proxy" not found
Sep 18 19:43:59.662074 ip-10-0-136-185 kubenswrapper[2585]: E0918 19:43:59.661861    2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f3c00964-2160-44d6-8699-354dae4ac4a5-insights-runtime-extractor-tls podName:f3c00964-2160-44d6-8699-354dae4ac4a5 nodeName:}" failed. No retries permitted until 2026-09-18 19:44:07.661844386 +0000 UTC m=+289.186449791 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/f3c00964-2160-44d6-8699-354dae4ac4a5-insights-runtime-extractor-tls") pod "insights-runtime-extractor-7czbj" (UID: "f3c00964-2160-44d6-8699-354dae4ac4a5") : secret "insights-runtime-extractor-tls" not found
Sep 18 19:44:01.189387 ip-10-0-136-185 kubenswrapper[2585]: E0918 19:44:01.189338    2585 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-dns/dns-default-grfwt" podUID="13a57b4c-73f9-4960-89d4-e9c25d392279"
Sep 18 19:44:01.568769 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:44:01.568692    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-grfwt"
Sep 18 19:44:03.293286 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:44:03.293250    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/79747afc-194b-4ded-bdd1-d6332f8412b7-registry-tls\") pod \"image-registry-75d9fb895b-85gqr\" (UID: \"79747afc-194b-4ded-bdd1-d6332f8412b7\") " pod="openshift-image-registry/image-registry-75d9fb895b-85gqr"
Sep 18 19:44:03.295694 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:44:03.295663    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/79747afc-194b-4ded-bdd1-d6332f8412b7-registry-tls\") pod \"image-registry-75d9fb895b-85gqr\" (UID: \"79747afc-194b-4ded-bdd1-d6332f8412b7\") " pod="openshift-image-registry/image-registry-75d9fb895b-85gqr"
Sep 18 19:44:03.387528 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:44:03.387487    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-75d9fb895b-85gqr"
Sep 18 19:44:03.659446 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:44:03.659416    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-75d9fb895b-85gqr"]
Sep 18 19:44:03.662143 ip-10-0-136-185 kubenswrapper[2585]: W0918 19:44:03.662112    2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod79747afc_194b_4ded_bdd1_d6332f8412b7.slice/crio-5ef60cc6451c629c0357f3477e61b68725dad8b2fc63d0343b37df6681d99cbb WatchSource:0}: Error finding container 5ef60cc6451c629c0357f3477e61b68725dad8b2fc63d0343b37df6681d99cbb: Status 404 returned error can't find the container with id 5ef60cc6451c629c0357f3477e61b68725dad8b2fc63d0343b37df6681d99cbb
Sep 18 19:44:04.579183 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:44:04.579151    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-75d9fb895b-85gqr" event={"ID":"79747afc-194b-4ded-bdd1-d6332f8412b7","Type":"ContainerStarted","Data":"8ffae4b2577c47948578da505a586e8182a3f8cb684198b48d523d297c400649"}
Sep 18 19:44:04.579688 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:44:04.579205    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-75d9fb895b-85gqr" event={"ID":"79747afc-194b-4ded-bdd1-d6332f8412b7","Type":"ContainerStarted","Data":"5ef60cc6451c629c0357f3477e61b68725dad8b2fc63d0343b37df6681d99cbb"}
Sep 18 19:44:04.579688 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:44:04.579254    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-75d9fb895b-85gqr"
Sep 18 19:44:04.599675 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:44:04.599624    2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-75d9fb895b-85gqr" podStartSLOduration=17.599607133 podStartE2EDuration="17.599607133s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 19:43:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 19:44:04.599118082 +0000 UTC m=+286.123723505" watchObservedRunningTime="2026-09-18 19:44:04.599607133 +0000 UTC m=+286.124212598"
Sep 18 19:44:04.807972 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:44:04.807930    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/13a57b4c-73f9-4960-89d4-e9c25d392279-metrics-tls\") pod \"dns-default-grfwt\" (UID: \"13a57b4c-73f9-4960-89d4-e9c25d392279\") " pod="openshift-dns/dns-default-grfwt"
Sep 18 19:44:04.810264 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:44:04.810236    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/13a57b4c-73f9-4960-89d4-e9c25d392279-metrics-tls\") pod \"dns-default-grfwt\" (UID: \"13a57b4c-73f9-4960-89d4-e9c25d392279\") " pod="openshift-dns/dns-default-grfwt"
Sep 18 19:44:04.871728 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:44:04.871643    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-v8jjp\""
Sep 18 19:44:04.879593 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:44:04.879576    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-grfwt"
Sep 18 19:44:05.153284 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:44:05.153255    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-grfwt"]
Sep 18 19:44:05.156395 ip-10-0-136-185 kubenswrapper[2585]: W0918 19:44:05.156369    2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod13a57b4c_73f9_4960_89d4_e9c25d392279.slice/crio-cbe9a9a0d71c6f85b73379961611dee7276d7a3942db15f369ae24a612989420 WatchSource:0}: Error finding container cbe9a9a0d71c6f85b73379961611dee7276d7a3942db15f369ae24a612989420: Status 404 returned error can't find the container with id cbe9a9a0d71c6f85b73379961611dee7276d7a3942db15f369ae24a612989420
Sep 18 19:44:05.584761 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:44:05.584676    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-grfwt" event={"ID":"13a57b4c-73f9-4960-89d4-e9c25d392279","Type":"ContainerStarted","Data":"cbe9a9a0d71c6f85b73379961611dee7276d7a3942db15f369ae24a612989420"}
Sep 18 19:44:07.593236 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:44:07.593202    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-grfwt" event={"ID":"13a57b4c-73f9-4960-89d4-e9c25d392279","Type":"ContainerStarted","Data":"3734cd251060a04dae0c580fcb3bc4f66a59f555a1bc8cc4bc3bf4b5ffd1a0f5"}
Sep 18 19:44:07.593666 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:44:07.593252    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-grfwt" event={"ID":"13a57b4c-73f9-4960-89d4-e9c25d392279","Type":"ContainerStarted","Data":"97d888470383fddcd082e3aeaa8f454bc43ce858c0ae70999ff5b29425b04fbd"}
Sep 18 19:44:07.593666 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:44:07.593291    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-grfwt"
Sep 18 19:44:07.609920 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:44:07.609871    2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-grfwt" podStartSLOduration=252.22093078 podStartE2EDuration="4m13.609857594s" totalImagesPullingTime="1.388926814s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 19:39:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 19:44:07.609607256 +0000 UTC m=+289.134212682" watchObservedRunningTime="2026-09-18 19:44:07.609857594 +0000 UTC m=+289.134462996"
Sep 18 19:44:07.739436 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:44:07.739387    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/f3c00964-2160-44d6-8699-354dae4ac4a5-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-7czbj\" (UID: \"f3c00964-2160-44d6-8699-354dae4ac4a5\") " pod="openshift-insights/insights-runtime-extractor-7czbj"
Sep 18 19:44:07.739436 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:44:07.739449    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/f3c00964-2160-44d6-8699-354dae4ac4a5-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-7czbj\" (UID: \"f3c00964-2160-44d6-8699-354dae4ac4a5\") " pod="openshift-insights/insights-runtime-extractor-7czbj"
Sep 18 19:44:07.739912 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:44:07.739893    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/f3c00964-2160-44d6-8699-354dae4ac4a5-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-7czbj\" (UID: \"f3c00964-2160-44d6-8699-354dae4ac4a5\") " pod="openshift-insights/insights-runtime-extractor-7czbj"
Sep 18 19:44:07.741656 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:44:07.741639    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/f3c00964-2160-44d6-8699-354dae4ac4a5-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-7czbj\" (UID: \"f3c00964-2160-44d6-8699-354dae4ac4a5\") " pod="openshift-insights/insights-runtime-extractor-7czbj"
Sep 18 19:44:08.032373 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:44:08.032332    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-7czbj"
Sep 18 19:44:08.302449 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:44:08.302244    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-7czbj"]
Sep 18 19:44:08.304934 ip-10-0-136-185 kubenswrapper[2585]: W0918 19:44:08.304909    2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf3c00964_2160_44d6_8699_354dae4ac4a5.slice/crio-408fcc80afa41aeb79f7bbd9657e9b6d9bb2d30d36f03e07bad1e376ad596f03 WatchSource:0}: Error finding container 408fcc80afa41aeb79f7bbd9657e9b6d9bb2d30d36f03e07bad1e376ad596f03: Status 404 returned error can't find the container with id 408fcc80afa41aeb79f7bbd9657e9b6d9bb2d30d36f03e07bad1e376ad596f03
Sep 18 19:44:08.597589 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:44:08.597504    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-7czbj" event={"ID":"f3c00964-2160-44d6-8699-354dae4ac4a5","Type":"ContainerStarted","Data":"cb08a6ba510b1e94efc66da07587a08632db23cf6b28f749beb4757c4ca06458"}
Sep 18 19:44:08.597589 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:44:08.597562    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-7czbj" event={"ID":"f3c00964-2160-44d6-8699-354dae4ac4a5","Type":"ContainerStarted","Data":"408fcc80afa41aeb79f7bbd9657e9b6d9bb2d30d36f03e07bad1e376ad596f03"}
Sep 18 19:44:09.604186 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:44:09.604147    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-7czbj" event={"ID":"f3c00964-2160-44d6-8699-354dae4ac4a5","Type":"ContainerStarted","Data":"31788d618ffb60e24dbe37f5574a83e6f9a4fe5f90025909d4cb728605a9318e"}
Sep 18 19:44:10.609964 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:44:10.609875    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-7czbj" event={"ID":"f3c00964-2160-44d6-8699-354dae4ac4a5","Type":"ContainerStarted","Data":"d99a036a1da60c14f50ca9d4d5fbad36f468f5438c71a4d8099cf278ccef6930"}
Sep 18 19:44:10.629690 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:44:10.629644    2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-7czbj" podStartSLOduration=17.885527948 podStartE2EDuration="19.629629162s" totalImagesPullingTime="1.744101214s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-18 19:43:51 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 19:44:10.628873884 +0000 UTC m=+292.153479307" watchObservedRunningTime="2026-09-18 19:44:10.629629162 +0000 UTC m=+292.154234584"
Sep 18 19:44:13.657883 ip-10-0-136-185 kubenswrapper[2585]: E0918 19:44:13.657843    2585 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-certs], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-multus/network-metrics-daemon-4njfl" podUID="3eb3cd73-4ea1-4950-8bfd-82b5fa60fd09"
Sep 18 19:44:14.892174 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:44:14.892138    2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-75d9fb895b-85gqr"]
Sep 18 19:44:14.914040 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:44:14.914010    2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-5c6747dc7b-59t4f"]
Sep 18 19:44:14.915959 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:44:14.915944    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-5c6747dc7b-59t4f"
Sep 18 19:44:14.919134 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:44:14.919112    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-tls\""
Sep 18 19:44:14.919275 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:44:14.919184    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-dockercfg-r9r5c\""
Sep 18 19:44:14.932957 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:44:14.932933    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-5c6747dc7b-59t4f"]
Sep 18 19:44:14.935925 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:44:14.935899    2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/downloads-6594f4c75-p7gk4"]
Sep 18 19:44:14.938138 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:44:14.938110    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-6594f4c75-p7gk4"
Sep 18 19:44:14.940939 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:44:14.940917    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"default-dockercfg-84529\""
Sep 18 19:44:14.941171 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:44:14.941156    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\""
Sep 18 19:44:14.941171 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:44:14.941164    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\""
Sep 18 19:44:14.957284 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:44:14.957259    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-6594f4c75-p7gk4"]
Sep 18 19:44:14.991532 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:44:14.991502    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/bddff53b-d5cc-4cf2-8e76-9ab8ef1dd7ef-tls-certificates\") pod \"prometheus-operator-admission-webhook-5c6747dc7b-59t4f\" (UID: \"bddff53b-d5cc-4cf2-8e76-9ab8ef1dd7ef\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-5c6747dc7b-59t4f"
Sep 18 19:44:14.991686 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:44:14.991542    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4pgvv\" (UniqueName: \"kubernetes.io/projected/1f4af1db-93c0-48d0-81df-868ea52c8a1e-kube-api-access-4pgvv\") pod \"downloads-6594f4c75-p7gk4\" (UID: \"1f4af1db-93c0-48d0-81df-868ea52c8a1e\") " pod="openshift-console/downloads-6594f4c75-p7gk4"
Sep 18 19:44:15.092724 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:44:15.092686    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4pgvv\" (UniqueName: \"kubernetes.io/projected/1f4af1db-93c0-48d0-81df-868ea52c8a1e-kube-api-access-4pgvv\") pod \"downloads-6594f4c75-p7gk4\" (UID: \"1f4af1db-93c0-48d0-81df-868ea52c8a1e\") " pod="openshift-console/downloads-6594f4c75-p7gk4"
Sep 18 19:44:15.092894 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:44:15.092835    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/bddff53b-d5cc-4cf2-8e76-9ab8ef1dd7ef-tls-certificates\") pod \"prometheus-operator-admission-webhook-5c6747dc7b-59t4f\" (UID: \"bddff53b-d5cc-4cf2-8e76-9ab8ef1dd7ef\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-5c6747dc7b-59t4f"
Sep 18 19:44:15.095345 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:44:15.095310    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/bddff53b-d5cc-4cf2-8e76-9ab8ef1dd7ef-tls-certificates\") pod \"prometheus-operator-admission-webhook-5c6747dc7b-59t4f\" (UID: \"bddff53b-d5cc-4cf2-8e76-9ab8ef1dd7ef\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-5c6747dc7b-59t4f"
Sep 18 19:44:15.104525 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:44:15.104496    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4pgvv\" (UniqueName: \"kubernetes.io/projected/1f4af1db-93c0-48d0-81df-868ea52c8a1e-kube-api-access-4pgvv\") pod \"downloads-6594f4c75-p7gk4\" (UID: \"1f4af1db-93c0-48d0-81df-868ea52c8a1e\") " pod="openshift-console/downloads-6594f4c75-p7gk4"
Sep 18 19:44:15.225200 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:44:15.225111    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-5c6747dc7b-59t4f"
Sep 18 19:44:15.247169 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:44:15.247141    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-6594f4c75-p7gk4"
Sep 18 19:44:15.495026 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:44:15.494957    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-5c6747dc7b-59t4f"]
Sep 18 19:44:15.497820 ip-10-0-136-185 kubenswrapper[2585]: W0918 19:44:15.497776    2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbddff53b_d5cc_4cf2_8e76_9ab8ef1dd7ef.slice/crio-50477ed3e3396ca139636a3339fb38733c6ad726b32e69841dce2ed9c960e758 WatchSource:0}: Error finding container 50477ed3e3396ca139636a3339fb38733c6ad726b32e69841dce2ed9c960e758: Status 404 returned error can't find the container with id 50477ed3e3396ca139636a3339fb38733c6ad726b32e69841dce2ed9c960e758
Sep 18 19:44:15.526902 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:44:15.526867    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-6594f4c75-p7gk4"]
Sep 18 19:44:15.529237 ip-10-0-136-185 kubenswrapper[2585]: W0918 19:44:15.529200    2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1f4af1db_93c0_48d0_81df_868ea52c8a1e.slice/crio-fe2c1772254f1660e2ef241866ffaf6a07dc2706b1e200ada01ab1c92285c5a4 WatchSource:0}: Error finding container fe2c1772254f1660e2ef241866ffaf6a07dc2706b1e200ada01ab1c92285c5a4: Status 404 returned error can't find the container with id fe2c1772254f1660e2ef241866ffaf6a07dc2706b1e200ada01ab1c92285c5a4
Sep 18 19:44:15.627451 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:44:15.627413    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-5c6747dc7b-59t4f" event={"ID":"bddff53b-d5cc-4cf2-8e76-9ab8ef1dd7ef","Type":"ContainerStarted","Data":"50477ed3e3396ca139636a3339fb38733c6ad726b32e69841dce2ed9c960e758"}
Sep 18 19:44:15.628512 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:44:15.628488    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-6594f4c75-p7gk4" event={"ID":"1f4af1db-93c0-48d0-81df-868ea52c8a1e","Type":"ContainerStarted","Data":"fe2c1772254f1660e2ef241866ffaf6a07dc2706b1e200ada01ab1c92285c5a4"}
Sep 18 19:44:17.600567 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:44:17.600540    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-grfwt"
Sep 18 19:44:17.639694 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:44:17.639657    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-5c6747dc7b-59t4f" event={"ID":"bddff53b-d5cc-4cf2-8e76-9ab8ef1dd7ef","Type":"ContainerStarted","Data":"ec3851150872e7a602d967934f4a02213721252a893e75ace16a173f719f5d1c"}
Sep 18 19:44:17.639885 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:44:17.639835    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-5c6747dc7b-59t4f"
Sep 18 19:44:17.645905 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:44:17.645872    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-5c6747dc7b-59t4f"
Sep 18 19:44:17.659780 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:44:17.659727    2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-admission-webhook-5c6747dc7b-59t4f" podStartSLOduration=2.50553053 podStartE2EDuration="3.659709933s" totalImagesPullingTime="1.154179403s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 19:44:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 19:44:17.658206626 +0000 UTC m=+299.182812065" watchObservedRunningTime="2026-09-18 19:44:17.659709933 +0000 UTC m=+299.184315356"
Sep 18 19:44:20.527268 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:44:20.527239    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-rctld_cc0e0de2-75d5-43a1-b64b-b071427cd8ce/ovn-acl-logging/0.log"
Sep 18 19:44:20.528352 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:44:20.528326    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-rctld_cc0e0de2-75d5-43a1-b64b-b071427cd8ce/ovn-acl-logging/0.log"
Sep 18 19:44:20.533478 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:44:20.533453    2585 kubelet.go:1648] "Image garbage collection succeeded"
Sep 18 19:44:24.008622 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:44:24.008585    2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/openshift-state-metrics-86c5bc79d6-q45br"]
Sep 18 19:44:24.012566 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:44:24.012539    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-86c5bc79d6-q45br"
Sep 18 19:44:24.015245 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:44:24.015216    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-tls\""
Sep 18 19:44:24.016566 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:44:24.015417    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-dockercfg-rrlrv\""
Sep 18 19:44:24.016566 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:44:24.015673    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Sep 18 19:44:24.016566 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:44:24.015981    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Sep 18 19:44:24.016566 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:44:24.016213    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Sep 18 19:44:24.016566 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:44:24.016391    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-kube-rbac-proxy-config\""
Sep 18 19:44:24.027312 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:44:24.027225    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-86c5bc79d6-q45br"]
Sep 18 19:44:24.042899 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:44:24.041103    2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-m9c72"]
Sep 18 19:44:24.045359 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:44:24.045332    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-m9c72"
Sep 18 19:44:24.048763 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:44:24.047500    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Sep 18 19:44:24.048763 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:44:24.047809    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Sep 18 19:44:24.048763 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:44:24.048038    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Sep 18 19:44:24.048763 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:44:24.048380    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-4wzv5\""
Sep 18 19:44:24.176607 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:44:24.176552    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/d6506887-aec3-4244-911e-b0a4fd696601-openshift-state-metrics-tls\") pod \"openshift-state-metrics-86c5bc79d6-q45br\" (UID: \"d6506887-aec3-4244-911e-b0a4fd696601\") " pod="openshift-monitoring/openshift-state-metrics-86c5bc79d6-q45br"
Sep 18 19:44:24.176774 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:44:24.176612    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/d6506887-aec3-4244-911e-b0a4fd696601-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-86c5bc79d6-q45br\" (UID: \"d6506887-aec3-4244-911e-b0a4fd696601\") " pod="openshift-monitoring/openshift-state-metrics-86c5bc79d6-q45br"
Sep 18 19:44:24.176774 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:44:24.176664    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9ztnd\" (UniqueName: \"kubernetes.io/projected/d6506887-aec3-4244-911e-b0a4fd696601-kube-api-access-9ztnd\") pod \"openshift-state-metrics-86c5bc79d6-q45br\" (UID: \"d6506887-aec3-4244-911e-b0a4fd696601\") " pod="openshift-monitoring/openshift-state-metrics-86c5bc79d6-q45br"
Sep 18 19:44:24.176774 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:44:24.176690    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/ff4afcd9-ff15-4e03-a63a-7927c4b458f4-node-exporter-textfile\") pod \"node-exporter-m9c72\" (UID: \"ff4afcd9-ff15-4e03-a63a-7927c4b458f4\") " pod="openshift-monitoring/node-exporter-m9c72"
Sep 18 19:44:24.176774 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:44:24.176711    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/ff4afcd9-ff15-4e03-a63a-7927c4b458f4-node-exporter-wtmp\") pod \"node-exporter-m9c72\" (UID: \"ff4afcd9-ff15-4e03-a63a-7927c4b458f4\") " pod="openshift-monitoring/node-exporter-m9c72"
Sep 18 19:44:24.176774 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:44:24.176733    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6qzmp\" (UniqueName: \"kubernetes.io/projected/ff4afcd9-ff15-4e03-a63a-7927c4b458f4-kube-api-access-6qzmp\") pod \"node-exporter-m9c72\" (UID: \"ff4afcd9-ff15-4e03-a63a-7927c4b458f4\") " pod="openshift-monitoring/node-exporter-m9c72"
Sep 18 19:44:24.177088 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:44:24.176830    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/ff4afcd9-ff15-4e03-a63a-7927c4b458f4-node-exporter-tls\") pod \"node-exporter-m9c72\" (UID: \"ff4afcd9-ff15-4e03-a63a-7927c4b458f4\") " pod="openshift-monitoring/node-exporter-m9c72"
Sep 18 19:44:24.177088 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:44:24.176905    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/ff4afcd9-ff15-4e03-a63a-7927c4b458f4-sys\") pod \"node-exporter-m9c72\" (UID: \"ff4afcd9-ff15-4e03-a63a-7927c4b458f4\") " pod="openshift-monitoring/node-exporter-m9c72"
Sep 18 19:44:24.177088 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:44:24.176935    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/ff4afcd9-ff15-4e03-a63a-7927c4b458f4-root\") pod \"node-exporter-m9c72\" (UID: \"ff4afcd9-ff15-4e03-a63a-7927c4b458f4\") " pod="openshift-monitoring/node-exporter-m9c72"
Sep 18 19:44:24.177088 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:44:24.176970    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ff4afcd9-ff15-4e03-a63a-7927c4b458f4-metrics-client-ca\") pod \"node-exporter-m9c72\" (UID: \"ff4afcd9-ff15-4e03-a63a-7927c4b458f4\") " pod="openshift-monitoring/node-exporter-m9c72"
Sep 18 19:44:24.177088 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:44:24.176997    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/ff4afcd9-ff15-4e03-a63a-7927c4b458f4-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-m9c72\" (UID: \"ff4afcd9-ff15-4e03-a63a-7927c4b458f4\") " pod="openshift-monitoring/node-exporter-m9c72"
Sep 18 19:44:24.177088 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:44:24.177029    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/ff4afcd9-ff15-4e03-a63a-7927c4b458f4-node-exporter-accelerators-collector-config\") pod \"node-exporter-m9c72\" (UID: \"ff4afcd9-ff15-4e03-a63a-7927c4b458f4\") " pod="openshift-monitoring/node-exporter-m9c72"
Sep 18 19:44:24.177088 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:44:24.177057    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d6506887-aec3-4244-911e-b0a4fd696601-metrics-client-ca\") pod \"openshift-state-metrics-86c5bc79d6-q45br\" (UID: \"d6506887-aec3-4244-911e-b0a4fd696601\") " pod="openshift-monitoring/openshift-state-metrics-86c5bc79d6-q45br"
Sep 18 19:44:24.278261 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:44:24.277769    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/d6506887-aec3-4244-911e-b0a4fd696601-openshift-state-metrics-tls\") pod \"openshift-state-metrics-86c5bc79d6-q45br\" (UID: \"d6506887-aec3-4244-911e-b0a4fd696601\") " pod="openshift-monitoring/openshift-state-metrics-86c5bc79d6-q45br"
Sep 18 19:44:24.278261 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:44:24.277853    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/d6506887-aec3-4244-911e-b0a4fd696601-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-86c5bc79d6-q45br\" (UID: \"d6506887-aec3-4244-911e-b0a4fd696601\") " pod="openshift-monitoring/openshift-state-metrics-86c5bc79d6-q45br"
Sep 18 19:44:24.278261 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:44:24.277908    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9ztnd\" (UniqueName: \"kubernetes.io/projected/d6506887-aec3-4244-911e-b0a4fd696601-kube-api-access-9ztnd\") pod \"openshift-state-metrics-86c5bc79d6-q45br\" (UID: \"d6506887-aec3-4244-911e-b0a4fd696601\") " pod="openshift-monitoring/openshift-state-metrics-86c5bc79d6-q45br"
Sep 18 19:44:24.278261 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:44:24.277935    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/ff4afcd9-ff15-4e03-a63a-7927c4b458f4-node-exporter-textfile\") pod \"node-exporter-m9c72\" (UID: \"ff4afcd9-ff15-4e03-a63a-7927c4b458f4\") " pod="openshift-monitoring/node-exporter-m9c72"
Sep 18 19:44:24.278261 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:44:24.277962    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/ff4afcd9-ff15-4e03-a63a-7927c4b458f4-node-exporter-wtmp\") pod \"node-exporter-m9c72\" (UID: \"ff4afcd9-ff15-4e03-a63a-7927c4b458f4\") " pod="openshift-monitoring/node-exporter-m9c72"
Sep 18 19:44:24.278261 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:44:24.277985    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6qzmp\" (UniqueName: \"kubernetes.io/projected/ff4afcd9-ff15-4e03-a63a-7927c4b458f4-kube-api-access-6qzmp\") pod \"node-exporter-m9c72\" (UID: \"ff4afcd9-ff15-4e03-a63a-7927c4b458f4\") " pod="openshift-monitoring/node-exporter-m9c72"
Sep 18 19:44:24.278261 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:44:24.278019    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/ff4afcd9-ff15-4e03-a63a-7927c4b458f4-node-exporter-tls\") pod \"node-exporter-m9c72\" (UID: \"ff4afcd9-ff15-4e03-a63a-7927c4b458f4\") " pod="openshift-monitoring/node-exporter-m9c72"
Sep 18 19:44:24.278261 ip-10-0-136-185 kubenswrapper[2585]: E0918 19:44:24.277935    2585 secret.go:189] Couldn't get secret openshift-monitoring/openshift-state-metrics-tls: secret "openshift-state-metrics-tls" not found
Sep 18 19:44:24.278261 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:44:24.278047    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/ff4afcd9-ff15-4e03-a63a-7927c4b458f4-sys\") pod \"node-exporter-m9c72\" (UID: \"ff4afcd9-ff15-4e03-a63a-7927c4b458f4\") " pod="openshift-monitoring/node-exporter-m9c72"
Sep 18 19:44:24.278261 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:44:24.278074    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/ff4afcd9-ff15-4e03-a63a-7927c4b458f4-root\") pod \"node-exporter-m9c72\" (UID: \"ff4afcd9-ff15-4e03-a63a-7927c4b458f4\") " pod="openshift-monitoring/node-exporter-m9c72"
Sep 18 19:44:24.278261 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:44:24.278101    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ff4afcd9-ff15-4e03-a63a-7927c4b458f4-metrics-client-ca\") pod \"node-exporter-m9c72\" (UID: \"ff4afcd9-ff15-4e03-a63a-7927c4b458f4\") " pod="openshift-monitoring/node-exporter-m9c72"
Sep 18 19:44:24.278261 ip-10-0-136-185 kubenswrapper[2585]: E0918 19:44:24.278132    2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d6506887-aec3-4244-911e-b0a4fd696601-openshift-state-metrics-tls podName:d6506887-aec3-4244-911e-b0a4fd696601 nodeName:}" failed. No retries permitted until 2026-09-18 19:44:24.778103446 +0000 UTC m=+306.302708862 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "openshift-state-metrics-tls" (UniqueName: "kubernetes.io/secret/d6506887-aec3-4244-911e-b0a4fd696601-openshift-state-metrics-tls") pod "openshift-state-metrics-86c5bc79d6-q45br" (UID: "d6506887-aec3-4244-911e-b0a4fd696601") : secret "openshift-state-metrics-tls" not found
Sep 18 19:44:24.278261 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:44:24.278173    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/ff4afcd9-ff15-4e03-a63a-7927c4b458f4-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-m9c72\" (UID: \"ff4afcd9-ff15-4e03-a63a-7927c4b458f4\") " pod="openshift-monitoring/node-exporter-m9c72"
Sep 18 19:44:24.278261 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:44:24.278208    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/ff4afcd9-ff15-4e03-a63a-7927c4b458f4-node-exporter-accelerators-collector-config\") pod \"node-exporter-m9c72\" (UID: \"ff4afcd9-ff15-4e03-a63a-7927c4b458f4\") " pod="openshift-monitoring/node-exporter-m9c72"
Sep 18 19:44:24.278261 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:44:24.278242    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d6506887-aec3-4244-911e-b0a4fd696601-metrics-client-ca\") pod \"openshift-state-metrics-86c5bc79d6-q45br\" (UID: \"d6506887-aec3-4244-911e-b0a4fd696601\") " pod="openshift-monitoring/openshift-state-metrics-86c5bc79d6-q45br"
Sep 18 19:44:24.279140 ip-10-0-136-185 kubenswrapper[2585]: E0918 19:44:24.278421    2585 secret.go:189] Couldn't get secret openshift-monitoring/node-exporter-tls: secret "node-exporter-tls" not found
Sep 18 19:44:24.279140 ip-10-0-136-185 kubenswrapper[2585]: E0918 19:44:24.278478    2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ff4afcd9-ff15-4e03-a63a-7927c4b458f4-node-exporter-tls podName:ff4afcd9-ff15-4e03-a63a-7927c4b458f4 nodeName:}" failed. No retries permitted until 2026-09-18 19:44:24.778460889 +0000 UTC m=+306.303066296 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "node-exporter-tls" (UniqueName: "kubernetes.io/secret/ff4afcd9-ff15-4e03-a63a-7927c4b458f4-node-exporter-tls") pod "node-exporter-m9c72" (UID: "ff4afcd9-ff15-4e03-a63a-7927c4b458f4") : secret "node-exporter-tls" not found
Sep 18 19:44:24.279140 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:44:24.278590    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/ff4afcd9-ff15-4e03-a63a-7927c4b458f4-node-exporter-wtmp\") pod \"node-exporter-m9c72\" (UID: \"ff4afcd9-ff15-4e03-a63a-7927c4b458f4\") " pod="openshift-monitoring/node-exporter-m9c72"
Sep 18 19:44:24.279140 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:44:24.278678    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ff4afcd9-ff15-4e03-a63a-7927c4b458f4-metrics-client-ca\") pod \"node-exporter-m9c72\" (UID: \"ff4afcd9-ff15-4e03-a63a-7927c4b458f4\") " pod="openshift-monitoring/node-exporter-m9c72"
Sep 18 19:44:24.279140 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:44:24.278750    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/ff4afcd9-ff15-4e03-a63a-7927c4b458f4-sys\") pod \"node-exporter-m9c72\" (UID: \"ff4afcd9-ff15-4e03-a63a-7927c4b458f4\") " pod="openshift-monitoring/node-exporter-m9c72"
Sep 18 19:44:24.279140 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:44:24.278812    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/ff4afcd9-ff15-4e03-a63a-7927c4b458f4-root\") pod \"node-exporter-m9c72\" (UID: \"ff4afcd9-ff15-4e03-a63a-7927c4b458f4\") " pod="openshift-monitoring/node-exporter-m9c72"
Sep 18 19:44:24.279449 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:44:24.279144    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/ff4afcd9-ff15-4e03-a63a-7927c4b458f4-node-exporter-textfile\") pod \"node-exporter-m9c72\" (UID: \"ff4afcd9-ff15-4e03-a63a-7927c4b458f4\") " pod="openshift-monitoring/node-exporter-m9c72"
Sep 18 19:44:24.279504 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:44:24.279442    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d6506887-aec3-4244-911e-b0a4fd696601-metrics-client-ca\") pod \"openshift-state-metrics-86c5bc79d6-q45br\" (UID: \"d6506887-aec3-4244-911e-b0a4fd696601\") " pod="openshift-monitoring/openshift-state-metrics-86c5bc79d6-q45br"
Sep 18 19:44:24.279617 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:44:24.279557    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/ff4afcd9-ff15-4e03-a63a-7927c4b458f4-node-exporter-accelerators-collector-config\") pod \"node-exporter-m9c72\" (UID: \"ff4afcd9-ff15-4e03-a63a-7927c4b458f4\") " pod="openshift-monitoring/node-exporter-m9c72"
Sep 18 19:44:24.284602 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:44:24.284574    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/ff4afcd9-ff15-4e03-a63a-7927c4b458f4-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-m9c72\" (UID: \"ff4afcd9-ff15-4e03-a63a-7927c4b458f4\") " pod="openshift-monitoring/node-exporter-m9c72"
Sep 18 19:44:24.284753 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:44:24.284622    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/d6506887-aec3-4244-911e-b0a4fd696601-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-86c5bc79d6-q45br\" (UID: \"d6506887-aec3-4244-911e-b0a4fd696601\") " pod="openshift-monitoring/openshift-state-metrics-86c5bc79d6-q45br"
Sep 18 19:44:24.288831 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:44:24.288708    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6qzmp\" (UniqueName: \"kubernetes.io/projected/ff4afcd9-ff15-4e03-a63a-7927c4b458f4-kube-api-access-6qzmp\") pod \"node-exporter-m9c72\" (UID: \"ff4afcd9-ff15-4e03-a63a-7927c4b458f4\") " pod="openshift-monitoring/node-exporter-m9c72"
Sep 18 19:44:24.292808 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:44:24.292746    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9ztnd\" (UniqueName: \"kubernetes.io/projected/d6506887-aec3-4244-911e-b0a4fd696601-kube-api-access-9ztnd\") pod \"openshift-state-metrics-86c5bc79d6-q45br\" (UID: \"d6506887-aec3-4244-911e-b0a4fd696601\") " pod="openshift-monitoring/openshift-state-metrics-86c5bc79d6-q45br"
Sep 18 19:44:24.783103 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:44:24.783071    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/d6506887-aec3-4244-911e-b0a4fd696601-openshift-state-metrics-tls\") pod \"openshift-state-metrics-86c5bc79d6-q45br\" (UID: \"d6506887-aec3-4244-911e-b0a4fd696601\") " pod="openshift-monitoring/openshift-state-metrics-86c5bc79d6-q45br"
Sep 18 19:44:24.783313 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:44:24.783135    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/ff4afcd9-ff15-4e03-a63a-7927c4b458f4-node-exporter-tls\") pod \"node-exporter-m9c72\" (UID: \"ff4afcd9-ff15-4e03-a63a-7927c4b458f4\") " pod="openshift-monitoring/node-exporter-m9c72"
Sep 18 19:44:24.785939 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:44:24.785912    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/ff4afcd9-ff15-4e03-a63a-7927c4b458f4-node-exporter-tls\") pod \"node-exporter-m9c72\" (UID: \"ff4afcd9-ff15-4e03-a63a-7927c4b458f4\") " pod="openshift-monitoring/node-exporter-m9c72"
Sep 18 19:44:24.786071 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:44:24.786002    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/d6506887-aec3-4244-911e-b0a4fd696601-openshift-state-metrics-tls\") pod \"openshift-state-metrics-86c5bc79d6-q45br\" (UID: \"d6506887-aec3-4244-911e-b0a4fd696601\") " pod="openshift-monitoring/openshift-state-metrics-86c5bc79d6-q45br"
Sep 18 19:44:24.898766 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:44:24.898737    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-75d9fb895b-85gqr"
Sep 18 19:44:24.927142 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:44:24.927110    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-86c5bc79d6-q45br"
Sep 18 19:44:24.959625 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:44:24.959407    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-m9c72"
Sep 18 19:44:24.996250 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:44:24.996136    2585 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 18 19:44:25.243452 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:44:25.243356    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-86c5bc79d6-q45br"]
Sep 18 19:44:25.246360 ip-10-0-136-185 kubenswrapper[2585]: W0918 19:44:25.246328    2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd6506887_aec3_4244_911e_b0a4fd696601.slice/crio-24c35fc60d88f2285746ff55b088586db1fce784b7d0f2f7ffdb49b07986eed2 WatchSource:0}: Error finding container 24c35fc60d88f2285746ff55b088586db1fce784b7d0f2f7ffdb49b07986eed2: Status 404 returned error can't find the container with id 24c35fc60d88f2285746ff55b088586db1fce784b7d0f2f7ffdb49b07986eed2
Sep 18 19:44:25.671474 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:44:25.671430    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-86c5bc79d6-q45br" event={"ID":"d6506887-aec3-4244-911e-b0a4fd696601","Type":"ContainerStarted","Data":"acdb916b028e858c8ff861366c512d8e2078bf00dcf6b7f56f39e1e36d1c5d38"}
Sep 18 19:44:25.671674 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:44:25.671512    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-86c5bc79d6-q45br" event={"ID":"d6506887-aec3-4244-911e-b0a4fd696601","Type":"ContainerStarted","Data":"06549cb00e4735eec4db5c685f04890a1c42c94c40d5532f1adf2b52004a0721"}
Sep 18 19:44:25.671674 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:44:25.671544    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-86c5bc79d6-q45br" event={"ID":"d6506887-aec3-4244-911e-b0a4fd696601","Type":"ContainerStarted","Data":"24c35fc60d88f2285746ff55b088586db1fce784b7d0f2f7ffdb49b07986eed2"}
Sep 18 19:44:25.672861 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:44:25.672829    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-m9c72" event={"ID":"ff4afcd9-ff15-4e03-a63a-7927c4b458f4","Type":"ContainerStarted","Data":"bb4288253934eacfd99b017100c6844c509651d1c0f063fe0603ec7e2a6d4e86"}
Sep 18 19:44:26.656245 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:44:26.656209    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4njfl"
Sep 18 19:44:28.840390 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:44:28.840356    2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/monitoring-plugin-5984459ff8-dmghk"]
Sep 18 19:44:28.865965 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:44:28.865935    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-5984459ff8-dmghk"]
Sep 18 19:44:28.866112 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:44:28.866043    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-5984459ff8-dmghk"
Sep 18 19:44:28.868615 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:44:28.868591    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-dockercfg-nfkjh\""
Sep 18 19:44:28.868753 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:44:28.868627    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-cert\""
Sep 18 19:44:29.027637 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:44:29.027591    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/4cf62754-c2f2-4fb8-81bb-5302f58e0cf5-monitoring-plugin-cert\") pod \"monitoring-plugin-5984459ff8-dmghk\" (UID: \"4cf62754-c2f2-4fb8-81bb-5302f58e0cf5\") " pod="openshift-monitoring/monitoring-plugin-5984459ff8-dmghk"
Sep 18 19:44:29.027892 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:44:29.027742    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2j8p4\" (UniqueName: \"kubernetes.io/projected/4cf62754-c2f2-4fb8-81bb-5302f58e0cf5-kube-api-access-2j8p4\") pod \"monitoring-plugin-5984459ff8-dmghk\" (UID: \"4cf62754-c2f2-4fb8-81bb-5302f58e0cf5\") " pod="openshift-monitoring/monitoring-plugin-5984459ff8-dmghk"
Sep 18 19:44:29.128612 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:44:29.128520    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/4cf62754-c2f2-4fb8-81bb-5302f58e0cf5-monitoring-plugin-cert\") pod \"monitoring-plugin-5984459ff8-dmghk\" (UID: \"4cf62754-c2f2-4fb8-81bb-5302f58e0cf5\") " pod="openshift-monitoring/monitoring-plugin-5984459ff8-dmghk"
Sep 18 19:44:29.128809 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:44:29.128690    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2j8p4\" (UniqueName: \"kubernetes.io/projected/4cf62754-c2f2-4fb8-81bb-5302f58e0cf5-kube-api-access-2j8p4\") pod \"monitoring-plugin-5984459ff8-dmghk\" (UID: \"4cf62754-c2f2-4fb8-81bb-5302f58e0cf5\") " pod="openshift-monitoring/monitoring-plugin-5984459ff8-dmghk"
Sep 18 19:44:29.131558 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:44:29.131527    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/4cf62754-c2f2-4fb8-81bb-5302f58e0cf5-monitoring-plugin-cert\") pod \"monitoring-plugin-5984459ff8-dmghk\" (UID: \"4cf62754-c2f2-4fb8-81bb-5302f58e0cf5\") " pod="openshift-monitoring/monitoring-plugin-5984459ff8-dmghk"
Sep 18 19:44:29.145767 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:44:29.145746    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2j8p4\" (UniqueName: \"kubernetes.io/projected/4cf62754-c2f2-4fb8-81bb-5302f58e0cf5-kube-api-access-2j8p4\") pod \"monitoring-plugin-5984459ff8-dmghk\" (UID: \"4cf62754-c2f2-4fb8-81bb-5302f58e0cf5\") " pod="openshift-monitoring/monitoring-plugin-5984459ff8-dmghk"
Sep 18 19:44:29.177631 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:44:29.177558    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-5984459ff8-dmghk"
Sep 18 19:44:30.227544 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:44:30.227513    2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 18 19:44:30.248635 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:44:30.248597    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 19:44:30.248816 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:44:30.248701    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 18 19:44:30.252347 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:44:30.252322    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\""
Sep 18 19:44:30.252482 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:44:30.252401    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\""
Sep 18 19:44:30.252749 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:44:30.252699    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\""
Sep 18 19:44:30.252749 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:44:30.252709    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\""
Sep 18 19:44:30.252960 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:44:30.252758    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\""
Sep 18 19:44:30.253056 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:44:30.253019    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\""
Sep 18 19:44:30.253491 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:44:30.253357    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\""
Sep 18 19:44:30.253592 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:44:30.253555    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\""
Sep 18 19:44:30.253691 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:44:30.253666    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\""
Sep 18 19:44:30.253864 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:44:30.253836    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\""
Sep 18 19:44:30.253939 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:44:30.253916    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-3cv80vk60q20o\""
Sep 18 19:44:30.255386 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:44:30.255363    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\""
Sep 18 19:44:30.255386 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:44:30.255375    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\""
Sep 18 19:44:30.255545 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:44:30.255391    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-htqkl\""
Sep 18 19:44:30.255826 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:44:30.255765    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\""
Sep 18 19:44:30.258820 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:44:30.258427    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\""
Sep 18 19:44:30.261003 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:44:30.260975    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\""
Sep 18 19:44:30.441023 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:44:30.440983    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/41d2256f-9766-4a77-af4c-22d08bd3f3eb-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"41d2256f-9766-4a77-af4c-22d08bd3f3eb\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 19:44:30.441023 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:44:30.441028    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/41d2256f-9766-4a77-af4c-22d08bd3f3eb-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"41d2256f-9766-4a77-af4c-22d08bd3f3eb\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 19:44:30.441264 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:44:30.441050    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/41d2256f-9766-4a77-af4c-22d08bd3f3eb-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"41d2256f-9766-4a77-af4c-22d08bd3f3eb\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 19:44:30.441264 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:44:30.441068    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/41d2256f-9766-4a77-af4c-22d08bd3f3eb-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"41d2256f-9766-4a77-af4c-22d08bd3f3eb\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 19:44:30.441264 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:44:30.441186    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zg6p4\" (UniqueName: \"kubernetes.io/projected/41d2256f-9766-4a77-af4c-22d08bd3f3eb-kube-api-access-zg6p4\") pod \"prometheus-k8s-0\" (UID: \"41d2256f-9766-4a77-af4c-22d08bd3f3eb\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 19:44:30.441264 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:44:30.441228    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/41d2256f-9766-4a77-af4c-22d08bd3f3eb-config\") pod \"prometheus-k8s-0\" (UID: \"41d2256f-9766-4a77-af4c-22d08bd3f3eb\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 19:44:30.441264 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:44:30.441251    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/41d2256f-9766-4a77-af4c-22d08bd3f3eb-web-config\") pod \"prometheus-k8s-0\" (UID: \"41d2256f-9766-4a77-af4c-22d08bd3f3eb\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 19:44:30.441468 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:44:30.441278    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/41d2256f-9766-4a77-af4c-22d08bd3f3eb-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"41d2256f-9766-4a77-af4c-22d08bd3f3eb\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 19:44:30.441468 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:44:30.441303    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/41d2256f-9766-4a77-af4c-22d08bd3f3eb-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"41d2256f-9766-4a77-af4c-22d08bd3f3eb\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 19:44:30.441468 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:44:30.441358    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/41d2256f-9766-4a77-af4c-22d08bd3f3eb-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"41d2256f-9766-4a77-af4c-22d08bd3f3eb\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 19:44:30.441468 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:44:30.441395    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/41d2256f-9766-4a77-af4c-22d08bd3f3eb-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"41d2256f-9766-4a77-af4c-22d08bd3f3eb\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 19:44:30.441468 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:44:30.441436    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/41d2256f-9766-4a77-af4c-22d08bd3f3eb-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"41d2256f-9766-4a77-af4c-22d08bd3f3eb\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 19:44:30.441688 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:44:30.441476    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/41d2256f-9766-4a77-af4c-22d08bd3f3eb-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"41d2256f-9766-4a77-af4c-22d08bd3f3eb\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 19:44:30.441688 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:44:30.441504    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/41d2256f-9766-4a77-af4c-22d08bd3f3eb-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"41d2256f-9766-4a77-af4c-22d08bd3f3eb\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 19:44:30.441688 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:44:30.441518    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/41d2256f-9766-4a77-af4c-22d08bd3f3eb-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"41d2256f-9766-4a77-af4c-22d08bd3f3eb\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 19:44:30.441688 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:44:30.441545    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/41d2256f-9766-4a77-af4c-22d08bd3f3eb-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"41d2256f-9766-4a77-af4c-22d08bd3f3eb\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 19:44:30.441688 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:44:30.441642    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/41d2256f-9766-4a77-af4c-22d08bd3f3eb-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"41d2256f-9766-4a77-af4c-22d08bd3f3eb\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 19:44:30.441688 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:44:30.441684    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/41d2256f-9766-4a77-af4c-22d08bd3f3eb-config-out\") pod \"prometheus-k8s-0\" (UID: \"41d2256f-9766-4a77-af4c-22d08bd3f3eb\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 19:44:30.441962 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:44:30.441708    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/41d2256f-9766-4a77-af4c-22d08bd3f3eb-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"41d2256f-9766-4a77-af4c-22d08bd3f3eb\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 19:44:30.441962 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:44:30.441733    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/41d2256f-9766-4a77-af4c-22d08bd3f3eb-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"41d2256f-9766-4a77-af4c-22d08bd3f3eb\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 19:44:30.543137 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:44:30.543052    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/41d2256f-9766-4a77-af4c-22d08bd3f3eb-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"41d2256f-9766-4a77-af4c-22d08bd3f3eb\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 19:44:30.543137 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:44:30.543112    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/41d2256f-9766-4a77-af4c-22d08bd3f3eb-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"41d2256f-9766-4a77-af4c-22d08bd3f3eb\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 19:44:30.543137 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:44:30.543139    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/41d2256f-9766-4a77-af4c-22d08bd3f3eb-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"41d2256f-9766-4a77-af4c-22d08bd3f3eb\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 19:44:30.543426 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:44:30.543170    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/41d2256f-9766-4a77-af4c-22d08bd3f3eb-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"41d2256f-9766-4a77-af4c-22d08bd3f3eb\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 19:44:30.543426 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:44:30.543225    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/41d2256f-9766-4a77-af4c-22d08bd3f3eb-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"41d2256f-9766-4a77-af4c-22d08bd3f3eb\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 19:44:30.543426 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:44:30.543256    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/41d2256f-9766-4a77-af4c-22d08bd3f3eb-config-out\") pod \"prometheus-k8s-0\" (UID: \"41d2256f-9766-4a77-af4c-22d08bd3f3eb\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 19:44:30.543426 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:44:30.543281    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/41d2256f-9766-4a77-af4c-22d08bd3f3eb-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"41d2256f-9766-4a77-af4c-22d08bd3f3eb\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 19:44:30.543426 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:44:30.543305    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/41d2256f-9766-4a77-af4c-22d08bd3f3eb-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"41d2256f-9766-4a77-af4c-22d08bd3f3eb\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 19:44:30.543426 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:44:30.543332    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/41d2256f-9766-4a77-af4c-22d08bd3f3eb-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"41d2256f-9766-4a77-af4c-22d08bd3f3eb\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 19:44:30.543426 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:44:30.543358    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/41d2256f-9766-4a77-af4c-22d08bd3f3eb-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"41d2256f-9766-4a77-af4c-22d08bd3f3eb\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 19:44:30.543426 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:44:30.543386    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/41d2256f-9766-4a77-af4c-22d08bd3f3eb-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"41d2256f-9766-4a77-af4c-22d08bd3f3eb\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 19:44:30.543426 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:44:30.543410    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/41d2256f-9766-4a77-af4c-22d08bd3f3eb-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"41d2256f-9766-4a77-af4c-22d08bd3f3eb\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 19:44:30.543910 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:44:30.543452    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zg6p4\" (UniqueName: \"kubernetes.io/projected/41d2256f-9766-4a77-af4c-22d08bd3f3eb-kube-api-access-zg6p4\") pod \"prometheus-k8s-0\" (UID: \"41d2256f-9766-4a77-af4c-22d08bd3f3eb\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 19:44:30.543910 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:44:30.543477    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/41d2256f-9766-4a77-af4c-22d08bd3f3eb-config\") pod \"prometheus-k8s-0\" (UID: \"41d2256f-9766-4a77-af4c-22d08bd3f3eb\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 19:44:30.543910 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:44:30.543499    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/41d2256f-9766-4a77-af4c-22d08bd3f3eb-web-config\") pod \"prometheus-k8s-0\" (UID: \"41d2256f-9766-4a77-af4c-22d08bd3f3eb\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 19:44:30.543910 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:44:30.543527    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/41d2256f-9766-4a77-af4c-22d08bd3f3eb-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"41d2256f-9766-4a77-af4c-22d08bd3f3eb\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 19:44:30.543910 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:44:30.543551    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/41d2256f-9766-4a77-af4c-22d08bd3f3eb-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"41d2256f-9766-4a77-af4c-22d08bd3f3eb\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 19:44:30.543910 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:44:30.543577    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/41d2256f-9766-4a77-af4c-22d08bd3f3eb-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"41d2256f-9766-4a77-af4c-22d08bd3f3eb\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 19:44:30.543910 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:44:30.543600    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/41d2256f-9766-4a77-af4c-22d08bd3f3eb-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"41d2256f-9766-4a77-af4c-22d08bd3f3eb\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 19:44:30.543910 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:44:30.543635    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/41d2256f-9766-4a77-af4c-22d08bd3f3eb-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"41d2256f-9766-4a77-af4c-22d08bd3f3eb\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 19:44:30.543910 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:44:30.543821    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/41d2256f-9766-4a77-af4c-22d08bd3f3eb-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"41d2256f-9766-4a77-af4c-22d08bd3f3eb\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 19:44:30.544355 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:44:30.544099    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/41d2256f-9766-4a77-af4c-22d08bd3f3eb-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"41d2256f-9766-4a77-af4c-22d08bd3f3eb\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 19:44:30.544598 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:44:30.544571    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/41d2256f-9766-4a77-af4c-22d08bd3f3eb-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"41d2256f-9766-4a77-af4c-22d08bd3f3eb\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 19:44:30.544598 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:44:30.544578    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/41d2256f-9766-4a77-af4c-22d08bd3f3eb-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"41d2256f-9766-4a77-af4c-22d08bd3f3eb\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 19:44:30.545857 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:44:30.545535    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/41d2256f-9766-4a77-af4c-22d08bd3f3eb-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"41d2256f-9766-4a77-af4c-22d08bd3f3eb\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 19:44:30.547544 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:44:30.546665    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/41d2256f-9766-4a77-af4c-22d08bd3f3eb-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"41d2256f-9766-4a77-af4c-22d08bd3f3eb\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 19:44:30.547544 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:44:30.547101    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/41d2256f-9766-4a77-af4c-22d08bd3f3eb-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"41d2256f-9766-4a77-af4c-22d08bd3f3eb\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 19:44:30.547544 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:44:30.547239    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/41d2256f-9766-4a77-af4c-22d08bd3f3eb-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"41d2256f-9766-4a77-af4c-22d08bd3f3eb\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 19:44:30.547544 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:44:30.547262    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/41d2256f-9766-4a77-af4c-22d08bd3f3eb-config\") pod \"prometheus-k8s-0\" (UID: \"41d2256f-9766-4a77-af4c-22d08bd3f3eb\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 19:44:30.547544 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:44:30.547414    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/41d2256f-9766-4a77-af4c-22d08bd3f3eb-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"41d2256f-9766-4a77-af4c-22d08bd3f3eb\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 19:44:30.548358 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:44:30.548054    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/41d2256f-9766-4a77-af4c-22d08bd3f3eb-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"41d2256f-9766-4a77-af4c-22d08bd3f3eb\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 19:44:30.548451 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:44:30.548363    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/41d2256f-9766-4a77-af4c-22d08bd3f3eb-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"41d2256f-9766-4a77-af4c-22d08bd3f3eb\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 19:44:30.548451 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:44:30.548432    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/41d2256f-9766-4a77-af4c-22d08bd3f3eb-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"41d2256f-9766-4a77-af4c-22d08bd3f3eb\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 19:44:30.548701 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:44:30.548658    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/41d2256f-9766-4a77-af4c-22d08bd3f3eb-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"41d2256f-9766-4a77-af4c-22d08bd3f3eb\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 19:44:30.549676 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:44:30.549649    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/41d2256f-9766-4a77-af4c-22d08bd3f3eb-config-out\") pod \"prometheus-k8s-0\" (UID: \"41d2256f-9766-4a77-af4c-22d08bd3f3eb\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 19:44:30.550140 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:44:30.550101    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/41d2256f-9766-4a77-af4c-22d08bd3f3eb-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"41d2256f-9766-4a77-af4c-22d08bd3f3eb\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 19:44:30.550648 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:44:30.550623    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/41d2256f-9766-4a77-af4c-22d08bd3f3eb-web-config\") pod \"prometheus-k8s-0\" (UID: \"41d2256f-9766-4a77-af4c-22d08bd3f3eb\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 19:44:30.550883 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:44:30.550863    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/41d2256f-9766-4a77-af4c-22d08bd3f3eb-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"41d2256f-9766-4a77-af4c-22d08bd3f3eb\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 19:44:30.551348 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:44:30.551301    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/41d2256f-9766-4a77-af4c-22d08bd3f3eb-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"41d2256f-9766-4a77-af4c-22d08bd3f3eb\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 19:44:30.553647 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:44:30.553626    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zg6p4\" (UniqueName: \"kubernetes.io/projected/41d2256f-9766-4a77-af4c-22d08bd3f3eb-kube-api-access-zg6p4\") pod \"prometheus-k8s-0\" (UID: \"41d2256f-9766-4a77-af4c-22d08bd3f3eb\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 19:44:30.561516 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:44:30.561494    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 19:44:33.064824 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:44:33.064696    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-5984459ff8-dmghk"]
Sep 18 19:44:33.080607 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:44:33.080569    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 18 19:44:33.278334 ip-10-0-136-185 kubenswrapper[2585]: W0918 19:44:33.278293    2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4cf62754_c2f2_4fb8_81bb_5302f58e0cf5.slice/crio-49332908ce700956b78f8fd2a75a18f79a0726b53e9726c024201c1ed2b9ea42 WatchSource:0}: Error finding container 49332908ce700956b78f8fd2a75a18f79a0726b53e9726c024201c1ed2b9ea42: Status 404 returned error can't find the container with id 49332908ce700956b78f8fd2a75a18f79a0726b53e9726c024201c1ed2b9ea42
Sep 18 19:44:33.278961 ip-10-0-136-185 kubenswrapper[2585]: W0918 19:44:33.278927    2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41d2256f_9766_4a77_af4c_22d08bd3f3eb.slice/crio-86fa1a7cd6208e1d29ba9c0628d6c17b9c540549e6eccc5bf457bc2374e974a8 WatchSource:0}: Error finding container 86fa1a7cd6208e1d29ba9c0628d6c17b9c540549e6eccc5bf457bc2374e974a8: Status 404 returned error can't find the container with id 86fa1a7cd6208e1d29ba9c0628d6c17b9c540549e6eccc5bf457bc2374e974a8
Sep 18 19:44:33.707150 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:44:33.707111    2585 generic.go:356] "Generic (PLEG): container finished" podID="ff4afcd9-ff15-4e03-a63a-7927c4b458f4" containerID="ffa0c3c10e5eff1938019ae2a484bfaea66501f098b605424d76911d421de5ee" exitCode=0
Sep 18 19:44:33.707339 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:44:33.707237    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-m9c72" event={"ID":"ff4afcd9-ff15-4e03-a63a-7927c4b458f4","Type":"ContainerDied","Data":"ffa0c3c10e5eff1938019ae2a484bfaea66501f098b605424d76911d421de5ee"}
Sep 18 19:44:33.708758 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:44:33.708727    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-5984459ff8-dmghk" event={"ID":"4cf62754-c2f2-4fb8-81bb-5302f58e0cf5","Type":"ContainerStarted","Data":"49332908ce700956b78f8fd2a75a18f79a0726b53e9726c024201c1ed2b9ea42"}
Sep 18 19:44:33.711585 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:44:33.711443    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-86c5bc79d6-q45br" event={"ID":"d6506887-aec3-4244-911e-b0a4fd696601","Type":"ContainerStarted","Data":"4edf7f38ce31f4cc560ba39f139bab74f1ae229b6a845b512df91d77f3806896"}
Sep 18 19:44:33.713236 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:44:33.713209    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"41d2256f-9766-4a77-af4c-22d08bd3f3eb","Type":"ContainerStarted","Data":"86fa1a7cd6208e1d29ba9c0628d6c17b9c540549e6eccc5bf457bc2374e974a8"}
Sep 18 19:44:33.715060 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:44:33.715017    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-6594f4c75-p7gk4" event={"ID":"1f4af1db-93c0-48d0-81df-868ea52c8a1e","Type":"ContainerStarted","Data":"cf848e35bd921e6dfcc8a3219b180a9c43b3ae3df129cbf53581e135121ea8d0"}
Sep 18 19:44:33.715748 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:44:33.715727    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/downloads-6594f4c75-p7gk4"
Sep 18 19:44:33.729268 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:44:33.729242    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/downloads-6594f4c75-p7gk4"
Sep 18 19:44:33.746464 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:44:33.746419    2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/openshift-state-metrics-86c5bc79d6-q45br" podStartSLOduration=2.941463847 podStartE2EDuration="10.746400941s" totalImagesPullingTime="7.804937094s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 19:44:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 19:44:33.745119341 +0000 UTC m=+315.269724765" watchObservedRunningTime="2026-09-18 19:44:33.746400941 +0000 UTC m=+315.271006364"
Sep 18 19:44:33.761989 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:44:33.761937    2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/downloads-6594f4c75-p7gk4" podStartSLOduration=2.451198032 podStartE2EDuration="19.761920415s" totalImagesPullingTime="17.310722383s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 19:44:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 19:44:33.761515415 +0000 UTC m=+315.286120838" watchObservedRunningTime="2026-09-18 19:44:33.761920415 +0000 UTC m=+315.286525838"
Sep 18 19:44:34.723773 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:44:34.723557    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-m9c72" event={"ID":"ff4afcd9-ff15-4e03-a63a-7927c4b458f4","Type":"ContainerStarted","Data":"3a47c7bc994027df29eebaf21307f34870b45c35abd43adbcec5411b98f67478"}
Sep 18 19:44:34.723773 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:44:34.723737    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-m9c72" event={"ID":"ff4afcd9-ff15-4e03-a63a-7927c4b458f4","Type":"ContainerStarted","Data":"04a4af548c72cd989913fc814727b1f60f89c35960b69e13331ae9e00b30c7af"}
Sep 18 19:44:34.742829 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:44:34.742384    2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-m9c72" podStartSLOduration=2.8725396549999997 podStartE2EDuration="10.742363092s" totalImagesPullingTime="7.788645489s" totalInitContainerRuntime="81.177948ms" isStatefulPod=true podCreationTimestamp="2026-09-18 19:44:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 19:44:34.742299568 +0000 UTC m=+316.266904992" watchObservedRunningTime="2026-09-18 19:44:34.742363092 +0000 UTC m=+316.266968516"
Sep 18 19:44:35.729654 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:44:35.729554    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-5984459ff8-dmghk" event={"ID":"4cf62754-c2f2-4fb8-81bb-5302f58e0cf5","Type":"ContainerStarted","Data":"3796b00cf5f5fa8025dc93e3639ea5d8884c2cac29a8e79745f9661364625683"}
Sep 18 19:44:35.730148 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:44:35.729865    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/monitoring-plugin-5984459ff8-dmghk"
Sep 18 19:44:35.731615 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:44:35.731581    2585 generic.go:356] "Generic (PLEG): container finished" podID="41d2256f-9766-4a77-af4c-22d08bd3f3eb" containerID="b2c744a360d6f3c074c55c47bbd633c70e7f10efa7780b29ee14e9221c9654b4" exitCode=0
Sep 18 19:44:35.731861 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:44:35.731835    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"41d2256f-9766-4a77-af4c-22d08bd3f3eb","Type":"ContainerDied","Data":"b2c744a360d6f3c074c55c47bbd633c70e7f10efa7780b29ee14e9221c9654b4"}
Sep 18 19:44:35.735828 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:44:35.735806    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/monitoring-plugin-5984459ff8-dmghk"
Sep 18 19:44:35.744633 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:44:35.744593    2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/monitoring-plugin-5984459ff8-dmghk" podStartSLOduration=5.5543765579999995 podStartE2EDuration="7.744574553s" totalImagesPullingTime="2.190197995s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 19:44:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 19:44:35.743814111 +0000 UTC m=+317.268419536" watchObservedRunningTime="2026-09-18 19:44:35.744574553 +0000 UTC m=+317.269179976"
Sep 18 19:44:39.912101 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:44:39.911896    2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-image-registry/image-registry-75d9fb895b-85gqr" podUID="79747afc-194b-4ded-bdd1-d6332f8412b7" containerName="registry" containerID="cri-o://8ffae4b2577c47948578da505a586e8182a3f8cb684198b48d523d297c400649" gracePeriod=30
Sep 18 19:44:40.231412 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:44:40.231378    2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-75d9fb895b-85gqr"
Sep 18 19:44:40.335683 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:44:40.335594    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/79747afc-194b-4ded-bdd1-d6332f8412b7-installation-pull-secrets\") pod \"79747afc-194b-4ded-bdd1-d6332f8412b7\" (UID: \"79747afc-194b-4ded-bdd1-d6332f8412b7\") "
Sep 18 19:44:40.335683 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:44:40.335657    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/79747afc-194b-4ded-bdd1-d6332f8412b7-trusted-ca\") pod \"79747afc-194b-4ded-bdd1-d6332f8412b7\" (UID: \"79747afc-194b-4ded-bdd1-d6332f8412b7\") "
Sep 18 19:44:40.335683 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:44:40.335686    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/79747afc-194b-4ded-bdd1-d6332f8412b7-bound-sa-token\") pod \"79747afc-194b-4ded-bdd1-d6332f8412b7\" (UID: \"79747afc-194b-4ded-bdd1-d6332f8412b7\") "
Sep 18 19:44:40.335996 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:44:40.335830    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/79747afc-194b-4ded-bdd1-d6332f8412b7-registry-tls\") pod \"79747afc-194b-4ded-bdd1-d6332f8412b7\" (UID: \"79747afc-194b-4ded-bdd1-d6332f8412b7\") "
Sep 18 19:44:40.335996 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:44:40.335870    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/79747afc-194b-4ded-bdd1-d6332f8412b7-image-registry-private-configuration\") pod \"79747afc-194b-4ded-bdd1-d6332f8412b7\" (UID: \"79747afc-194b-4ded-bdd1-d6332f8412b7\") "
Sep 18 19:44:40.335996 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:44:40.335914    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/79747afc-194b-4ded-bdd1-d6332f8412b7-ca-trust-extracted\") pod \"79747afc-194b-4ded-bdd1-d6332f8412b7\" (UID: \"79747afc-194b-4ded-bdd1-d6332f8412b7\") "
Sep 18 19:44:40.335996 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:44:40.335957    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/79747afc-194b-4ded-bdd1-d6332f8412b7-registry-certificates\") pod \"79747afc-194b-4ded-bdd1-d6332f8412b7\" (UID: \"79747afc-194b-4ded-bdd1-d6332f8412b7\") "
Sep 18 19:44:40.335996 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:44:40.335985    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-59cdq\" (UniqueName: \"kubernetes.io/projected/79747afc-194b-4ded-bdd1-d6332f8412b7-kube-api-access-59cdq\") pod \"79747afc-194b-4ded-bdd1-d6332f8412b7\" (UID: \"79747afc-194b-4ded-bdd1-d6332f8412b7\") "
Sep 18 19:44:40.336280 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:44:40.336100    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/79747afc-194b-4ded-bdd1-d6332f8412b7-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "79747afc-194b-4ded-bdd1-d6332f8412b7" (UID: "79747afc-194b-4ded-bdd1-d6332f8412b7"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 19:44:40.336383 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:44:40.336336    2585 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/79747afc-194b-4ded-bdd1-d6332f8412b7-trusted-ca\") on node \"ip-10-0-136-185.ec2.internal\" DevicePath \"\""
Sep 18 19:44:40.336457 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:44:40.336418    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/79747afc-194b-4ded-bdd1-d6332f8412b7-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "79747afc-194b-4ded-bdd1-d6332f8412b7" (UID: "79747afc-194b-4ded-bdd1-d6332f8412b7"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 19:44:40.338559 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:44:40.338531    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/79747afc-194b-4ded-bdd1-d6332f8412b7-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "79747afc-194b-4ded-bdd1-d6332f8412b7" (UID: "79747afc-194b-4ded-bdd1-d6332f8412b7"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 19:44:40.339083 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:44:40.339052    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/79747afc-194b-4ded-bdd1-d6332f8412b7-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "79747afc-194b-4ded-bdd1-d6332f8412b7" (UID: "79747afc-194b-4ded-bdd1-d6332f8412b7"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 19:44:40.339083 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:44:40.339079    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/79747afc-194b-4ded-bdd1-d6332f8412b7-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "79747afc-194b-4ded-bdd1-d6332f8412b7" (UID: "79747afc-194b-4ded-bdd1-d6332f8412b7"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 19:44:40.339391 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:44:40.339369    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/79747afc-194b-4ded-bdd1-d6332f8412b7-kube-api-access-59cdq" (OuterVolumeSpecName: "kube-api-access-59cdq") pod "79747afc-194b-4ded-bdd1-d6332f8412b7" (UID: "79747afc-194b-4ded-bdd1-d6332f8412b7"). InnerVolumeSpecName "kube-api-access-59cdq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 19:44:40.339584 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:44:40.339542    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/79747afc-194b-4ded-bdd1-d6332f8412b7-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "79747afc-194b-4ded-bdd1-d6332f8412b7" (UID: "79747afc-194b-4ded-bdd1-d6332f8412b7"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 19:44:40.345296 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:44:40.345265    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/79747afc-194b-4ded-bdd1-d6332f8412b7-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "79747afc-194b-4ded-bdd1-d6332f8412b7" (UID: "79747afc-194b-4ded-bdd1-d6332f8412b7"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 19:44:40.436827 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:44:40.436761    2585 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/79747afc-194b-4ded-bdd1-d6332f8412b7-bound-sa-token\") on node \"ip-10-0-136-185.ec2.internal\" DevicePath \"\""
Sep 18 19:44:40.436827 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:44:40.436824    2585 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/79747afc-194b-4ded-bdd1-d6332f8412b7-registry-tls\") on node \"ip-10-0-136-185.ec2.internal\" DevicePath \"\""
Sep 18 19:44:40.437043 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:44:40.436841    2585 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/79747afc-194b-4ded-bdd1-d6332f8412b7-image-registry-private-configuration\") on node \"ip-10-0-136-185.ec2.internal\" DevicePath \"\""
Sep 18 19:44:40.437043 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:44:40.436857    2585 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/79747afc-194b-4ded-bdd1-d6332f8412b7-ca-trust-extracted\") on node \"ip-10-0-136-185.ec2.internal\" DevicePath \"\""
Sep 18 19:44:40.437043 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:44:40.436871    2585 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/79747afc-194b-4ded-bdd1-d6332f8412b7-registry-certificates\") on node \"ip-10-0-136-185.ec2.internal\" DevicePath \"\""
Sep 18 19:44:40.437043 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:44:40.436887    2585 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-59cdq\" (UniqueName: \"kubernetes.io/projected/79747afc-194b-4ded-bdd1-d6332f8412b7-kube-api-access-59cdq\") on node \"ip-10-0-136-185.ec2.internal\" DevicePath \"\""
Sep 18 19:44:40.437043 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:44:40.436902    2585 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/79747afc-194b-4ded-bdd1-d6332f8412b7-installation-pull-secrets\") on node \"ip-10-0-136-185.ec2.internal\" DevicePath \"\""
Sep 18 19:44:40.759645 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:44:40.759606    2585 generic.go:356] "Generic (PLEG): container finished" podID="79747afc-194b-4ded-bdd1-d6332f8412b7" containerID="8ffae4b2577c47948578da505a586e8182a3f8cb684198b48d523d297c400649" exitCode=0
Sep 18 19:44:40.759837 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:44:40.759799    2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-75d9fb895b-85gqr"
Sep 18 19:44:40.760456 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:44:40.760129    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-75d9fb895b-85gqr" event={"ID":"79747afc-194b-4ded-bdd1-d6332f8412b7","Type":"ContainerDied","Data":"8ffae4b2577c47948578da505a586e8182a3f8cb684198b48d523d297c400649"}
Sep 18 19:44:40.760456 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:44:40.760248    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-75d9fb895b-85gqr" event={"ID":"79747afc-194b-4ded-bdd1-d6332f8412b7","Type":"ContainerDied","Data":"5ef60cc6451c629c0357f3477e61b68725dad8b2fc63d0343b37df6681d99cbb"}
Sep 18 19:44:40.760456 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:44:40.760276    2585 scope.go:117] "RemoveContainer" containerID="8ffae4b2577c47948578da505a586e8182a3f8cb684198b48d523d297c400649"
Sep 18 19:44:40.765242 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:44:40.765213    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"41d2256f-9766-4a77-af4c-22d08bd3f3eb","Type":"ContainerStarted","Data":"a4335bc302a43d97f61ffeef2a6daa7c085b6081ae8f5f86d8e6454f37a20cde"}
Sep 18 19:44:40.765376 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:44:40.765357    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"41d2256f-9766-4a77-af4c-22d08bd3f3eb","Type":"ContainerStarted","Data":"c964b1efea9fdec097ed230ae8ec386223ef81b1f2e5307e33def9ddc2d1e336"}
Sep 18 19:44:40.773006 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:44:40.772979    2585 scope.go:117] "RemoveContainer" containerID="8ffae4b2577c47948578da505a586e8182a3f8cb684198b48d523d297c400649"
Sep 18 19:44:40.773490 ip-10-0-136-185 kubenswrapper[2585]: E0918 19:44:40.773460    2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8ffae4b2577c47948578da505a586e8182a3f8cb684198b48d523d297c400649\": container with ID starting with 8ffae4b2577c47948578da505a586e8182a3f8cb684198b48d523d297c400649 not found: ID does not exist" containerID="8ffae4b2577c47948578da505a586e8182a3f8cb684198b48d523d297c400649"
Sep 18 19:44:40.773602 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:44:40.773501    2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8ffae4b2577c47948578da505a586e8182a3f8cb684198b48d523d297c400649"} err="failed to get container status \"8ffae4b2577c47948578da505a586e8182a3f8cb684198b48d523d297c400649\": rpc error: code = NotFound desc = could not find container \"8ffae4b2577c47948578da505a586e8182a3f8cb684198b48d523d297c400649\": container with ID starting with 8ffae4b2577c47948578da505a586e8182a3f8cb684198b48d523d297c400649 not found: ID does not exist"
Sep 18 19:44:40.777656 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:44:40.777628    2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-75d9fb895b-85gqr"]
Sep 18 19:44:40.783134 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:44:40.783107    2585 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-75d9fb895b-85gqr"]
Sep 18 19:44:42.661100 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:44:42.661050    2585 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="79747afc-194b-4ded-bdd1-d6332f8412b7" path="/var/lib/kubelet/pods/79747afc-194b-4ded-bdd1-d6332f8412b7/volumes"
Sep 18 19:44:42.778558 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:44:42.778528    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"41d2256f-9766-4a77-af4c-22d08bd3f3eb","Type":"ContainerStarted","Data":"87c91cb8554a54434d8ed7e7ca7d9749cdbce91c86a3ffa78fb541c442c16e56"}
Sep 18 19:44:42.778717 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:44:42.778592    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"41d2256f-9766-4a77-af4c-22d08bd3f3eb","Type":"ContainerStarted","Data":"bd672fd65b3cc3916a3fbc16c82ee83e25b934c9c3d8362f16ce8c12b6f1286d"}
Sep 18 19:44:42.778717 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:44:42.778618    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"41d2256f-9766-4a77-af4c-22d08bd3f3eb","Type":"ContainerStarted","Data":"28389f299be1d3204aec9230b0766a75ee721d436d24ae7d37bf7477c9bd0a42"}
Sep 18 19:44:43.785086 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:44:43.785051    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"41d2256f-9766-4a77-af4c-22d08bd3f3eb","Type":"ContainerStarted","Data":"f3b4855a741ffa1c79b46340a8d87cfc5136b9594cd653e43204b65c372212db"}
Sep 18 19:44:43.816851 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:44:43.816780    2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=5.269492953 podStartE2EDuration="13.816762747s" totalImagesPullingTime="8.528316126s" totalInitContainerRuntime="18.953668ms" isStatefulPod=false podCreationTimestamp="2026-09-18 19:44:30 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 19:44:43.815868758 +0000 UTC m=+325.340474182" watchObservedRunningTime="2026-09-18 19:44:43.816762747 +0000 UTC m=+325.341368169"
Sep 18 19:44:45.561918 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:44:45.561883    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 19:45:01.853620 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:45:01.853585    2585 generic.go:356] "Generic (PLEG): container finished" podID="a9344bdb-53d7-4066-a8ef-7d28173a0617" containerID="5f3e7d9d557dcc0dbe73ab04ba3b748892424d1ef374ede43a167d0ed4951e1f" exitCode=0
Sep 18 19:45:01.854118 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:45:01.853627    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-57cdc4b768-xpd8n" event={"ID":"a9344bdb-53d7-4066-a8ef-7d28173a0617","Type":"ContainerDied","Data":"5f3e7d9d557dcc0dbe73ab04ba3b748892424d1ef374ede43a167d0ed4951e1f"}
Sep 18 19:45:01.854118 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:45:01.853954    2585 scope.go:117] "RemoveContainer" containerID="5f3e7d9d557dcc0dbe73ab04ba3b748892424d1ef374ede43a167d0ed4951e1f"
Sep 18 19:45:02.859439 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:45:02.859402    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-57cdc4b768-xpd8n" event={"ID":"a9344bdb-53d7-4066-a8ef-7d28173a0617","Type":"ContainerStarted","Data":"08299033ab1166436f055194a62004be1481e6cff78e49fbf814e6e89221f5f5"}
Sep 18 19:45:20.934451 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:45:20.934414    2585 generic.go:356] "Generic (PLEG): container finished" podID="d6560c03-8e80-4d69-b52c-73e63032f2ec" containerID="e294a4c944c9a9de9266c3a9fcc6264209ec84b375ba66a42eedec9d325d6611" exitCode=0
Sep 18 19:45:20.934863 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:45:20.934487    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-6f9c644dd9-6bc55" event={"ID":"d6560c03-8e80-4d69-b52c-73e63032f2ec","Type":"ContainerDied","Data":"e294a4c944c9a9de9266c3a9fcc6264209ec84b375ba66a42eedec9d325d6611"}
Sep 18 19:45:20.934863 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:45:20.934840    2585 scope.go:117] "RemoveContainer" containerID="e294a4c944c9a9de9266c3a9fcc6264209ec84b375ba66a42eedec9d325d6611"
Sep 18 19:45:21.940594 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:45:21.940561    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-6f9c644dd9-6bc55" event={"ID":"d6560c03-8e80-4d69-b52c-73e63032f2ec","Type":"ContainerStarted","Data":"90639989fc0bdc5b6ee0c45ae145b7273465c272b16a51087491c9f79ae1c4cc"}
Sep 18 19:45:30.561656 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:45:30.561604    2585 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 19:45:30.580322 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:45:30.580295    2585 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 19:45:30.990960 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:45:30.990934    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 19:45:34.485169 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:45:34.484573    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3eb3cd73-4ea1-4950-8bfd-82b5fa60fd09-metrics-certs\") pod \"network-metrics-daemon-4njfl\" (UID: \"3eb3cd73-4ea1-4950-8bfd-82b5fa60fd09\") " pod="openshift-multus/network-metrics-daemon-4njfl"
Sep 18 19:45:34.488527 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:45:34.488465    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3eb3cd73-4ea1-4950-8bfd-82b5fa60fd09-metrics-certs\") pod \"network-metrics-daemon-4njfl\" (UID: \"3eb3cd73-4ea1-4950-8bfd-82b5fa60fd09\") " pod="openshift-multus/network-metrics-daemon-4njfl"
Sep 18 19:45:34.759458 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:45:34.759376    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-m9tpx\""
Sep 18 19:45:34.767481 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:45:34.767442    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4njfl"
Sep 18 19:45:35.057365 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:45:35.057271    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-4njfl"]
Sep 18 19:45:35.062185 ip-10-0-136-185 kubenswrapper[2585]: W0918 19:45:35.062151    2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3eb3cd73_4ea1_4950_8bfd_82b5fa60fd09.slice/crio-95db09e8f7f349e535ea73079710e5781f3d5f226cb109bbfd0c97308e038b29 WatchSource:0}: Error finding container 95db09e8f7f349e535ea73079710e5781f3d5f226cb109bbfd0c97308e038b29: Status 404 returned error can't find the container with id 95db09e8f7f349e535ea73079710e5781f3d5f226cb109bbfd0c97308e038b29
Sep 18 19:45:35.996600 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:45:35.996559    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-4njfl" event={"ID":"3eb3cd73-4ea1-4950-8bfd-82b5fa60fd09","Type":"ContainerStarted","Data":"95db09e8f7f349e535ea73079710e5781f3d5f226cb109bbfd0c97308e038b29"}
Sep 18 19:45:37.002267 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:45:37.002233    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-4njfl" event={"ID":"3eb3cd73-4ea1-4950-8bfd-82b5fa60fd09","Type":"ContainerStarted","Data":"197054c5af97a81a7c9a44f41583e7ab0dfb3693e4e51c5ed98cd1e092ab7607"}
Sep 18 19:45:37.002647 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:45:37.002285    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-4njfl" event={"ID":"3eb3cd73-4ea1-4950-8bfd-82b5fa60fd09","Type":"ContainerStarted","Data":"fa8e1e2499ada74a4e52e1e49dc83604222d249fc825c77e9ae790dd23d1a0e2"}
Sep 18 19:45:37.018373 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:45:37.018320    2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-4njfl" podStartSLOduration=375.919355525 podStartE2EDuration="6m17.018304223s" totalImagesPullingTime="1.098948698s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 19:39:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 19:45:37.017685213 +0000 UTC m=+378.542290635" watchObservedRunningTime="2026-09-18 19:45:37.018304223 +0000 UTC m=+378.542909647"
Sep 18 19:45:48.623994 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:45:48.623913    2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 18 19:45:48.624409 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:45:48.624298    2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="41d2256f-9766-4a77-af4c-22d08bd3f3eb" containerName="kube-rbac-proxy-web" containerID="cri-o://bd672fd65b3cc3916a3fbc16c82ee83e25b934c9c3d8362f16ce8c12b6f1286d" gracePeriod=600
Sep 18 19:45:48.624409 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:45:48.624328    2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="41d2256f-9766-4a77-af4c-22d08bd3f3eb" containerName="kube-rbac-proxy-thanos" containerID="cri-o://f3b4855a741ffa1c79b46340a8d87cfc5136b9594cd653e43204b65c372212db" gracePeriod=600
Sep 18 19:45:48.624409 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:45:48.624322    2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="41d2256f-9766-4a77-af4c-22d08bd3f3eb" containerName="config-reloader" containerID="cri-o://a4335bc302a43d97f61ffeef2a6daa7c085b6081ae8f5f86d8e6454f37a20cde" gracePeriod=600
Sep 18 19:45:48.624409 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:45:48.624270    2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="41d2256f-9766-4a77-af4c-22d08bd3f3eb" containerName="prometheus" containerID="cri-o://c964b1efea9fdec097ed230ae8ec386223ef81b1f2e5307e33def9ddc2d1e336" gracePeriod=600
Sep 18 19:45:48.624409 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:45:48.624278    2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="41d2256f-9766-4a77-af4c-22d08bd3f3eb" containerName="kube-rbac-proxy" containerID="cri-o://87c91cb8554a54434d8ed7e7ca7d9749cdbce91c86a3ffa78fb541c442c16e56" gracePeriod=600
Sep 18 19:45:48.624623 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:45:48.624282    2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="41d2256f-9766-4a77-af4c-22d08bd3f3eb" containerName="thanos-sidecar" containerID="cri-o://28389f299be1d3204aec9230b0766a75ee721d436d24ae7d37bf7477c9bd0a42" gracePeriod=600
Sep 18 19:45:49.049311 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:45:49.049279    2585 generic.go:356] "Generic (PLEG): container finished" podID="41d2256f-9766-4a77-af4c-22d08bd3f3eb" containerID="f3b4855a741ffa1c79b46340a8d87cfc5136b9594cd653e43204b65c372212db" exitCode=0
Sep 18 19:45:49.049311 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:45:49.049303    2585 generic.go:356] "Generic (PLEG): container finished" podID="41d2256f-9766-4a77-af4c-22d08bd3f3eb" containerID="87c91cb8554a54434d8ed7e7ca7d9749cdbce91c86a3ffa78fb541c442c16e56" exitCode=0
Sep 18 19:45:49.049311 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:45:49.049309    2585 generic.go:356] "Generic (PLEG): container finished" podID="41d2256f-9766-4a77-af4c-22d08bd3f3eb" containerID="28389f299be1d3204aec9230b0766a75ee721d436d24ae7d37bf7477c9bd0a42" exitCode=0
Sep 18 19:45:49.049311 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:45:49.049314    2585 generic.go:356] "Generic (PLEG): container finished" podID="41d2256f-9766-4a77-af4c-22d08bd3f3eb" containerID="a4335bc302a43d97f61ffeef2a6daa7c085b6081ae8f5f86d8e6454f37a20cde" exitCode=0
Sep 18 19:45:49.049311 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:45:49.049327    2585 generic.go:356] "Generic (PLEG): container finished" podID="41d2256f-9766-4a77-af4c-22d08bd3f3eb" containerID="c964b1efea9fdec097ed230ae8ec386223ef81b1f2e5307e33def9ddc2d1e336" exitCode=0
Sep 18 19:45:49.049610 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:45:49.049354    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"41d2256f-9766-4a77-af4c-22d08bd3f3eb","Type":"ContainerDied","Data":"f3b4855a741ffa1c79b46340a8d87cfc5136b9594cd653e43204b65c372212db"}
Sep 18 19:45:49.049610 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:45:49.049421    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"41d2256f-9766-4a77-af4c-22d08bd3f3eb","Type":"ContainerDied","Data":"87c91cb8554a54434d8ed7e7ca7d9749cdbce91c86a3ffa78fb541c442c16e56"}
Sep 18 19:45:49.049610 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:45:49.049448    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"41d2256f-9766-4a77-af4c-22d08bd3f3eb","Type":"ContainerDied","Data":"28389f299be1d3204aec9230b0766a75ee721d436d24ae7d37bf7477c9bd0a42"}
Sep 18 19:45:49.049610 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:45:49.049478    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"41d2256f-9766-4a77-af4c-22d08bd3f3eb","Type":"ContainerDied","Data":"a4335bc302a43d97f61ffeef2a6daa7c085b6081ae8f5f86d8e6454f37a20cde"}
Sep 18 19:45:49.049610 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:45:49.049505    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"41d2256f-9766-4a77-af4c-22d08bd3f3eb","Type":"ContainerDied","Data":"c964b1efea9fdec097ed230ae8ec386223ef81b1f2e5307e33def9ddc2d1e336"}
Sep 18 19:45:49.863211 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:45:49.863187    2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 19:45:49.909930 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:45:49.909896    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/41d2256f-9766-4a77-af4c-22d08bd3f3eb-config\") pod \"41d2256f-9766-4a77-af4c-22d08bd3f3eb\" (UID: \"41d2256f-9766-4a77-af4c-22d08bd3f3eb\") "
Sep 18 19:45:49.909930 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:45:49.909932    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/41d2256f-9766-4a77-af4c-22d08bd3f3eb-secret-metrics-client-certs\") pod \"41d2256f-9766-4a77-af4c-22d08bd3f3eb\" (UID: \"41d2256f-9766-4a77-af4c-22d08bd3f3eb\") "
Sep 18 19:45:49.910162 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:45:49.909948    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/41d2256f-9766-4a77-af4c-22d08bd3f3eb-secret-grpc-tls\") pod \"41d2256f-9766-4a77-af4c-22d08bd3f3eb\" (UID: \"41d2256f-9766-4a77-af4c-22d08bd3f3eb\") "
Sep 18 19:45:49.910162 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:45:49.909973    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/41d2256f-9766-4a77-af4c-22d08bd3f3eb-web-config\") pod \"41d2256f-9766-4a77-af4c-22d08bd3f3eb\" (UID: \"41d2256f-9766-4a77-af4c-22d08bd3f3eb\") "
Sep 18 19:45:49.910162 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:45:49.909990    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/41d2256f-9766-4a77-af4c-22d08bd3f3eb-configmap-kubelet-serving-ca-bundle\") pod \"41d2256f-9766-4a77-af4c-22d08bd3f3eb\" (UID: \"41d2256f-9766-4a77-af4c-22d08bd3f3eb\") "
Sep 18 19:45:49.910162 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:45:49.910026    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/41d2256f-9766-4a77-af4c-22d08bd3f3eb-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"41d2256f-9766-4a77-af4c-22d08bd3f3eb\" (UID: \"41d2256f-9766-4a77-af4c-22d08bd3f3eb\") "
Sep 18 19:45:49.910162 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:45:49.910050    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/41d2256f-9766-4a77-af4c-22d08bd3f3eb-thanos-prometheus-http-client-file\") pod \"41d2256f-9766-4a77-af4c-22d08bd3f3eb\" (UID: \"41d2256f-9766-4a77-af4c-22d08bd3f3eb\") "
Sep 18 19:45:49.910162 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:45:49.910082    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zg6p4\" (UniqueName: \"kubernetes.io/projected/41d2256f-9766-4a77-af4c-22d08bd3f3eb-kube-api-access-zg6p4\") pod \"41d2256f-9766-4a77-af4c-22d08bd3f3eb\" (UID: \"41d2256f-9766-4a77-af4c-22d08bd3f3eb\") "
Sep 18 19:45:49.910162 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:45:49.910111    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/41d2256f-9766-4a77-af4c-22d08bd3f3eb-configmap-serving-certs-ca-bundle\") pod \"41d2256f-9766-4a77-af4c-22d08bd3f3eb\" (UID: \"41d2256f-9766-4a77-af4c-22d08bd3f3eb\") "
Sep 18 19:45:49.910162 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:45:49.910135    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/41d2256f-9766-4a77-af4c-22d08bd3f3eb-prometheus-trusted-ca-bundle\") pod \"41d2256f-9766-4a77-af4c-22d08bd3f3eb\" (UID: \"41d2256f-9766-4a77-af4c-22d08bd3f3eb\") "
Sep 18 19:45:49.910162 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:45:49.910163    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/41d2256f-9766-4a77-af4c-22d08bd3f3eb-prometheus-k8s-rulefiles-0\") pod \"41d2256f-9766-4a77-af4c-22d08bd3f3eb\" (UID: \"41d2256f-9766-4a77-af4c-22d08bd3f3eb\") "
Sep 18 19:45:49.910581 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:45:49.910192    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/41d2256f-9766-4a77-af4c-22d08bd3f3eb-secret-prometheus-k8s-tls\") pod \"41d2256f-9766-4a77-af4c-22d08bd3f3eb\" (UID: \"41d2256f-9766-4a77-af4c-22d08bd3f3eb\") "
Sep 18 19:45:49.910581 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:45:49.910221    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/41d2256f-9766-4a77-af4c-22d08bd3f3eb-tls-assets\") pod \"41d2256f-9766-4a77-af4c-22d08bd3f3eb\" (UID: \"41d2256f-9766-4a77-af4c-22d08bd3f3eb\") "
Sep 18 19:45:49.910581 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:45:49.910273    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/41d2256f-9766-4a77-af4c-22d08bd3f3eb-prometheus-k8s-rulefiles-1\") pod \"41d2256f-9766-4a77-af4c-22d08bd3f3eb\" (UID: \"41d2256f-9766-4a77-af4c-22d08bd3f3eb\") "
Sep 18 19:45:49.910581 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:45:49.910301    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/41d2256f-9766-4a77-af4c-22d08bd3f3eb-prometheus-k8s-db\") pod \"41d2256f-9766-4a77-af4c-22d08bd3f3eb\" (UID: \"41d2256f-9766-4a77-af4c-22d08bd3f3eb\") "
Sep 18 19:45:49.910581 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:45:49.910351    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/41d2256f-9766-4a77-af4c-22d08bd3f3eb-secret-kube-rbac-proxy\") pod \"41d2256f-9766-4a77-af4c-22d08bd3f3eb\" (UID: \"41d2256f-9766-4a77-af4c-22d08bd3f3eb\") "
Sep 18 19:45:49.910581 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:45:49.910381    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/41d2256f-9766-4a77-af4c-22d08bd3f3eb-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"41d2256f-9766-4a77-af4c-22d08bd3f3eb\" (UID: \"41d2256f-9766-4a77-af4c-22d08bd3f3eb\") "
Sep 18 19:45:49.910581 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:45:49.910430    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/41d2256f-9766-4a77-af4c-22d08bd3f3eb-configmap-metrics-client-ca\") pod \"41d2256f-9766-4a77-af4c-22d08bd3f3eb\" (UID: \"41d2256f-9766-4a77-af4c-22d08bd3f3eb\") "
Sep 18 19:45:49.910581 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:45:49.910459    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/41d2256f-9766-4a77-af4c-22d08bd3f3eb-prometheus-k8s-rulefiles-2\") pod \"41d2256f-9766-4a77-af4c-22d08bd3f3eb\" (UID: \"41d2256f-9766-4a77-af4c-22d08bd3f3eb\") "
Sep 18 19:45:49.910581 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:45:49.910490    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/41d2256f-9766-4a77-af4c-22d08bd3f3eb-config-out\") pod \"41d2256f-9766-4a77-af4c-22d08bd3f3eb\" (UID: \"41d2256f-9766-4a77-af4c-22d08bd3f3eb\") "
Sep 18 19:45:49.912469 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:45:49.910712    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/41d2256f-9766-4a77-af4c-22d08bd3f3eb-prometheus-trusted-ca-bundle" (OuterVolumeSpecName: "prometheus-trusted-ca-bundle") pod "41d2256f-9766-4a77-af4c-22d08bd3f3eb" (UID: "41d2256f-9766-4a77-af4c-22d08bd3f3eb"). InnerVolumeSpecName "prometheus-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 19:45:49.912469 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:45:49.911033    2585 reconciler_common.go:299] "Volume detached for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/41d2256f-9766-4a77-af4c-22d08bd3f3eb-prometheus-trusted-ca-bundle\") on node \"ip-10-0-136-185.ec2.internal\" DevicePath \"\""
Sep 18 19:45:49.913478 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:45:49.913426    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/41d2256f-9766-4a77-af4c-22d08bd3f3eb-prometheus-k8s-rulefiles-0" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-0") pod "41d2256f-9766-4a77-af4c-22d08bd3f3eb" (UID: "41d2256f-9766-4a77-af4c-22d08bd3f3eb"). InnerVolumeSpecName "prometheus-k8s-rulefiles-0". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 19:45:49.914893 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:45:49.914819    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/41d2256f-9766-4a77-af4c-22d08bd3f3eb-configmap-serving-certs-ca-bundle" (OuterVolumeSpecName: "configmap-serving-certs-ca-bundle") pod "41d2256f-9766-4a77-af4c-22d08bd3f3eb" (UID: "41d2256f-9766-4a77-af4c-22d08bd3f3eb"). InnerVolumeSpecName "configmap-serving-certs-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 19:45:49.915284 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:45:49.915234    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/41d2256f-9766-4a77-af4c-22d08bd3f3eb-configmap-metrics-client-ca" (OuterVolumeSpecName: "configmap-metrics-client-ca") pod "41d2256f-9766-4a77-af4c-22d08bd3f3eb" (UID: "41d2256f-9766-4a77-af4c-22d08bd3f3eb"). InnerVolumeSpecName "configmap-metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 19:45:49.915491 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:45:49.915376    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/41d2256f-9766-4a77-af4c-22d08bd3f3eb-prometheus-k8s-rulefiles-2" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-2") pod "41d2256f-9766-4a77-af4c-22d08bd3f3eb" (UID: "41d2256f-9766-4a77-af4c-22d08bd3f3eb"). InnerVolumeSpecName "prometheus-k8s-rulefiles-2". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 19:45:49.916042 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:45:49.915994    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/41d2256f-9766-4a77-af4c-22d08bd3f3eb-prometheus-k8s-rulefiles-1" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-1") pod "41d2256f-9766-4a77-af4c-22d08bd3f3eb" (UID: "41d2256f-9766-4a77-af4c-22d08bd3f3eb"). InnerVolumeSpecName "prometheus-k8s-rulefiles-1". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 19:45:49.916217 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:45:49.916191    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/41d2256f-9766-4a77-af4c-22d08bd3f3eb-configmap-kubelet-serving-ca-bundle" (OuterVolumeSpecName: "configmap-kubelet-serving-ca-bundle") pod "41d2256f-9766-4a77-af4c-22d08bd3f3eb" (UID: "41d2256f-9766-4a77-af4c-22d08bd3f3eb"). InnerVolumeSpecName "configmap-kubelet-serving-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 19:45:49.917100 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:45:49.917064    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/41d2256f-9766-4a77-af4c-22d08bd3f3eb-prometheus-k8s-db" (OuterVolumeSpecName: "prometheus-k8s-db") pod "41d2256f-9766-4a77-af4c-22d08bd3f3eb" (UID: "41d2256f-9766-4a77-af4c-22d08bd3f3eb"). InnerVolumeSpecName "prometheus-k8s-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 19:45:49.917515 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:45:49.917469    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/41d2256f-9766-4a77-af4c-22d08bd3f3eb-secret-metrics-client-certs" (OuterVolumeSpecName: "secret-metrics-client-certs") pod "41d2256f-9766-4a77-af4c-22d08bd3f3eb" (UID: "41d2256f-9766-4a77-af4c-22d08bd3f3eb"). InnerVolumeSpecName "secret-metrics-client-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 19:45:49.917889 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:45:49.917849    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/41d2256f-9766-4a77-af4c-22d08bd3f3eb-thanos-prometheus-http-client-file" (OuterVolumeSpecName: "thanos-prometheus-http-client-file") pod "41d2256f-9766-4a77-af4c-22d08bd3f3eb" (UID: "41d2256f-9766-4a77-af4c-22d08bd3f3eb"). InnerVolumeSpecName "thanos-prometheus-http-client-file". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 19:45:49.918688 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:45:49.918662    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/41d2256f-9766-4a77-af4c-22d08bd3f3eb-config-out" (OuterVolumeSpecName: "config-out") pod "41d2256f-9766-4a77-af4c-22d08bd3f3eb" (UID: "41d2256f-9766-4a77-af4c-22d08bd3f3eb"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 19:45:49.919003 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:45:49.918975    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/41d2256f-9766-4a77-af4c-22d08bd3f3eb-config" (OuterVolumeSpecName: "config") pod "41d2256f-9766-4a77-af4c-22d08bd3f3eb" (UID: "41d2256f-9766-4a77-af4c-22d08bd3f3eb"). InnerVolumeSpecName "config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 19:45:49.919080 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:45:49.919023    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/41d2256f-9766-4a77-af4c-22d08bd3f3eb-kube-api-access-zg6p4" (OuterVolumeSpecName: "kube-api-access-zg6p4") pod "41d2256f-9766-4a77-af4c-22d08bd3f3eb" (UID: "41d2256f-9766-4a77-af4c-22d08bd3f3eb"). InnerVolumeSpecName "kube-api-access-zg6p4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 19:45:49.919233 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:45:49.919107    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/41d2256f-9766-4a77-af4c-22d08bd3f3eb-secret-prometheus-k8s-tls" (OuterVolumeSpecName: "secret-prometheus-k8s-tls") pod "41d2256f-9766-4a77-af4c-22d08bd3f3eb" (UID: "41d2256f-9766-4a77-af4c-22d08bd3f3eb"). InnerVolumeSpecName "secret-prometheus-k8s-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 19:45:49.919613 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:45:49.919569    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/41d2256f-9766-4a77-af4c-22d08bd3f3eb-secret-prometheus-k8s-thanos-sidecar-tls" (OuterVolumeSpecName: "secret-prometheus-k8s-thanos-sidecar-tls") pod "41d2256f-9766-4a77-af4c-22d08bd3f3eb" (UID: "41d2256f-9766-4a77-af4c-22d08bd3f3eb"). InnerVolumeSpecName "secret-prometheus-k8s-thanos-sidecar-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 19:45:49.919903 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:45:49.919879    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/41d2256f-9766-4a77-af4c-22d08bd3f3eb-tls-assets" (OuterVolumeSpecName: "tls-assets") pod "41d2256f-9766-4a77-af4c-22d08bd3f3eb" (UID: "41d2256f-9766-4a77-af4c-22d08bd3f3eb"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 19:45:49.920411 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:45:49.920388    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/41d2256f-9766-4a77-af4c-22d08bd3f3eb-secret-grpc-tls" (OuterVolumeSpecName: "secret-grpc-tls") pod "41d2256f-9766-4a77-af4c-22d08bd3f3eb" (UID: "41d2256f-9766-4a77-af4c-22d08bd3f3eb"). InnerVolumeSpecName "secret-grpc-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 19:45:49.920664 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:45:49.920633    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/41d2256f-9766-4a77-af4c-22d08bd3f3eb-secret-prometheus-k8s-kube-rbac-proxy-web" (OuterVolumeSpecName: "secret-prometheus-k8s-kube-rbac-proxy-web") pod "41d2256f-9766-4a77-af4c-22d08bd3f3eb" (UID: "41d2256f-9766-4a77-af4c-22d08bd3f3eb"). InnerVolumeSpecName "secret-prometheus-k8s-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 19:45:49.921169 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:45:49.921138    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/41d2256f-9766-4a77-af4c-22d08bd3f3eb-secret-kube-rbac-proxy" (OuterVolumeSpecName: "secret-kube-rbac-proxy") pod "41d2256f-9766-4a77-af4c-22d08bd3f3eb" (UID: "41d2256f-9766-4a77-af4c-22d08bd3f3eb"). InnerVolumeSpecName "secret-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 19:45:49.926382 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:45:49.926359    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/41d2256f-9766-4a77-af4c-22d08bd3f3eb-web-config" (OuterVolumeSpecName: "web-config") pod "41d2256f-9766-4a77-af4c-22d08bd3f3eb" (UID: "41d2256f-9766-4a77-af4c-22d08bd3f3eb"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 19:45:50.012203 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:45:50.012166    2585 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/41d2256f-9766-4a77-af4c-22d08bd3f3eb-prometheus-k8s-rulefiles-1\") on node \"ip-10-0-136-185.ec2.internal\" DevicePath \"\""
Sep 18 19:45:50.012203 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:45:50.012197    2585 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/41d2256f-9766-4a77-af4c-22d08bd3f3eb-prometheus-k8s-db\") on node \"ip-10-0-136-185.ec2.internal\" DevicePath \"\""
Sep 18 19:45:50.012203 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:45:50.012207    2585 reconciler_common.go:299] "Volume detached for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/41d2256f-9766-4a77-af4c-22d08bd3f3eb-secret-kube-rbac-proxy\") on node \"ip-10-0-136-185.ec2.internal\" DevicePath \"\""
Sep 18 19:45:50.012434 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:45:50.012218    2585 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/41d2256f-9766-4a77-af4c-22d08bd3f3eb-secret-prometheus-k8s-kube-rbac-proxy-web\") on node \"ip-10-0-136-185.ec2.internal\" DevicePath \"\""
Sep 18 19:45:50.012434 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:45:50.012228    2585 reconciler_common.go:299] "Volume detached for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/41d2256f-9766-4a77-af4c-22d08bd3f3eb-configmap-metrics-client-ca\") on node \"ip-10-0-136-185.ec2.internal\" DevicePath \"\""
Sep 18 19:45:50.012434 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:45:50.012236    2585 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/41d2256f-9766-4a77-af4c-22d08bd3f3eb-prometheus-k8s-rulefiles-2\") on node \"ip-10-0-136-185.ec2.internal\" DevicePath \"\""
Sep 18 19:45:50.012434 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:45:50.012244    2585 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/41d2256f-9766-4a77-af4c-22d08bd3f3eb-config-out\") on node \"ip-10-0-136-185.ec2.internal\" DevicePath \"\""
Sep 18 19:45:50.012434 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:45:50.012253    2585 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/secret/41d2256f-9766-4a77-af4c-22d08bd3f3eb-config\") on node \"ip-10-0-136-185.ec2.internal\" DevicePath \"\""
Sep 18 19:45:50.012434 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:45:50.012260    2585 reconciler_common.go:299] "Volume detached for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/41d2256f-9766-4a77-af4c-22d08bd3f3eb-secret-metrics-client-certs\") on node \"ip-10-0-136-185.ec2.internal\" DevicePath \"\""
Sep 18 19:45:50.012434 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:45:50.012268    2585 reconciler_common.go:299] "Volume detached for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/41d2256f-9766-4a77-af4c-22d08bd3f3eb-secret-grpc-tls\") on node \"ip-10-0-136-185.ec2.internal\" DevicePath \"\""
Sep 18 19:45:50.012434 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:45:50.012276    2585 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/41d2256f-9766-4a77-af4c-22d08bd3f3eb-web-config\") on node \"ip-10-0-136-185.ec2.internal\" DevicePath \"\""
Sep 18 19:45:50.012434 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:45:50.012283    2585 reconciler_common.go:299] "Volume detached for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/41d2256f-9766-4a77-af4c-22d08bd3f3eb-configmap-kubelet-serving-ca-bundle\") on node \"ip-10-0-136-185.ec2.internal\" DevicePath \"\""
Sep 18 19:45:50.012434 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:45:50.012292    2585 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/41d2256f-9766-4a77-af4c-22d08bd3f3eb-secret-prometheus-k8s-thanos-sidecar-tls\") on node \"ip-10-0-136-185.ec2.internal\" DevicePath \"\""
Sep 18 19:45:50.012434 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:45:50.012302    2585 reconciler_common.go:299] "Volume detached for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/41d2256f-9766-4a77-af4c-22d08bd3f3eb-thanos-prometheus-http-client-file\") on node \"ip-10-0-136-185.ec2.internal\" DevicePath \"\""
Sep 18 19:45:50.012434 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:45:50.012311    2585 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zg6p4\" (UniqueName: \"kubernetes.io/projected/41d2256f-9766-4a77-af4c-22d08bd3f3eb-kube-api-access-zg6p4\") on node \"ip-10-0-136-185.ec2.internal\" DevicePath \"\""
Sep 18 19:45:50.012434 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:45:50.012319    2585 reconciler_common.go:299] "Volume detached for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/41d2256f-9766-4a77-af4c-22d08bd3f3eb-configmap-serving-certs-ca-bundle\") on node \"ip-10-0-136-185.ec2.internal\" DevicePath \"\""
Sep 18 19:45:50.012434 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:45:50.012328    2585 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/41d2256f-9766-4a77-af4c-22d08bd3f3eb-prometheus-k8s-rulefiles-0\") on node \"ip-10-0-136-185.ec2.internal\" DevicePath \"\""
Sep 18 19:45:50.012434 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:45:50.012338    2585 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/41d2256f-9766-4a77-af4c-22d08bd3f3eb-secret-prometheus-k8s-tls\") on node \"ip-10-0-136-185.ec2.internal\" DevicePath \"\""
Sep 18 19:45:50.012434 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:45:50.012347    2585 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/41d2256f-9766-4a77-af4c-22d08bd3f3eb-tls-assets\") on node \"ip-10-0-136-185.ec2.internal\" DevicePath \"\""
Sep 18 19:45:50.056718 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:45:50.056685    2585 generic.go:356] "Generic (PLEG): container finished" podID="41d2256f-9766-4a77-af4c-22d08bd3f3eb" containerID="bd672fd65b3cc3916a3fbc16c82ee83e25b934c9c3d8362f16ce8c12b6f1286d" exitCode=0
Sep 18 19:45:50.056871 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:45:50.056774    2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 19:45:50.056871 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:45:50.056768    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"41d2256f-9766-4a77-af4c-22d08bd3f3eb","Type":"ContainerDied","Data":"bd672fd65b3cc3916a3fbc16c82ee83e25b934c9c3d8362f16ce8c12b6f1286d"}
Sep 18 19:45:50.056871 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:45:50.056852    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"41d2256f-9766-4a77-af4c-22d08bd3f3eb","Type":"ContainerDied","Data":"86fa1a7cd6208e1d29ba9c0628d6c17b9c540549e6eccc5bf457bc2374e974a8"}
Sep 18 19:45:50.056973 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:45:50.056884    2585 scope.go:117] "RemoveContainer" containerID="f3b4855a741ffa1c79b46340a8d87cfc5136b9594cd653e43204b65c372212db"
Sep 18 19:45:50.064707 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:45:50.064549    2585 scope.go:117] "RemoveContainer" containerID="87c91cb8554a54434d8ed7e7ca7d9749cdbce91c86a3ffa78fb541c442c16e56"
Sep 18 19:45:50.072339 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:45:50.072321    2585 scope.go:117] "RemoveContainer" containerID="bd672fd65b3cc3916a3fbc16c82ee83e25b934c9c3d8362f16ce8c12b6f1286d"
Sep 18 19:45:50.079101 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:45:50.079051    2585 scope.go:117] "RemoveContainer" containerID="28389f299be1d3204aec9230b0766a75ee721d436d24ae7d37bf7477c9bd0a42"
Sep 18 19:45:50.081074 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:45:50.081051    2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 18 19:45:50.086765 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:45:50.086745    2585 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 18 19:45:50.086838 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:45:50.086781    2585 scope.go:117] "RemoveContainer" containerID="a4335bc302a43d97f61ffeef2a6daa7c085b6081ae8f5f86d8e6454f37a20cde"
Sep 18 19:45:50.093372 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:45:50.093352    2585 scope.go:117] "RemoveContainer" containerID="c964b1efea9fdec097ed230ae8ec386223ef81b1f2e5307e33def9ddc2d1e336"
Sep 18 19:45:50.100236 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:45:50.100221    2585 scope.go:117] "RemoveContainer" containerID="b2c744a360d6f3c074c55c47bbd633c70e7f10efa7780b29ee14e9221c9654b4"
Sep 18 19:45:50.106668 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:45:50.106654    2585 scope.go:117] "RemoveContainer" containerID="f3b4855a741ffa1c79b46340a8d87cfc5136b9594cd653e43204b65c372212db"
Sep 18 19:45:50.106940 ip-10-0-136-185 kubenswrapper[2585]: E0918 19:45:50.106923    2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f3b4855a741ffa1c79b46340a8d87cfc5136b9594cd653e43204b65c372212db\": container with ID starting with f3b4855a741ffa1c79b46340a8d87cfc5136b9594cd653e43204b65c372212db not found: ID does not exist" containerID="f3b4855a741ffa1c79b46340a8d87cfc5136b9594cd653e43204b65c372212db"
Sep 18 19:45:50.106982 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:45:50.106950    2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f3b4855a741ffa1c79b46340a8d87cfc5136b9594cd653e43204b65c372212db"} err="failed to get container status \"f3b4855a741ffa1c79b46340a8d87cfc5136b9594cd653e43204b65c372212db\": rpc error: code = NotFound desc = could not find container \"f3b4855a741ffa1c79b46340a8d87cfc5136b9594cd653e43204b65c372212db\": container with ID starting with f3b4855a741ffa1c79b46340a8d87cfc5136b9594cd653e43204b65c372212db not found: ID does not exist"
Sep 18 19:45:50.106982 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:45:50.106967    2585 scope.go:117] "RemoveContainer" containerID="87c91cb8554a54434d8ed7e7ca7d9749cdbce91c86a3ffa78fb541c442c16e56"
Sep 18 19:45:50.107216 ip-10-0-136-185 kubenswrapper[2585]: E0918 19:45:50.107196    2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"87c91cb8554a54434d8ed7e7ca7d9749cdbce91c86a3ffa78fb541c442c16e56\": container with ID starting with 87c91cb8554a54434d8ed7e7ca7d9749cdbce91c86a3ffa78fb541c442c16e56 not found: ID does not exist" containerID="87c91cb8554a54434d8ed7e7ca7d9749cdbce91c86a3ffa78fb541c442c16e56"
Sep 18 19:45:50.107257 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:45:50.107223    2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"87c91cb8554a54434d8ed7e7ca7d9749cdbce91c86a3ffa78fb541c442c16e56"} err="failed to get container status \"87c91cb8554a54434d8ed7e7ca7d9749cdbce91c86a3ffa78fb541c442c16e56\": rpc error: code = NotFound desc = could not find container \"87c91cb8554a54434d8ed7e7ca7d9749cdbce91c86a3ffa78fb541c442c16e56\": container with ID starting with 87c91cb8554a54434d8ed7e7ca7d9749cdbce91c86a3ffa78fb541c442c16e56 not found: ID does not exist"
Sep 18 19:45:50.107257 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:45:50.107241    2585 scope.go:117] "RemoveContainer" containerID="bd672fd65b3cc3916a3fbc16c82ee83e25b934c9c3d8362f16ce8c12b6f1286d"
Sep 18 19:45:50.107490 ip-10-0-136-185 kubenswrapper[2585]: E0918 19:45:50.107473    2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bd672fd65b3cc3916a3fbc16c82ee83e25b934c9c3d8362f16ce8c12b6f1286d\": container with ID starting with bd672fd65b3cc3916a3fbc16c82ee83e25b934c9c3d8362f16ce8c12b6f1286d not found: ID does not exist" containerID="bd672fd65b3cc3916a3fbc16c82ee83e25b934c9c3d8362f16ce8c12b6f1286d"
Sep 18 19:45:50.107528 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:45:50.107497    2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bd672fd65b3cc3916a3fbc16c82ee83e25b934c9c3d8362f16ce8c12b6f1286d"} err="failed to get container status \"bd672fd65b3cc3916a3fbc16c82ee83e25b934c9c3d8362f16ce8c12b6f1286d\": rpc error: code = NotFound desc = could not find container \"bd672fd65b3cc3916a3fbc16c82ee83e25b934c9c3d8362f16ce8c12b6f1286d\": container with ID starting with bd672fd65b3cc3916a3fbc16c82ee83e25b934c9c3d8362f16ce8c12b6f1286d not found: ID does not exist"
Sep 18 19:45:50.107528 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:45:50.107513    2585 scope.go:117] "RemoveContainer" containerID="28389f299be1d3204aec9230b0766a75ee721d436d24ae7d37bf7477c9bd0a42"
Sep 18 19:45:50.107749 ip-10-0-136-185 kubenswrapper[2585]: E0918 19:45:50.107733    2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"28389f299be1d3204aec9230b0766a75ee721d436d24ae7d37bf7477c9bd0a42\": container with ID starting with 28389f299be1d3204aec9230b0766a75ee721d436d24ae7d37bf7477c9bd0a42 not found: ID does not exist" containerID="28389f299be1d3204aec9230b0766a75ee721d436d24ae7d37bf7477c9bd0a42"
Sep 18 19:45:50.107805 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:45:50.107756    2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"28389f299be1d3204aec9230b0766a75ee721d436d24ae7d37bf7477c9bd0a42"} err="failed to get container status \"28389f299be1d3204aec9230b0766a75ee721d436d24ae7d37bf7477c9bd0a42\": rpc error: code = NotFound desc = could not find container \"28389f299be1d3204aec9230b0766a75ee721d436d24ae7d37bf7477c9bd0a42\": container with ID starting with 28389f299be1d3204aec9230b0766a75ee721d436d24ae7d37bf7477c9bd0a42 not found: ID does not exist"
Sep 18 19:45:50.107805 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:45:50.107771    2585 scope.go:117] "RemoveContainer" containerID="a4335bc302a43d97f61ffeef2a6daa7c085b6081ae8f5f86d8e6454f37a20cde"
Sep 18 19:45:50.108059 ip-10-0-136-185 kubenswrapper[2585]: E0918 19:45:50.108043    2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a4335bc302a43d97f61ffeef2a6daa7c085b6081ae8f5f86d8e6454f37a20cde\": container with ID starting with a4335bc302a43d97f61ffeef2a6daa7c085b6081ae8f5f86d8e6454f37a20cde not found: ID does not exist" containerID="a4335bc302a43d97f61ffeef2a6daa7c085b6081ae8f5f86d8e6454f37a20cde"
Sep 18 19:45:50.108109 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:45:50.108064    2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a4335bc302a43d97f61ffeef2a6daa7c085b6081ae8f5f86d8e6454f37a20cde"} err="failed to get container status \"a4335bc302a43d97f61ffeef2a6daa7c085b6081ae8f5f86d8e6454f37a20cde\": rpc error: code = NotFound desc = could not find container \"a4335bc302a43d97f61ffeef2a6daa7c085b6081ae8f5f86d8e6454f37a20cde\": container with ID starting with a4335bc302a43d97f61ffeef2a6daa7c085b6081ae8f5f86d8e6454f37a20cde not found: ID does not exist"
Sep 18 19:45:50.108109 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:45:50.108077    2585 scope.go:117] "RemoveContainer" containerID="c964b1efea9fdec097ed230ae8ec386223ef81b1f2e5307e33def9ddc2d1e336"
Sep 18 19:45:50.108308 ip-10-0-136-185 kubenswrapper[2585]: E0918 19:45:50.108293    2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c964b1efea9fdec097ed230ae8ec386223ef81b1f2e5307e33def9ddc2d1e336\": container with ID starting with c964b1efea9fdec097ed230ae8ec386223ef81b1f2e5307e33def9ddc2d1e336 not found: ID does not exist" containerID="c964b1efea9fdec097ed230ae8ec386223ef81b1f2e5307e33def9ddc2d1e336"
Sep 18 19:45:50.108350 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:45:50.108313    2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c964b1efea9fdec097ed230ae8ec386223ef81b1f2e5307e33def9ddc2d1e336"} err="failed to get container status \"c964b1efea9fdec097ed230ae8ec386223ef81b1f2e5307e33def9ddc2d1e336\": rpc error: code = NotFound desc = could not find container \"c964b1efea9fdec097ed230ae8ec386223ef81b1f2e5307e33def9ddc2d1e336\": container with ID starting with c964b1efea9fdec097ed230ae8ec386223ef81b1f2e5307e33def9ddc2d1e336 not found: ID does not exist"
Sep 18 19:45:50.108350 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:45:50.108328    2585 scope.go:117] "RemoveContainer" containerID="b2c744a360d6f3c074c55c47bbd633c70e7f10efa7780b29ee14e9221c9654b4"
Sep 18 19:45:50.108545 ip-10-0-136-185 kubenswrapper[2585]: E0918 19:45:50.108530    2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b2c744a360d6f3c074c55c47bbd633c70e7f10efa7780b29ee14e9221c9654b4\": container with ID starting with b2c744a360d6f3c074c55c47bbd633c70e7f10efa7780b29ee14e9221c9654b4 not found: ID does not exist" containerID="b2c744a360d6f3c074c55c47bbd633c70e7f10efa7780b29ee14e9221c9654b4"
Sep 18 19:45:50.108584 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:45:50.108548    2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b2c744a360d6f3c074c55c47bbd633c70e7f10efa7780b29ee14e9221c9654b4"} err="failed to get container status \"b2c744a360d6f3c074c55c47bbd633c70e7f10efa7780b29ee14e9221c9654b4\": rpc error: code = NotFound desc = could not find container \"b2c744a360d6f3c074c55c47bbd633c70e7f10efa7780b29ee14e9221c9654b4\": container with ID starting with b2c744a360d6f3c074c55c47bbd633c70e7f10efa7780b29ee14e9221c9654b4 not found: ID does not exist"
Sep 18 19:45:50.113904 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:45:50.113885    2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 18 19:45:50.114151 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:45:50.114139    2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="41d2256f-9766-4a77-af4c-22d08bd3f3eb" containerName="kube-rbac-proxy-thanos"
Sep 18 19:45:50.114188 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:45:50.114154    2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="41d2256f-9766-4a77-af4c-22d08bd3f3eb" containerName="kube-rbac-proxy-thanos"
Sep 18 19:45:50.114188 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:45:50.114165    2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="41d2256f-9766-4a77-af4c-22d08bd3f3eb" containerName="prometheus"
Sep 18 19:45:50.114188 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:45:50.114171    2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="41d2256f-9766-4a77-af4c-22d08bd3f3eb" containerName="prometheus"
Sep 18 19:45:50.114188 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:45:50.114178    2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="41d2256f-9766-4a77-af4c-22d08bd3f3eb" containerName="kube-rbac-proxy-web"
Sep 18 19:45:50.114188 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:45:50.114183    2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="41d2256f-9766-4a77-af4c-22d08bd3f3eb" containerName="kube-rbac-proxy-web"
Sep 18 19:45:50.114333 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:45:50.114193    2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="41d2256f-9766-4a77-af4c-22d08bd3f3eb" containerName="init-config-reloader"
Sep 18 19:45:50.114333 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:45:50.114198    2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="41d2256f-9766-4a77-af4c-22d08bd3f3eb" containerName="init-config-reloader"
Sep 18 19:45:50.114333 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:45:50.114205    2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="41d2256f-9766-4a77-af4c-22d08bd3f3eb" containerName="kube-rbac-proxy"
Sep 18 19:45:50.114333 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:45:50.114210    2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="41d2256f-9766-4a77-af4c-22d08bd3f3eb" containerName="kube-rbac-proxy"
Sep 18 19:45:50.114333 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:45:50.114218    2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="41d2256f-9766-4a77-af4c-22d08bd3f3eb" containerName="config-reloader"
Sep 18 19:45:50.114333 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:45:50.114224    2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="41d2256f-9766-4a77-af4c-22d08bd3f3eb" containerName="config-reloader"
Sep 18 19:45:50.114333 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:45:50.114232    2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="79747afc-194b-4ded-bdd1-d6332f8412b7" containerName="registry"
Sep 18 19:45:50.114333 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:45:50.114238    2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="79747afc-194b-4ded-bdd1-d6332f8412b7" containerName="registry"
Sep 18 19:45:50.114333 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:45:50.114246    2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="41d2256f-9766-4a77-af4c-22d08bd3f3eb" containerName="thanos-sidecar"
Sep 18 19:45:50.114333 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:45:50.114251    2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="41d2256f-9766-4a77-af4c-22d08bd3f3eb" containerName="thanos-sidecar"
Sep 18 19:45:50.114333 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:45:50.114293    2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="41d2256f-9766-4a77-af4c-22d08bd3f3eb" containerName="prometheus"
Sep 18 19:45:50.114333 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:45:50.114300    2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="41d2256f-9766-4a77-af4c-22d08bd3f3eb" containerName="kube-rbac-proxy-web"
Sep 18 19:45:50.114333 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:45:50.114307    2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="79747afc-194b-4ded-bdd1-d6332f8412b7" containerName="registry"
Sep 18 19:45:50.114333 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:45:50.114312    2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="41d2256f-9766-4a77-af4c-22d08bd3f3eb" containerName="config-reloader"
Sep 18 19:45:50.114333 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:45:50.114318    2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="41d2256f-9766-4a77-af4c-22d08bd3f3eb" containerName="thanos-sidecar"
Sep 18 19:45:50.114333 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:45:50.114323    2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="41d2256f-9766-4a77-af4c-22d08bd3f3eb" containerName="kube-rbac-proxy-thanos"
Sep 18 19:45:50.114333 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:45:50.114328    2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="41d2256f-9766-4a77-af4c-22d08bd3f3eb" containerName="kube-rbac-proxy"
Sep 18 19:45:50.117774 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:45:50.117760    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 19:45:50.120229 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:45:50.120209    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\""
Sep 18 19:45:50.120436 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:45:50.120420    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\""
Sep 18 19:45:50.120508 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:45:50.120491    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\""
Sep 18 19:45:50.120567 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:45:50.120510    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\""
Sep 18 19:45:50.120619 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:45:50.120591    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\""
Sep 18 19:45:50.120835 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:45:50.120803    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\""
Sep 18 19:45:50.120946 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:45:50.120866    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\""
Sep 18 19:45:50.120946 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:45:50.120933    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\""
Sep 18 19:45:50.121041 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:45:50.120943    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\""
Sep 18 19:45:50.121171 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:45:50.121158    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\""
Sep 18 19:45:50.121228 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:45:50.121214    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-3cv80vk60q20o\""
Sep 18 19:45:50.121272 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:45:50.121224    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\""
Sep 18 19:45:50.121333 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:45:50.121312    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\""
Sep 18 19:45:50.121375 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:45:50.121334    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\""
Sep 18 19:45:50.121630 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:45:50.121617    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-htqkl\""
Sep 18 19:45:50.128390 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:45:50.128365    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\""
Sep 18 19:45:50.131155 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:45:50.131135    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\""
Sep 18 19:45:50.136396 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:45:50.136376    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 18 19:45:50.214104 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:45:50.214029    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/a1205f9f-d121-45f4-baf3-b56d70ccb24b-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"a1205f9f-d121-45f4-baf3-b56d70ccb24b\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 19:45:50.214104 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:45:50.214070    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/a1205f9f-d121-45f4-baf3-b56d70ccb24b-web-config\") pod \"prometheus-k8s-0\" (UID: \"a1205f9f-d121-45f4-baf3-b56d70ccb24b\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 19:45:50.214104 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:45:50.214096    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/a1205f9f-d121-45f4-baf3-b56d70ccb24b-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"a1205f9f-d121-45f4-baf3-b56d70ccb24b\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 19:45:50.214306 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:45:50.214122    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/a1205f9f-d121-45f4-baf3-b56d70ccb24b-config\") pod \"prometheus-k8s-0\" (UID: \"a1205f9f-d121-45f4-baf3-b56d70ccb24b\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 19:45:50.214306 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:45:50.214181    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a1205f9f-d121-45f4-baf3-b56d70ccb24b-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"a1205f9f-d121-45f4-baf3-b56d70ccb24b\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 19:45:50.214306 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:45:50.214211    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a1205f9f-d121-45f4-baf3-b56d70ccb24b-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"a1205f9f-d121-45f4-baf3-b56d70ccb24b\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 19:45:50.214306 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:45:50.214244    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/a1205f9f-d121-45f4-baf3-b56d70ccb24b-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"a1205f9f-d121-45f4-baf3-b56d70ccb24b\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 19:45:50.214306 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:45:50.214261    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/a1205f9f-d121-45f4-baf3-b56d70ccb24b-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"a1205f9f-d121-45f4-baf3-b56d70ccb24b\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 19:45:50.214306 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:45:50.214274    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/a1205f9f-d121-45f4-baf3-b56d70ccb24b-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"a1205f9f-d121-45f4-baf3-b56d70ccb24b\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 19:45:50.214306 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:45:50.214292    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/a1205f9f-d121-45f4-baf3-b56d70ccb24b-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"a1205f9f-d121-45f4-baf3-b56d70ccb24b\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 19:45:50.214306 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:45:50.214307    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/a1205f9f-d121-45f4-baf3-b56d70ccb24b-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"a1205f9f-d121-45f4-baf3-b56d70ccb24b\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 19:45:50.214534 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:45:50.214357    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/a1205f9f-d121-45f4-baf3-b56d70ccb24b-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"a1205f9f-d121-45f4-baf3-b56d70ccb24b\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 19:45:50.214534 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:45:50.214401    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/a1205f9f-d121-45f4-baf3-b56d70ccb24b-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"a1205f9f-d121-45f4-baf3-b56d70ccb24b\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 19:45:50.214534 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:45:50.214421    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a1205f9f-d121-45f4-baf3-b56d70ccb24b-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"a1205f9f-d121-45f4-baf3-b56d70ccb24b\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 19:45:50.214534 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:45:50.214467    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/a1205f9f-d121-45f4-baf3-b56d70ccb24b-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"a1205f9f-d121-45f4-baf3-b56d70ccb24b\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 19:45:50.214534 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:45:50.214486    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/a1205f9f-d121-45f4-baf3-b56d70ccb24b-config-out\") pod \"prometheus-k8s-0\" (UID: \"a1205f9f-d121-45f4-baf3-b56d70ccb24b\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 19:45:50.214534 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:45:50.214503    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/a1205f9f-d121-45f4-baf3-b56d70ccb24b-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"a1205f9f-d121-45f4-baf3-b56d70ccb24b\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 19:45:50.214534 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:45:50.214521    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qg8sd\" (UniqueName: \"kubernetes.io/projected/a1205f9f-d121-45f4-baf3-b56d70ccb24b-kube-api-access-qg8sd\") pod \"prometheus-k8s-0\" (UID: \"a1205f9f-d121-45f4-baf3-b56d70ccb24b\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 19:45:50.214777 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:45:50.214572    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/a1205f9f-d121-45f4-baf3-b56d70ccb24b-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"a1205f9f-d121-45f4-baf3-b56d70ccb24b\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 19:45:50.214777 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:45:50.214590    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a1205f9f-d121-45f4-baf3-b56d70ccb24b-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"a1205f9f-d121-45f4-baf3-b56d70ccb24b\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 19:45:50.315322 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:45:50.315267    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/a1205f9f-d121-45f4-baf3-b56d70ccb24b-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"a1205f9f-d121-45f4-baf3-b56d70ccb24b\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 19:45:50.315495 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:45:50.315336    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/a1205f9f-d121-45f4-baf3-b56d70ccb24b-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"a1205f9f-d121-45f4-baf3-b56d70ccb24b\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 19:45:50.315495 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:45:50.315395    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/a1205f9f-d121-45f4-baf3-b56d70ccb24b-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"a1205f9f-d121-45f4-baf3-b56d70ccb24b\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 19:45:50.315495 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:45:50.315429    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/a1205f9f-d121-45f4-baf3-b56d70ccb24b-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"a1205f9f-d121-45f4-baf3-b56d70ccb24b\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 19:45:50.315495 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:45:50.315450    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/a1205f9f-d121-45f4-baf3-b56d70ccb24b-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"a1205f9f-d121-45f4-baf3-b56d70ccb24b\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 19:45:50.315707 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:45:50.315511    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a1205f9f-d121-45f4-baf3-b56d70ccb24b-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"a1205f9f-d121-45f4-baf3-b56d70ccb24b\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 19:45:50.315707 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:45:50.315557    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/a1205f9f-d121-45f4-baf3-b56d70ccb24b-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"a1205f9f-d121-45f4-baf3-b56d70ccb24b\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 19:45:50.315707 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:45:50.315586    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/a1205f9f-d121-45f4-baf3-b56d70ccb24b-config-out\") pod \"prometheus-k8s-0\" (UID: \"a1205f9f-d121-45f4-baf3-b56d70ccb24b\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 19:45:50.315707 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:45:50.315604    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/a1205f9f-d121-45f4-baf3-b56d70ccb24b-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"a1205f9f-d121-45f4-baf3-b56d70ccb24b\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 19:45:50.315707 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:45:50.315628    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qg8sd\" (UniqueName: \"kubernetes.io/projected/a1205f9f-d121-45f4-baf3-b56d70ccb24b-kube-api-access-qg8sd\") pod \"prometheus-k8s-0\" (UID: \"a1205f9f-d121-45f4-baf3-b56d70ccb24b\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 19:45:50.315707 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:45:50.315664    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/a1205f9f-d121-45f4-baf3-b56d70ccb24b-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"a1205f9f-d121-45f4-baf3-b56d70ccb24b\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 19:45:50.315707 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:45:50.315686    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a1205f9f-d121-45f4-baf3-b56d70ccb24b-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"a1205f9f-d121-45f4-baf3-b56d70ccb24b\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 19:45:50.316065 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:45:50.315712    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/a1205f9f-d121-45f4-baf3-b56d70ccb24b-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"a1205f9f-d121-45f4-baf3-b56d70ccb24b\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 19:45:50.316065 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:45:50.315758    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/a1205f9f-d121-45f4-baf3-b56d70ccb24b-web-config\") pod \"prometheus-k8s-0\" (UID: \"a1205f9f-d121-45f4-baf3-b56d70ccb24b\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 19:45:50.316065 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:45:50.315809    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/a1205f9f-d121-45f4-baf3-b56d70ccb24b-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"a1205f9f-d121-45f4-baf3-b56d70ccb24b\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 19:45:50.316065 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:45:50.315837    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/a1205f9f-d121-45f4-baf3-b56d70ccb24b-config\") pod \"prometheus-k8s-0\" (UID: \"a1205f9f-d121-45f4-baf3-b56d70ccb24b\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 19:45:50.316065 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:45:50.315870    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a1205f9f-d121-45f4-baf3-b56d70ccb24b-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"a1205f9f-d121-45f4-baf3-b56d70ccb24b\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 19:45:50.316065 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:45:50.315898    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a1205f9f-d121-45f4-baf3-b56d70ccb24b-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"a1205f9f-d121-45f4-baf3-b56d70ccb24b\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 19:45:50.316065 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:45:50.315951    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/a1205f9f-d121-45f4-baf3-b56d70ccb24b-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"a1205f9f-d121-45f4-baf3-b56d70ccb24b\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 19:45:50.316065 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:45:50.315977    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/a1205f9f-d121-45f4-baf3-b56d70ccb24b-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"a1205f9f-d121-45f4-baf3-b56d70ccb24b\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 19:45:50.316460 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:45:50.316150    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/a1205f9f-d121-45f4-baf3-b56d70ccb24b-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"a1205f9f-d121-45f4-baf3-b56d70ccb24b\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 19:45:50.316460 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:45:50.316335    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a1205f9f-d121-45f4-baf3-b56d70ccb24b-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"a1205f9f-d121-45f4-baf3-b56d70ccb24b\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 19:45:50.319113 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:45:50.318080    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a1205f9f-d121-45f4-baf3-b56d70ccb24b-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"a1205f9f-d121-45f4-baf3-b56d70ccb24b\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 19:45:50.319113 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:45:50.318249    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/a1205f9f-d121-45f4-baf3-b56d70ccb24b-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"a1205f9f-d121-45f4-baf3-b56d70ccb24b\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 19:45:50.319113 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:45:50.318321    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a1205f9f-d121-45f4-baf3-b56d70ccb24b-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"a1205f9f-d121-45f4-baf3-b56d70ccb24b\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 19:45:50.319113 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:45:50.318586    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/a1205f9f-d121-45f4-baf3-b56d70ccb24b-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"a1205f9f-d121-45f4-baf3-b56d70ccb24b\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 19:45:50.319113 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:45:50.318627    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/a1205f9f-d121-45f4-baf3-b56d70ccb24b-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"a1205f9f-d121-45f4-baf3-b56d70ccb24b\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 19:45:50.319113 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:45:50.318685    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a1205f9f-d121-45f4-baf3-b56d70ccb24b-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"a1205f9f-d121-45f4-baf3-b56d70ccb24b\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 19:45:50.319113 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:45:50.319049    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/a1205f9f-d121-45f4-baf3-b56d70ccb24b-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"a1205f9f-d121-45f4-baf3-b56d70ccb24b\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 19:45:50.319113 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:45:50.319051    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/a1205f9f-d121-45f4-baf3-b56d70ccb24b-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"a1205f9f-d121-45f4-baf3-b56d70ccb24b\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 19:45:50.319113 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:45:50.319096    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/a1205f9f-d121-45f4-baf3-b56d70ccb24b-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"a1205f9f-d121-45f4-baf3-b56d70ccb24b\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 19:45:50.319661 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:45:50.319618    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/a1205f9f-d121-45f4-baf3-b56d70ccb24b-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"a1205f9f-d121-45f4-baf3-b56d70ccb24b\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 19:45:50.319980 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:45:50.319938    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/a1205f9f-d121-45f4-baf3-b56d70ccb24b-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"a1205f9f-d121-45f4-baf3-b56d70ccb24b\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 19:45:50.320167 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:45:50.320145    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/a1205f9f-d121-45f4-baf3-b56d70ccb24b-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"a1205f9f-d121-45f4-baf3-b56d70ccb24b\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 19:45:50.320492 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:45:50.320470    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/a1205f9f-d121-45f4-baf3-b56d70ccb24b-config\") pod \"prometheus-k8s-0\" (UID: \"a1205f9f-d121-45f4-baf3-b56d70ccb24b\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 19:45:50.320599 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:45:50.320580    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/a1205f9f-d121-45f4-baf3-b56d70ccb24b-config-out\") pod \"prometheus-k8s-0\" (UID: \"a1205f9f-d121-45f4-baf3-b56d70ccb24b\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 19:45:50.321008 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:45:50.320988    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/a1205f9f-d121-45f4-baf3-b56d70ccb24b-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"a1205f9f-d121-45f4-baf3-b56d70ccb24b\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 19:45:50.321337 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:45:50.321309    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/a1205f9f-d121-45f4-baf3-b56d70ccb24b-web-config\") pod \"prometheus-k8s-0\" (UID: \"a1205f9f-d121-45f4-baf3-b56d70ccb24b\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 19:45:50.321666 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:45:50.321647    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/a1205f9f-d121-45f4-baf3-b56d70ccb24b-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"a1205f9f-d121-45f4-baf3-b56d70ccb24b\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 19:45:50.324834 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:45:50.324767    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qg8sd\" (UniqueName: \"kubernetes.io/projected/a1205f9f-d121-45f4-baf3-b56d70ccb24b-kube-api-access-qg8sd\") pod \"prometheus-k8s-0\" (UID: \"a1205f9f-d121-45f4-baf3-b56d70ccb24b\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 19:45:50.431539 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:45:50.431492    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 19:45:50.662017 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:45:50.661983    2585 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="41d2256f-9766-4a77-af4c-22d08bd3f3eb" path="/var/lib/kubelet/pods/41d2256f-9766-4a77-af4c-22d08bd3f3eb/volumes"
Sep 18 19:45:50.711495 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:45:50.711461    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 18 19:45:50.713847 ip-10-0-136-185 kubenswrapper[2585]: W0918 19:45:50.713820    2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda1205f9f_d121_45f4_baf3_b56d70ccb24b.slice/crio-0640cdb8658f0e5eaebd0cb9d6fe5fb98f236976f42305b2a4076135e83a7f2a WatchSource:0}: Error finding container 0640cdb8658f0e5eaebd0cb9d6fe5fb98f236976f42305b2a4076135e83a7f2a: Status 404 returned error can't find the container with id 0640cdb8658f0e5eaebd0cb9d6fe5fb98f236976f42305b2a4076135e83a7f2a
Sep 18 19:45:51.063317 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:45:51.063231    2585 generic.go:356] "Generic (PLEG): container finished" podID="a1205f9f-d121-45f4-baf3-b56d70ccb24b" containerID="a8878e199ca4c218a0a48923ba27fb9cfba3aae1b7636b1d3fa2cb7e00391b17" exitCode=0
Sep 18 19:45:51.063693 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:45:51.063312    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"a1205f9f-d121-45f4-baf3-b56d70ccb24b","Type":"ContainerDied","Data":"a8878e199ca4c218a0a48923ba27fb9cfba3aae1b7636b1d3fa2cb7e00391b17"}
Sep 18 19:45:51.063693 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:45:51.063380    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"a1205f9f-d121-45f4-baf3-b56d70ccb24b","Type":"ContainerStarted","Data":"0640cdb8658f0e5eaebd0cb9d6fe5fb98f236976f42305b2a4076135e83a7f2a"}
Sep 18 19:45:52.071105 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:45:52.071012    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"a1205f9f-d121-45f4-baf3-b56d70ccb24b","Type":"ContainerStarted","Data":"26379b924fcf275ac86b803516409a41a1606172f74702e0cd429930834bb352"}
Sep 18 19:45:52.071105 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:45:52.071089    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"a1205f9f-d121-45f4-baf3-b56d70ccb24b","Type":"ContainerStarted","Data":"e1106de0a30b14643024959f0197837b53dea1dcc9ca544c6bb0a9324a46a07e"}
Sep 18 19:45:52.071626 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:45:52.071128    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"a1205f9f-d121-45f4-baf3-b56d70ccb24b","Type":"ContainerStarted","Data":"6d3f23c2a82a355ac35988dfd2d602a6a93ef203f7f57d42339fe4397d622b4a"}
Sep 18 19:45:52.071626 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:45:52.071169    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"a1205f9f-d121-45f4-baf3-b56d70ccb24b","Type":"ContainerStarted","Data":"2726ac83790b33bec60e860e14aa620798944bdc6b3fc3ab59e79b355bd00335"}
Sep 18 19:45:52.071626 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:45:52.071207    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"a1205f9f-d121-45f4-baf3-b56d70ccb24b","Type":"ContainerStarted","Data":"21e4e38380e460ec276dcef17d7550ee9c4c7572cc6d9f0004ba89c872963222"}
Sep 18 19:45:52.071626 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:45:52.071284    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"a1205f9f-d121-45f4-baf3-b56d70ccb24b","Type":"ContainerStarted","Data":"d5dec0c97bc75fbfa30a7ebd7ee20b8a89bcd76304c21407d6ace8c860b20d83"}
Sep 18 19:45:52.101559 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:45:52.101501    2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=2.078581488 podStartE2EDuration="2.10148526s" totalImagesPullingTime="0s" totalInitContainerRuntime="22.903772ms" isStatefulPod=false podCreationTimestamp="2026-09-18 19:45:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 19:45:52.100833628 +0000 UTC m=+393.625439051" watchObservedRunningTime="2026-09-18 19:45:52.10148526 +0000 UTC m=+393.626090680"
Sep 18 19:45:55.431646 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:45:55.431610    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 19:46:50.432677 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:46:50.432581    2585 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 19:46:50.448209 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:46:50.448183    2585 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 19:46:51.313754 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:46:51.313727    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 19:47:16.950523 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:47:16.950482    2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-bd4fcf9c7-qlrdn"]
Sep 18 19:47:16.953915 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:47:16.953891    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-bd4fcf9c7-qlrdn"
Sep 18 19:47:16.956827 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:47:16.956782    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\""
Sep 18 19:47:16.957001 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:47:16.956874    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\""
Sep 18 19:47:16.957181 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:47:16.957158    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\""
Sep 18 19:47:16.957270 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:47:16.957258    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-bdx4v\""
Sep 18 19:47:16.957443 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:47:16.957426    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\""
Sep 18 19:47:16.984750 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:47:16.984711    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-bd4fcf9c7-qlrdn"]
Sep 18 19:47:16.994582 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:47:16.994552    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/66b774a3-01a1-4190-979b-1a956d8b5049-tmp\") pod \"opendatahub-operator-controller-manager-bd4fcf9c7-qlrdn\" (UID: \"66b774a3-01a1-4190-979b-1a956d8b5049\") " pod="opendatahub/opendatahub-operator-controller-manager-bd4fcf9c7-qlrdn"
Sep 18 19:47:16.994729 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:47:16.994595    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/66b774a3-01a1-4190-979b-1a956d8b5049-manifests\") pod \"opendatahub-operator-controller-manager-bd4fcf9c7-qlrdn\" (UID: \"66b774a3-01a1-4190-979b-1a956d8b5049\") " pod="opendatahub/opendatahub-operator-controller-manager-bd4fcf9c7-qlrdn"
Sep 18 19:47:16.994729 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:47:16.994632    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/66b774a3-01a1-4190-979b-1a956d8b5049-webhook-cert\") pod \"opendatahub-operator-controller-manager-bd4fcf9c7-qlrdn\" (UID: \"66b774a3-01a1-4190-979b-1a956d8b5049\") " pod="opendatahub/opendatahub-operator-controller-manager-bd4fcf9c7-qlrdn"
Sep 18 19:47:16.994729 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:47:16.994653    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b6ld7\" (UniqueName: \"kubernetes.io/projected/66b774a3-01a1-4190-979b-1a956d8b5049-kube-api-access-b6ld7\") pod \"opendatahub-operator-controller-manager-bd4fcf9c7-qlrdn\" (UID: \"66b774a3-01a1-4190-979b-1a956d8b5049\") " pod="opendatahub/opendatahub-operator-controller-manager-bd4fcf9c7-qlrdn"
Sep 18 19:47:16.994729 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:47:16.994672    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/66b774a3-01a1-4190-979b-1a956d8b5049-apiservice-cert\") pod \"opendatahub-operator-controller-manager-bd4fcf9c7-qlrdn\" (UID: \"66b774a3-01a1-4190-979b-1a956d8b5049\") " pod="opendatahub/opendatahub-operator-controller-manager-bd4fcf9c7-qlrdn"
Sep 18 19:47:17.095738 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:47:17.095695    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/66b774a3-01a1-4190-979b-1a956d8b5049-tmp\") pod \"opendatahub-operator-controller-manager-bd4fcf9c7-qlrdn\" (UID: \"66b774a3-01a1-4190-979b-1a956d8b5049\") " pod="opendatahub/opendatahub-operator-controller-manager-bd4fcf9c7-qlrdn"
Sep 18 19:47:17.095955 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:47:17.095752    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/66b774a3-01a1-4190-979b-1a956d8b5049-manifests\") pod \"opendatahub-operator-controller-manager-bd4fcf9c7-qlrdn\" (UID: \"66b774a3-01a1-4190-979b-1a956d8b5049\") " pod="opendatahub/opendatahub-operator-controller-manager-bd4fcf9c7-qlrdn"
Sep 18 19:47:17.095955 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:47:17.095810    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/66b774a3-01a1-4190-979b-1a956d8b5049-webhook-cert\") pod \"opendatahub-operator-controller-manager-bd4fcf9c7-qlrdn\" (UID: \"66b774a3-01a1-4190-979b-1a956d8b5049\") " pod="opendatahub/opendatahub-operator-controller-manager-bd4fcf9c7-qlrdn"
Sep 18 19:47:17.095955 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:47:17.095836    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b6ld7\" (UniqueName: \"kubernetes.io/projected/66b774a3-01a1-4190-979b-1a956d8b5049-kube-api-access-b6ld7\") pod \"opendatahub-operator-controller-manager-bd4fcf9c7-qlrdn\" (UID: \"66b774a3-01a1-4190-979b-1a956d8b5049\") " pod="opendatahub/opendatahub-operator-controller-manager-bd4fcf9c7-qlrdn"
Sep 18 19:47:17.095955 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:47:17.095868    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/66b774a3-01a1-4190-979b-1a956d8b5049-apiservice-cert\") pod \"opendatahub-operator-controller-manager-bd4fcf9c7-qlrdn\" (UID: \"66b774a3-01a1-4190-979b-1a956d8b5049\") " pod="opendatahub/opendatahub-operator-controller-manager-bd4fcf9c7-qlrdn"
Sep 18 19:47:17.096266 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:47:17.096245    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/66b774a3-01a1-4190-979b-1a956d8b5049-manifests\") pod \"opendatahub-operator-controller-manager-bd4fcf9c7-qlrdn\" (UID: \"66b774a3-01a1-4190-979b-1a956d8b5049\") " pod="opendatahub/opendatahub-operator-controller-manager-bd4fcf9c7-qlrdn"
Sep 18 19:47:17.098010 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:47:17.097980    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/66b774a3-01a1-4190-979b-1a956d8b5049-tmp\") pod \"opendatahub-operator-controller-manager-bd4fcf9c7-qlrdn\" (UID: \"66b774a3-01a1-4190-979b-1a956d8b5049\") " pod="opendatahub/opendatahub-operator-controller-manager-bd4fcf9c7-qlrdn"
Sep 18 19:47:17.098303 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:47:17.098286    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/66b774a3-01a1-4190-979b-1a956d8b5049-apiservice-cert\") pod \"opendatahub-operator-controller-manager-bd4fcf9c7-qlrdn\" (UID: \"66b774a3-01a1-4190-979b-1a956d8b5049\") " pod="opendatahub/opendatahub-operator-controller-manager-bd4fcf9c7-qlrdn"
Sep 18 19:47:17.098346 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:47:17.098335    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/66b774a3-01a1-4190-979b-1a956d8b5049-webhook-cert\") pod \"opendatahub-operator-controller-manager-bd4fcf9c7-qlrdn\" (UID: \"66b774a3-01a1-4190-979b-1a956d8b5049\") " pod="opendatahub/opendatahub-operator-controller-manager-bd4fcf9c7-qlrdn"
Sep 18 19:47:17.109091 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:47:17.109067    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-b6ld7\" (UniqueName: \"kubernetes.io/projected/66b774a3-01a1-4190-979b-1a956d8b5049-kube-api-access-b6ld7\") pod \"opendatahub-operator-controller-manager-bd4fcf9c7-qlrdn\" (UID: \"66b774a3-01a1-4190-979b-1a956d8b5049\") " pod="opendatahub/opendatahub-operator-controller-manager-bd4fcf9c7-qlrdn"
Sep 18 19:47:17.264453 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:47:17.264364    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-bd4fcf9c7-qlrdn"
Sep 18 19:47:17.547622 ip-10-0-136-185 kubenswrapper[2585]: W0918 19:47:17.547567    2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod66b774a3_01a1_4190_979b_1a956d8b5049.slice/crio-e0ba1e950f43b18a4ec96bc0e32806df041e93ac879c4f68ca1c53b96fbd4b58 WatchSource:0}: Error finding container e0ba1e950f43b18a4ec96bc0e32806df041e93ac879c4f68ca1c53b96fbd4b58: Status 404 returned error can't find the container with id e0ba1e950f43b18a4ec96bc0e32806df041e93ac879c4f68ca1c53b96fbd4b58
Sep 18 19:47:17.550184 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:47:17.550062    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-bd4fcf9c7-qlrdn"]
Sep 18 19:47:18.409014 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:47:18.408976    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-bd4fcf9c7-qlrdn" event={"ID":"66b774a3-01a1-4190-979b-1a956d8b5049","Type":"ContainerStarted","Data":"e0ba1e950f43b18a4ec96bc0e32806df041e93ac879c4f68ca1c53b96fbd4b58"}
Sep 18 19:47:21.423848 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:47:21.423813    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-bd4fcf9c7-qlrdn" event={"ID":"66b774a3-01a1-4190-979b-1a956d8b5049","Type":"ContainerStarted","Data":"dfe32f4eab67cf73e04c7eb641edf83c2f24891035d5e341dba776098f5481ea"}
Sep 18 19:47:22.429954 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:47:22.429919    2585 generic.go:356] "Generic (PLEG): container finished" podID="66b774a3-01a1-4190-979b-1a956d8b5049" containerID="dfe32f4eab67cf73e04c7eb641edf83c2f24891035d5e341dba776098f5481ea" exitCode=0
Sep 18 19:47:22.430415 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:47:22.430002    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-bd4fcf9c7-qlrdn" event={"ID":"66b774a3-01a1-4190-979b-1a956d8b5049","Type":"ContainerDied","Data":"dfe32f4eab67cf73e04c7eb641edf83c2f24891035d5e341dba776098f5481ea"}
Sep 18 19:47:23.436249 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:47:23.436212    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-bd4fcf9c7-qlrdn" event={"ID":"66b774a3-01a1-4190-979b-1a956d8b5049","Type":"ContainerStarted","Data":"e17ee8d3d739259414585dad3b9eae76002efddfd96383a5485f69e5c7d1b136"}
Sep 18 19:47:23.436732 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:47:23.436450    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-bd4fcf9c7-qlrdn"
Sep 18 19:47:23.456757 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:47:23.456697    2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-bd4fcf9c7-qlrdn" podStartSLOduration=3.219516572 podStartE2EDuration="7.45667608s" totalImagesPullingTime="3.220839979s" totalInitContainerRuntime="1.016319529s" isStatefulPod=false podCreationTimestamp="2026-09-18 19:47:16 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 19:47:23.455363579 +0000 UTC m=+484.979969002" watchObservedRunningTime="2026-09-18 19:47:23.45667608 +0000 UTC m=+484.981281504"
Sep 18 19:47:34.443141 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:47:34.443111    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-bd4fcf9c7-qlrdn"
Sep 18 19:47:36.285719 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:47:36.285682    2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-6bc78fcbdb-tr229"]
Sep 18 19:47:36.288964 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:47:36.288941    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-6bc78fcbdb-tr229"
Sep 18 19:47:36.292489 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:47:36.292456    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\""
Sep 18 19:47:36.292626 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:47:36.292500    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\""
Sep 18 19:47:36.292626 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:47:36.292536    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
Sep 18 19:47:36.293357 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:47:36.293337    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
Sep 18 19:47:36.293454 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:47:36.293337    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-s2f98\""
Sep 18 19:47:36.307604 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:47:36.307580    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-6bc78fcbdb-tr229"]
Sep 18 19:47:36.469339 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:47:36.469297    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-85cnm\" (UniqueName: \"kubernetes.io/projected/5d1ac900-f718-4110-b637-069335895e44-kube-api-access-85cnm\") pod \"kube-auth-proxy-6bc78fcbdb-tr229\" (UID: \"5d1ac900-f718-4110-b637-069335895e44\") " pod="openshift-ingress/kube-auth-proxy-6bc78fcbdb-tr229"
Sep 18 19:47:36.469520 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:47:36.469360    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/5d1ac900-f718-4110-b637-069335895e44-tmp\") pod \"kube-auth-proxy-6bc78fcbdb-tr229\" (UID: \"5d1ac900-f718-4110-b637-069335895e44\") " pod="openshift-ingress/kube-auth-proxy-6bc78fcbdb-tr229"
Sep 18 19:47:36.469520 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:47:36.469426    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5d1ac900-f718-4110-b637-069335895e44-tls-certs\") pod \"kube-auth-proxy-6bc78fcbdb-tr229\" (UID: \"5d1ac900-f718-4110-b637-069335895e44\") " pod="openshift-ingress/kube-auth-proxy-6bc78fcbdb-tr229"
Sep 18 19:47:36.570341 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:47:36.570253    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5d1ac900-f718-4110-b637-069335895e44-tls-certs\") pod \"kube-auth-proxy-6bc78fcbdb-tr229\" (UID: \"5d1ac900-f718-4110-b637-069335895e44\") " pod="openshift-ingress/kube-auth-proxy-6bc78fcbdb-tr229"
Sep 18 19:47:36.570341 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:47:36.570300    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-85cnm\" (UniqueName: \"kubernetes.io/projected/5d1ac900-f718-4110-b637-069335895e44-kube-api-access-85cnm\") pod \"kube-auth-proxy-6bc78fcbdb-tr229\" (UID: \"5d1ac900-f718-4110-b637-069335895e44\") " pod="openshift-ingress/kube-auth-proxy-6bc78fcbdb-tr229"
Sep 18 19:47:36.570341 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:47:36.570331    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/5d1ac900-f718-4110-b637-069335895e44-tmp\") pod \"kube-auth-proxy-6bc78fcbdb-tr229\" (UID: \"5d1ac900-f718-4110-b637-069335895e44\") " pod="openshift-ingress/kube-auth-proxy-6bc78fcbdb-tr229"
Sep 18 19:47:36.572604 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:47:36.572572    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/5d1ac900-f718-4110-b637-069335895e44-tmp\") pod \"kube-auth-proxy-6bc78fcbdb-tr229\" (UID: \"5d1ac900-f718-4110-b637-069335895e44\") " pod="openshift-ingress/kube-auth-proxy-6bc78fcbdb-tr229"
Sep 18 19:47:36.572810 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:47:36.572777    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5d1ac900-f718-4110-b637-069335895e44-tls-certs\") pod \"kube-auth-proxy-6bc78fcbdb-tr229\" (UID: \"5d1ac900-f718-4110-b637-069335895e44\") " pod="openshift-ingress/kube-auth-proxy-6bc78fcbdb-tr229"
Sep 18 19:47:36.583580 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:47:36.583556    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-85cnm\" (UniqueName: \"kubernetes.io/projected/5d1ac900-f718-4110-b637-069335895e44-kube-api-access-85cnm\") pod \"kube-auth-proxy-6bc78fcbdb-tr229\" (UID: \"5d1ac900-f718-4110-b637-069335895e44\") " pod="openshift-ingress/kube-auth-proxy-6bc78fcbdb-tr229"
Sep 18 19:47:36.598508 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:47:36.598484    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-6bc78fcbdb-tr229"
Sep 18 19:47:36.868073 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:47:36.867991    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-6bc78fcbdb-tr229"]
Sep 18 19:47:37.099462 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:47:37.099415    2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/lws-controller-manager-5947d9d788-w2hwk"]
Sep 18 19:47:37.103485 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:47:37.103467    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-5947d9d788-w2hwk"
Sep 18 19:47:37.106744 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:47:37.106712    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\""
Sep 18 19:47:37.106744 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:47:37.106737    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\""
Sep 18 19:47:37.106957 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:47:37.106755    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"lws-controller-manager-dockercfg-9nshn\""
Sep 18 19:47:37.106957 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:47:37.106713    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"metrics-server-cert\""
Sep 18 19:47:37.106957 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:47:37.106841    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"lws-manager-config\""
Sep 18 19:47:37.106957 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:47:37.106712    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"webhook-server-cert\""
Sep 18 19:47:37.113206 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:47:37.113188    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-5947d9d788-w2hwk"]
Sep 18 19:47:37.276502 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:47:37.276473    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/799a1d4a-ff39-4bba-84c2-893ef3b7fbf7-cert\") pod \"lws-controller-manager-5947d9d788-w2hwk\" (UID: \"799a1d4a-ff39-4bba-84c2-893ef3b7fbf7\") " pod="openshift-lws-operator/lws-controller-manager-5947d9d788-w2hwk"
Sep 18 19:47:37.276677 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:47:37.276520    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/799a1d4a-ff39-4bba-84c2-893ef3b7fbf7-manager-config\") pod \"lws-controller-manager-5947d9d788-w2hwk\" (UID: \"799a1d4a-ff39-4bba-84c2-893ef3b7fbf7\") " pod="openshift-lws-operator/lws-controller-manager-5947d9d788-w2hwk"
Sep 18 19:47:37.276677 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:47:37.276606    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/799a1d4a-ff39-4bba-84c2-893ef3b7fbf7-metrics-cert\") pod \"lws-controller-manager-5947d9d788-w2hwk\" (UID: \"799a1d4a-ff39-4bba-84c2-893ef3b7fbf7\") " pod="openshift-lws-operator/lws-controller-manager-5947d9d788-w2hwk"
Sep 18 19:47:37.276677 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:47:37.276647    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-drcmf\" (UniqueName: \"kubernetes.io/projected/799a1d4a-ff39-4bba-84c2-893ef3b7fbf7-kube-api-access-drcmf\") pod \"lws-controller-manager-5947d9d788-w2hwk\" (UID: \"799a1d4a-ff39-4bba-84c2-893ef3b7fbf7\") " pod="openshift-lws-operator/lws-controller-manager-5947d9d788-w2hwk"
Sep 18 19:47:37.377479 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:47:37.377439    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/799a1d4a-ff39-4bba-84c2-893ef3b7fbf7-metrics-cert\") pod \"lws-controller-manager-5947d9d788-w2hwk\" (UID: \"799a1d4a-ff39-4bba-84c2-893ef3b7fbf7\") " pod="openshift-lws-operator/lws-controller-manager-5947d9d788-w2hwk"
Sep 18 19:47:37.377479 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:47:37.377484    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-drcmf\" (UniqueName: \"kubernetes.io/projected/799a1d4a-ff39-4bba-84c2-893ef3b7fbf7-kube-api-access-drcmf\") pod \"lws-controller-manager-5947d9d788-w2hwk\" (UID: \"799a1d4a-ff39-4bba-84c2-893ef3b7fbf7\") " pod="openshift-lws-operator/lws-controller-manager-5947d9d788-w2hwk"
Sep 18 19:47:37.377960 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:47:37.377522    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/799a1d4a-ff39-4bba-84c2-893ef3b7fbf7-cert\") pod \"lws-controller-manager-5947d9d788-w2hwk\" (UID: \"799a1d4a-ff39-4bba-84c2-893ef3b7fbf7\") " pod="openshift-lws-operator/lws-controller-manager-5947d9d788-w2hwk"
Sep 18 19:47:37.377960 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:47:37.377572    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/799a1d4a-ff39-4bba-84c2-893ef3b7fbf7-manager-config\") pod \"lws-controller-manager-5947d9d788-w2hwk\" (UID: \"799a1d4a-ff39-4bba-84c2-893ef3b7fbf7\") " pod="openshift-lws-operator/lws-controller-manager-5947d9d788-w2hwk"
Sep 18 19:47:37.378292 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:47:37.378273    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/799a1d4a-ff39-4bba-84c2-893ef3b7fbf7-manager-config\") pod \"lws-controller-manager-5947d9d788-w2hwk\" (UID: \"799a1d4a-ff39-4bba-84c2-893ef3b7fbf7\") " pod="openshift-lws-operator/lws-controller-manager-5947d9d788-w2hwk"
Sep 18 19:47:37.379907 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:47:37.379880    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/799a1d4a-ff39-4bba-84c2-893ef3b7fbf7-cert\") pod \"lws-controller-manager-5947d9d788-w2hwk\" (UID: \"799a1d4a-ff39-4bba-84c2-893ef3b7fbf7\") " pod="openshift-lws-operator/lws-controller-manager-5947d9d788-w2hwk"
Sep 18 19:47:37.380010 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:47:37.379956    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/799a1d4a-ff39-4bba-84c2-893ef3b7fbf7-metrics-cert\") pod \"lws-controller-manager-5947d9d788-w2hwk\" (UID: \"799a1d4a-ff39-4bba-84c2-893ef3b7fbf7\") " pod="openshift-lws-operator/lws-controller-manager-5947d9d788-w2hwk"
Sep 18 19:47:37.385465 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:47:37.385447    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-drcmf\" (UniqueName: \"kubernetes.io/projected/799a1d4a-ff39-4bba-84c2-893ef3b7fbf7-kube-api-access-drcmf\") pod \"lws-controller-manager-5947d9d788-w2hwk\" (UID: \"799a1d4a-ff39-4bba-84c2-893ef3b7fbf7\") " pod="openshift-lws-operator/lws-controller-manager-5947d9d788-w2hwk"
Sep 18 19:47:37.413494 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:47:37.413467    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-5947d9d788-w2hwk"
Sep 18 19:47:37.497238 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:47:37.497204    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-6bc78fcbdb-tr229" event={"ID":"5d1ac900-f718-4110-b637-069335895e44","Type":"ContainerStarted","Data":"2e4791ee7560e7845c6b8f790130045c549a029ff2358f57b864d0b647aa3314"}
Sep 18 19:47:37.685576 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:47:37.685543    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-5947d9d788-w2hwk"]
Sep 18 19:47:37.688298 ip-10-0-136-185 kubenswrapper[2585]: W0918 19:47:37.688266    2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod799a1d4a_ff39_4bba_84c2_893ef3b7fbf7.slice/crio-3f23efd23c38f0723d43411d8813aac366683de5f920d46136d17a995a31632b WatchSource:0}: Error finding container 3f23efd23c38f0723d43411d8813aac366683de5f920d46136d17a995a31632b: Status 404 returned error can't find the container with id 3f23efd23c38f0723d43411d8813aac366683de5f920d46136d17a995a31632b
Sep 18 19:47:38.507801 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:47:38.507702    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-5947d9d788-w2hwk" event={"ID":"799a1d4a-ff39-4bba-84c2-893ef3b7fbf7","Type":"ContainerStarted","Data":"3f23efd23c38f0723d43411d8813aac366683de5f920d46136d17a995a31632b"}
Sep 18 19:47:41.522599 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:47:41.522560    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-6bc78fcbdb-tr229" event={"ID":"5d1ac900-f718-4110-b637-069335895e44","Type":"ContainerStarted","Data":"af4974c49bbe000d9f9a71e5fb050cdeb6636a94074119e6cb27ac971a30b9b6"}
Sep 18 19:47:41.524062 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:47:41.524040    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-5947d9d788-w2hwk" event={"ID":"799a1d4a-ff39-4bba-84c2-893ef3b7fbf7","Type":"ContainerStarted","Data":"ba13a68379faf3428920fe5f91107f55c1041e9c744889d3ed20e3858cfd0cba"}
Sep 18 19:47:41.524128 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:47:41.524092    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-lws-operator/lws-controller-manager-5947d9d788-w2hwk"
Sep 18 19:47:41.538696 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:47:41.538643    2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-6bc78fcbdb-tr229" podStartSLOduration=1.503951482 podStartE2EDuration="5.538630382s" totalImagesPullingTime="4.0346789s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 19:47:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 19:47:41.536918065 +0000 UTC m=+503.061523488" watchObservedRunningTime="2026-09-18 19:47:41.538630382 +0000 UTC m=+503.063235836"
Sep 18 19:47:41.552435 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:47:41.552382    2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/lws-controller-manager-5947d9d788-w2hwk" podStartSLOduration=1.282711801 podStartE2EDuration="4.552363434s" totalImagesPullingTime="3.269651633s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 19:47:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 19:47:41.551766237 +0000 UTC m=+503.076371677" watchObservedRunningTime="2026-09-18 19:47:41.552363434 +0000 UTC m=+503.076968858"
Sep 18 19:47:52.531111 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:47:52.531081    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-lws-operator/lws-controller-manager-5947d9d788-w2hwk"
Sep 18 19:49:20.559766 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:49:20.559736    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-rctld_cc0e0de2-75d5-43a1-b64b-b071427cd8ce/ovn-acl-logging/0.log"
Sep 18 19:49:20.560305 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:49:20.559892    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-rctld_cc0e0de2-75d5-43a1-b64b-b071427cd8ce/ovn-acl-logging/0.log"
Sep 18 19:49:24.771643 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:49:24.771611    2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-74495c69f8-nj8ds"]
Sep 18 19:49:24.774985 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:49:24.774969    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-nj8ds"
Sep 18 19:49:24.777567 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:49:24.777549    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\""
Sep 18 19:49:24.777689 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:49:24.777598    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\""
Sep 18 19:49:24.778356 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:49:24.778341    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"limitador-operator-controller-manager-dockercfg-f6w7j\""
Sep 18 19:49:24.786591 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:49:24.786568    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-74495c69f8-nj8ds"]
Sep 18 19:49:24.804817 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:49:24.804766    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-46prn\" (UniqueName: \"kubernetes.io/projected/3b8173f6-26db-4ca9-b53b-471b9692b149-kube-api-access-46prn\") pod \"limitador-operator-controller-manager-74495c69f8-nj8ds\" (UID: \"3b8173f6-26db-4ca9-b53b-471b9692b149\") " pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-nj8ds"
Sep 18 19:49:24.906130 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:49:24.906092    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-46prn\" (UniqueName: \"kubernetes.io/projected/3b8173f6-26db-4ca9-b53b-471b9692b149-kube-api-access-46prn\") pod \"limitador-operator-controller-manager-74495c69f8-nj8ds\" (UID: \"3b8173f6-26db-4ca9-b53b-471b9692b149\") " pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-nj8ds"
Sep 18 19:49:24.915297 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:49:24.915267    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-46prn\" (UniqueName: \"kubernetes.io/projected/3b8173f6-26db-4ca9-b53b-471b9692b149-kube-api-access-46prn\") pod \"limitador-operator-controller-manager-74495c69f8-nj8ds\" (UID: \"3b8173f6-26db-4ca9-b53b-471b9692b149\") " pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-nj8ds"
Sep 18 19:49:25.086400 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:49:25.086297    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-nj8ds"
Sep 18 19:49:25.378467 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:49:25.378442    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-74495c69f8-nj8ds"]
Sep 18 19:49:25.380671 ip-10-0-136-185 kubenswrapper[2585]: W0918 19:49:25.380646    2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3b8173f6_26db_4ca9_b53b_471b9692b149.slice/crio-31f12b4ea9a21ef3e414b39e5fdf64af2d639842fb8d6614ab2d3d659155236a WatchSource:0}: Error finding container 31f12b4ea9a21ef3e414b39e5fdf64af2d639842fb8d6614ab2d3d659155236a: Status 404 returned error can't find the container with id 31f12b4ea9a21ef3e414b39e5fdf64af2d639842fb8d6614ab2d3d659155236a
Sep 18 19:49:25.383341 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:49:25.383318    2585 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 18 19:49:25.968968 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:49:25.968930    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-nj8ds" event={"ID":"3b8173f6-26db-4ca9-b53b-471b9692b149","Type":"ContainerStarted","Data":"31f12b4ea9a21ef3e414b39e5fdf64af2d639842fb8d6614ab2d3d659155236a"}
Sep 18 19:49:27.980364 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:49:27.980270    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-nj8ds" event={"ID":"3b8173f6-26db-4ca9-b53b-471b9692b149","Type":"ContainerStarted","Data":"f13f66c98ad906ea88c83bd2d6333c2cab420dd0920885a2e646c7ed51906258"}
Sep 18 19:49:27.980716 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:49:27.980398    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-nj8ds"
Sep 18 19:49:27.997802 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:49:27.997748    2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-nj8ds" podStartSLOduration=1.6951314659999999 podStartE2EDuration="3.997731785s" totalImagesPullingTime="2.302600319s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 19:49:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 19:49:27.996290416 +0000 UTC m=+609.520895842" watchObservedRunningTime="2026-09-18 19:49:27.997731785 +0000 UTC m=+609.522337208"
Sep 18 19:49:38.987239 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:49:38.987158    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-nj8ds"
Sep 18 19:49:59.369860 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:49:59.369819    2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-xmwh7"]
Sep 18 19:49:59.378240 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:49:59.378216    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-84994f7ffc-xmwh7"
Sep 18 19:49:59.380630 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:49:59.380599    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\""
Sep 18 19:49:59.380964 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:49:59.380724    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-vg27v\""
Sep 18 19:49:59.381842 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:49:59.381818    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-xmwh7"]
Sep 18 19:49:59.394099 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:49:59.394071    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-xmwh7"]
Sep 18 19:49:59.420992 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:49:59.420968    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r84ng\" (UniqueName: \"kubernetes.io/projected/d1b251b4-2bbc-43d9-9ea1-5fc90e16f6e7-kube-api-access-r84ng\") pod \"limitador-limitador-84994f7ffc-xmwh7\" (UID: \"d1b251b4-2bbc-43d9-9ea1-5fc90e16f6e7\") " pod="kuadrant-system/limitador-limitador-84994f7ffc-xmwh7"
Sep 18 19:49:59.421106 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:49:59.421022    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/d1b251b4-2bbc-43d9-9ea1-5fc90e16f6e7-config-file\") pod \"limitador-limitador-84994f7ffc-xmwh7\" (UID: \"d1b251b4-2bbc-43d9-9ea1-5fc90e16f6e7\") " pod="kuadrant-system/limitador-limitador-84994f7ffc-xmwh7"
Sep 18 19:49:59.521535 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:49:59.521498    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r84ng\" (UniqueName: \"kubernetes.io/projected/d1b251b4-2bbc-43d9-9ea1-5fc90e16f6e7-kube-api-access-r84ng\") pod \"limitador-limitador-84994f7ffc-xmwh7\" (UID: \"d1b251b4-2bbc-43d9-9ea1-5fc90e16f6e7\") " pod="kuadrant-system/limitador-limitador-84994f7ffc-xmwh7"
Sep 18 19:49:59.521711 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:49:59.521558    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/d1b251b4-2bbc-43d9-9ea1-5fc90e16f6e7-config-file\") pod \"limitador-limitador-84994f7ffc-xmwh7\" (UID: \"d1b251b4-2bbc-43d9-9ea1-5fc90e16f6e7\") " pod="kuadrant-system/limitador-limitador-84994f7ffc-xmwh7"
Sep 18 19:49:59.522183 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:49:59.522164    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/d1b251b4-2bbc-43d9-9ea1-5fc90e16f6e7-config-file\") pod \"limitador-limitador-84994f7ffc-xmwh7\" (UID: \"d1b251b4-2bbc-43d9-9ea1-5fc90e16f6e7\") " pod="kuadrant-system/limitador-limitador-84994f7ffc-xmwh7"
Sep 18 19:49:59.532002 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:49:59.531981    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-r84ng\" (UniqueName: \"kubernetes.io/projected/d1b251b4-2bbc-43d9-9ea1-5fc90e16f6e7-kube-api-access-r84ng\") pod \"limitador-limitador-84994f7ffc-xmwh7\" (UID: \"d1b251b4-2bbc-43d9-9ea1-5fc90e16f6e7\") " pod="kuadrant-system/limitador-limitador-84994f7ffc-xmwh7"
Sep 18 19:49:59.689626 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:49:59.689590    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-84994f7ffc-xmwh7"
Sep 18 19:49:59.846710 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:49:59.846674    2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-59c974bd48-hcj2c"]
Sep 18 19:49:59.850600 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:49:59.850584    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-59c974bd48-hcj2c"
Sep 18 19:49:59.852941 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:49:59.852922    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-fssg2\""
Sep 18 19:49:59.861178 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:49:59.861153    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-59c974bd48-hcj2c"]
Sep 18 19:49:59.924387 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:49:59.924354    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qxn45\" (UniqueName: \"kubernetes.io/projected/15cb46ba-374b-4563-aec8-dbd5adde9744-kube-api-access-qxn45\") pod \"authorino-59c974bd48-hcj2c\" (UID: \"15cb46ba-374b-4563-aec8-dbd5adde9744\") " pod="kuadrant-system/authorino-59c974bd48-hcj2c"
Sep 18 19:49:59.959490 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:49:59.959410    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-xmwh7"]
Sep 18 19:49:59.962675 ip-10-0-136-185 kubenswrapper[2585]: W0918 19:49:59.962647    2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd1b251b4_2bbc_43d9_9ea1_5fc90e16f6e7.slice/crio-3f41c1d6bf9848a3b96ba283f1e1778fe45dcc018e171d6481c3d1adbef53083 WatchSource:0}: Error finding container 3f41c1d6bf9848a3b96ba283f1e1778fe45dcc018e171d6481c3d1adbef53083: Status 404 returned error can't find the container with id 3f41c1d6bf9848a3b96ba283f1e1778fe45dcc018e171d6481c3d1adbef53083
Sep 18 19:50:00.025887 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:50:00.025281    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qxn45\" (UniqueName: \"kubernetes.io/projected/15cb46ba-374b-4563-aec8-dbd5adde9744-kube-api-access-qxn45\") pod \"authorino-59c974bd48-hcj2c\" (UID: \"15cb46ba-374b-4563-aec8-dbd5adde9744\") " pod="kuadrant-system/authorino-59c974bd48-hcj2c"
Sep 18 19:50:00.038471 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:50:00.038426    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qxn45\" (UniqueName: \"kubernetes.io/projected/15cb46ba-374b-4563-aec8-dbd5adde9744-kube-api-access-qxn45\") pod \"authorino-59c974bd48-hcj2c\" (UID: \"15cb46ba-374b-4563-aec8-dbd5adde9744\") " pod="kuadrant-system/authorino-59c974bd48-hcj2c"
Sep 18 19:50:00.118521 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:50:00.118480    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-84994f7ffc-xmwh7" event={"ID":"d1b251b4-2bbc-43d9-9ea1-5fc90e16f6e7","Type":"ContainerStarted","Data":"3f41c1d6bf9848a3b96ba283f1e1778fe45dcc018e171d6481c3d1adbef53083"}
Sep 18 19:50:00.160127 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:50:00.160089    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-59c974bd48-hcj2c"
Sep 18 19:50:00.434527 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:50:00.434497    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-59c974bd48-hcj2c"]
Sep 18 19:50:00.437062 ip-10-0-136-185 kubenswrapper[2585]: W0918 19:50:00.437037    2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod15cb46ba_374b_4563_aec8_dbd5adde9744.slice/crio-38461b55cf43a6faaaf98eab81ac41495575ab4b6c538e09294bc52e70f34827 WatchSource:0}: Error finding container 38461b55cf43a6faaaf98eab81ac41495575ab4b6c538e09294bc52e70f34827: Status 404 returned error can't find the container with id 38461b55cf43a6faaaf98eab81ac41495575ab4b6c538e09294bc52e70f34827
Sep 18 19:50:01.124505 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:50:01.124472    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-59c974bd48-hcj2c" event={"ID":"15cb46ba-374b-4563-aec8-dbd5adde9744","Type":"ContainerStarted","Data":"38461b55cf43a6faaaf98eab81ac41495575ab4b6c538e09294bc52e70f34827"}
Sep 18 19:50:02.130770 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:50:02.130736    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-84994f7ffc-xmwh7" event={"ID":"d1b251b4-2bbc-43d9-9ea1-5fc90e16f6e7","Type":"ContainerStarted","Data":"99a45e6d41b550ecdbeff25801a121b81813385fd6c9177ee0cf13ca89b93208"}
Sep 18 19:50:02.131167 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:50:02.130937    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-84994f7ffc-xmwh7"
Sep 18 19:50:02.146205 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:50:02.146155    2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-84994f7ffc-xmwh7" podStartSLOduration=1.9718934369999999 podStartE2EDuration="3.146139454s" totalImagesPullingTime="1.174246017s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 19:49:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 19:50:02.145343915 +0000 UTC m=+643.669949333" watchObservedRunningTime="2026-09-18 19:50:02.146139454 +0000 UTC m=+643.670744934"
Sep 18 19:50:03.137072 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:50:03.137036    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-59c974bd48-hcj2c" event={"ID":"15cb46ba-374b-4563-aec8-dbd5adde9744","Type":"ContainerStarted","Data":"8e4a6f6d31991016a4da005dd957c538cfae3bd6c8e8437282fd4380141ec77c"}
Sep 18 19:50:03.150461 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:50:03.150411    2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-59c974bd48-hcj2c" podStartSLOduration=2.413972729 podStartE2EDuration="4.150396611s" totalImagesPullingTime="1.736423882s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 19:49:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 19:50:03.14960247 +0000 UTC m=+644.674207890" watchObservedRunningTime="2026-09-18 19:50:03.150396611 +0000 UTC m=+644.675002033"
Sep 18 19:50:13.138869 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:50:13.138832    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-84994f7ffc-xmwh7"
Sep 18 19:50:27.513804 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:50:27.513756    2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-557f876f69-6v8km"]
Sep 18 19:50:27.517302 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:50:27.517280    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-557f876f69-6v8km"
Sep 18 19:50:27.523444 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:50:27.523416    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-557f876f69-6v8km"]
Sep 18 19:50:27.677663 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:50:27.677629    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hspt5\" (UniqueName: \"kubernetes.io/projected/d4780205-66a3-4d6f-8b02-46c7d9b4f07d-kube-api-access-hspt5\") pod \"authorino-557f876f69-6v8km\" (UID: \"d4780205-66a3-4d6f-8b02-46c7d9b4f07d\") " pod="kuadrant-system/authorino-557f876f69-6v8km"
Sep 18 19:50:27.752894 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:50:27.752859    2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-557f876f69-6v8km"]
Sep 18 19:50:27.753132 ip-10-0-136-185 kubenswrapper[2585]: E0918 19:50:27.753115    2585 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kube-api-access-hspt5], unattached volumes=[], failed to process volumes=[]: context canceled" pod="kuadrant-system/authorino-557f876f69-6v8km" podUID="d4780205-66a3-4d6f-8b02-46c7d9b4f07d"
Sep 18 19:50:27.778438 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:50:27.778341    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hspt5\" (UniqueName: \"kubernetes.io/projected/d4780205-66a3-4d6f-8b02-46c7d9b4f07d-kube-api-access-hspt5\") pod \"authorino-557f876f69-6v8km\" (UID: \"d4780205-66a3-4d6f-8b02-46c7d9b4f07d\") " pod="kuadrant-system/authorino-557f876f69-6v8km"
Sep 18 19:50:27.786618 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:50:27.786593    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hspt5\" (UniqueName: \"kubernetes.io/projected/d4780205-66a3-4d6f-8b02-46c7d9b4f07d-kube-api-access-hspt5\") pod \"authorino-557f876f69-6v8km\" (UID: \"d4780205-66a3-4d6f-8b02-46c7d9b4f07d\") " pod="kuadrant-system/authorino-557f876f69-6v8km"
Sep 18 19:50:27.835344 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:50:27.835301    2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-54df5476d-rpd8t"]
Sep 18 19:50:27.838704 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:50:27.838680    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-54df5476d-rpd8t"
Sep 18 19:50:27.840910 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:50:27.840893    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-server-cert\""
Sep 18 19:50:27.845866 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:50:27.845845    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-54df5476d-rpd8t"]
Sep 18 19:50:27.980428 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:50:27.980370    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/03f3cd82-0e74-490c-b9c6-684865a4b4fc-tls-cert\") pod \"authorino-54df5476d-rpd8t\" (UID: \"03f3cd82-0e74-490c-b9c6-684865a4b4fc\") " pod="kuadrant-system/authorino-54df5476d-rpd8t"
Sep 18 19:50:27.980613 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:50:27.980459    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b7q5k\" (UniqueName: \"kubernetes.io/projected/03f3cd82-0e74-490c-b9c6-684865a4b4fc-kube-api-access-b7q5k\") pod \"authorino-54df5476d-rpd8t\" (UID: \"03f3cd82-0e74-490c-b9c6-684865a4b4fc\") " pod="kuadrant-system/authorino-54df5476d-rpd8t"
Sep 18 19:50:28.081974 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:50:28.081866    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b7q5k\" (UniqueName: \"kubernetes.io/projected/03f3cd82-0e74-490c-b9c6-684865a4b4fc-kube-api-access-b7q5k\") pod \"authorino-54df5476d-rpd8t\" (UID: \"03f3cd82-0e74-490c-b9c6-684865a4b4fc\") " pod="kuadrant-system/authorino-54df5476d-rpd8t"
Sep 18 19:50:28.081974 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:50:28.081960    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/03f3cd82-0e74-490c-b9c6-684865a4b4fc-tls-cert\") pod \"authorino-54df5476d-rpd8t\" (UID: \"03f3cd82-0e74-490c-b9c6-684865a4b4fc\") " pod="kuadrant-system/authorino-54df5476d-rpd8t"
Sep 18 19:50:28.084376 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:50:28.084353    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/03f3cd82-0e74-490c-b9c6-684865a4b4fc-tls-cert\") pod \"authorino-54df5476d-rpd8t\" (UID: \"03f3cd82-0e74-490c-b9c6-684865a4b4fc\") " pod="kuadrant-system/authorino-54df5476d-rpd8t"
Sep 18 19:50:28.089678 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:50:28.089653    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-b7q5k\" (UniqueName: \"kubernetes.io/projected/03f3cd82-0e74-490c-b9c6-684865a4b4fc-kube-api-access-b7q5k\") pod \"authorino-54df5476d-rpd8t\" (UID: \"03f3cd82-0e74-490c-b9c6-684865a4b4fc\") " pod="kuadrant-system/authorino-54df5476d-rpd8t"
Sep 18 19:50:28.148676 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:50:28.148638    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-54df5476d-rpd8t"
Sep 18 19:50:28.250456 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:50:28.250425    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-557f876f69-6v8km"
Sep 18 19:50:28.257689 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:50:28.257660    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-557f876f69-6v8km"
Sep 18 19:50:28.384760 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:50:28.384667    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hspt5\" (UniqueName: \"kubernetes.io/projected/d4780205-66a3-4d6f-8b02-46c7d9b4f07d-kube-api-access-hspt5\") pod \"d4780205-66a3-4d6f-8b02-46c7d9b4f07d\" (UID: \"d4780205-66a3-4d6f-8b02-46c7d9b4f07d\") "
Sep 18 19:50:28.386694 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:50:28.386671    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d4780205-66a3-4d6f-8b02-46c7d9b4f07d-kube-api-access-hspt5" (OuterVolumeSpecName: "kube-api-access-hspt5") pod "d4780205-66a3-4d6f-8b02-46c7d9b4f07d" (UID: "d4780205-66a3-4d6f-8b02-46c7d9b4f07d"). InnerVolumeSpecName "kube-api-access-hspt5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 19:50:28.415001 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:50:28.414975    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-54df5476d-rpd8t"]
Sep 18 19:50:28.417711 ip-10-0-136-185 kubenswrapper[2585]: W0918 19:50:28.417682    2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod03f3cd82_0e74_490c_b9c6_684865a4b4fc.slice/crio-7725dfbd79a14393abc1d6bed5294d11cce9b36469a1da04affebba231246d05 WatchSource:0}: Error finding container 7725dfbd79a14393abc1d6bed5294d11cce9b36469a1da04affebba231246d05: Status 404 returned error can't find the container with id 7725dfbd79a14393abc1d6bed5294d11cce9b36469a1da04affebba231246d05
Sep 18 19:50:28.486244 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:50:28.486212    2585 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hspt5\" (UniqueName: \"kubernetes.io/projected/d4780205-66a3-4d6f-8b02-46c7d9b4f07d-kube-api-access-hspt5\") on node \"ip-10-0-136-185.ec2.internal\" DevicePath \"\""
Sep 18 19:50:29.256970 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:50:29.256923    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-54df5476d-rpd8t" event={"ID":"03f3cd82-0e74-490c-b9c6-684865a4b4fc","Type":"ContainerStarted","Data":"54ef2946f0ea0529988ffd59e77881683fabdcf9082a881110c2f29ad731440f"}
Sep 18 19:50:29.256970 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:50:29.256975    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-54df5476d-rpd8t" event={"ID":"03f3cd82-0e74-490c-b9c6-684865a4b4fc","Type":"ContainerStarted","Data":"7725dfbd79a14393abc1d6bed5294d11cce9b36469a1da04affebba231246d05"}
Sep 18 19:50:29.257463 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:50:29.257035    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-557f876f69-6v8km"
Sep 18 19:50:29.271745 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:50:29.271690    2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-54df5476d-rpd8t" podStartSLOduration=1.621533877 podStartE2EDuration="2.27167531s" totalImagesPullingTime="650.141433ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 19:50:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 19:50:29.270307395 +0000 UTC m=+670.794912819" watchObservedRunningTime="2026-09-18 19:50:29.27167531 +0000 UTC m=+670.796280774"
Sep 18 19:50:29.292308 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:50:29.292276    2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-557f876f69-6v8km"]
Sep 18 19:50:29.298330 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:50:29.298302    2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-59c974bd48-hcj2c"]
Sep 18 19:50:29.298501 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:50:29.298481    2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-59c974bd48-hcj2c" podUID="15cb46ba-374b-4563-aec8-dbd5adde9744" containerName="authorino" containerID="cri-o://8e4a6f6d31991016a4da005dd957c538cfae3bd6c8e8437282fd4380141ec77c" gracePeriod=30
Sep 18 19:50:29.302608 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:50:29.302581    2585 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-557f876f69-6v8km"]
Sep 18 19:50:29.538050 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:50:29.538025    2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-59c974bd48-hcj2c"
Sep 18 19:50:29.696505 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:50:29.696420    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qxn45\" (UniqueName: \"kubernetes.io/projected/15cb46ba-374b-4563-aec8-dbd5adde9744-kube-api-access-qxn45\") pod \"15cb46ba-374b-4563-aec8-dbd5adde9744\" (UID: \"15cb46ba-374b-4563-aec8-dbd5adde9744\") "
Sep 18 19:50:29.698626 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:50:29.698597    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/15cb46ba-374b-4563-aec8-dbd5adde9744-kube-api-access-qxn45" (OuterVolumeSpecName: "kube-api-access-qxn45") pod "15cb46ba-374b-4563-aec8-dbd5adde9744" (UID: "15cb46ba-374b-4563-aec8-dbd5adde9744"). InnerVolumeSpecName "kube-api-access-qxn45". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 19:50:29.797948 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:50:29.797915    2585 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qxn45\" (UniqueName: \"kubernetes.io/projected/15cb46ba-374b-4563-aec8-dbd5adde9744-kube-api-access-qxn45\") on node \"ip-10-0-136-185.ec2.internal\" DevicePath \"\""
Sep 18 19:50:30.262825 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:50:30.262778    2585 generic.go:356] "Generic (PLEG): container finished" podID="15cb46ba-374b-4563-aec8-dbd5adde9744" containerID="8e4a6f6d31991016a4da005dd957c538cfae3bd6c8e8437282fd4380141ec77c" exitCode=0
Sep 18 19:50:30.262825 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:50:30.262831    2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-59c974bd48-hcj2c"
Sep 18 19:50:30.263346 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:50:30.262889    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-59c974bd48-hcj2c" event={"ID":"15cb46ba-374b-4563-aec8-dbd5adde9744","Type":"ContainerDied","Data":"8e4a6f6d31991016a4da005dd957c538cfae3bd6c8e8437282fd4380141ec77c"}
Sep 18 19:50:30.263346 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:50:30.262923    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-59c974bd48-hcj2c" event={"ID":"15cb46ba-374b-4563-aec8-dbd5adde9744","Type":"ContainerDied","Data":"38461b55cf43a6faaaf98eab81ac41495575ab4b6c538e09294bc52e70f34827"}
Sep 18 19:50:30.263346 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:50:30.262942    2585 scope.go:117] "RemoveContainer" containerID="8e4a6f6d31991016a4da005dd957c538cfae3bd6c8e8437282fd4380141ec77c"
Sep 18 19:50:30.271559 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:50:30.271539    2585 scope.go:117] "RemoveContainer" containerID="8e4a6f6d31991016a4da005dd957c538cfae3bd6c8e8437282fd4380141ec77c"
Sep 18 19:50:30.271853 ip-10-0-136-185 kubenswrapper[2585]: E0918 19:50:30.271829    2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8e4a6f6d31991016a4da005dd957c538cfae3bd6c8e8437282fd4380141ec77c\": container with ID starting with 8e4a6f6d31991016a4da005dd957c538cfae3bd6c8e8437282fd4380141ec77c not found: ID does not exist" containerID="8e4a6f6d31991016a4da005dd957c538cfae3bd6c8e8437282fd4380141ec77c"
Sep 18 19:50:30.271933 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:50:30.271865    2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8e4a6f6d31991016a4da005dd957c538cfae3bd6c8e8437282fd4380141ec77c"} err="failed to get container status \"8e4a6f6d31991016a4da005dd957c538cfae3bd6c8e8437282fd4380141ec77c\": rpc error: code = NotFound desc = could not find container \"8e4a6f6d31991016a4da005dd957c538cfae3bd6c8e8437282fd4380141ec77c\": container with ID starting with 8e4a6f6d31991016a4da005dd957c538cfae3bd6c8e8437282fd4380141ec77c not found: ID does not exist"
Sep 18 19:50:30.282646 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:50:30.282624    2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-59c974bd48-hcj2c"]
Sep 18 19:50:30.284805 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:50:30.284773    2585 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-59c974bd48-hcj2c"]
Sep 18 19:50:30.661052 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:50:30.661024    2585 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="15cb46ba-374b-4563-aec8-dbd5adde9744" path="/var/lib/kubelet/pods/15cb46ba-374b-4563-aec8-dbd5adde9744/volumes"
Sep 18 19:50:30.661360 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:50:30.661347    2585 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d4780205-66a3-4d6f-8b02-46c7d9b4f07d" path="/var/lib/kubelet/pods/d4780205-66a3-4d6f-8b02-46c7d9b4f07d/volumes"
Sep 18 19:51:11.533903 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:51:11.533864    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-xmwh7"]
Sep 18 19:51:33.372889 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:51:33.372833    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-xmwh7"]
Sep 18 19:51:37.170694 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:51:37.170652    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-xmwh7"]
Sep 18 19:51:42.469770 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:51:42.469737    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-xmwh7"]
Sep 18 19:52:07.775916 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:52:07.775882    2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-598466548d-96nlc"]
Sep 18 19:52:07.776362 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:52:07.776263    2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="15cb46ba-374b-4563-aec8-dbd5adde9744" containerName="authorino"
Sep 18 19:52:07.776362 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:52:07.776279    2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="15cb46ba-374b-4563-aec8-dbd5adde9744" containerName="authorino"
Sep 18 19:52:07.776362 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:52:07.776336    2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="15cb46ba-374b-4563-aec8-dbd5adde9744" containerName="authorino"
Sep 18 19:52:07.778255 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:52:07.778231    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-96nlc"
Sep 18 19:52:07.780495 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:52:07.780475    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-2md6r\""
Sep 18 19:52:07.780619 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:52:07.780483    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\""
Sep 18 19:52:07.781374 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:52:07.781353    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\""
Sep 18 19:52:07.781493 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:52:07.781355    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-trlp-test-simulated-kserve-self-signed-certs\""
Sep 18 19:52:07.789707 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:52:07.789684    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-598466548d-96nlc"]
Sep 18 19:52:07.942266 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:52:07.942230    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j5xdn\" (UniqueName: \"kubernetes.io/projected/407f9e49-a524-4758-ac91-66120cb274ea-kube-api-access-j5xdn\") pod \"e2e-trlp-test-simulated-kserve-598466548d-96nlc\" (UID: \"407f9e49-a524-4758-ac91-66120cb274ea\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-96nlc"
Sep 18 19:52:07.942266 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:52:07.942276    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/407f9e49-a524-4758-ac91-66120cb274ea-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-598466548d-96nlc\" (UID: \"407f9e49-a524-4758-ac91-66120cb274ea\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-96nlc"
Sep 18 19:52:07.942487 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:52:07.942295    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/407f9e49-a524-4758-ac91-66120cb274ea-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-598466548d-96nlc\" (UID: \"407f9e49-a524-4758-ac91-66120cb274ea\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-96nlc"
Sep 18 19:52:07.942487 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:52:07.942390    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/407f9e49-a524-4758-ac91-66120cb274ea-home\") pod \"e2e-trlp-test-simulated-kserve-598466548d-96nlc\" (UID: \"407f9e49-a524-4758-ac91-66120cb274ea\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-96nlc"
Sep 18 19:52:07.942487 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:52:07.942442    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/407f9e49-a524-4758-ac91-66120cb274ea-model-cache\") pod \"e2e-trlp-test-simulated-kserve-598466548d-96nlc\" (UID: \"407f9e49-a524-4758-ac91-66120cb274ea\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-96nlc"
Sep 18 19:52:07.942581 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:52:07.942506    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/407f9e49-a524-4758-ac91-66120cb274ea-dshm\") pod \"e2e-trlp-test-simulated-kserve-598466548d-96nlc\" (UID: \"407f9e49-a524-4758-ac91-66120cb274ea\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-96nlc"
Sep 18 19:52:08.043106 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:52:08.043013    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/407f9e49-a524-4758-ac91-66120cb274ea-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-598466548d-96nlc\" (UID: \"407f9e49-a524-4758-ac91-66120cb274ea\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-96nlc"
Sep 18 19:52:08.043106 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:52:08.043053    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/407f9e49-a524-4758-ac91-66120cb274ea-home\") pod \"e2e-trlp-test-simulated-kserve-598466548d-96nlc\" (UID: \"407f9e49-a524-4758-ac91-66120cb274ea\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-96nlc"
Sep 18 19:52:08.043106 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:52:08.043086    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/407f9e49-a524-4758-ac91-66120cb274ea-model-cache\") pod \"e2e-trlp-test-simulated-kserve-598466548d-96nlc\" (UID: \"407f9e49-a524-4758-ac91-66120cb274ea\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-96nlc"
Sep 18 19:52:08.043106 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:52:08.043106    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/407f9e49-a524-4758-ac91-66120cb274ea-dshm\") pod \"e2e-trlp-test-simulated-kserve-598466548d-96nlc\" (UID: \"407f9e49-a524-4758-ac91-66120cb274ea\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-96nlc"
Sep 18 19:52:08.043460 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:52:08.043150    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j5xdn\" (UniqueName: \"kubernetes.io/projected/407f9e49-a524-4758-ac91-66120cb274ea-kube-api-access-j5xdn\") pod \"e2e-trlp-test-simulated-kserve-598466548d-96nlc\" (UID: \"407f9e49-a524-4758-ac91-66120cb274ea\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-96nlc"
Sep 18 19:52:08.043460 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:52:08.043173    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/407f9e49-a524-4758-ac91-66120cb274ea-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-598466548d-96nlc\" (UID: \"407f9e49-a524-4758-ac91-66120cb274ea\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-96nlc"
Sep 18 19:52:08.043586 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:52:08.043555    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/407f9e49-a524-4758-ac91-66120cb274ea-model-cache\") pod \"e2e-trlp-test-simulated-kserve-598466548d-96nlc\" (UID: \"407f9e49-a524-4758-ac91-66120cb274ea\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-96nlc"
Sep 18 19:52:08.043586 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:52:08.043568    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/407f9e49-a524-4758-ac91-66120cb274ea-home\") pod \"e2e-trlp-test-simulated-kserve-598466548d-96nlc\" (UID: \"407f9e49-a524-4758-ac91-66120cb274ea\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-96nlc"
Sep 18 19:52:08.043699 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:52:08.043622    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/407f9e49-a524-4758-ac91-66120cb274ea-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-598466548d-96nlc\" (UID: \"407f9e49-a524-4758-ac91-66120cb274ea\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-96nlc"
Sep 18 19:52:08.045377 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:52:08.045354    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/407f9e49-a524-4758-ac91-66120cb274ea-dshm\") pod \"e2e-trlp-test-simulated-kserve-598466548d-96nlc\" (UID: \"407f9e49-a524-4758-ac91-66120cb274ea\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-96nlc"
Sep 18 19:52:08.045497 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:52:08.045418    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/407f9e49-a524-4758-ac91-66120cb274ea-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-598466548d-96nlc\" (UID: \"407f9e49-a524-4758-ac91-66120cb274ea\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-96nlc"
Sep 18 19:52:08.052174 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:52:08.052149    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-j5xdn\" (UniqueName: \"kubernetes.io/projected/407f9e49-a524-4758-ac91-66120cb274ea-kube-api-access-j5xdn\") pod \"e2e-trlp-test-simulated-kserve-598466548d-96nlc\" (UID: \"407f9e49-a524-4758-ac91-66120cb274ea\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-96nlc"
Sep 18 19:52:08.088806 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:52:08.088748    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-96nlc"
Sep 18 19:52:08.356962 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:52:08.356937    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-598466548d-96nlc"]
Sep 18 19:52:08.359065 ip-10-0-136-185 kubenswrapper[2585]: W0918 19:52:08.359034    2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod407f9e49_a524_4758_ac91_66120cb274ea.slice/crio-d876ad24a80a95354954bafe1d1e2161bb2a8a4f6ee556bbce44cf91db23d2d5 WatchSource:0}: Error finding container d876ad24a80a95354954bafe1d1e2161bb2a8a4f6ee556bbce44cf91db23d2d5: Status 404 returned error can't find the container with id d876ad24a80a95354954bafe1d1e2161bb2a8a4f6ee556bbce44cf91db23d2d5
Sep 18 19:52:08.701827 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:52:08.701771    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-96nlc" event={"ID":"407f9e49-a524-4758-ac91-66120cb274ea","Type":"ContainerStarted","Data":"d876ad24a80a95354954bafe1d1e2161bb2a8a4f6ee556bbce44cf91db23d2d5"}
Sep 18 19:52:08.973566 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:52:08.973481    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-xmwh7"]
Sep 18 19:52:09.709051 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:52:09.709009    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-96nlc" event={"ID":"407f9e49-a524-4758-ac91-66120cb274ea","Type":"ContainerStarted","Data":"f98f9d1fa9b767c35ab4ce2d7fba2c49bc18aee09da60ed4a49fd5d9269d3113"}
Sep 18 19:52:09.730982 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:52:09.730926    2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-96nlc" podStartSLOduration=1.725077875 podStartE2EDuration="2.730890673s" totalImagesPullingTime="1.005812798s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 19:52:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 19:52:09.729958162 +0000 UTC m=+771.254563606" watchObservedRunningTime="2026-09-18 19:52:09.730890673 +0000 UTC m=+771.255496118"
Sep 18 19:52:18.089970 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:52:18.089932    2585 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-96nlc"
Sep 18 19:52:18.090458 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:52:18.090003    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-96nlc"
Sep 18 19:52:18.102589 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:52:18.102559    2585 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-96nlc"
Sep 18 19:52:18.759692 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:52:18.759664    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-96nlc"
Sep 18 19:52:26.172725 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:52:26.172690    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-xmwh7"]
Sep 18 19:52:36.954029 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:52:36.953945    2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-wvw2g"]
Sep 18 19:52:36.957525 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:52:36.957504    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-wvw2g"
Sep 18 19:52:36.959832 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:52:36.959811    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"facebook-opt-125m-simulated-kserve-self-signed-certs\""
Sep 18 19:52:36.968275 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:52:36.968249    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-wvw2g"]
Sep 18 19:52:37.092143 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:52:37.092109    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1220608c-ad1c-46cd-8fa7-cb983458aa4e-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-wvw2g\" (UID: \"1220608c-ad1c-46cd-8fa7-cb983458aa4e\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-wvw2g"
Sep 18 19:52:37.092143 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:52:37.092142    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1220608c-ad1c-46cd-8fa7-cb983458aa4e-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-wvw2g\" (UID: \"1220608c-ad1c-46cd-8fa7-cb983458aa4e\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-wvw2g"
Sep 18 19:52:37.092403 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:52:37.092170    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1220608c-ad1c-46cd-8fa7-cb983458aa4e-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-wvw2g\" (UID: \"1220608c-ad1c-46cd-8fa7-cb983458aa4e\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-wvw2g"
Sep 18 19:52:37.092403 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:52:37.092196    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1220608c-ad1c-46cd-8fa7-cb983458aa4e-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-wvw2g\" (UID: \"1220608c-ad1c-46cd-8fa7-cb983458aa4e\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-wvw2g"
Sep 18 19:52:37.092403 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:52:37.092300    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zjt2b\" (UniqueName: \"kubernetes.io/projected/1220608c-ad1c-46cd-8fa7-cb983458aa4e-kube-api-access-zjt2b\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-wvw2g\" (UID: \"1220608c-ad1c-46cd-8fa7-cb983458aa4e\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-wvw2g"
Sep 18 19:52:37.092403 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:52:37.092338    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1220608c-ad1c-46cd-8fa7-cb983458aa4e-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-wvw2g\" (UID: \"1220608c-ad1c-46cd-8fa7-cb983458aa4e\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-wvw2g"
Sep 18 19:52:37.192848 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:52:37.192807    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1220608c-ad1c-46cd-8fa7-cb983458aa4e-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-wvw2g\" (UID: \"1220608c-ad1c-46cd-8fa7-cb983458aa4e\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-wvw2g"
Sep 18 19:52:37.193011 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:52:37.192876    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1220608c-ad1c-46cd-8fa7-cb983458aa4e-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-wvw2g\" (UID: \"1220608c-ad1c-46cd-8fa7-cb983458aa4e\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-wvw2g"
Sep 18 19:52:37.193011 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:52:37.192894    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1220608c-ad1c-46cd-8fa7-cb983458aa4e-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-wvw2g\" (UID: \"1220608c-ad1c-46cd-8fa7-cb983458aa4e\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-wvw2g"
Sep 18 19:52:37.193011 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:52:37.192919    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1220608c-ad1c-46cd-8fa7-cb983458aa4e-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-wvw2g\" (UID: \"1220608c-ad1c-46cd-8fa7-cb983458aa4e\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-wvw2g"
Sep 18 19:52:37.193011 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:52:37.192941    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1220608c-ad1c-46cd-8fa7-cb983458aa4e-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-wvw2g\" (UID: \"1220608c-ad1c-46cd-8fa7-cb983458aa4e\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-wvw2g"
Sep 18 19:52:37.193011 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:52:37.192994    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zjt2b\" (UniqueName: \"kubernetes.io/projected/1220608c-ad1c-46cd-8fa7-cb983458aa4e-kube-api-access-zjt2b\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-wvw2g\" (UID: \"1220608c-ad1c-46cd-8fa7-cb983458aa4e\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-wvw2g"
Sep 18 19:52:37.193327 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:52:37.193306    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1220608c-ad1c-46cd-8fa7-cb983458aa4e-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-wvw2g\" (UID: \"1220608c-ad1c-46cd-8fa7-cb983458aa4e\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-wvw2g"
Sep 18 19:52:37.193401 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:52:37.193340    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1220608c-ad1c-46cd-8fa7-cb983458aa4e-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-wvw2g\" (UID: \"1220608c-ad1c-46cd-8fa7-cb983458aa4e\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-wvw2g"
Sep 18 19:52:37.193440 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:52:37.193400    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1220608c-ad1c-46cd-8fa7-cb983458aa4e-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-wvw2g\" (UID: \"1220608c-ad1c-46cd-8fa7-cb983458aa4e\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-wvw2g"
Sep 18 19:52:37.195200 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:52:37.195180    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1220608c-ad1c-46cd-8fa7-cb983458aa4e-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-wvw2g\" (UID: \"1220608c-ad1c-46cd-8fa7-cb983458aa4e\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-wvw2g"
Sep 18 19:52:37.195477 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:52:37.195458    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1220608c-ad1c-46cd-8fa7-cb983458aa4e-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-wvw2g\" (UID: \"1220608c-ad1c-46cd-8fa7-cb983458aa4e\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-wvw2g"
Sep 18 19:52:37.201040 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:52:37.201020    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zjt2b\" (UniqueName: \"kubernetes.io/projected/1220608c-ad1c-46cd-8fa7-cb983458aa4e-kube-api-access-zjt2b\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-wvw2g\" (UID: \"1220608c-ad1c-46cd-8fa7-cb983458aa4e\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-wvw2g"
Sep 18 19:52:37.267971 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:52:37.267880    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-wvw2g"
Sep 18 19:52:37.542542 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:52:37.542517    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-wvw2g"]
Sep 18 19:52:37.544639 ip-10-0-136-185 kubenswrapper[2585]: W0918 19:52:37.544609    2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1220608c_ad1c_46cd_8fa7_cb983458aa4e.slice/crio-82d2bc1c01de64018bea6ef89b7162fa7fd42a40951645a33ee67040a47aa521 WatchSource:0}: Error finding container 82d2bc1c01de64018bea6ef89b7162fa7fd42a40951645a33ee67040a47aa521: Status 404 returned error can't find the container with id 82d2bc1c01de64018bea6ef89b7162fa7fd42a40951645a33ee67040a47aa521
Sep 18 19:52:37.831476 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:52:37.831398    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-wvw2g" event={"ID":"1220608c-ad1c-46cd-8fa7-cb983458aa4e","Type":"ContainerStarted","Data":"0485c7c4a5e226fc4ca0e1ca6597b40b3e5b422f87b16db6c60ffd7f9ef65740"}
Sep 18 19:52:37.831476 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:52:37.831465    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-wvw2g" event={"ID":"1220608c-ad1c-46cd-8fa7-cb983458aa4e","Type":"ContainerStarted","Data":"82d2bc1c01de64018bea6ef89b7162fa7fd42a40951645a33ee67040a47aa521"}
Sep 18 19:52:37.846599 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:52:37.846537    2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-wvw2g" podStartSLOduration=1.6751322910000002 podStartE2EDuration="1.84651776s" totalImagesPullingTime="171.385469ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 19:52:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 19:52:37.846415101 +0000 UTC m=+799.371020548" watchObservedRunningTime="2026-09-18 19:52:37.84651776 +0000 UTC m=+799.371123183"
Sep 18 19:52:39.110888 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:52:39.110850    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-xmwh7"]
Sep 18 19:52:39.673641 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:52:39.673605    2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-57fc66f697-kmnjt"]
Sep 18 19:52:39.677502 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:52:39.677468    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-kmnjt"
Sep 18 19:52:39.680428 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:52:39.680401    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"premium-simulated-simulated-premium-kserve-self-signed-certs\""
Sep 18 19:52:39.686155 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:52:39.686126    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-57fc66f697-kmnjt"]
Sep 18 19:52:39.817617 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:52:39.817576    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ed909c73-5b8a-4187-9369-4d321ce59a9e-model-cache\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-kmnjt\" (UID: \"ed909c73-5b8a-4187-9369-4d321ce59a9e\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-kmnjt"
Sep 18 19:52:39.817617 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:52:39.817616    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ed909c73-5b8a-4187-9369-4d321ce59a9e-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-kmnjt\" (UID: \"ed909c73-5b8a-4187-9369-4d321ce59a9e\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-kmnjt"
Sep 18 19:52:39.817908 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:52:39.817745    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ed909c73-5b8a-4187-9369-4d321ce59a9e-dshm\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-kmnjt\" (UID: \"ed909c73-5b8a-4187-9369-4d321ce59a9e\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-kmnjt"
Sep 18 19:52:39.817908 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:52:39.817775    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ed909c73-5b8a-4187-9369-4d321ce59a9e-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-kmnjt\" (UID: \"ed909c73-5b8a-4187-9369-4d321ce59a9e\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-kmnjt"
Sep 18 19:52:39.817908 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:52:39.817809    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ed909c73-5b8a-4187-9369-4d321ce59a9e-home\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-kmnjt\" (UID: \"ed909c73-5b8a-4187-9369-4d321ce59a9e\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-kmnjt"
Sep 18 19:52:39.817908 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:52:39.817844    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hxgvl\" (UniqueName: \"kubernetes.io/projected/ed909c73-5b8a-4187-9369-4d321ce59a9e-kube-api-access-hxgvl\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-kmnjt\" (UID: \"ed909c73-5b8a-4187-9369-4d321ce59a9e\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-kmnjt"
Sep 18 19:52:39.918506 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:52:39.918469    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ed909c73-5b8a-4187-9369-4d321ce59a9e-dshm\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-kmnjt\" (UID: \"ed909c73-5b8a-4187-9369-4d321ce59a9e\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-kmnjt"
Sep 18 19:52:39.918506 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:52:39.918506    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ed909c73-5b8a-4187-9369-4d321ce59a9e-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-kmnjt\" (UID: \"ed909c73-5b8a-4187-9369-4d321ce59a9e\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-kmnjt"
Sep 18 19:52:39.918724 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:52:39.918526    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ed909c73-5b8a-4187-9369-4d321ce59a9e-home\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-kmnjt\" (UID: \"ed909c73-5b8a-4187-9369-4d321ce59a9e\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-kmnjt"
Sep 18 19:52:39.918724 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:52:39.918546    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hxgvl\" (UniqueName: \"kubernetes.io/projected/ed909c73-5b8a-4187-9369-4d321ce59a9e-kube-api-access-hxgvl\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-kmnjt\" (UID: \"ed909c73-5b8a-4187-9369-4d321ce59a9e\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-kmnjt"
Sep 18 19:52:39.918724 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:52:39.918603    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ed909c73-5b8a-4187-9369-4d321ce59a9e-model-cache\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-kmnjt\" (UID: \"ed909c73-5b8a-4187-9369-4d321ce59a9e\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-kmnjt"
Sep 18 19:52:39.918724 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:52:39.918619    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ed909c73-5b8a-4187-9369-4d321ce59a9e-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-kmnjt\" (UID: \"ed909c73-5b8a-4187-9369-4d321ce59a9e\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-kmnjt"
Sep 18 19:52:39.919040 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:52:39.919013    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ed909c73-5b8a-4187-9369-4d321ce59a9e-home\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-kmnjt\" (UID: \"ed909c73-5b8a-4187-9369-4d321ce59a9e\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-kmnjt"
Sep 18 19:52:39.919123 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:52:39.919023    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ed909c73-5b8a-4187-9369-4d321ce59a9e-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-kmnjt\" (UID: \"ed909c73-5b8a-4187-9369-4d321ce59a9e\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-kmnjt"
Sep 18 19:52:39.919123 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:52:39.919057    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ed909c73-5b8a-4187-9369-4d321ce59a9e-model-cache\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-kmnjt\" (UID: \"ed909c73-5b8a-4187-9369-4d321ce59a9e\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-kmnjt"
Sep 18 19:52:39.920835 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:52:39.920816    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ed909c73-5b8a-4187-9369-4d321ce59a9e-dshm\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-kmnjt\" (UID: \"ed909c73-5b8a-4187-9369-4d321ce59a9e\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-kmnjt"
Sep 18 19:52:39.921104 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:52:39.921085    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ed909c73-5b8a-4187-9369-4d321ce59a9e-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-kmnjt\" (UID: \"ed909c73-5b8a-4187-9369-4d321ce59a9e\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-kmnjt"
Sep 18 19:52:39.926223 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:52:39.926169    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hxgvl\" (UniqueName: \"kubernetes.io/projected/ed909c73-5b8a-4187-9369-4d321ce59a9e-kube-api-access-hxgvl\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-kmnjt\" (UID: \"ed909c73-5b8a-4187-9369-4d321ce59a9e\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-kmnjt"
Sep 18 19:52:39.990533 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:52:39.990489    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-kmnjt"
Sep 18 19:52:40.264527 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:52:40.264504    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-57fc66f697-kmnjt"]
Sep 18 19:52:40.266594 ip-10-0-136-185 kubenswrapper[2585]: W0918 19:52:40.266567    2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poded909c73_5b8a_4187_9369_4d321ce59a9e.slice/crio-1adfe0e86d1a62a9ee6bef3765834d6dbce5c12920fb2fc336458d67f32ebcab WatchSource:0}: Error finding container 1adfe0e86d1a62a9ee6bef3765834d6dbce5c12920fb2fc336458d67f32ebcab: Status 404 returned error can't find the container with id 1adfe0e86d1a62a9ee6bef3765834d6dbce5c12920fb2fc336458d67f32ebcab
Sep 18 19:52:40.844986 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:52:40.844950    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-kmnjt" event={"ID":"ed909c73-5b8a-4187-9369-4d321ce59a9e","Type":"ContainerStarted","Data":"56d463a9d50a2e46395a136bac0848d93fb1d5d8823c501dedccfbd902b93407"}
Sep 18 19:52:40.844986 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:52:40.844991    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-kmnjt" event={"ID":"ed909c73-5b8a-4187-9369-4d321ce59a9e","Type":"ContainerStarted","Data":"1adfe0e86d1a62a9ee6bef3765834d6dbce5c12920fb2fc336458d67f32ebcab"}
Sep 18 19:52:40.860879 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:52:40.860833    2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-kmnjt" podStartSLOduration=1.691913876 podStartE2EDuration="1.860817151s" totalImagesPullingTime="168.903275ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 19:52:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 19:52:40.859527119 +0000 UTC m=+802.384132543" watchObservedRunningTime="2026-09-18 19:52:40.860817151 +0000 UTC m=+802.385422571"
Sep 18 19:52:45.968321 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:52:45.968288    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-xmwh7"]
Sep 18 19:52:47.268894 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:52:47.268853    2585 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-wvw2g"
Sep 18 19:52:47.269322 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:52:47.268946    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-wvw2g"
Sep 18 19:52:47.281551 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:52:47.281522    2585 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-wvw2g"
Sep 18 19:52:47.888561 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:52:47.888530    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-wvw2g"
Sep 18 19:52:49.990629 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:52:49.990594    2585 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-kmnjt"
Sep 18 19:52:49.990629 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:52:49.990638    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-kmnjt"
Sep 18 19:52:50.003031 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:52:50.003006    2585 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-kmnjt"
Sep 18 19:52:50.901594 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:52:50.901565    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-kmnjt"
Sep 18 19:53:59.164432 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:53:59.164401    2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-54df5476d-rpd8t"]
Sep 18 19:53:59.164942 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:53:59.164616    2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-54df5476d-rpd8t" podUID="03f3cd82-0e74-490c-b9c6-684865a4b4fc" containerName="authorino" containerID="cri-o://54ef2946f0ea0529988ffd59e77881683fabdcf9082a881110c2f29ad731440f" gracePeriod=30
Sep 18 19:53:59.417380 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:53:59.417319    2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-54df5476d-rpd8t"
Sep 18 19:53:59.511033 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:53:59.510983    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/03f3cd82-0e74-490c-b9c6-684865a4b4fc-tls-cert\") pod \"03f3cd82-0e74-490c-b9c6-684865a4b4fc\" (UID: \"03f3cd82-0e74-490c-b9c6-684865a4b4fc\") "
Sep 18 19:53:59.511033 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:53:59.511046    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-b7q5k\" (UniqueName: \"kubernetes.io/projected/03f3cd82-0e74-490c-b9c6-684865a4b4fc-kube-api-access-b7q5k\") pod \"03f3cd82-0e74-490c-b9c6-684865a4b4fc\" (UID: \"03f3cd82-0e74-490c-b9c6-684865a4b4fc\") "
Sep 18 19:53:59.513091 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:53:59.513062    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/03f3cd82-0e74-490c-b9c6-684865a4b4fc-kube-api-access-b7q5k" (OuterVolumeSpecName: "kube-api-access-b7q5k") pod "03f3cd82-0e74-490c-b9c6-684865a4b4fc" (UID: "03f3cd82-0e74-490c-b9c6-684865a4b4fc"). InnerVolumeSpecName "kube-api-access-b7q5k". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 19:53:59.520828 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:53:59.520802    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/03f3cd82-0e74-490c-b9c6-684865a4b4fc-tls-cert" (OuterVolumeSpecName: "tls-cert") pod "03f3cd82-0e74-490c-b9c6-684865a4b4fc" (UID: "03f3cd82-0e74-490c-b9c6-684865a4b4fc"). InnerVolumeSpecName "tls-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 19:53:59.611894 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:53:59.611862    2585 reconciler_common.go:299] "Volume detached for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/03f3cd82-0e74-490c-b9c6-684865a4b4fc-tls-cert\") on node \"ip-10-0-136-185.ec2.internal\" DevicePath \"\""
Sep 18 19:53:59.611894 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:53:59.611891    2585 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b7q5k\" (UniqueName: \"kubernetes.io/projected/03f3cd82-0e74-490c-b9c6-684865a4b4fc-kube-api-access-b7q5k\") on node \"ip-10-0-136-185.ec2.internal\" DevicePath \"\""
Sep 18 19:54:00.222934 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:54:00.222893    2585 generic.go:356] "Generic (PLEG): container finished" podID="03f3cd82-0e74-490c-b9c6-684865a4b4fc" containerID="54ef2946f0ea0529988ffd59e77881683fabdcf9082a881110c2f29ad731440f" exitCode=0
Sep 18 19:54:00.223408 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:54:00.222961    2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-54df5476d-rpd8t"
Sep 18 19:54:00.223408 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:54:00.222964    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-54df5476d-rpd8t" event={"ID":"03f3cd82-0e74-490c-b9c6-684865a4b4fc","Type":"ContainerDied","Data":"54ef2946f0ea0529988ffd59e77881683fabdcf9082a881110c2f29ad731440f"}
Sep 18 19:54:00.223408 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:54:00.223003    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-54df5476d-rpd8t" event={"ID":"03f3cd82-0e74-490c-b9c6-684865a4b4fc","Type":"ContainerDied","Data":"7725dfbd79a14393abc1d6bed5294d11cce9b36469a1da04affebba231246d05"}
Sep 18 19:54:00.223408 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:54:00.223025    2585 scope.go:117] "RemoveContainer" containerID="54ef2946f0ea0529988ffd59e77881683fabdcf9082a881110c2f29ad731440f"
Sep 18 19:54:00.232977 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:54:00.232953    2585 scope.go:117] "RemoveContainer" containerID="54ef2946f0ea0529988ffd59e77881683fabdcf9082a881110c2f29ad731440f"
Sep 18 19:54:00.233253 ip-10-0-136-185 kubenswrapper[2585]: E0918 19:54:00.233237    2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"54ef2946f0ea0529988ffd59e77881683fabdcf9082a881110c2f29ad731440f\": container with ID starting with 54ef2946f0ea0529988ffd59e77881683fabdcf9082a881110c2f29ad731440f not found: ID does not exist" containerID="54ef2946f0ea0529988ffd59e77881683fabdcf9082a881110c2f29ad731440f"
Sep 18 19:54:00.233304 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:54:00.233262    2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"54ef2946f0ea0529988ffd59e77881683fabdcf9082a881110c2f29ad731440f"} err="failed to get container status \"54ef2946f0ea0529988ffd59e77881683fabdcf9082a881110c2f29ad731440f\": rpc error: code = NotFound desc = could not find container \"54ef2946f0ea0529988ffd59e77881683fabdcf9082a881110c2f29ad731440f\": container with ID starting with 54ef2946f0ea0529988ffd59e77881683fabdcf9082a881110c2f29ad731440f not found: ID does not exist"
Sep 18 19:54:00.243533 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:54:00.243502    2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-54df5476d-rpd8t"]
Sep 18 19:54:00.250041 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:54:00.250015    2585 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-54df5476d-rpd8t"]
Sep 18 19:54:00.661959 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:54:00.661931    2585 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="03f3cd82-0e74-490c-b9c6-684865a4b4fc" path="/var/lib/kubelet/pods/03f3cd82-0e74-490c-b9c6-684865a4b4fc/volumes"
Sep 18 19:54:19.981969 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:54:19.981939    2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7534-62d427"]
Sep 18 19:54:19.982450 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:54:19.982266    2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="03f3cd82-0e74-490c-b9c6-684865a4b4fc" containerName="authorino"
Sep 18 19:54:19.982450 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:54:19.982276    2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="03f3cd82-0e74-490c-b9c6-684865a4b4fc" containerName="authorino"
Sep 18 19:54:19.982450 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:54:19.982370    2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="03f3cd82-0e74-490c-b9c6-684865a4b4fc" containerName="authorino"
Sep 18 19:54:19.985239 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:54:19.985223    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7534-62d427"
Sep 18 19:54:19.988412 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:54:19.988382    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-ls85n\""
Sep 18 19:54:20.000547 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:54:20.000514    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7534-62d427"]
Sep 18 19:54:20.096445 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:54:20.096405    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kjtf9\" (UniqueName: \"kubernetes.io/projected/dac278bc-e8d0-4e8a-aff2-0af34c54c43f-kube-api-access-kjtf9\") pod \"test-curl-7534-62d427\" (UID: \"dac278bc-e8d0-4e8a-aff2-0af34c54c43f\") " pod="opendatahub/test-curl-7534-62d427"
Sep 18 19:54:20.197554 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:54:20.197501    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kjtf9\" (UniqueName: \"kubernetes.io/projected/dac278bc-e8d0-4e8a-aff2-0af34c54c43f-kube-api-access-kjtf9\") pod \"test-curl-7534-62d427\" (UID: \"dac278bc-e8d0-4e8a-aff2-0af34c54c43f\") " pod="opendatahub/test-curl-7534-62d427"
Sep 18 19:54:20.205244 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:54:20.205215    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kjtf9\" (UniqueName: \"kubernetes.io/projected/dac278bc-e8d0-4e8a-aff2-0af34c54c43f-kube-api-access-kjtf9\") pod \"test-curl-7534-62d427\" (UID: \"dac278bc-e8d0-4e8a-aff2-0af34c54c43f\") " pod="opendatahub/test-curl-7534-62d427"
Sep 18 19:54:20.297369 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:54:20.297267    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7534-62d427"
Sep 18 19:54:20.566275 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:54:20.566247    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7534-62d427"]
Sep 18 19:54:20.569401 ip-10-0-136-185 kubenswrapper[2585]: W0918 19:54:20.569369    2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poddac278bc_e8d0_4e8a_aff2_0af34c54c43f.slice/crio-20165d95aa733dfcecd56621c54eee67187da41975fddd939c7c6326fee2e835 WatchSource:0}: Error finding container 20165d95aa733dfcecd56621c54eee67187da41975fddd939c7c6326fee2e835: Status 404 returned error can't find the container with id 20165d95aa733dfcecd56621c54eee67187da41975fddd939c7c6326fee2e835
Sep 18 19:54:20.589480 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:54:20.589454    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-rctld_cc0e0de2-75d5-43a1-b64b-b071427cd8ce/ovn-acl-logging/0.log"
Sep 18 19:54:20.590965 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:54:20.590940    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-rctld_cc0e0de2-75d5-43a1-b64b-b071427cd8ce/ovn-acl-logging/0.log"
Sep 18 19:54:21.326096 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:54:21.326056    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7534-62d427" event={"ID":"dac278bc-e8d0-4e8a-aff2-0af34c54c43f","Type":"ContainerStarted","Data":"20165d95aa733dfcecd56621c54eee67187da41975fddd939c7c6326fee2e835"}
Sep 18 19:54:21.807328 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:54:21.807289    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-xmwh7"]
Sep 18 19:54:23.336663 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:54:23.336623    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7534-62d427" event={"ID":"dac278bc-e8d0-4e8a-aff2-0af34c54c43f","Type":"ContainerStarted","Data":"f4e6226315b75a12c2f23fbf7a812e907a0f14e5780c6f2b4b7fc59c282503d0"}
Sep 18 19:54:23.354370 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:54:23.354317    2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-7534-62d427" podStartSLOduration=2.365599726 podStartE2EDuration="4.354303074s" totalImagesPullingTime="1.988703348s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 19:54:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 19:54:23.351956842 +0000 UTC m=+904.876562263" watchObservedRunningTime="2026-09-18 19:54:23.354303074 +0000 UTC m=+904.878908497"
Sep 18 19:54:23.957558 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:54:23.957523    2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7534-62d427"]
Sep 18 19:54:23.961694 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:54:23.961661    2585 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7534-62d427"]
Sep 18 19:54:24.077981 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:54:24.077896    2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7534-25e910"]
Sep 18 19:54:24.078231 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:54:24.078218    2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dac278bc-e8d0-4e8a-aff2-0af34c54c43f" containerName="test-curl-7534-62d427"
Sep 18 19:54:24.078287 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:54:24.078232    2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="dac278bc-e8d0-4e8a-aff2-0af34c54c43f" containerName="test-curl-7534-62d427"
Sep 18 19:54:24.078323 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:54:24.078304    2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="dac278bc-e8d0-4e8a-aff2-0af34c54c43f" containerName="test-curl-7534-62d427"
Sep 18 19:54:24.081389 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:54:24.081372    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7534-25e910"
Sep 18 19:54:24.087554 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:54:24.087525    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7534-25e910"]
Sep 18 19:54:24.137364 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:54:24.137307    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5xtw7\" (UniqueName: \"kubernetes.io/projected/0cd71fd5-6554-4b1a-88b8-b53532bb27fe-kube-api-access-5xtw7\") pod \"test-curl-7534-25e910\" (UID: \"0cd71fd5-6554-4b1a-88b8-b53532bb27fe\") " pod="opendatahub/test-curl-7534-25e910"
Sep 18 19:54:24.238933 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:54:24.238900    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5xtw7\" (UniqueName: \"kubernetes.io/projected/0cd71fd5-6554-4b1a-88b8-b53532bb27fe-kube-api-access-5xtw7\") pod \"test-curl-7534-25e910\" (UID: \"0cd71fd5-6554-4b1a-88b8-b53532bb27fe\") " pod="opendatahub/test-curl-7534-25e910"
Sep 18 19:54:24.247070 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:54:24.247036    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5xtw7\" (UniqueName: \"kubernetes.io/projected/0cd71fd5-6554-4b1a-88b8-b53532bb27fe-kube-api-access-5xtw7\") pod \"test-curl-7534-25e910\" (UID: \"0cd71fd5-6554-4b1a-88b8-b53532bb27fe\") " pod="opendatahub/test-curl-7534-25e910"
Sep 18 19:54:24.341719 ip-10-0-136-185 kubenswrapper[2585]: E0918 19:54:24.341636    2585 kuberuntime_manager.go:649] "Missing actuated resource record" pod="opendatahub/test-curl-7534-62d427" container="test-curl-7534-62d427"
Sep 18 19:54:24.391650 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:54:24.391618    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7534-25e910"
Sep 18 19:54:24.672584 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:54:24.672546    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7534-25e910"]
Sep 18 19:54:24.676353 ip-10-0-136-185 kubenswrapper[2585]: W0918 19:54:24.676314    2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod0cd71fd5_6554_4b1a_88b8_b53532bb27fe.slice/crio-513cdcec4448fb06816fe741372192627ee01d986c664f01ad29e776af7f55cb WatchSource:0}: Error finding container 513cdcec4448fb06816fe741372192627ee01d986c664f01ad29e776af7f55cb: Status 404 returned error can't find the container with id 513cdcec4448fb06816fe741372192627ee01d986c664f01ad29e776af7f55cb
Sep 18 19:54:25.347667 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:54:25.347631    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7534-25e910" event={"ID":"0cd71fd5-6554-4b1a-88b8-b53532bb27fe","Type":"ContainerStarted","Data":"485e840a0f5946aee9d3273a96f4bdbca81f8a5b58a06113ac53b15d8678a98c"}
Sep 18 19:54:25.347667 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:54:25.347674    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7534-25e910" event={"ID":"0cd71fd5-6554-4b1a-88b8-b53532bb27fe","Type":"ContainerStarted","Data":"513cdcec4448fb06816fe741372192627ee01d986c664f01ad29e776af7f55cb"}
Sep 18 19:54:25.349939 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:54:25.347757    2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-7534-62d427" podUID="dac278bc-e8d0-4e8a-aff2-0af34c54c43f" containerName="test-curl-7534-62d427" containerID="cri-o://f4e6226315b75a12c2f23fbf7a812e907a0f14e5780c6f2b4b7fc59c282503d0" gracePeriod=2
Sep 18 19:54:25.361481 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:54:25.361434    2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-7534-25e910" podStartSLOduration=1.361418749 podStartE2EDuration="1.361418749s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 19:54:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 19:54:25.360114397 +0000 UTC m=+906.884719822" watchObservedRunningTime="2026-09-18 19:54:25.361418749 +0000 UTC m=+906.886024169"
Sep 18 19:54:25.391896 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:54:25.391861    2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-674263-565d4c87c9-wpr88"]
Sep 18 19:54:25.395177 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:54:25.395161    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w4-674263-565d4c87c9-wpr88"
Sep 18 19:54:25.397507 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:54:25.397487    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w4-674263-dockercfg-9zfvk\""
Sep 18 19:54:25.397831 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:54:25.397811    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w4-674263\""
Sep 18 19:54:25.405857 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:54:25.405837    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-674263-565d4c87c9-wpr88"]
Sep 18 19:54:25.448767 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:54:25.448737    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9fqqq\" (UniqueName: \"kubernetes.io/projected/e8a9b3b2-b551-48de-90fe-e30eddd5f9a4-kube-api-access-9fqqq\") pod \"payload-processing-e2e-worker-w4-674263-565d4c87c9-wpr88\" (UID: \"e8a9b3b2-b551-48de-90fe-e30eddd5f9a4\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-674263-565d4c87c9-wpr88"
Sep 18 19:54:25.448932 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:54:25.448779    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e8a9b3b2-b551-48de-90fe-e30eddd5f9a4-plugins-config-volume\") pod \"payload-processing-e2e-worker-w4-674263-565d4c87c9-wpr88\" (UID: \"e8a9b3b2-b551-48de-90fe-e30eddd5f9a4\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-674263-565d4c87c9-wpr88"
Sep 18 19:54:25.549722 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:54:25.549692    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e8a9b3b2-b551-48de-90fe-e30eddd5f9a4-plugins-config-volume\") pod \"payload-processing-e2e-worker-w4-674263-565d4c87c9-wpr88\" (UID: \"e8a9b3b2-b551-48de-90fe-e30eddd5f9a4\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-674263-565d4c87c9-wpr88"
Sep 18 19:54:25.549929 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:54:25.549781    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9fqqq\" (UniqueName: \"kubernetes.io/projected/e8a9b3b2-b551-48de-90fe-e30eddd5f9a4-kube-api-access-9fqqq\") pod \"payload-processing-e2e-worker-w4-674263-565d4c87c9-wpr88\" (UID: \"e8a9b3b2-b551-48de-90fe-e30eddd5f9a4\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-674263-565d4c87c9-wpr88"
Sep 18 19:54:25.550491 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:54:25.550467    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e8a9b3b2-b551-48de-90fe-e30eddd5f9a4-plugins-config-volume\") pod \"payload-processing-e2e-worker-w4-674263-565d4c87c9-wpr88\" (UID: \"e8a9b3b2-b551-48de-90fe-e30eddd5f9a4\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-674263-565d4c87c9-wpr88"
Sep 18 19:54:25.559210 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:54:25.559184    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9fqqq\" (UniqueName: \"kubernetes.io/projected/e8a9b3b2-b551-48de-90fe-e30eddd5f9a4-kube-api-access-9fqqq\") pod \"payload-processing-e2e-worker-w4-674263-565d4c87c9-wpr88\" (UID: \"e8a9b3b2-b551-48de-90fe-e30eddd5f9a4\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-674263-565d4c87c9-wpr88"
Sep 18 19:54:25.705439 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:54:25.705405    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w4-674263-565d4c87c9-wpr88"
Sep 18 19:54:25.961933 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:54:25.961861    2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7534-25e910"]
Sep 18 19:54:25.964265 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:54:25.964141    2585 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7534-25e910"]
Sep 18 19:54:25.977171 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:54:25.977140    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-674263-565d4c87c9-wpr88"]
Sep 18 19:54:25.981139 ip-10-0-136-185 kubenswrapper[2585]: W0918 19:54:25.981102    2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode8a9b3b2_b551_48de_90fe_e30eddd5f9a4.slice/crio-2d848bb7d612901cbff21ce3d4598ecc9b31f81d756d5e3f075e18f4553a098e WatchSource:0}: Error finding container 2d848bb7d612901cbff21ce3d4598ecc9b31f81d756d5e3f075e18f4553a098e: Status 404 returned error can't find the container with id 2d848bb7d612901cbff21ce3d4598ecc9b31f81d756d5e3f075e18f4553a098e
Sep 18 19:54:25.983837 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:54:25.983819    2585 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 18 19:54:26.353686 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:54:26.353594    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w4-674263-565d4c87c9-wpr88" event={"ID":"e8a9b3b2-b551-48de-90fe-e30eddd5f9a4","Type":"ContainerStarted","Data":"2d848bb7d612901cbff21ce3d4598ecc9b31f81d756d5e3f075e18f4553a098e"}
Sep 18 19:54:26.354059 ip-10-0-136-185 kubenswrapper[2585]: E0918 19:54:26.353760    2585 kuberuntime_manager.go:649] "Missing actuated resource record" pod="opendatahub/test-curl-7534-25e910" container="test-curl-7534-25e910"
Sep 18 19:54:26.506077 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:54:26.505258    2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-d94b9e-57d4f777b6-tslhj"]
Sep 18 19:54:26.506257 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:54:26.506156    2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0cd71fd5-6554-4b1a-88b8-b53532bb27fe" containerName="test-curl-7534-25e910"
Sep 18 19:54:26.506257 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:54:26.506175    2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="0cd71fd5-6554-4b1a-88b8-b53532bb27fe" containerName="test-curl-7534-25e910"
Sep 18 19:54:26.506426 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:54:26.506406    2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="0cd71fd5-6554-4b1a-88b8-b53532bb27fe" containerName="test-curl-7534-25e910"
Sep 18 19:54:26.510290 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:54:26.510249    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-d94b9e-57d4f777b6-tslhj"
Sep 18 19:54:26.516146 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:54:26.516119    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w0-d94b9e\""
Sep 18 19:54:26.516327 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:54:26.516310    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w0-d94b9e-dockercfg-2sn89\""
Sep 18 19:54:26.526632 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:54:26.526604    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-d94b9e-57d4f777b6-tslhj"]
Sep 18 19:54:26.529839 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:54:26.529812    2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-d94b9e-cd84d77-hd88x"]
Sep 18 19:54:26.533601 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:54:26.533581    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-d94b9e-cd84d77-hd88x"
Sep 18 19:54:26.545853 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:54:26.545769    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-d94b9e-cd84d77-hd88x"]
Sep 18 19:54:26.658408 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:54:26.658365    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/661ede6f-264a-4fbe-9037-9c61a63697e4-plugins-config-volume\") pod \"payload-processing-e2e-worker-w0-d94b9e-cd84d77-hd88x\" (UID: \"661ede6f-264a-4fbe-9037-9c61a63697e4\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-d94b9e-cd84d77-hd88x"
Sep 18 19:54:26.658610 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:54:26.658470    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c67b483b-0a36-4990-97a3-3227c41fcca4-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w0-d94b9e-57d4f777b6-tslhj\" (UID: \"c67b483b-0a36-4990-97a3-3227c41fcca4\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-d94b9e-57d4f777b6-tslhj"
Sep 18 19:54:26.658610 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:54:26.658501    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d7k4q\" (UniqueName: \"kubernetes.io/projected/c67b483b-0a36-4990-97a3-3227c41fcca4-kube-api-access-d7k4q\") pod \"payload-pre-processing-e2e-worker-w0-d94b9e-57d4f777b6-tslhj\" (UID: \"c67b483b-0a36-4990-97a3-3227c41fcca4\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-d94b9e-57d4f777b6-tslhj"
Sep 18 19:54:26.658610 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:54:26.658563    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4pjkz\" (UniqueName: \"kubernetes.io/projected/661ede6f-264a-4fbe-9037-9c61a63697e4-kube-api-access-4pjkz\") pod \"payload-processing-e2e-worker-w0-d94b9e-cd84d77-hd88x\" (UID: \"661ede6f-264a-4fbe-9037-9c61a63697e4\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-d94b9e-cd84d77-hd88x"
Sep 18 19:54:26.759663 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:54:26.759611    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4pjkz\" (UniqueName: \"kubernetes.io/projected/661ede6f-264a-4fbe-9037-9c61a63697e4-kube-api-access-4pjkz\") pod \"payload-processing-e2e-worker-w0-d94b9e-cd84d77-hd88x\" (UID: \"661ede6f-264a-4fbe-9037-9c61a63697e4\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-d94b9e-cd84d77-hd88x"
Sep 18 19:54:26.759862 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:54:26.759728    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/661ede6f-264a-4fbe-9037-9c61a63697e4-plugins-config-volume\") pod \"payload-processing-e2e-worker-w0-d94b9e-cd84d77-hd88x\" (UID: \"661ede6f-264a-4fbe-9037-9c61a63697e4\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-d94b9e-cd84d77-hd88x"
Sep 18 19:54:26.759862 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:54:26.759837    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c67b483b-0a36-4990-97a3-3227c41fcca4-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w0-d94b9e-57d4f777b6-tslhj\" (UID: \"c67b483b-0a36-4990-97a3-3227c41fcca4\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-d94b9e-57d4f777b6-tslhj"
Sep 18 19:54:26.760009 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:54:26.759868    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d7k4q\" (UniqueName: \"kubernetes.io/projected/c67b483b-0a36-4990-97a3-3227c41fcca4-kube-api-access-d7k4q\") pod \"payload-pre-processing-e2e-worker-w0-d94b9e-57d4f777b6-tslhj\" (UID: \"c67b483b-0a36-4990-97a3-3227c41fcca4\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-d94b9e-57d4f777b6-tslhj"
Sep 18 19:54:26.760690 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:54:26.760663    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/661ede6f-264a-4fbe-9037-9c61a63697e4-plugins-config-volume\") pod \"payload-processing-e2e-worker-w0-d94b9e-cd84d77-hd88x\" (UID: \"661ede6f-264a-4fbe-9037-9c61a63697e4\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-d94b9e-cd84d77-hd88x"
Sep 18 19:54:26.760690 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:54:26.760679    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c67b483b-0a36-4990-97a3-3227c41fcca4-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w0-d94b9e-57d4f777b6-tslhj\" (UID: \"c67b483b-0a36-4990-97a3-3227c41fcca4\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-d94b9e-57d4f777b6-tslhj"
Sep 18 19:54:26.782773 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:54:26.782738    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4pjkz\" (UniqueName: \"kubernetes.io/projected/661ede6f-264a-4fbe-9037-9c61a63697e4-kube-api-access-4pjkz\") pod \"payload-processing-e2e-worker-w0-d94b9e-cd84d77-hd88x\" (UID: \"661ede6f-264a-4fbe-9037-9c61a63697e4\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-d94b9e-cd84d77-hd88x"
Sep 18 19:54:26.782922 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:54:26.782865    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-d7k4q\" (UniqueName: \"kubernetes.io/projected/c67b483b-0a36-4990-97a3-3227c41fcca4-kube-api-access-d7k4q\") pod \"payload-pre-processing-e2e-worker-w0-d94b9e-57d4f777b6-tslhj\" (UID: \"c67b483b-0a36-4990-97a3-3227c41fcca4\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-d94b9e-57d4f777b6-tslhj"
Sep 18 19:54:26.825301 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:54:26.825194    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-d94b9e-57d4f777b6-tslhj"
Sep 18 19:54:26.847628 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:54:26.847600    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-d94b9e-cd84d77-hd88x"
Sep 18 19:54:27.109420 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:54:27.109385    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-d94b9e-57d4f777b6-tslhj"]
Sep 18 19:54:27.111040 ip-10-0-136-185 kubenswrapper[2585]: W0918 19:54:27.111011    2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc67b483b_0a36_4990_97a3_3227c41fcca4.slice/crio-9b8e07b76e2f4a4262590adde7317721b53937368d86fa66270346d61a5c625f WatchSource:0}: Error finding container 9b8e07b76e2f4a4262590adde7317721b53937368d86fa66270346d61a5c625f: Status 404 returned error can't find the container with id 9b8e07b76e2f4a4262590adde7317721b53937368d86fa66270346d61a5c625f
Sep 18 19:54:27.134008 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:54:27.133983    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-d94b9e-cd84d77-hd88x"]
Sep 18 19:54:27.136010 ip-10-0-136-185 kubenswrapper[2585]: W0918 19:54:27.135985    2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod661ede6f_264a_4fbe_9037_9c61a63697e4.slice/crio-e192f95c7e34f6803c72a059fbac01b8aa86dabada7e415e1d45e3304ff7ae7f WatchSource:0}: Error finding container e192f95c7e34f6803c72a059fbac01b8aa86dabada7e415e1d45e3304ff7ae7f: Status 404 returned error can't find the container with id e192f95c7e34f6803c72a059fbac01b8aa86dabada7e415e1d45e3304ff7ae7f
Sep 18 19:54:27.360242 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:54:27.360203    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-d94b9e-cd84d77-hd88x" event={"ID":"661ede6f-264a-4fbe-9037-9c61a63697e4","Type":"ContainerStarted","Data":"e192f95c7e34f6803c72a059fbac01b8aa86dabada7e415e1d45e3304ff7ae7f"}
Sep 18 19:54:27.361673 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:54:27.361646    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-d94b9e-57d4f777b6-tslhj" event={"ID":"c67b483b-0a36-4990-97a3-3227c41fcca4","Type":"ContainerStarted","Data":"9b8e07b76e2f4a4262590adde7317721b53937368d86fa66270346d61a5c625f"}
Sep 18 19:54:27.361779 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:54:27.361719    2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-7534-25e910" podUID="0cd71fd5-6554-4b1a-88b8-b53532bb27fe" containerName="test-curl-7534-25e910" containerID="cri-o://485e840a0f5946aee9d3273a96f4bdbca81f8a5b58a06113ac53b15d8678a98c" gracePeriod=2
Sep 18 19:54:27.521272 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:54:27.521242    2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7534-62d427"
Sep 18 19:54:27.668269 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:54:27.667701    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kjtf9\" (UniqueName: \"kubernetes.io/projected/dac278bc-e8d0-4e8a-aff2-0af34c54c43f-kube-api-access-kjtf9\") pod \"dac278bc-e8d0-4e8a-aff2-0af34c54c43f\" (UID: \"dac278bc-e8d0-4e8a-aff2-0af34c54c43f\") "
Sep 18 19:54:27.672448 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:54:27.672405    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dac278bc-e8d0-4e8a-aff2-0af34c54c43f-kube-api-access-kjtf9" (OuterVolumeSpecName: "kube-api-access-kjtf9") pod "dac278bc-e8d0-4e8a-aff2-0af34c54c43f" (UID: "dac278bc-e8d0-4e8a-aff2-0af34c54c43f"). InnerVolumeSpecName "kube-api-access-kjtf9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 19:54:27.768589 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:54:27.768516    2585 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kjtf9\" (UniqueName: \"kubernetes.io/projected/dac278bc-e8d0-4e8a-aff2-0af34c54c43f-kube-api-access-kjtf9\") on node \"ip-10-0-136-185.ec2.internal\" DevicePath \"\""
Sep 18 19:54:28.372466 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:54:28.371759    2585 generic.go:356] "Generic (PLEG): container finished" podID="dac278bc-e8d0-4e8a-aff2-0af34c54c43f" containerID="f4e6226315b75a12c2f23fbf7a812e907a0f14e5780c6f2b4b7fc59c282503d0" exitCode=137
Sep 18 19:54:28.372466 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:54:28.371899    2585 scope.go:117] "RemoveContainer" containerID="f4e6226315b75a12c2f23fbf7a812e907a0f14e5780c6f2b4b7fc59c282503d0"
Sep 18 19:54:28.372466 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:54:28.372051    2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7534-62d427"
Sep 18 19:54:28.386261 ip-10-0-136-185 kubenswrapper[2585]: E0918 19:54:28.386227    2585 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7534-62d427\" is forbidden: User \"system:node:ip-10-0-136-185.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-136-185.ec2.internal' and this object" podUID="dac278bc-e8d0-4e8a-aff2-0af34c54c43f" pod="opendatahub/test-curl-7534-62d427"
Sep 18 19:54:28.662559 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:54:28.662524    2585 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="dac278bc-e8d0-4e8a-aff2-0af34c54c43f" path="/var/lib/kubelet/pods/dac278bc-e8d0-4e8a-aff2-0af34c54c43f/volumes"
Sep 18 19:54:28.973588 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:54:28.973563    2585 scope.go:117] "RemoveContainer" containerID="f4e6226315b75a12c2f23fbf7a812e907a0f14e5780c6f2b4b7fc59c282503d0"
Sep 18 19:54:28.973966 ip-10-0-136-185 kubenswrapper[2585]: E0918 19:54:28.973940    2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f4e6226315b75a12c2f23fbf7a812e907a0f14e5780c6f2b4b7fc59c282503d0\": container with ID starting with f4e6226315b75a12c2f23fbf7a812e907a0f14e5780c6f2b4b7fc59c282503d0 not found: ID does not exist" containerID="f4e6226315b75a12c2f23fbf7a812e907a0f14e5780c6f2b4b7fc59c282503d0"
Sep 18 19:54:28.974084 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:54:28.973981    2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f4e6226315b75a12c2f23fbf7a812e907a0f14e5780c6f2b4b7fc59c282503d0"} err="failed to get container status \"f4e6226315b75a12c2f23fbf7a812e907a0f14e5780c6f2b4b7fc59c282503d0\": rpc error: code = NotFound desc = could not find container \"f4e6226315b75a12c2f23fbf7a812e907a0f14e5780c6f2b4b7fc59c282503d0\": container with ID starting with f4e6226315b75a12c2f23fbf7a812e907a0f14e5780c6f2b4b7fc59c282503d0 not found: ID does not exist"
Sep 18 19:54:29.379589 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:54:29.379558    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w4-674263-565d4c87c9-wpr88" event={"ID":"e8a9b3b2-b551-48de-90fe-e30eddd5f9a4","Type":"ContainerStarted","Data":"dde5bbab0ce5ac2a4ea802d3c73816b8d371f80bedf7751d1ca50f17bc90b219"}
Sep 18 19:54:29.379989 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:54:29.379728    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w4-674263-565d4c87c9-wpr88"
Sep 18 19:54:29.381213 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:54:29.381184    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-d94b9e-57d4f777b6-tslhj" event={"ID":"c67b483b-0a36-4990-97a3-3227c41fcca4","Type":"ContainerStarted","Data":"bf0d0c10c5b90a71048153959d776c338ee5ca4f08951b0f8808a70a9a925383"}
Sep 18 19:54:29.381354 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:54:29.381254    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-d94b9e-57d4f777b6-tslhj"
Sep 18 19:54:29.383752 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:54:29.383730    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-d94b9e-cd84d77-hd88x" event={"ID":"661ede6f-264a-4fbe-9037-9c61a63697e4","Type":"ContainerStarted","Data":"5abbfc123c29f3e80c8336cc8a9fd2e87b926c1521e41736062608431b5c3718"}
Sep 18 19:54:29.383876 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:54:29.383819    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w0-d94b9e-cd84d77-hd88x"
Sep 18 19:54:29.401428 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:54:29.401320    2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w4-674263-565d4c87c9-wpr88" podStartSLOduration=1.362322653 podStartE2EDuration="4.401297785s" totalImagesPullingTime="3.038975132s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 19:54:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 19:54:29.399570771 +0000 UTC m=+910.924176192" watchObservedRunningTime="2026-09-18 19:54:29.401297785 +0000 UTC m=+910.925903205"
Sep 18 19:54:29.420002 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:54:29.419953    2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w0-d94b9e-cd84d77-hd88x" podStartSLOduration=1.4910973699999999 podStartE2EDuration="3.419934705s" totalImagesPullingTime="1.928837335s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 19:54:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 19:54:29.418186498 +0000 UTC m=+910.942791924" watchObservedRunningTime="2026-09-18 19:54:29.419934705 +0000 UTC m=+910.944540165"
Sep 18 19:54:29.443685 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:54:29.443618    2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-d94b9e-57d4f777b6-tslhj" podStartSLOduration=1.464421913 podStartE2EDuration="3.443594474s" totalImagesPullingTime="1.979172561s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 19:54:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 19:54:29.440931652 +0000 UTC m=+910.965537074" watchObservedRunningTime="2026-09-18 19:54:29.443594474 +0000 UTC m=+910.968199900"
Sep 18 19:54:29.538900 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:54:29.538870    2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7534-25e910"
Sep 18 19:54:29.688121 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:54:29.688084    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5xtw7\" (UniqueName: \"kubernetes.io/projected/0cd71fd5-6554-4b1a-88b8-b53532bb27fe-kube-api-access-5xtw7\") pod \"0cd71fd5-6554-4b1a-88b8-b53532bb27fe\" (UID: \"0cd71fd5-6554-4b1a-88b8-b53532bb27fe\") "
Sep 18 19:54:29.690208 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:54:29.690178    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0cd71fd5-6554-4b1a-88b8-b53532bb27fe-kube-api-access-5xtw7" (OuterVolumeSpecName: "kube-api-access-5xtw7") pod "0cd71fd5-6554-4b1a-88b8-b53532bb27fe" (UID: "0cd71fd5-6554-4b1a-88b8-b53532bb27fe"). InnerVolumeSpecName "kube-api-access-5xtw7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 19:54:29.789105 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:54:29.789068    2585 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5xtw7\" (UniqueName: \"kubernetes.io/projected/0cd71fd5-6554-4b1a-88b8-b53532bb27fe-kube-api-access-5xtw7\") on node \"ip-10-0-136-185.ec2.internal\" DevicePath \"\""
Sep 18 19:54:30.390764 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:54:30.390731    2585 generic.go:356] "Generic (PLEG): container finished" podID="0cd71fd5-6554-4b1a-88b8-b53532bb27fe" containerID="485e840a0f5946aee9d3273a96f4bdbca81f8a5b58a06113ac53b15d8678a98c" exitCode=137
Sep 18 19:54:30.391234 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:54:30.390813    2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7534-25e910"
Sep 18 19:54:30.391234 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:54:30.390873    2585 scope.go:117] "RemoveContainer" containerID="485e840a0f5946aee9d3273a96f4bdbca81f8a5b58a06113ac53b15d8678a98c"
Sep 18 19:54:30.399473 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:54:30.399294    2585 scope.go:117] "RemoveContainer" containerID="485e840a0f5946aee9d3273a96f4bdbca81f8a5b58a06113ac53b15d8678a98c"
Sep 18 19:54:30.399606 ip-10-0-136-185 kubenswrapper[2585]: E0918 19:54:30.399588    2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"485e840a0f5946aee9d3273a96f4bdbca81f8a5b58a06113ac53b15d8678a98c\": container with ID starting with 485e840a0f5946aee9d3273a96f4bdbca81f8a5b58a06113ac53b15d8678a98c not found: ID does not exist" containerID="485e840a0f5946aee9d3273a96f4bdbca81f8a5b58a06113ac53b15d8678a98c"
Sep 18 19:54:30.399650 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:54:30.399618    2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"485e840a0f5946aee9d3273a96f4bdbca81f8a5b58a06113ac53b15d8678a98c"} err="failed to get container status \"485e840a0f5946aee9d3273a96f4bdbca81f8a5b58a06113ac53b15d8678a98c\": rpc error: code = NotFound desc = could not find container \"485e840a0f5946aee9d3273a96f4bdbca81f8a5b58a06113ac53b15d8678a98c\": container with ID starting with 485e840a0f5946aee9d3273a96f4bdbca81f8a5b58a06113ac53b15d8678a98c not found: ID does not exist"
Sep 18 19:54:30.403268 ip-10-0-136-185 kubenswrapper[2585]: E0918 19:54:30.403239    2585 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7534-25e910\" is forbidden: User \"system:node:ip-10-0-136-185.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-136-185.ec2.internal' and this object" podUID="0cd71fd5-6554-4b1a-88b8-b53532bb27fe" pod="opendatahub/test-curl-7534-25e910"
Sep 18 19:54:30.661282 ip-10-0-136-185 kubenswrapper[2585]: E0918 19:54:30.661189    2585 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7534-25e910\" is forbidden: User \"system:node:ip-10-0-136-185.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-136-185.ec2.internal' and this object" podUID="0cd71fd5-6554-4b1a-88b8-b53532bb27fe" pod="opendatahub/test-curl-7534-25e910"
Sep 18 19:54:30.662036 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:54:30.662010    2585 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0cd71fd5-6554-4b1a-88b8-b53532bb27fe" path="/var/lib/kubelet/pods/0cd71fd5-6554-4b1a-88b8-b53532bb27fe/volumes"
Sep 18 19:54:33.310525 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:54:33.310490    2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-1d073e-86f694cfb6-xnhf9"]
Sep 18 19:54:33.315179 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:54:33.315148    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-autores-1d073e-86f694cfb6-xnhf9"
Sep 18 19:54:33.317846 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:54:33.317823    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-autores-1d073e\""
Sep 18 19:54:33.335928 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:54:33.335899    2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-1d073e-dfd798d4f-gtgfh"]
Sep 18 19:54:33.339135 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:54:33.339119    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-autores-1d073e-dfd798d4f-gtgfh"
Sep 18 19:54:33.342369 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:54:33.342344    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-autores-1d073e-dockercfg-z2mjv\""
Sep 18 19:54:33.345110 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:54:33.345087    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-1d073e-86f694cfb6-xnhf9"]
Sep 18 19:54:33.376654 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:54:33.376618    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-1d073e-dfd798d4f-gtgfh"]
Sep 18 19:54:33.422035 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:54:33.421999    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/522fc879-ad65-4bf9-8018-ead5194d612d-plugins-config-volume\") pod \"payload-pre-processing-e2e-autores-1d073e-86f694cfb6-xnhf9\" (UID: \"522fc879-ad65-4bf9-8018-ead5194d612d\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-1d073e-86f694cfb6-xnhf9"
Sep 18 19:54:33.422035 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:54:33.422047    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gmvsg\" (UniqueName: \"kubernetes.io/projected/522fc879-ad65-4bf9-8018-ead5194d612d-kube-api-access-gmvsg\") pod \"payload-pre-processing-e2e-autores-1d073e-86f694cfb6-xnhf9\" (UID: \"522fc879-ad65-4bf9-8018-ead5194d612d\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-1d073e-86f694cfb6-xnhf9"
Sep 18 19:54:33.422244 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:54:33.422064    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0e6c1362-1133-4772-86c9-b9a0b4378cb5-plugins-config-volume\") pod \"payload-processing-e2e-autores-1d073e-dfd798d4f-gtgfh\" (UID: \"0e6c1362-1133-4772-86c9-b9a0b4378cb5\") " pod="openshift-ingress/payload-processing-e2e-autores-1d073e-dfd798d4f-gtgfh"
Sep 18 19:54:33.422244 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:54:33.422194    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-98dfl\" (UniqueName: \"kubernetes.io/projected/0e6c1362-1133-4772-86c9-b9a0b4378cb5-kube-api-access-98dfl\") pod \"payload-processing-e2e-autores-1d073e-dfd798d4f-gtgfh\" (UID: \"0e6c1362-1133-4772-86c9-b9a0b4378cb5\") " pod="openshift-ingress/payload-processing-e2e-autores-1d073e-dfd798d4f-gtgfh"
Sep 18 19:54:33.523419 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:54:33.523382    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/522fc879-ad65-4bf9-8018-ead5194d612d-plugins-config-volume\") pod \"payload-pre-processing-e2e-autores-1d073e-86f694cfb6-xnhf9\" (UID: \"522fc879-ad65-4bf9-8018-ead5194d612d\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-1d073e-86f694cfb6-xnhf9"
Sep 18 19:54:33.523621 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:54:33.523428    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gmvsg\" (UniqueName: \"kubernetes.io/projected/522fc879-ad65-4bf9-8018-ead5194d612d-kube-api-access-gmvsg\") pod \"payload-pre-processing-e2e-autores-1d073e-86f694cfb6-xnhf9\" (UID: \"522fc879-ad65-4bf9-8018-ead5194d612d\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-1d073e-86f694cfb6-xnhf9"
Sep 18 19:54:33.523621 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:54:33.523447    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0e6c1362-1133-4772-86c9-b9a0b4378cb5-plugins-config-volume\") pod \"payload-processing-e2e-autores-1d073e-dfd798d4f-gtgfh\" (UID: \"0e6c1362-1133-4772-86c9-b9a0b4378cb5\") " pod="openshift-ingress/payload-processing-e2e-autores-1d073e-dfd798d4f-gtgfh"
Sep 18 19:54:33.523621 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:54:33.523507    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-98dfl\" (UniqueName: \"kubernetes.io/projected/0e6c1362-1133-4772-86c9-b9a0b4378cb5-kube-api-access-98dfl\") pod \"payload-processing-e2e-autores-1d073e-dfd798d4f-gtgfh\" (UID: \"0e6c1362-1133-4772-86c9-b9a0b4378cb5\") " pod="openshift-ingress/payload-processing-e2e-autores-1d073e-dfd798d4f-gtgfh"
Sep 18 19:54:33.524238 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:54:33.524211    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/522fc879-ad65-4bf9-8018-ead5194d612d-plugins-config-volume\") pod \"payload-pre-processing-e2e-autores-1d073e-86f694cfb6-xnhf9\" (UID: \"522fc879-ad65-4bf9-8018-ead5194d612d\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-1d073e-86f694cfb6-xnhf9"
Sep 18 19:54:33.524329 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:54:33.524256    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0e6c1362-1133-4772-86c9-b9a0b4378cb5-plugins-config-volume\") pod \"payload-processing-e2e-autores-1d073e-dfd798d4f-gtgfh\" (UID: \"0e6c1362-1133-4772-86c9-b9a0b4378cb5\") " pod="openshift-ingress/payload-processing-e2e-autores-1d073e-dfd798d4f-gtgfh"
Sep 18 19:54:33.532256 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:54:33.532233    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-98dfl\" (UniqueName: \"kubernetes.io/projected/0e6c1362-1133-4772-86c9-b9a0b4378cb5-kube-api-access-98dfl\") pod \"payload-processing-e2e-autores-1d073e-dfd798d4f-gtgfh\" (UID: \"0e6c1362-1133-4772-86c9-b9a0b4378cb5\") " pod="openshift-ingress/payload-processing-e2e-autores-1d073e-dfd798d4f-gtgfh"
Sep 18 19:54:33.533835 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:54:33.533812    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gmvsg\" (UniqueName: \"kubernetes.io/projected/522fc879-ad65-4bf9-8018-ead5194d612d-kube-api-access-gmvsg\") pod \"payload-pre-processing-e2e-autores-1d073e-86f694cfb6-xnhf9\" (UID: \"522fc879-ad65-4bf9-8018-ead5194d612d\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-1d073e-86f694cfb6-xnhf9"
Sep 18 19:54:33.625060 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:54:33.624959    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-autores-1d073e-86f694cfb6-xnhf9"
Sep 18 19:54:33.650979 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:54:33.650933    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-autores-1d073e-dfd798d4f-gtgfh"
Sep 18 19:54:33.899287 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:54:33.899260    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-1d073e-86f694cfb6-xnhf9"]
Sep 18 19:54:33.901181 ip-10-0-136-185 kubenswrapper[2585]: W0918 19:54:33.901157    2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod522fc879_ad65_4bf9_8018_ead5194d612d.slice/crio-b53f95a0284f2052e94a2d2664b46f09af7314256ebf262aeb8e66de3a34870c WatchSource:0}: Error finding container b53f95a0284f2052e94a2d2664b46f09af7314256ebf262aeb8e66de3a34870c: Status 404 returned error can't find the container with id b53f95a0284f2052e94a2d2664b46f09af7314256ebf262aeb8e66de3a34870c
Sep 18 19:54:33.946530 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:54:33.946500    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-1d073e-dfd798d4f-gtgfh"]
Sep 18 19:54:33.947612 ip-10-0-136-185 kubenswrapper[2585]: W0918 19:54:33.947591    2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0e6c1362_1133_4772_86c9_b9a0b4378cb5.slice/crio-2783aab9a5947a1005f8fbdf1ba1889107002438a8dbdd39d2994e94df670b23 WatchSource:0}: Error finding container 2783aab9a5947a1005f8fbdf1ba1889107002438a8dbdd39d2994e94df670b23: Status 404 returned error can't find the container with id 2783aab9a5947a1005f8fbdf1ba1889107002438a8dbdd39d2994e94df670b23
Sep 18 19:54:34.412147 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:54:34.412111    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-autores-1d073e-dfd798d4f-gtgfh" event={"ID":"0e6c1362-1133-4772-86c9-b9a0b4378cb5","Type":"ContainerStarted","Data":"baef9a1d360a0a33607fc99c4fcef2ee628f9ae86a87df812b8c26cb056b99fd"}
Sep 18 19:54:34.412555 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:54:34.412173    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-autores-1d073e-dfd798d4f-gtgfh" event={"ID":"0e6c1362-1133-4772-86c9-b9a0b4378cb5","Type":"ContainerStarted","Data":"2783aab9a5947a1005f8fbdf1ba1889107002438a8dbdd39d2994e94df670b23"}
Sep 18 19:54:34.412555 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:54:34.412222    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-autores-1d073e-dfd798d4f-gtgfh"
Sep 18 19:54:34.414095 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:54:34.414062    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-autores-1d073e-86f694cfb6-xnhf9" event={"ID":"522fc879-ad65-4bf9-8018-ead5194d612d","Type":"ContainerStarted","Data":"b954c8c66fd99d2b85df79253194228865631811d840b81def83b75b26690950"}
Sep 18 19:54:34.414095 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:54:34.414097    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-autores-1d073e-86f694cfb6-xnhf9" event={"ID":"522fc879-ad65-4bf9-8018-ead5194d612d","Type":"ContainerStarted","Data":"b53f95a0284f2052e94a2d2664b46f09af7314256ebf262aeb8e66de3a34870c"}
Sep 18 19:54:34.414287 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:54:34.414207    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-autores-1d073e-86f694cfb6-xnhf9"
Sep 18 19:54:34.429761 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:54:34.429700    2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-autores-1d073e-dfd798d4f-gtgfh" podStartSLOduration=1.429678093 podStartE2EDuration="1.429678093s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 19:54:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 19:54:34.426996484 +0000 UTC m=+915.951601909" watchObservedRunningTime="2026-09-18 19:54:34.429678093 +0000 UTC m=+915.954283517"
Sep 18 19:54:34.443686 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:54:34.443640    2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-autores-1d073e-86f694cfb6-xnhf9" podStartSLOduration=1.4436250290000001 podStartE2EDuration="1.443625029s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 19:54:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 19:54:34.44183643 +0000 UTC m=+915.966441853" watchObservedRunningTime="2026-09-18 19:54:34.443625029 +0000 UTC m=+915.968230451"
Sep 18 19:54:35.116579 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:54:35.116544    2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-6c083038-65ffc97fb9-dpmw8"]
Sep 18 19:54:35.120660 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:54:35.120641    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-6c083038-65ffc97fb9-dpmw8"
Sep 18 19:54:35.123250 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:54:35.123229    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-6c083038-dockercfg-tl56t\""
Sep 18 19:54:35.123354 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:54:35.123246    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-6c083038\""
Sep 18 19:54:35.131052 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:54:35.131001    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-6c083038-65ffc97fb9-dpmw8"]
Sep 18 19:54:35.239574 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:54:35.239539    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3410b8a8-83cb-45a5-b6e6-c7fe016e9aa3-plugins-config-volume\") pod \"payload-processing-e2e-ait-6c083038-65ffc97fb9-dpmw8\" (UID: \"3410b8a8-83cb-45a5-b6e6-c7fe016e9aa3\") " pod="openshift-ingress/payload-processing-e2e-ait-6c083038-65ffc97fb9-dpmw8"
Sep 18 19:54:35.239752 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:54:35.239609    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bldtc\" (UniqueName: \"kubernetes.io/projected/3410b8a8-83cb-45a5-b6e6-c7fe016e9aa3-kube-api-access-bldtc\") pod \"payload-processing-e2e-ait-6c083038-65ffc97fb9-dpmw8\" (UID: \"3410b8a8-83cb-45a5-b6e6-c7fe016e9aa3\") " pod="openshift-ingress/payload-processing-e2e-ait-6c083038-65ffc97fb9-dpmw8"
Sep 18 19:54:35.340900 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:54:35.340857    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3410b8a8-83cb-45a5-b6e6-c7fe016e9aa3-plugins-config-volume\") pod \"payload-processing-e2e-ait-6c083038-65ffc97fb9-dpmw8\" (UID: \"3410b8a8-83cb-45a5-b6e6-c7fe016e9aa3\") " pod="openshift-ingress/payload-processing-e2e-ait-6c083038-65ffc97fb9-dpmw8"
Sep 18 19:54:35.341065 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:54:35.340922    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bldtc\" (UniqueName: \"kubernetes.io/projected/3410b8a8-83cb-45a5-b6e6-c7fe016e9aa3-kube-api-access-bldtc\") pod \"payload-processing-e2e-ait-6c083038-65ffc97fb9-dpmw8\" (UID: \"3410b8a8-83cb-45a5-b6e6-c7fe016e9aa3\") " pod="openshift-ingress/payload-processing-e2e-ait-6c083038-65ffc97fb9-dpmw8"
Sep 18 19:54:35.341613 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:54:35.341592    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3410b8a8-83cb-45a5-b6e6-c7fe016e9aa3-plugins-config-volume\") pod \"payload-processing-e2e-ait-6c083038-65ffc97fb9-dpmw8\" (UID: \"3410b8a8-83cb-45a5-b6e6-c7fe016e9aa3\") " pod="openshift-ingress/payload-processing-e2e-ait-6c083038-65ffc97fb9-dpmw8"
Sep 18 19:54:35.351331 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:54:35.351301    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bldtc\" (UniqueName: \"kubernetes.io/projected/3410b8a8-83cb-45a5-b6e6-c7fe016e9aa3-kube-api-access-bldtc\") pod \"payload-processing-e2e-ait-6c083038-65ffc97fb9-dpmw8\" (UID: \"3410b8a8-83cb-45a5-b6e6-c7fe016e9aa3\") " pod="openshift-ingress/payload-processing-e2e-ait-6c083038-65ffc97fb9-dpmw8"
Sep 18 19:54:35.433635 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:54:35.433608    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-6c083038-65ffc97fb9-dpmw8"
Sep 18 19:54:35.708375 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:54:35.708192    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-6c083038-65ffc97fb9-dpmw8"]
Sep 18 19:54:35.710667 ip-10-0-136-185 kubenswrapper[2585]: W0918 19:54:35.710634    2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3410b8a8_83cb_45a5_b6e6_c7fe016e9aa3.slice/crio-d82c89f0bb90b982f71129248e7435d1dca973c4848558788ab4c5d2a5764998 WatchSource:0}: Error finding container d82c89f0bb90b982f71129248e7435d1dca973c4848558788ab4c5d2a5764998: Status 404 returned error can't find the container with id d82c89f0bb90b982f71129248e7435d1dca973c4848558788ab4c5d2a5764998
Sep 18 19:54:36.427000 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:54:36.426966    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-6c083038-65ffc97fb9-dpmw8" event={"ID":"3410b8a8-83cb-45a5-b6e6-c7fe016e9aa3","Type":"ContainerStarted","Data":"dc56f5a130554f737d0a22667b054acb923a035ce25e18713bcb48217f0a4a04"}
Sep 18 19:54:36.427165 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:54:36.427019    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-6c083038-65ffc97fb9-dpmw8" event={"ID":"3410b8a8-83cb-45a5-b6e6-c7fe016e9aa3","Type":"ContainerStarted","Data":"d82c89f0bb90b982f71129248e7435d1dca973c4848558788ab4c5d2a5764998"}
Sep 18 19:54:36.427165 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:54:36.427040    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-6c083038-65ffc97fb9-dpmw8"
Sep 18 19:54:36.444514 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:54:36.444460    2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-6c083038-65ffc97fb9-dpmw8" podStartSLOduration=1.444443655 podStartE2EDuration="1.444443655s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 19:54:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 19:54:36.444093071 +0000 UTC m=+917.968698493" watchObservedRunningTime="2026-09-18 19:54:36.444443655 +0000 UTC m=+917.969049077"
Sep 18 19:54:39.788824 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:54:39.788778    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-xmwh7"]
Sep 18 19:54:40.393460 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:54:40.393349    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w4-674263-565d4c87c9-wpr88"
Sep 18 19:54:40.393835 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:54:40.393772    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-d94b9e-57d4f777b6-tslhj"
Sep 18 19:54:40.393945 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:54:40.393865    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w0-d94b9e-cd84d77-hd88x"
Sep 18 19:54:45.421526 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:54:45.421491    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-autores-1d073e-dfd798d4f-gtgfh"
Sep 18 19:54:45.421944 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:54:45.421567    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-autores-1d073e-86f694cfb6-xnhf9"
Sep 18 19:54:45.577194 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:54:45.577159    2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-d5f7da-cd6fff666-5fkmd"]
Sep 18 19:54:45.580537 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:54:45.580520    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-d5f7da-cd6fff666-5fkmd"
Sep 18 19:54:45.582829 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:54:45.582808    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-d5f7da\""
Sep 18 19:54:45.597518 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:54:45.597489    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-d5f7da-cd6fff666-5fkmd"]
Sep 18 19:54:45.733975 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:54:45.733892    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8b34fcc9-59f3-4642-aa96-2cbc3790cda5-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-d5f7da-cd6fff666-5fkmd\" (UID: \"8b34fcc9-59f3-4642-aa96-2cbc3790cda5\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-d5f7da-cd6fff666-5fkmd"
Sep 18 19:54:45.733975 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:54:45.733970    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w9mpq\" (UniqueName: \"kubernetes.io/projected/8b34fcc9-59f3-4642-aa96-2cbc3790cda5-kube-api-access-w9mpq\") pod \"payload-pre-processing-e2e-inf-b-d5f7da-cd6fff666-5fkmd\" (UID: \"8b34fcc9-59f3-4642-aa96-2cbc3790cda5\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-d5f7da-cd6fff666-5fkmd"
Sep 18 19:54:45.835373 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:54:45.835337    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8b34fcc9-59f3-4642-aa96-2cbc3790cda5-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-d5f7da-cd6fff666-5fkmd\" (UID: \"8b34fcc9-59f3-4642-aa96-2cbc3790cda5\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-d5f7da-cd6fff666-5fkmd"
Sep 18 19:54:45.835542 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:54:45.835404    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w9mpq\" (UniqueName: \"kubernetes.io/projected/8b34fcc9-59f3-4642-aa96-2cbc3790cda5-kube-api-access-w9mpq\") pod \"payload-pre-processing-e2e-inf-b-d5f7da-cd6fff666-5fkmd\" (UID: \"8b34fcc9-59f3-4642-aa96-2cbc3790cda5\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-d5f7da-cd6fff666-5fkmd"
Sep 18 19:54:45.836191 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:54:45.836170    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8b34fcc9-59f3-4642-aa96-2cbc3790cda5-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-d5f7da-cd6fff666-5fkmd\" (UID: \"8b34fcc9-59f3-4642-aa96-2cbc3790cda5\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-d5f7da-cd6fff666-5fkmd"
Sep 18 19:54:45.844414 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:54:45.844384    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-w9mpq\" (UniqueName: \"kubernetes.io/projected/8b34fcc9-59f3-4642-aa96-2cbc3790cda5-kube-api-access-w9mpq\") pod \"payload-pre-processing-e2e-inf-b-d5f7da-cd6fff666-5fkmd\" (UID: \"8b34fcc9-59f3-4642-aa96-2cbc3790cda5\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-d5f7da-cd6fff666-5fkmd"
Sep 18 19:54:45.890129 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:54:45.890095    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-d5f7da-cd6fff666-5fkmd"
Sep 18 19:54:46.168172 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:54:46.168144    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-d5f7da-cd6fff666-5fkmd"]
Sep 18 19:54:46.170642 ip-10-0-136-185 kubenswrapper[2585]: W0918 19:54:46.170617    2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8b34fcc9_59f3_4642_aa96_2cbc3790cda5.slice/crio-3510e87116481567581ecac99da34b6fb8934231f6f84c8010815cadd5683a62 WatchSource:0}: Error finding container 3510e87116481567581ecac99da34b6fb8934231f6f84c8010815cadd5683a62: Status 404 returned error can't find the container with id 3510e87116481567581ecac99da34b6fb8934231f6f84c8010815cadd5683a62
Sep 18 19:54:46.479921 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:54:46.479831    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-d5f7da-cd6fff666-5fkmd" event={"ID":"8b34fcc9-59f3-4642-aa96-2cbc3790cda5","Type":"ContainerStarted","Data":"aad4dc39dc415da8d4800ca9e9e57e3b7eec818216bbe7b2968068083846a89a"}
Sep 18 19:54:46.479921 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:54:46.479888    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-d5f7da-cd6fff666-5fkmd" event={"ID":"8b34fcc9-59f3-4642-aa96-2cbc3790cda5","Type":"ContainerStarted","Data":"3510e87116481567581ecac99da34b6fb8934231f6f84c8010815cadd5683a62"}
Sep 18 19:54:46.480342 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:54:46.479929    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-d5f7da-cd6fff666-5fkmd"
Sep 18 19:54:46.499360 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:54:46.499307    2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-d5f7da-cd6fff666-5fkmd" podStartSLOduration=1.499289687 podStartE2EDuration="1.499289687s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 19:54:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 19:54:46.496397401 +0000 UTC m=+928.021002823" watchObservedRunningTime="2026-09-18 19:54:46.499289687 +0000 UTC m=+928.023895108"
Sep 18 19:54:47.433374 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:54:47.433323    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-6c083038-65ffc97fb9-dpmw8"
Sep 18 19:54:54.751963 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:54:54.751934    2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6c083038-8pflc"]
Sep 18 19:54:54.760521 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:54:54.760501    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6c083038-8pflc"
Sep 18 19:54:54.763067 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:54:54.763039    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 18 19:54:54.763222 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:54:54.763040    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-5ttnn\""
Sep 18 19:54:54.764196 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:54:54.764177    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6c083038-8pflc"]
Sep 18 19:54:54.817999 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:54:54.817966    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4acc541d-283d-4fff-bf9e-b700857ee78f-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-6c083038-8pflc\" (UID: \"4acc541d-283d-4fff-bf9e-b700857ee78f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6c083038-8pflc"
Sep 18 19:54:54.918954 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:54:54.918919    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4acc541d-283d-4fff-bf9e-b700857ee78f-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-6c083038-8pflc\" (UID: \"4acc541d-283d-4fff-bf9e-b700857ee78f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6c083038-8pflc"
Sep 18 19:54:54.919570 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:54:54.919542    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4acc541d-283d-4fff-bf9e-b700857ee78f-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-6c083038-8pflc\" (UID: \"4acc541d-283d-4fff-bf9e-b700857ee78f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6c083038-8pflc"
Sep 18 19:54:55.073224 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:54:55.073141    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6c083038-8pflc"
Sep 18 19:54:55.341031 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:54:55.341004    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6c083038-8pflc"]
Sep 18 19:54:55.342692 ip-10-0-136-185 kubenswrapper[2585]: W0918 19:54:55.342660    2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4acc541d_283d_4fff_bf9e_b700857ee78f.slice/crio-4e6bc3f47b7e4ec64e4c886cc67ffbfd93a3bf90bde4459742772819c73b0881 WatchSource:0}: Error finding container 4e6bc3f47b7e4ec64e4c886cc67ffbfd93a3bf90bde4459742772819c73b0881: Status 404 returned error can't find the container with id 4e6bc3f47b7e4ec64e4c886cc67ffbfd93a3bf90bde4459742772819c73b0881
Sep 18 19:54:55.525632 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:54:55.525598    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6c083038-8pflc" event={"ID":"4acc541d-283d-4fff-bf9e-b700857ee78f","Type":"ContainerStarted","Data":"4e6bc3f47b7e4ec64e4c886cc67ffbfd93a3bf90bde4459742772819c73b0881"}
Sep 18 19:54:57.488850 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:54:57.488819    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-d5f7da-cd6fff666-5fkmd"
Sep 18 19:54:58.543925 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:54:58.543895    2585 generic.go:356] "Generic (PLEG): container finished" podID="4acc541d-283d-4fff-bf9e-b700857ee78f" containerID="d3534b450bf252f47ed48e3397a5bbbe1c758344b559b30c5e722f2076ba0092" exitCode=0
Sep 18 19:54:58.544300 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:54:58.543985    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6c083038-8pflc" event={"ID":"4acc541d-283d-4fff-bf9e-b700857ee78f","Type":"ContainerDied","Data":"d3534b450bf252f47ed48e3397a5bbbe1c758344b559b30c5e722f2076ba0092"}
Sep 18 19:54:59.674880 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:54:59.674857    2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6c083038-8pflc"
Sep 18 19:54:59.764635 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:54:59.764599    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4acc541d-283d-4fff-bf9e-b700857ee78f-maas-api-service-ca\") pod \"4acc541d-283d-4fff-bf9e-b700857ee78f\" (UID: \"4acc541d-283d-4fff-bf9e-b700857ee78f\") "
Sep 18 19:54:59.765039 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:54:59.765020    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4acc541d-283d-4fff-bf9e-b700857ee78f-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "4acc541d-283d-4fff-bf9e-b700857ee78f" (UID: "4acc541d-283d-4fff-bf9e-b700857ee78f"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 19:54:59.865952 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:54:59.865869    2585 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4acc541d-283d-4fff-bf9e-b700857ee78f-maas-api-service-ca\") on node \"ip-10-0-136-185.ec2.internal\" DevicePath \"\""
Sep 18 19:55:00.555701 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:55:00.555665    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6c083038-8pflc" event={"ID":"4acc541d-283d-4fff-bf9e-b700857ee78f","Type":"ContainerDied","Data":"4e6bc3f47b7e4ec64e4c886cc67ffbfd93a3bf90bde4459742772819c73b0881"}
Sep 18 19:55:00.555701 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:55:00.555695    2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6c083038-8pflc"
Sep 18 19:55:00.555988 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:55:00.555751    2585 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4e6bc3f47b7e4ec64e4c886cc67ffbfd93a3bf90bde4459742772819c73b0881"
Sep 18 19:55:01.071780 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:55:01.071743    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-xmwh7"]
Sep 18 19:55:11.303936 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:55:11.303901    2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-6c083038-65ffc97fb9-dpmw8"]
Sep 18 19:55:11.304401 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:55:11.304121    2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-6c083038-65ffc97fb9-dpmw8" podUID="3410b8a8-83cb-45a5-b6e6-c7fe016e9aa3" containerName="payload-processing" containerID="cri-o://dc56f5a130554f737d0a22667b054acb923a035ce25e18713bcb48217f0a4a04" gracePeriod=30
Sep 18 19:55:11.524957 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:55:11.524920    2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6c083038-8pflc"]
Sep 18 19:55:11.532904 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:55:11.532875    2585 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6c083038-8pflc"]
Sep 18 19:55:11.562183 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:55:11.562129    2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-6c083038-65ffc97fb9-dpmw8"
Sep 18 19:55:11.616326 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:55:11.616291    2585 generic.go:356] "Generic (PLEG): container finished" podID="3410b8a8-83cb-45a5-b6e6-c7fe016e9aa3" containerID="dc56f5a130554f737d0a22667b054acb923a035ce25e18713bcb48217f0a4a04" exitCode=0
Sep 18 19:55:11.616488 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:55:11.616332    2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-6c083038-65ffc97fb9-dpmw8"
Sep 18 19:55:11.616488 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:55:11.616341    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-6c083038-65ffc97fb9-dpmw8" event={"ID":"3410b8a8-83cb-45a5-b6e6-c7fe016e9aa3","Type":"ContainerDied","Data":"dc56f5a130554f737d0a22667b054acb923a035ce25e18713bcb48217f0a4a04"}
Sep 18 19:55:11.616488 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:55:11.616401    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-6c083038-65ffc97fb9-dpmw8" event={"ID":"3410b8a8-83cb-45a5-b6e6-c7fe016e9aa3","Type":"ContainerDied","Data":"d82c89f0bb90b982f71129248e7435d1dca973c4848558788ab4c5d2a5764998"}
Sep 18 19:55:11.616488 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:55:11.616429    2585 scope.go:117] "RemoveContainer" containerID="dc56f5a130554f737d0a22667b054acb923a035ce25e18713bcb48217f0a4a04"
Sep 18 19:55:11.626202 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:55:11.626177    2585 scope.go:117] "RemoveContainer" containerID="dc56f5a130554f737d0a22667b054acb923a035ce25e18713bcb48217f0a4a04"
Sep 18 19:55:11.626613 ip-10-0-136-185 kubenswrapper[2585]: E0918 19:55:11.626585    2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dc56f5a130554f737d0a22667b054acb923a035ce25e18713bcb48217f0a4a04\": container with ID starting with dc56f5a130554f737d0a22667b054acb923a035ce25e18713bcb48217f0a4a04 not found: ID does not exist" containerID="dc56f5a130554f737d0a22667b054acb923a035ce25e18713bcb48217f0a4a04"
Sep 18 19:55:11.626723 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:55:11.626620    2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dc56f5a130554f737d0a22667b054acb923a035ce25e18713bcb48217f0a4a04"} err="failed to get container status \"dc56f5a130554f737d0a22667b054acb923a035ce25e18713bcb48217f0a4a04\": rpc error: code = NotFound desc = could not find container \"dc56f5a130554f737d0a22667b054acb923a035ce25e18713bcb48217f0a4a04\": container with ID starting with dc56f5a130554f737d0a22667b054acb923a035ce25e18713bcb48217f0a4a04 not found: ID does not exist"
Sep 18 19:55:11.670134 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:55:11.670102    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3410b8a8-83cb-45a5-b6e6-c7fe016e9aa3-plugins-config-volume\") pod \"3410b8a8-83cb-45a5-b6e6-c7fe016e9aa3\" (UID: \"3410b8a8-83cb-45a5-b6e6-c7fe016e9aa3\") "
Sep 18 19:55:11.670303 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:55:11.670177    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bldtc\" (UniqueName: \"kubernetes.io/projected/3410b8a8-83cb-45a5-b6e6-c7fe016e9aa3-kube-api-access-bldtc\") pod \"3410b8a8-83cb-45a5-b6e6-c7fe016e9aa3\" (UID: \"3410b8a8-83cb-45a5-b6e6-c7fe016e9aa3\") "
Sep 18 19:55:11.670466 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:55:11.670442    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3410b8a8-83cb-45a5-b6e6-c7fe016e9aa3-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "3410b8a8-83cb-45a5-b6e6-c7fe016e9aa3" (UID: "3410b8a8-83cb-45a5-b6e6-c7fe016e9aa3"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 19:55:11.672366 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:55:11.672339    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3410b8a8-83cb-45a5-b6e6-c7fe016e9aa3-kube-api-access-bldtc" (OuterVolumeSpecName: "kube-api-access-bldtc") pod "3410b8a8-83cb-45a5-b6e6-c7fe016e9aa3" (UID: "3410b8a8-83cb-45a5-b6e6-c7fe016e9aa3"). InnerVolumeSpecName "kube-api-access-bldtc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 19:55:11.770974 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:55:11.770937    2585 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3410b8a8-83cb-45a5-b6e6-c7fe016e9aa3-plugins-config-volume\") on node \"ip-10-0-136-185.ec2.internal\" DevicePath \"\""
Sep 18 19:55:11.770974 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:55:11.770967    2585 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bldtc\" (UniqueName: \"kubernetes.io/projected/3410b8a8-83cb-45a5-b6e6-c7fe016e9aa3-kube-api-access-bldtc\") on node \"ip-10-0-136-185.ec2.internal\" DevicePath \"\""
Sep 18 19:55:11.945143 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:55:11.945109    2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-6c083038-65ffc97fb9-dpmw8"]
Sep 18 19:55:11.951718 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:55:11.951694    2585 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-6c083038-65ffc97fb9-dpmw8"]
Sep 18 19:55:12.662704 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:55:12.662670    2585 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3410b8a8-83cb-45a5-b6e6-c7fe016e9aa3" path="/var/lib/kubelet/pods/3410b8a8-83cb-45a5-b6e6-c7fe016e9aa3/volumes"
Sep 18 19:55:12.663149 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:55:12.663059    2585 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4acc541d-283d-4fff-bf9e-b700857ee78f" path="/var/lib/kubelet/pods/4acc541d-283d-4fff-bf9e-b700857ee78f/volumes"
Sep 18 19:55:17.085017 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:55:17.084970    2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-64cc2d85-774f947b78-bt69s"]
Sep 18 19:55:17.085679 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:55:17.085652    2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3410b8a8-83cb-45a5-b6e6-c7fe016e9aa3" containerName="payload-processing"
Sep 18 19:55:17.085679 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:55:17.085680    2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="3410b8a8-83cb-45a5-b6e6-c7fe016e9aa3" containerName="payload-processing"
Sep 18 19:55:17.085884 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:55:17.085694    2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4acc541d-283d-4fff-bf9e-b700857ee78f" containerName="revoke-keys"
Sep 18 19:55:17.085884 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:55:17.085702    2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="4acc541d-283d-4fff-bf9e-b700857ee78f" containerName="revoke-keys"
Sep 18 19:55:17.085884 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:55:17.085838    2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="4acc541d-283d-4fff-bf9e-b700857ee78f" containerName="revoke-keys"
Sep 18 19:55:17.085884 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:55:17.085855    2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="3410b8a8-83cb-45a5-b6e6-c7fe016e9aa3" containerName="payload-processing"
Sep 18 19:55:17.090838 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:55:17.090811    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-64cc2d85-774f947b78-bt69s"
Sep 18 19:55:17.093546 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:55:17.093356    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-migrate-64cc2d85-serving-cert\""
Sep 18 19:55:17.093919 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:55:17.093775    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-jqjhf\""
Sep 18 19:55:17.094255 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:55:17.094227    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 18 19:55:17.094883 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:55:17.094669    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\""
Sep 18 19:55:17.094883 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:55:17.094775    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Sep 18 19:55:17.104186 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:55:17.103549    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-64cc2d85-774f947b78-bt69s"]
Sep 18 19:55:17.223378 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:55:17.223338    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n26b2\" (UniqueName: \"kubernetes.io/projected/3bc9bad8-f44f-4286-a267-69a521e9453b-kube-api-access-n26b2\") pod \"maas-api-e2e-migrate-64cc2d85-774f947b78-bt69s\" (UID: \"3bc9bad8-f44f-4286-a267-69a521e9453b\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-64cc2d85-774f947b78-bt69s"
Sep 18 19:55:17.223552 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:55:17.223388    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3bc9bad8-f44f-4286-a267-69a521e9453b-maas-api-tls\") pod \"maas-api-e2e-migrate-64cc2d85-774f947b78-bt69s\" (UID: \"3bc9bad8-f44f-4286-a267-69a521e9453b\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-64cc2d85-774f947b78-bt69s"
Sep 18 19:55:17.223552 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:55:17.223511    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3bc9bad8-f44f-4286-a267-69a521e9453b-metrics-certs\") pod \"maas-api-e2e-migrate-64cc2d85-774f947b78-bt69s\" (UID: \"3bc9bad8-f44f-4286-a267-69a521e9453b\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-64cc2d85-774f947b78-bt69s"
Sep 18 19:55:17.223639 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:55:17.223561    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3bc9bad8-f44f-4286-a267-69a521e9453b-openshift-service-ca\") pod \"maas-api-e2e-migrate-64cc2d85-774f947b78-bt69s\" (UID: \"3bc9bad8-f44f-4286-a267-69a521e9453b\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-64cc2d85-774f947b78-bt69s"
Sep 18 19:55:17.324935 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:55:17.324900    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3bc9bad8-f44f-4286-a267-69a521e9453b-metrics-certs\") pod \"maas-api-e2e-migrate-64cc2d85-774f947b78-bt69s\" (UID: \"3bc9bad8-f44f-4286-a267-69a521e9453b\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-64cc2d85-774f947b78-bt69s"
Sep 18 19:55:17.325135 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:55:17.324957    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3bc9bad8-f44f-4286-a267-69a521e9453b-openshift-service-ca\") pod \"maas-api-e2e-migrate-64cc2d85-774f947b78-bt69s\" (UID: \"3bc9bad8-f44f-4286-a267-69a521e9453b\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-64cc2d85-774f947b78-bt69s"
Sep 18 19:55:17.325135 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:55:17.325007    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n26b2\" (UniqueName: \"kubernetes.io/projected/3bc9bad8-f44f-4286-a267-69a521e9453b-kube-api-access-n26b2\") pod \"maas-api-e2e-migrate-64cc2d85-774f947b78-bt69s\" (UID: \"3bc9bad8-f44f-4286-a267-69a521e9453b\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-64cc2d85-774f947b78-bt69s"
Sep 18 19:55:17.325135 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:55:17.325037    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3bc9bad8-f44f-4286-a267-69a521e9453b-maas-api-tls\") pod \"maas-api-e2e-migrate-64cc2d85-774f947b78-bt69s\" (UID: \"3bc9bad8-f44f-4286-a267-69a521e9453b\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-64cc2d85-774f947b78-bt69s"
Sep 18 19:55:17.325309 ip-10-0-136-185 kubenswrapper[2585]: E0918 19:55:17.325136    2585 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-migrate-64cc2d85-serving-cert: secret "maas-api-e2e-migrate-64cc2d85-serving-cert" not found
Sep 18 19:55:17.325309 ip-10-0-136-185 kubenswrapper[2585]: E0918 19:55:17.325217    2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3bc9bad8-f44f-4286-a267-69a521e9453b-maas-api-tls podName:3bc9bad8-f44f-4286-a267-69a521e9453b nodeName:}" failed. No retries permitted until 2026-09-18 19:55:17.825196532 +0000 UTC m=+959.349801946 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/3bc9bad8-f44f-4286-a267-69a521e9453b-maas-api-tls") pod "maas-api-e2e-migrate-64cc2d85-774f947b78-bt69s" (UID: "3bc9bad8-f44f-4286-a267-69a521e9453b") : secret "maas-api-e2e-migrate-64cc2d85-serving-cert" not found
Sep 18 19:55:17.325677 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:55:17.325657    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3bc9bad8-f44f-4286-a267-69a521e9453b-openshift-service-ca\") pod \"maas-api-e2e-migrate-64cc2d85-774f947b78-bt69s\" (UID: \"3bc9bad8-f44f-4286-a267-69a521e9453b\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-64cc2d85-774f947b78-bt69s"
Sep 18 19:55:17.327478 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:55:17.327459    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3bc9bad8-f44f-4286-a267-69a521e9453b-metrics-certs\") pod \"maas-api-e2e-migrate-64cc2d85-774f947b78-bt69s\" (UID: \"3bc9bad8-f44f-4286-a267-69a521e9453b\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-64cc2d85-774f947b78-bt69s"
Sep 18 19:55:17.338698 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:55:17.338636    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-n26b2\" (UniqueName: \"kubernetes.io/projected/3bc9bad8-f44f-4286-a267-69a521e9453b-kube-api-access-n26b2\") pod \"maas-api-e2e-migrate-64cc2d85-774f947b78-bt69s\" (UID: \"3bc9bad8-f44f-4286-a267-69a521e9453b\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-64cc2d85-774f947b78-bt69s"
Sep 18 19:55:17.683448 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:55:17.683406    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-xmwh7"]
Sep 18 19:55:17.828884 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:55:17.828848    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3bc9bad8-f44f-4286-a267-69a521e9453b-maas-api-tls\") pod \"maas-api-e2e-migrate-64cc2d85-774f947b78-bt69s\" (UID: \"3bc9bad8-f44f-4286-a267-69a521e9453b\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-64cc2d85-774f947b78-bt69s"
Sep 18 19:55:17.831279 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:55:17.831254    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3bc9bad8-f44f-4286-a267-69a521e9453b-maas-api-tls\") pod \"maas-api-e2e-migrate-64cc2d85-774f947b78-bt69s\" (UID: \"3bc9bad8-f44f-4286-a267-69a521e9453b\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-64cc2d85-774f947b78-bt69s"
Sep 18 19:55:18.006456 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:55:18.006362    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-64cc2d85-774f947b78-bt69s"
Sep 18 19:55:18.290027 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:55:18.290000    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-64cc2d85-774f947b78-bt69s"]
Sep 18 19:55:18.292511 ip-10-0-136-185 kubenswrapper[2585]: W0918 19:55:18.292482    2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3bc9bad8_f44f_4286_a267_69a521e9453b.slice/crio-a4132bd10f7f7517b752ddcf58b48ce33869dd92759c313ce1df89e286947734 WatchSource:0}: Error finding container a4132bd10f7f7517b752ddcf58b48ce33869dd92759c313ce1df89e286947734: Status 404 returned error can't find the container with id a4132bd10f7f7517b752ddcf58b48ce33869dd92759c313ce1df89e286947734
Sep 18 19:55:18.660869 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:55:18.660827    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-64cc2d85-774f947b78-bt69s" event={"ID":"3bc9bad8-f44f-4286-a267-69a521e9453b","Type":"ContainerStarted","Data":"a4132bd10f7f7517b752ddcf58b48ce33869dd92759c313ce1df89e286947734"}
Sep 18 19:55:20.666210 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:55:20.666172    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-64cc2d85-774f947b78-bt69s" event={"ID":"3bc9bad8-f44f-4286-a267-69a521e9453b","Type":"ContainerStarted","Data":"a1b294abfc054f7adb919f2687b6867f9751a77b1803b1d6d1fa469bbba7e104"}
Sep 18 19:55:20.666553 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:55:20.666242    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-64cc2d85-774f947b78-bt69s"
Sep 18 19:55:20.684842 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:55:20.684780    2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-64cc2d85-774f947b78-bt69s" podStartSLOduration=1.5364756819999998 podStartE2EDuration="3.684762908s" totalImagesPullingTime="2.148287226s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 19:55:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 19:55:20.684334927 +0000 UTC m=+962.208940349" watchObservedRunningTime="2026-09-18 19:55:20.684762908 +0000 UTC m=+962.209368330"
Sep 18 19:55:25.776764 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:55:25.776728    2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-674263/simulator-w4-674263-kserve-78dd9c6fcf-zq5lx"]
Sep 18 19:55:25.781648 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:55:25.781630    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-674263/simulator-w4-674263-kserve-78dd9c6fcf-zq5lx"
Sep 18 19:55:25.783970 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:55:25.783946    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-674263\"/\"openshift-service-ca.crt\""
Sep 18 19:55:25.784819 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:55:25.784799    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-674263\"/\"default-dockercfg-nlxpd\""
Sep 18 19:55:25.784918 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:55:25.784835    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-674263\"/\"simulator-w4-674263-kserve-self-signed-certs\""
Sep 18 19:55:25.784918 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:55:25.784835    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-674263\"/\"kube-root-ca.crt\""
Sep 18 19:55:25.790568 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:55:25.790534    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-674263/simulator-w4-674263-kserve-78dd9c6fcf-zq5lx"]
Sep 18 19:55:25.901904 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:55:25.901869    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/51c71218-f838-4b23-a5bf-c5e274fa9c3d-model-cache\") pod \"simulator-w4-674263-kserve-78dd9c6fcf-zq5lx\" (UID: \"51c71218-f838-4b23-a5bf-c5e274fa9c3d\") " pod="e2e-models-e2e-worker-w4-674263/simulator-w4-674263-kserve-78dd9c6fcf-zq5lx"
Sep 18 19:55:25.902099 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:55:25.901911    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nn9s9\" (UniqueName: \"kubernetes.io/projected/51c71218-f838-4b23-a5bf-c5e274fa9c3d-kube-api-access-nn9s9\") pod \"simulator-w4-674263-kserve-78dd9c6fcf-zq5lx\" (UID: \"51c71218-f838-4b23-a5bf-c5e274fa9c3d\") " pod="e2e-models-e2e-worker-w4-674263/simulator-w4-674263-kserve-78dd9c6fcf-zq5lx"
Sep 18 19:55:25.902099 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:55:25.902050    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/51c71218-f838-4b23-a5bf-c5e274fa9c3d-dshm\") pod \"simulator-w4-674263-kserve-78dd9c6fcf-zq5lx\" (UID: \"51c71218-f838-4b23-a5bf-c5e274fa9c3d\") " pod="e2e-models-e2e-worker-w4-674263/simulator-w4-674263-kserve-78dd9c6fcf-zq5lx"
Sep 18 19:55:25.902099 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:55:25.902083    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/51c71218-f838-4b23-a5bf-c5e274fa9c3d-tmp-dir\") pod \"simulator-w4-674263-kserve-78dd9c6fcf-zq5lx\" (UID: \"51c71218-f838-4b23-a5bf-c5e274fa9c3d\") " pod="e2e-models-e2e-worker-w4-674263/simulator-w4-674263-kserve-78dd9c6fcf-zq5lx"
Sep 18 19:55:25.902229 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:55:25.902113    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/51c71218-f838-4b23-a5bf-c5e274fa9c3d-home\") pod \"simulator-w4-674263-kserve-78dd9c6fcf-zq5lx\" (UID: \"51c71218-f838-4b23-a5bf-c5e274fa9c3d\") " pod="e2e-models-e2e-worker-w4-674263/simulator-w4-674263-kserve-78dd9c6fcf-zq5lx"
Sep 18 19:55:25.902229 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:55:25.902190    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/51c71218-f838-4b23-a5bf-c5e274fa9c3d-tls-certs\") pod \"simulator-w4-674263-kserve-78dd9c6fcf-zq5lx\" (UID: \"51c71218-f838-4b23-a5bf-c5e274fa9c3d\") " pod="e2e-models-e2e-worker-w4-674263/simulator-w4-674263-kserve-78dd9c6fcf-zq5lx"
Sep 18 19:55:26.002957 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:55:26.002911    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/51c71218-f838-4b23-a5bf-c5e274fa9c3d-dshm\") pod \"simulator-w4-674263-kserve-78dd9c6fcf-zq5lx\" (UID: \"51c71218-f838-4b23-a5bf-c5e274fa9c3d\") " pod="e2e-models-e2e-worker-w4-674263/simulator-w4-674263-kserve-78dd9c6fcf-zq5lx"
Sep 18 19:55:26.002957 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:55:26.002951    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/51c71218-f838-4b23-a5bf-c5e274fa9c3d-tmp-dir\") pod \"simulator-w4-674263-kserve-78dd9c6fcf-zq5lx\" (UID: \"51c71218-f838-4b23-a5bf-c5e274fa9c3d\") " pod="e2e-models-e2e-worker-w4-674263/simulator-w4-674263-kserve-78dd9c6fcf-zq5lx"
Sep 18 19:55:26.003212 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:55:26.002976    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/51c71218-f838-4b23-a5bf-c5e274fa9c3d-home\") pod \"simulator-w4-674263-kserve-78dd9c6fcf-zq5lx\" (UID: \"51c71218-f838-4b23-a5bf-c5e274fa9c3d\") " pod="e2e-models-e2e-worker-w4-674263/simulator-w4-674263-kserve-78dd9c6fcf-zq5lx"
Sep 18 19:55:26.003212 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:55:26.003025    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/51c71218-f838-4b23-a5bf-c5e274fa9c3d-tls-certs\") pod \"simulator-w4-674263-kserve-78dd9c6fcf-zq5lx\" (UID: \"51c71218-f838-4b23-a5bf-c5e274fa9c3d\") " pod="e2e-models-e2e-worker-w4-674263/simulator-w4-674263-kserve-78dd9c6fcf-zq5lx"
Sep 18 19:55:26.003212 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:55:26.003069    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/51c71218-f838-4b23-a5bf-c5e274fa9c3d-model-cache\") pod \"simulator-w4-674263-kserve-78dd9c6fcf-zq5lx\" (UID: \"51c71218-f838-4b23-a5bf-c5e274fa9c3d\") " pod="e2e-models-e2e-worker-w4-674263/simulator-w4-674263-kserve-78dd9c6fcf-zq5lx"
Sep 18 19:55:26.003212 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:55:26.003109    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nn9s9\" (UniqueName: \"kubernetes.io/projected/51c71218-f838-4b23-a5bf-c5e274fa9c3d-kube-api-access-nn9s9\") pod \"simulator-w4-674263-kserve-78dd9c6fcf-zq5lx\" (UID: \"51c71218-f838-4b23-a5bf-c5e274fa9c3d\") " pod="e2e-models-e2e-worker-w4-674263/simulator-w4-674263-kserve-78dd9c6fcf-zq5lx"
Sep 18 19:55:26.003406 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:55:26.003383    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/51c71218-f838-4b23-a5bf-c5e274fa9c3d-tmp-dir\") pod \"simulator-w4-674263-kserve-78dd9c6fcf-zq5lx\" (UID: \"51c71218-f838-4b23-a5bf-c5e274fa9c3d\") " pod="e2e-models-e2e-worker-w4-674263/simulator-w4-674263-kserve-78dd9c6fcf-zq5lx"
Sep 18 19:55:26.003457 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:55:26.003429    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/51c71218-f838-4b23-a5bf-c5e274fa9c3d-home\") pod \"simulator-w4-674263-kserve-78dd9c6fcf-zq5lx\" (UID: \"51c71218-f838-4b23-a5bf-c5e274fa9c3d\") " pod="e2e-models-e2e-worker-w4-674263/simulator-w4-674263-kserve-78dd9c6fcf-zq5lx"
Sep 18 19:55:26.003517 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:55:26.003497    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/51c71218-f838-4b23-a5bf-c5e274fa9c3d-model-cache\") pod \"simulator-w4-674263-kserve-78dd9c6fcf-zq5lx\" (UID: \"51c71218-f838-4b23-a5bf-c5e274fa9c3d\") " pod="e2e-models-e2e-worker-w4-674263/simulator-w4-674263-kserve-78dd9c6fcf-zq5lx"
Sep 18 19:55:26.005308 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:55:26.005271    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/51c71218-f838-4b23-a5bf-c5e274fa9c3d-dshm\") pod \"simulator-w4-674263-kserve-78dd9c6fcf-zq5lx\" (UID: \"51c71218-f838-4b23-a5bf-c5e274fa9c3d\") " pod="e2e-models-e2e-worker-w4-674263/simulator-w4-674263-kserve-78dd9c6fcf-zq5lx"
Sep 18 19:55:26.005486 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:55:26.005468    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/51c71218-f838-4b23-a5bf-c5e274fa9c3d-tls-certs\") pod \"simulator-w4-674263-kserve-78dd9c6fcf-zq5lx\" (UID: \"51c71218-f838-4b23-a5bf-c5e274fa9c3d\") " pod="e2e-models-e2e-worker-w4-674263/simulator-w4-674263-kserve-78dd9c6fcf-zq5lx"
Sep 18 19:55:26.014456 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:55:26.014431    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nn9s9\" (UniqueName: \"kubernetes.io/projected/51c71218-f838-4b23-a5bf-c5e274fa9c3d-kube-api-access-nn9s9\") pod \"simulator-w4-674263-kserve-78dd9c6fcf-zq5lx\" (UID: \"51c71218-f838-4b23-a5bf-c5e274fa9c3d\") " pod="e2e-models-e2e-worker-w4-674263/simulator-w4-674263-kserve-78dd9c6fcf-zq5lx"
Sep 18 19:55:26.091584 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:55:26.091502    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-674263/simulator-w4-674263-kserve-78dd9c6fcf-zq5lx"
Sep 18 19:55:26.362567 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:55:26.362539    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-674263/simulator-w4-674263-kserve-78dd9c6fcf-zq5lx"]
Sep 18 19:55:26.364831 ip-10-0-136-185 kubenswrapper[2585]: W0918 19:55:26.364808    2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod51c71218_f838_4b23_a5bf_c5e274fa9c3d.slice/crio-e87f6529b00081ac8561743c4509ad463b10a55999cea884df96c8fe303aecaf WatchSource:0}: Error finding container e87f6529b00081ac8561743c4509ad463b10a55999cea884df96c8fe303aecaf: Status 404 returned error can't find the container with id e87f6529b00081ac8561743c4509ad463b10a55999cea884df96c8fe303aecaf
Sep 18 19:55:26.677098 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:55:26.677072    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-64cc2d85-774f947b78-bt69s"
Sep 18 19:55:26.696161 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:55:26.696130    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-674263/simulator-w4-674263-kserve-78dd9c6fcf-zq5lx" event={"ID":"51c71218-f838-4b23-a5bf-c5e274fa9c3d","Type":"ContainerStarted","Data":"2c1dd408b960480306c4e5853af2aeb0031b7170579759234de6e0e4f725da5b"}
Sep 18 19:55:26.696161 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:55:26.696168    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-674263/simulator-w4-674263-kserve-78dd9c6fcf-zq5lx" event={"ID":"51c71218-f838-4b23-a5bf-c5e274fa9c3d","Type":"ContainerStarted","Data":"e87f6529b00081ac8561743c4509ad463b10a55999cea884df96c8fe303aecaf"}
Sep 18 19:55:26.714890 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:55:26.714827    2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-674263/simulator-w4-674263-kserve-78dd9c6fcf-zq5lx" podStartSLOduration=1.714780419 podStartE2EDuration="1.714780419s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 19:55:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 19:55:26.71289968 +0000 UTC m=+968.237505094" watchObservedRunningTime="2026-09-18 19:55:26.714780419 +0000 UTC m=+968.239385843"
Sep 18 19:55:34.475217 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:55:34.475137    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-xmwh7"]
Sep 18 19:55:36.092313 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:55:36.092271    2585 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-674263/simulator-w4-674263-kserve-78dd9c6fcf-zq5lx"
Sep 18 19:55:36.092737 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:55:36.092361    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-674263/simulator-w4-674263-kserve-78dd9c6fcf-zq5lx"
Sep 18 19:55:36.104955 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:55:36.104928    2585 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-674263/simulator-w4-674263-kserve-78dd9c6fcf-zq5lx"
Sep 18 19:55:36.757331 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:55:36.757292    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-674263/simulator-w4-674263-kserve-78dd9c6fcf-zq5lx"
Sep 18 19:55:40.770883 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:55:40.770849    2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-d94b9e/simulator-w0-d94b9e-kserve-7669969b8-6p4m8"]
Sep 18 19:55:40.774336 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:55:40.774318    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-d94b9e/simulator-w0-d94b9e-kserve-7669969b8-6p4m8"
Sep 18 19:55:40.777484 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:55:40.777458    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-d94b9e\"/\"simulator-w0-d94b9e-kserve-self-signed-certs\""
Sep 18 19:55:40.777604 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:55:40.777483    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-d94b9e\"/\"kube-root-ca.crt\""
Sep 18 19:55:40.777604 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:55:40.777503    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-d94b9e\"/\"openshift-service-ca.crt\""
Sep 18 19:55:40.777604 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:55:40.777511    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-d94b9e\"/\"default-dockercfg-8wprj\""
Sep 18 19:55:40.782751 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:55:40.782728    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-d94b9e/simulator-w0-d94b9e-kserve-7669969b8-6p4m8"]
Sep 18 19:55:40.839854 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:55:40.839825    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cc71b9f8-e3bb-4704-98f7-0d07a7f6fe9b-model-cache\") pod \"simulator-w0-d94b9e-kserve-7669969b8-6p4m8\" (UID: \"cc71b9f8-e3bb-4704-98f7-0d07a7f6fe9b\") " pod="e2e-models-e2e-worker-w0-d94b9e/simulator-w0-d94b9e-kserve-7669969b8-6p4m8"
Sep 18 19:55:40.840030 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:55:40.839871    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mwwjm\" (UniqueName: \"kubernetes.io/projected/cc71b9f8-e3bb-4704-98f7-0d07a7f6fe9b-kube-api-access-mwwjm\") pod \"simulator-w0-d94b9e-kserve-7669969b8-6p4m8\" (UID: \"cc71b9f8-e3bb-4704-98f7-0d07a7f6fe9b\") " pod="e2e-models-e2e-worker-w0-d94b9e/simulator-w0-d94b9e-kserve-7669969b8-6p4m8"
Sep 18 19:55:40.840030 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:55:40.839891    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cc71b9f8-e3bb-4704-98f7-0d07a7f6fe9b-home\") pod \"simulator-w0-d94b9e-kserve-7669969b8-6p4m8\" (UID: \"cc71b9f8-e3bb-4704-98f7-0d07a7f6fe9b\") " pod="e2e-models-e2e-worker-w0-d94b9e/simulator-w0-d94b9e-kserve-7669969b8-6p4m8"
Sep 18 19:55:40.840030 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:55:40.839969    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cc71b9f8-e3bb-4704-98f7-0d07a7f6fe9b-dshm\") pod \"simulator-w0-d94b9e-kserve-7669969b8-6p4m8\" (UID: \"cc71b9f8-e3bb-4704-98f7-0d07a7f6fe9b\") " pod="e2e-models-e2e-worker-w0-d94b9e/simulator-w0-d94b9e-kserve-7669969b8-6p4m8"
Sep 18 19:55:40.840030 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:55:40.840001    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cc71b9f8-e3bb-4704-98f7-0d07a7f6fe9b-tmp-dir\") pod \"simulator-w0-d94b9e-kserve-7669969b8-6p4m8\" (UID: \"cc71b9f8-e3bb-4704-98f7-0d07a7f6fe9b\") " pod="e2e-models-e2e-worker-w0-d94b9e/simulator-w0-d94b9e-kserve-7669969b8-6p4m8"
Sep 18 19:55:40.840211 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:55:40.840056    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cc71b9f8-e3bb-4704-98f7-0d07a7f6fe9b-tls-certs\") pod \"simulator-w0-d94b9e-kserve-7669969b8-6p4m8\" (UID: \"cc71b9f8-e3bb-4704-98f7-0d07a7f6fe9b\") " pod="e2e-models-e2e-worker-w0-d94b9e/simulator-w0-d94b9e-kserve-7669969b8-6p4m8"
Sep 18 19:55:40.940979 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:55:40.940942    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cc71b9f8-e3bb-4704-98f7-0d07a7f6fe9b-tls-certs\") pod \"simulator-w0-d94b9e-kserve-7669969b8-6p4m8\" (UID: \"cc71b9f8-e3bb-4704-98f7-0d07a7f6fe9b\") " pod="e2e-models-e2e-worker-w0-d94b9e/simulator-w0-d94b9e-kserve-7669969b8-6p4m8"
Sep 18 19:55:40.941161 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:55:40.941014    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cc71b9f8-e3bb-4704-98f7-0d07a7f6fe9b-model-cache\") pod \"simulator-w0-d94b9e-kserve-7669969b8-6p4m8\" (UID: \"cc71b9f8-e3bb-4704-98f7-0d07a7f6fe9b\") " pod="e2e-models-e2e-worker-w0-d94b9e/simulator-w0-d94b9e-kserve-7669969b8-6p4m8"
Sep 18 19:55:40.941161 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:55:40.941068    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mwwjm\" (UniqueName: \"kubernetes.io/projected/cc71b9f8-e3bb-4704-98f7-0d07a7f6fe9b-kube-api-access-mwwjm\") pod \"simulator-w0-d94b9e-kserve-7669969b8-6p4m8\" (UID: \"cc71b9f8-e3bb-4704-98f7-0d07a7f6fe9b\") " pod="e2e-models-e2e-worker-w0-d94b9e/simulator-w0-d94b9e-kserve-7669969b8-6p4m8"
Sep 18 19:55:40.941161 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:55:40.941094    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cc71b9f8-e3bb-4704-98f7-0d07a7f6fe9b-home\") pod \"simulator-w0-d94b9e-kserve-7669969b8-6p4m8\" (UID: \"cc71b9f8-e3bb-4704-98f7-0d07a7f6fe9b\") " pod="e2e-models-e2e-worker-w0-d94b9e/simulator-w0-d94b9e-kserve-7669969b8-6p4m8"
Sep 18 19:55:40.941161 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:55:40.941142    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cc71b9f8-e3bb-4704-98f7-0d07a7f6fe9b-dshm\") pod \"simulator-w0-d94b9e-kserve-7669969b8-6p4m8\" (UID: \"cc71b9f8-e3bb-4704-98f7-0d07a7f6fe9b\") " pod="e2e-models-e2e-worker-w0-d94b9e/simulator-w0-d94b9e-kserve-7669969b8-6p4m8"
Sep 18 19:55:40.941362 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:55:40.941168    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cc71b9f8-e3bb-4704-98f7-0d07a7f6fe9b-tmp-dir\") pod \"simulator-w0-d94b9e-kserve-7669969b8-6p4m8\" (UID: \"cc71b9f8-e3bb-4704-98f7-0d07a7f6fe9b\") " pod="e2e-models-e2e-worker-w0-d94b9e/simulator-w0-d94b9e-kserve-7669969b8-6p4m8"
Sep 18 19:55:40.941499 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:55:40.941480    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cc71b9f8-e3bb-4704-98f7-0d07a7f6fe9b-model-cache\") pod \"simulator-w0-d94b9e-kserve-7669969b8-6p4m8\" (UID: \"cc71b9f8-e3bb-4704-98f7-0d07a7f6fe9b\") " pod="e2e-models-e2e-worker-w0-d94b9e/simulator-w0-d94b9e-kserve-7669969b8-6p4m8"
Sep 18 19:55:40.941499 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:55:40.941489    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cc71b9f8-e3bb-4704-98f7-0d07a7f6fe9b-home\") pod \"simulator-w0-d94b9e-kserve-7669969b8-6p4m8\" (UID: \"cc71b9f8-e3bb-4704-98f7-0d07a7f6fe9b\") " pod="e2e-models-e2e-worker-w0-d94b9e/simulator-w0-d94b9e-kserve-7669969b8-6p4m8"
Sep 18 19:55:40.941608 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:55:40.941534    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cc71b9f8-e3bb-4704-98f7-0d07a7f6fe9b-tmp-dir\") pod \"simulator-w0-d94b9e-kserve-7669969b8-6p4m8\" (UID: \"cc71b9f8-e3bb-4704-98f7-0d07a7f6fe9b\") " pod="e2e-models-e2e-worker-w0-d94b9e/simulator-w0-d94b9e-kserve-7669969b8-6p4m8"
Sep 18 19:55:40.943449 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:55:40.943433    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cc71b9f8-e3bb-4704-98f7-0d07a7f6fe9b-dshm\") pod \"simulator-w0-d94b9e-kserve-7669969b8-6p4m8\" (UID: \"cc71b9f8-e3bb-4704-98f7-0d07a7f6fe9b\") " pod="e2e-models-e2e-worker-w0-d94b9e/simulator-w0-d94b9e-kserve-7669969b8-6p4m8"
Sep 18 19:55:40.943630 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:55:40.943615    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cc71b9f8-e3bb-4704-98f7-0d07a7f6fe9b-tls-certs\") pod \"simulator-w0-d94b9e-kserve-7669969b8-6p4m8\" (UID: \"cc71b9f8-e3bb-4704-98f7-0d07a7f6fe9b\") " pod="e2e-models-e2e-worker-w0-d94b9e/simulator-w0-d94b9e-kserve-7669969b8-6p4m8"
Sep 18 19:55:40.952015 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:55:40.951987    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mwwjm\" (UniqueName: \"kubernetes.io/projected/cc71b9f8-e3bb-4704-98f7-0d07a7f6fe9b-kube-api-access-mwwjm\") pod \"simulator-w0-d94b9e-kserve-7669969b8-6p4m8\" (UID: \"cc71b9f8-e3bb-4704-98f7-0d07a7f6fe9b\") " pod="e2e-models-e2e-worker-w0-d94b9e/simulator-w0-d94b9e-kserve-7669969b8-6p4m8"
Sep 18 19:55:41.084248 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:55:41.084170    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-d94b9e/simulator-w0-d94b9e-kserve-7669969b8-6p4m8"
Sep 18 19:55:41.354740 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:55:41.354713    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-d94b9e/simulator-w0-d94b9e-kserve-7669969b8-6p4m8"]
Sep 18 19:55:41.357020 ip-10-0-136-185 kubenswrapper[2585]: W0918 19:55:41.356984    2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podcc71b9f8_e3bb_4704_98f7_0d07a7f6fe9b.slice/crio-1017d7175b6c07535113e05ccc8e1c3ae46033ef879c4accb2772549a2c80106 WatchSource:0}: Error finding container 1017d7175b6c07535113e05ccc8e1c3ae46033ef879c4accb2772549a2c80106: Status 404 returned error can't find the container with id 1017d7175b6c07535113e05ccc8e1c3ae46033ef879c4accb2772549a2c80106
Sep 18 19:55:41.772126 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:55:41.772089    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-d94b9e/simulator-w0-d94b9e-kserve-7669969b8-6p4m8" event={"ID":"cc71b9f8-e3bb-4704-98f7-0d07a7f6fe9b","Type":"ContainerStarted","Data":"ec0eb1f0385975278f3cd3cb69fbf9771ca5b678b8806381a50185b95029d696"}
Sep 18 19:55:41.772126 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:55:41.772132    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-d94b9e/simulator-w0-d94b9e-kserve-7669969b8-6p4m8" event={"ID":"cc71b9f8-e3bb-4704-98f7-0d07a7f6fe9b","Type":"ContainerStarted","Data":"1017d7175b6c07535113e05ccc8e1c3ae46033ef879c4accb2772549a2c80106"}
Sep 18 19:55:41.787673 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:55:41.787620    2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-d94b9e/simulator-w0-d94b9e-kserve-7669969b8-6p4m8" podStartSLOduration=1.787603037 podStartE2EDuration="1.787603037s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 19:55:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 19:55:41.787170992 +0000 UTC m=+983.311776415" watchObservedRunningTime="2026-09-18 19:55:41.787603037 +0000 UTC m=+983.312208460"
Sep 18 19:55:50.084164 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:55:50.084123    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-xmwh7"]
Sep 18 19:55:51.084707 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:55:51.084672    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-d94b9e/simulator-w0-d94b9e-kserve-7669969b8-6p4m8"
Sep 18 19:55:51.085214 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:55:51.084823    2585 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-d94b9e/simulator-w0-d94b9e-kserve-7669969b8-6p4m8"
Sep 18 19:55:51.097381 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:55:51.097350    2585 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-d94b9e/simulator-w0-d94b9e-kserve-7669969b8-6p4m8"
Sep 18 19:55:51.834647 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:55:51.834619    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-d94b9e/simulator-w0-d94b9e-kserve-7669969b8-6p4m8"
Sep 18 19:55:56.845011 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:55:56.844974    2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-64cc2d85-774f947b78-bt69s"]
Sep 18 19:55:56.845907 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:55:56.845280    2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-64cc2d85-774f947b78-bt69s" podUID="3bc9bad8-f44f-4286-a267-69a521e9453b" containerName="maas-api" containerID="cri-o://a1b294abfc054f7adb919f2687b6867f9751a77b1803b1d6d1fa469bbba7e104" gracePeriod=30
Sep 18 19:55:57.098365 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:55:57.098301    2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-64cc2d85-774f947b78-bt69s"
Sep 18 19:55:57.188215 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:55:57.188177    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3bc9bad8-f44f-4286-a267-69a521e9453b-metrics-certs\") pod \"3bc9bad8-f44f-4286-a267-69a521e9453b\" (UID: \"3bc9bad8-f44f-4286-a267-69a521e9453b\") "
Sep 18 19:55:57.188418 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:55:57.188224    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3bc9bad8-f44f-4286-a267-69a521e9453b-maas-api-tls\") pod \"3bc9bad8-f44f-4286-a267-69a521e9453b\" (UID: \"3bc9bad8-f44f-4286-a267-69a521e9453b\") "
Sep 18 19:55:57.188418 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:55:57.188295    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3bc9bad8-f44f-4286-a267-69a521e9453b-openshift-service-ca\") pod \"3bc9bad8-f44f-4286-a267-69a521e9453b\" (UID: \"3bc9bad8-f44f-4286-a267-69a521e9453b\") "
Sep 18 19:55:57.188418 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:55:57.188321    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-n26b2\" (UniqueName: \"kubernetes.io/projected/3bc9bad8-f44f-4286-a267-69a521e9453b-kube-api-access-n26b2\") pod \"3bc9bad8-f44f-4286-a267-69a521e9453b\" (UID: \"3bc9bad8-f44f-4286-a267-69a521e9453b\") "
Sep 18 19:55:57.188834 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:55:57.188770    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3bc9bad8-f44f-4286-a267-69a521e9453b-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "3bc9bad8-f44f-4286-a267-69a521e9453b" (UID: "3bc9bad8-f44f-4286-a267-69a521e9453b"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 19:55:57.192388 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:55:57.192356    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3bc9bad8-f44f-4286-a267-69a521e9453b-kube-api-access-n26b2" (OuterVolumeSpecName: "kube-api-access-n26b2") pod "3bc9bad8-f44f-4286-a267-69a521e9453b" (UID: "3bc9bad8-f44f-4286-a267-69a521e9453b"). InnerVolumeSpecName "kube-api-access-n26b2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 19:55:57.193736 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:55:57.193704    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3bc9bad8-f44f-4286-a267-69a521e9453b-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "3bc9bad8-f44f-4286-a267-69a521e9453b" (UID: "3bc9bad8-f44f-4286-a267-69a521e9453b"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 19:55:57.193736 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:55:57.193721    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3bc9bad8-f44f-4286-a267-69a521e9453b-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "3bc9bad8-f44f-4286-a267-69a521e9453b" (UID: "3bc9bad8-f44f-4286-a267-69a521e9453b"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 19:55:57.289882 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:55:57.289843    2585 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3bc9bad8-f44f-4286-a267-69a521e9453b-openshift-service-ca\") on node \"ip-10-0-136-185.ec2.internal\" DevicePath \"\""
Sep 18 19:55:57.289882 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:55:57.289879    2585 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n26b2\" (UniqueName: \"kubernetes.io/projected/3bc9bad8-f44f-4286-a267-69a521e9453b-kube-api-access-n26b2\") on node \"ip-10-0-136-185.ec2.internal\" DevicePath \"\""
Sep 18 19:55:57.290093 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:55:57.289896    2585 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3bc9bad8-f44f-4286-a267-69a521e9453b-metrics-certs\") on node \"ip-10-0-136-185.ec2.internal\" DevicePath \"\""
Sep 18 19:55:57.290093 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:55:57.289911    2585 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3bc9bad8-f44f-4286-a267-69a521e9453b-maas-api-tls\") on node \"ip-10-0-136-185.ec2.internal\" DevicePath \"\""
Sep 18 19:55:57.855759 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:55:57.855722    2585 generic.go:356] "Generic (PLEG): container finished" podID="3bc9bad8-f44f-4286-a267-69a521e9453b" containerID="a1b294abfc054f7adb919f2687b6867f9751a77b1803b1d6d1fa469bbba7e104" exitCode=0
Sep 18 19:55:57.856175 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:55:57.855783    2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-64cc2d85-774f947b78-bt69s"
Sep 18 19:55:57.856175 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:55:57.855813    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-64cc2d85-774f947b78-bt69s" event={"ID":"3bc9bad8-f44f-4286-a267-69a521e9453b","Type":"ContainerDied","Data":"a1b294abfc054f7adb919f2687b6867f9751a77b1803b1d6d1fa469bbba7e104"}
Sep 18 19:55:57.856175 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:55:57.855928    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-64cc2d85-774f947b78-bt69s" event={"ID":"3bc9bad8-f44f-4286-a267-69a521e9453b","Type":"ContainerDied","Data":"a4132bd10f7f7517b752ddcf58b48ce33869dd92759c313ce1df89e286947734"}
Sep 18 19:55:57.856175 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:55:57.855950    2585 scope.go:117] "RemoveContainer" containerID="a1b294abfc054f7adb919f2687b6867f9751a77b1803b1d6d1fa469bbba7e104"
Sep 18 19:55:57.865990 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:55:57.865967    2585 scope.go:117] "RemoveContainer" containerID="a1b294abfc054f7adb919f2687b6867f9751a77b1803b1d6d1fa469bbba7e104"
Sep 18 19:55:57.866282 ip-10-0-136-185 kubenswrapper[2585]: E0918 19:55:57.866263    2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a1b294abfc054f7adb919f2687b6867f9751a77b1803b1d6d1fa469bbba7e104\": container with ID starting with a1b294abfc054f7adb919f2687b6867f9751a77b1803b1d6d1fa469bbba7e104 not found: ID does not exist" containerID="a1b294abfc054f7adb919f2687b6867f9751a77b1803b1d6d1fa469bbba7e104"
Sep 18 19:55:57.866351 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:55:57.866290    2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a1b294abfc054f7adb919f2687b6867f9751a77b1803b1d6d1fa469bbba7e104"} err="failed to get container status \"a1b294abfc054f7adb919f2687b6867f9751a77b1803b1d6d1fa469bbba7e104\": rpc error: code = NotFound desc = could not find container \"a1b294abfc054f7adb919f2687b6867f9751a77b1803b1d6d1fa469bbba7e104\": container with ID starting with a1b294abfc054f7adb919f2687b6867f9751a77b1803b1d6d1fa469bbba7e104 not found: ID does not exist"
Sep 18 19:55:57.881539 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:55:57.881515    2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-64cc2d85-774f947b78-bt69s"]
Sep 18 19:55:57.888150 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:55:57.888125    2585 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-64cc2d85-774f947b78-bt69s"]
Sep 18 19:55:58.661588 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:55:58.661552    2585 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3bc9bad8-f44f-4286-a267-69a521e9453b" path="/var/lib/kubelet/pods/3bc9bad8-f44f-4286-a267-69a521e9453b/volumes"
Sep 18 19:56:02.394880 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:56:02.394848    2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-5b1253e5-74b6b7c4c8-f8nmn"]
Sep 18 19:56:02.395283 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:56:02.395267    2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3bc9bad8-f44f-4286-a267-69a521e9453b" containerName="maas-api"
Sep 18 19:56:02.395326 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:56:02.395286    2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="3bc9bad8-f44f-4286-a267-69a521e9453b" containerName="maas-api"
Sep 18 19:56:02.395360 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:56:02.395349    2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="3bc9bad8-f44f-4286-a267-69a521e9453b" containerName="maas-api"
Sep 18 19:56:02.399964 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:56:02.399937    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-5b1253e5-74b6b7c4c8-f8nmn"
Sep 18 19:56:02.402437 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:56:02.402405    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-5b1253e5-serving-cert\""
Sep 18 19:56:02.402575 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:56:02.402479    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Sep 18 19:56:02.402575 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:56:02.402485    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\""
Sep 18 19:56:02.403278 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:56:02.403258    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 18 19:56:02.403420 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:56:02.403392    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-jqjhf\""
Sep 18 19:56:02.413822 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:56:02.413778    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-5b1253e5-74b6b7c4c8-f8nmn"]
Sep 18 19:56:02.539575 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:56:02.539540    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/326a12a2-68f5-45f8-b175-392f97f5e3c2-openshift-service-ca\") pod \"maas-api-e2e-ait-5b1253e5-74b6b7c4c8-f8nmn\" (UID: \"326a12a2-68f5-45f8-b175-392f97f5e3c2\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-5b1253e5-74b6b7c4c8-f8nmn"
Sep 18 19:56:02.539743 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:56:02.539633    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9bgs2\" (UniqueName: \"kubernetes.io/projected/326a12a2-68f5-45f8-b175-392f97f5e3c2-kube-api-access-9bgs2\") pod \"maas-api-e2e-ait-5b1253e5-74b6b7c4c8-f8nmn\" (UID: \"326a12a2-68f5-45f8-b175-392f97f5e3c2\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-5b1253e5-74b6b7c4c8-f8nmn"
Sep 18 19:56:02.539743 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:56:02.539705    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/326a12a2-68f5-45f8-b175-392f97f5e3c2-metrics-certs\") pod \"maas-api-e2e-ait-5b1253e5-74b6b7c4c8-f8nmn\" (UID: \"326a12a2-68f5-45f8-b175-392f97f5e3c2\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-5b1253e5-74b6b7c4c8-f8nmn"
Sep 18 19:56:02.539743 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:56:02.539736    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/326a12a2-68f5-45f8-b175-392f97f5e3c2-maas-api-tls\") pod \"maas-api-e2e-ait-5b1253e5-74b6b7c4c8-f8nmn\" (UID: \"326a12a2-68f5-45f8-b175-392f97f5e3c2\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-5b1253e5-74b6b7c4c8-f8nmn"
Sep 18 19:56:02.641063 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:56:02.641023    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/326a12a2-68f5-45f8-b175-392f97f5e3c2-maas-api-tls\") pod \"maas-api-e2e-ait-5b1253e5-74b6b7c4c8-f8nmn\" (UID: \"326a12a2-68f5-45f8-b175-392f97f5e3c2\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-5b1253e5-74b6b7c4c8-f8nmn"
Sep 18 19:56:02.641235 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:56:02.641081    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/326a12a2-68f5-45f8-b175-392f97f5e3c2-openshift-service-ca\") pod \"maas-api-e2e-ait-5b1253e5-74b6b7c4c8-f8nmn\" (UID: \"326a12a2-68f5-45f8-b175-392f97f5e3c2\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-5b1253e5-74b6b7c4c8-f8nmn"
Sep 18 19:56:02.641235 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:56:02.641138    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9bgs2\" (UniqueName: \"kubernetes.io/projected/326a12a2-68f5-45f8-b175-392f97f5e3c2-kube-api-access-9bgs2\") pod \"maas-api-e2e-ait-5b1253e5-74b6b7c4c8-f8nmn\" (UID: \"326a12a2-68f5-45f8-b175-392f97f5e3c2\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-5b1253e5-74b6b7c4c8-f8nmn"
Sep 18 19:56:02.641235 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:56:02.641192    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/326a12a2-68f5-45f8-b175-392f97f5e3c2-metrics-certs\") pod \"maas-api-e2e-ait-5b1253e5-74b6b7c4c8-f8nmn\" (UID: \"326a12a2-68f5-45f8-b175-392f97f5e3c2\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-5b1253e5-74b6b7c4c8-f8nmn"
Sep 18 19:56:02.641866 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:56:02.641847    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/326a12a2-68f5-45f8-b175-392f97f5e3c2-openshift-service-ca\") pod \"maas-api-e2e-ait-5b1253e5-74b6b7c4c8-f8nmn\" (UID: \"326a12a2-68f5-45f8-b175-392f97f5e3c2\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-5b1253e5-74b6b7c4c8-f8nmn"
Sep 18 19:56:02.643558 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:56:02.643539    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/326a12a2-68f5-45f8-b175-392f97f5e3c2-maas-api-tls\") pod \"maas-api-e2e-ait-5b1253e5-74b6b7c4c8-f8nmn\" (UID: \"326a12a2-68f5-45f8-b175-392f97f5e3c2\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-5b1253e5-74b6b7c4c8-f8nmn"
Sep 18 19:56:02.643675 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:56:02.643603    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/326a12a2-68f5-45f8-b175-392f97f5e3c2-metrics-certs\") pod \"maas-api-e2e-ait-5b1253e5-74b6b7c4c8-f8nmn\" (UID: \"326a12a2-68f5-45f8-b175-392f97f5e3c2\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-5b1253e5-74b6b7c4c8-f8nmn"
Sep 18 19:56:02.651117 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:56:02.651042    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9bgs2\" (UniqueName: \"kubernetes.io/projected/326a12a2-68f5-45f8-b175-392f97f5e3c2-kube-api-access-9bgs2\") pod \"maas-api-e2e-ait-5b1253e5-74b6b7c4c8-f8nmn\" (UID: \"326a12a2-68f5-45f8-b175-392f97f5e3c2\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-5b1253e5-74b6b7c4c8-f8nmn"
Sep 18 19:56:02.711682 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:56:02.711643    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-5b1253e5-74b6b7c4c8-f8nmn"
Sep 18 19:56:02.994001 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:56:02.993972    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-5b1253e5-74b6b7c4c8-f8nmn"]
Sep 18 19:56:02.995896 ip-10-0-136-185 kubenswrapper[2585]: W0918 19:56:02.995867    2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod326a12a2_68f5_45f8_b175_392f97f5e3c2.slice/crio-aa4302d735c400e43f57cc716d509bbee1a920df0a69503a36de98139955d089 WatchSource:0}: Error finding container aa4302d735c400e43f57cc716d509bbee1a920df0a69503a36de98139955d089: Status 404 returned error can't find the container with id aa4302d735c400e43f57cc716d509bbee1a920df0a69503a36de98139955d089
Sep 18 19:56:03.892846 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:56:03.892770    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-5b1253e5-74b6b7c4c8-f8nmn" event={"ID":"326a12a2-68f5-45f8-b175-392f97f5e3c2","Type":"ContainerStarted","Data":"4a35eab48b73560ba0d5ed12ca921ee867009baa9363e004656eac9f89b26433"}
Sep 18 19:56:03.893246 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:56:03.892853    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-5b1253e5-74b6b7c4c8-f8nmn"
Sep 18 19:56:03.893246 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:56:03.892879    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-5b1253e5-74b6b7c4c8-f8nmn" event={"ID":"326a12a2-68f5-45f8-b175-392f97f5e3c2","Type":"ContainerStarted","Data":"aa4302d735c400e43f57cc716d509bbee1a920df0a69503a36de98139955d089"}
Sep 18 19:56:03.918102 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:56:03.918049    2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-5b1253e5-74b6b7c4c8-f8nmn" podStartSLOduration=1.495858714 podStartE2EDuration="1.918031727s" totalImagesPullingTime="422.173013ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 19:56:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 19:56:03.91498909 +0000 UTC m=+1005.439594512" watchObservedRunningTime="2026-09-18 19:56:03.918031727 +0000 UTC m=+1005.442637149"
Sep 18 19:56:04.990630 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:56:04.990598    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-xmwh7"]
Sep 18 19:56:09.903849 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:56:09.903817    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-5b1253e5-74b6b7c4c8-f8nmn"
Sep 18 19:56:15.454011 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:56:15.453975    2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-autores-1d073e/autores-model-1d073e-kserve-685c979499-mb8sc"]
Sep 18 19:56:15.457752 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:56:15.457730    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-autores-1d073e/autores-model-1d073e-kserve-685c979499-mb8sc"
Sep 18 19:56:15.460041 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:56:15.460022    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-autores-1d073e\"/\"openshift-service-ca.crt\""
Sep 18 19:56:15.460654 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:56:15.460639    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-autores-1d073e\"/\"kube-root-ca.crt\""
Sep 18 19:56:15.460724 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:56:15.460706    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-autores-1d073e\"/\"autores-model-1d073e-kserve-self-signed-certs\""
Sep 18 19:56:15.460777 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:56:15.460730    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-autores-1d073e\"/\"default-dockercfg-gtnfv\""
Sep 18 19:56:15.467659 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:56:15.467635    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-autores-1d073e/autores-model-1d073e-kserve-685c979499-mb8sc"]
Sep 18 19:56:15.567482 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:56:15.567441    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4f5b104a-8f72-4cd1-a292-ac8faa8e6391-home\") pod \"autores-model-1d073e-kserve-685c979499-mb8sc\" (UID: \"4f5b104a-8f72-4cd1-a292-ac8faa8e6391\") " pod="ai-tenant-e2e-autores-1d073e/autores-model-1d073e-kserve-685c979499-mb8sc"
Sep 18 19:56:15.567482 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:56:15.567480    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4f5b104a-8f72-4cd1-a292-ac8faa8e6391-tmp-dir\") pod \"autores-model-1d073e-kserve-685c979499-mb8sc\" (UID: \"4f5b104a-8f72-4cd1-a292-ac8faa8e6391\") " pod="ai-tenant-e2e-autores-1d073e/autores-model-1d073e-kserve-685c979499-mb8sc"
Sep 18 19:56:15.567693 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:56:15.567518    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4f5b104a-8f72-4cd1-a292-ac8faa8e6391-model-cache\") pod \"autores-model-1d073e-kserve-685c979499-mb8sc\" (UID: \"4f5b104a-8f72-4cd1-a292-ac8faa8e6391\") " pod="ai-tenant-e2e-autores-1d073e/autores-model-1d073e-kserve-685c979499-mb8sc"
Sep 18 19:56:15.567693 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:56:15.567537    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4f5b104a-8f72-4cd1-a292-ac8faa8e6391-dshm\") pod \"autores-model-1d073e-kserve-685c979499-mb8sc\" (UID: \"4f5b104a-8f72-4cd1-a292-ac8faa8e6391\") " pod="ai-tenant-e2e-autores-1d073e/autores-model-1d073e-kserve-685c979499-mb8sc"
Sep 18 19:56:15.567693 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:56:15.567599    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4f5b104a-8f72-4cd1-a292-ac8faa8e6391-tls-certs\") pod \"autores-model-1d073e-kserve-685c979499-mb8sc\" (UID: \"4f5b104a-8f72-4cd1-a292-ac8faa8e6391\") " pod="ai-tenant-e2e-autores-1d073e/autores-model-1d073e-kserve-685c979499-mb8sc"
Sep 18 19:56:15.567693 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:56:15.567645    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jp6fz\" (UniqueName: \"kubernetes.io/projected/4f5b104a-8f72-4cd1-a292-ac8faa8e6391-kube-api-access-jp6fz\") pod \"autores-model-1d073e-kserve-685c979499-mb8sc\" (UID: \"4f5b104a-8f72-4cd1-a292-ac8faa8e6391\") " pod="ai-tenant-e2e-autores-1d073e/autores-model-1d073e-kserve-685c979499-mb8sc"
Sep 18 19:56:15.668875 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:56:15.668839    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4f5b104a-8f72-4cd1-a292-ac8faa8e6391-home\") pod \"autores-model-1d073e-kserve-685c979499-mb8sc\" (UID: \"4f5b104a-8f72-4cd1-a292-ac8faa8e6391\") " pod="ai-tenant-e2e-autores-1d073e/autores-model-1d073e-kserve-685c979499-mb8sc"
Sep 18 19:56:15.669015 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:56:15.668884    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4f5b104a-8f72-4cd1-a292-ac8faa8e6391-tmp-dir\") pod \"autores-model-1d073e-kserve-685c979499-mb8sc\" (UID: \"4f5b104a-8f72-4cd1-a292-ac8faa8e6391\") " pod="ai-tenant-e2e-autores-1d073e/autores-model-1d073e-kserve-685c979499-mb8sc"
Sep 18 19:56:15.669015 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:56:15.668913    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4f5b104a-8f72-4cd1-a292-ac8faa8e6391-model-cache\") pod \"autores-model-1d073e-kserve-685c979499-mb8sc\" (UID: \"4f5b104a-8f72-4cd1-a292-ac8faa8e6391\") " pod="ai-tenant-e2e-autores-1d073e/autores-model-1d073e-kserve-685c979499-mb8sc"
Sep 18 19:56:15.669015 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:56:15.668930    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4f5b104a-8f72-4cd1-a292-ac8faa8e6391-dshm\") pod \"autores-model-1d073e-kserve-685c979499-mb8sc\" (UID: \"4f5b104a-8f72-4cd1-a292-ac8faa8e6391\") " pod="ai-tenant-e2e-autores-1d073e/autores-model-1d073e-kserve-685c979499-mb8sc"
Sep 18 19:56:15.669015 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:56:15.668956    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4f5b104a-8f72-4cd1-a292-ac8faa8e6391-tls-certs\") pod \"autores-model-1d073e-kserve-685c979499-mb8sc\" (UID: \"4f5b104a-8f72-4cd1-a292-ac8faa8e6391\") " pod="ai-tenant-e2e-autores-1d073e/autores-model-1d073e-kserve-685c979499-mb8sc"
Sep 18 19:56:15.669015 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:56:15.668983    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jp6fz\" (UniqueName: \"kubernetes.io/projected/4f5b104a-8f72-4cd1-a292-ac8faa8e6391-kube-api-access-jp6fz\") pod \"autores-model-1d073e-kserve-685c979499-mb8sc\" (UID: \"4f5b104a-8f72-4cd1-a292-ac8faa8e6391\") " pod="ai-tenant-e2e-autores-1d073e/autores-model-1d073e-kserve-685c979499-mb8sc"
Sep 18 19:56:15.669321 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:56:15.669287    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4f5b104a-8f72-4cd1-a292-ac8faa8e6391-home\") pod \"autores-model-1d073e-kserve-685c979499-mb8sc\" (UID: \"4f5b104a-8f72-4cd1-a292-ac8faa8e6391\") " pod="ai-tenant-e2e-autores-1d073e/autores-model-1d073e-kserve-685c979499-mb8sc"
Sep 18 19:56:15.669399 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:56:15.669322    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4f5b104a-8f72-4cd1-a292-ac8faa8e6391-model-cache\") pod \"autores-model-1d073e-kserve-685c979499-mb8sc\" (UID: \"4f5b104a-8f72-4cd1-a292-ac8faa8e6391\") " pod="ai-tenant-e2e-autores-1d073e/autores-model-1d073e-kserve-685c979499-mb8sc"
Sep 18 19:56:15.669399 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:56:15.669389    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4f5b104a-8f72-4cd1-a292-ac8faa8e6391-tmp-dir\") pod \"autores-model-1d073e-kserve-685c979499-mb8sc\" (UID: \"4f5b104a-8f72-4cd1-a292-ac8faa8e6391\") " pod="ai-tenant-e2e-autores-1d073e/autores-model-1d073e-kserve-685c979499-mb8sc"
Sep 18 19:56:15.671322 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:56:15.671300    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4f5b104a-8f72-4cd1-a292-ac8faa8e6391-dshm\") pod \"autores-model-1d073e-kserve-685c979499-mb8sc\" (UID: \"4f5b104a-8f72-4cd1-a292-ac8faa8e6391\") " pod="ai-tenant-e2e-autores-1d073e/autores-model-1d073e-kserve-685c979499-mb8sc"
Sep 18 19:56:15.671671 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:56:15.671643    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4f5b104a-8f72-4cd1-a292-ac8faa8e6391-tls-certs\") pod \"autores-model-1d073e-kserve-685c979499-mb8sc\" (UID: \"4f5b104a-8f72-4cd1-a292-ac8faa8e6391\") " pod="ai-tenant-e2e-autores-1d073e/autores-model-1d073e-kserve-685c979499-mb8sc"
Sep 18 19:56:15.678196 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:56:15.678172    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jp6fz\" (UniqueName: \"kubernetes.io/projected/4f5b104a-8f72-4cd1-a292-ac8faa8e6391-kube-api-access-jp6fz\") pod \"autores-model-1d073e-kserve-685c979499-mb8sc\" (UID: \"4f5b104a-8f72-4cd1-a292-ac8faa8e6391\") " pod="ai-tenant-e2e-autores-1d073e/autores-model-1d073e-kserve-685c979499-mb8sc"
Sep 18 19:56:15.768301 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:56:15.768207    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-autores-1d073e/autores-model-1d073e-kserve-685c979499-mb8sc"
Sep 18 19:56:16.043743 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:56:16.043656    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-autores-1d073e/autores-model-1d073e-kserve-685c979499-mb8sc"]
Sep 18 19:56:16.047403 ip-10-0-136-185 kubenswrapper[2585]: W0918 19:56:16.047369    2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod4f5b104a_8f72_4cd1_a292_ac8faa8e6391.slice/crio-c218889067496a0dd3d0c94a4e13dd813d99a8e4495b5257655166958f46b722 WatchSource:0}: Error finding container c218889067496a0dd3d0c94a4e13dd813d99a8e4495b5257655166958f46b722: Status 404 returned error can't find the container with id c218889067496a0dd3d0c94a4e13dd813d99a8e4495b5257655166958f46b722
Sep 18 19:56:16.959777 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:56:16.959735    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-autores-1d073e/autores-model-1d073e-kserve-685c979499-mb8sc" event={"ID":"4f5b104a-8f72-4cd1-a292-ac8faa8e6391","Type":"ContainerStarted","Data":"e237481b0cec53b690db549852463a9384d5511f89104a4b6f391740d20e20fe"}
Sep 18 19:56:16.960283 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:56:16.959783    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-autores-1d073e/autores-model-1d073e-kserve-685c979499-mb8sc" event={"ID":"4f5b104a-8f72-4cd1-a292-ac8faa8e6391","Type":"ContainerStarted","Data":"c218889067496a0dd3d0c94a4e13dd813d99a8e4495b5257655166958f46b722"}
Sep 18 19:56:16.977549 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:56:16.977488    2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-autores-1d073e/autores-model-1d073e-kserve-685c979499-mb8sc" podStartSLOduration=1.977471343 podStartE2EDuration="1.977471343s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 19:56:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 19:56:16.975733424 +0000 UTC m=+1018.500338847" watchObservedRunningTime="2026-09-18 19:56:16.977471343 +0000 UTC m=+1018.502076767"
Sep 18 19:56:23.175646 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:56:23.175601    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-xmwh7"]
Sep 18 19:56:24.368962 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:56:24.368922    2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-a-d6291d/test-model-d6291d-kserve-57fd894cc7-lnllx"]
Sep 18 19:56:24.374224 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:56:24.374199    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-d6291d/test-model-d6291d-kserve-57fd894cc7-lnllx"
Sep 18 19:56:24.377301 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:56:24.377269    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-d6291d\"/\"test-model-d6291d-kserve-self-signed-certs\""
Sep 18 19:56:24.377507 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:56:24.377484    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-d6291d\"/\"openshift-service-ca.crt\""
Sep 18 19:56:24.377597 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:56:24.377406    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-d6291d\"/\"default-dockercfg-z9hh2\""
Sep 18 19:56:24.377597 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:56:24.377350    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-d6291d\"/\"kube-root-ca.crt\""
Sep 18 19:56:24.379831 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:56:24.379804    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-d6291d/test-model-d6291d-kserve-57fd894cc7-lnllx"]
Sep 18 19:56:24.466349 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:56:24.466308    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f8356480-0e3a-4748-aa8b-af089ce729ef-model-cache\") pod \"test-model-d6291d-kserve-57fd894cc7-lnllx\" (UID: \"f8356480-0e3a-4748-aa8b-af089ce729ef\") " pod="ai-tenant-e2e-inf-a-d6291d/test-model-d6291d-kserve-57fd894cc7-lnllx"
Sep 18 19:56:24.466349 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:56:24.466350    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f8356480-0e3a-4748-aa8b-af089ce729ef-dshm\") pod \"test-model-d6291d-kserve-57fd894cc7-lnllx\" (UID: \"f8356480-0e3a-4748-aa8b-af089ce729ef\") " pod="ai-tenant-e2e-inf-a-d6291d/test-model-d6291d-kserve-57fd894cc7-lnllx"
Sep 18 19:56:24.466571 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:56:24.466479    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f8356480-0e3a-4748-aa8b-af089ce729ef-tmp-dir\") pod \"test-model-d6291d-kserve-57fd894cc7-lnllx\" (UID: \"f8356480-0e3a-4748-aa8b-af089ce729ef\") " pod="ai-tenant-e2e-inf-a-d6291d/test-model-d6291d-kserve-57fd894cc7-lnllx"
Sep 18 19:56:24.466571 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:56:24.466499    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f8356480-0e3a-4748-aa8b-af089ce729ef-home\") pod \"test-model-d6291d-kserve-57fd894cc7-lnllx\" (UID: \"f8356480-0e3a-4748-aa8b-af089ce729ef\") " pod="ai-tenant-e2e-inf-a-d6291d/test-model-d6291d-kserve-57fd894cc7-lnllx"
Sep 18 19:56:24.466571 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:56:24.466520    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t62f2\" (UniqueName: \"kubernetes.io/projected/f8356480-0e3a-4748-aa8b-af089ce729ef-kube-api-access-t62f2\") pod \"test-model-d6291d-kserve-57fd894cc7-lnllx\" (UID: \"f8356480-0e3a-4748-aa8b-af089ce729ef\") " pod="ai-tenant-e2e-inf-a-d6291d/test-model-d6291d-kserve-57fd894cc7-lnllx"
Sep 18 19:56:24.466571 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:56:24.466548    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f8356480-0e3a-4748-aa8b-af089ce729ef-tls-certs\") pod \"test-model-d6291d-kserve-57fd894cc7-lnllx\" (UID: \"f8356480-0e3a-4748-aa8b-af089ce729ef\") " pod="ai-tenant-e2e-inf-a-d6291d/test-model-d6291d-kserve-57fd894cc7-lnllx"
Sep 18 19:56:24.567640 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:56:24.567601    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f8356480-0e3a-4748-aa8b-af089ce729ef-tmp-dir\") pod \"test-model-d6291d-kserve-57fd894cc7-lnllx\" (UID: \"f8356480-0e3a-4748-aa8b-af089ce729ef\") " pod="ai-tenant-e2e-inf-a-d6291d/test-model-d6291d-kserve-57fd894cc7-lnllx"
Sep 18 19:56:24.567640 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:56:24.567637    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f8356480-0e3a-4748-aa8b-af089ce729ef-home\") pod \"test-model-d6291d-kserve-57fd894cc7-lnllx\" (UID: \"f8356480-0e3a-4748-aa8b-af089ce729ef\") " pod="ai-tenant-e2e-inf-a-d6291d/test-model-d6291d-kserve-57fd894cc7-lnllx"
Sep 18 19:56:24.567936 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:56:24.567663    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t62f2\" (UniqueName: \"kubernetes.io/projected/f8356480-0e3a-4748-aa8b-af089ce729ef-kube-api-access-t62f2\") pod \"test-model-d6291d-kserve-57fd894cc7-lnllx\" (UID: \"f8356480-0e3a-4748-aa8b-af089ce729ef\") " pod="ai-tenant-e2e-inf-a-d6291d/test-model-d6291d-kserve-57fd894cc7-lnllx"
Sep 18 19:56:24.567936 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:56:24.567814    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f8356480-0e3a-4748-aa8b-af089ce729ef-tls-certs\") pod \"test-model-d6291d-kserve-57fd894cc7-lnllx\" (UID: \"f8356480-0e3a-4748-aa8b-af089ce729ef\") " pod="ai-tenant-e2e-inf-a-d6291d/test-model-d6291d-kserve-57fd894cc7-lnllx"
Sep 18 19:56:24.567936 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:56:24.567887    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f8356480-0e3a-4748-aa8b-af089ce729ef-model-cache\") pod \"test-model-d6291d-kserve-57fd894cc7-lnllx\" (UID: \"f8356480-0e3a-4748-aa8b-af089ce729ef\") " pod="ai-tenant-e2e-inf-a-d6291d/test-model-d6291d-kserve-57fd894cc7-lnllx"
Sep 18 19:56:24.567936 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:56:24.567910    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f8356480-0e3a-4748-aa8b-af089ce729ef-dshm\") pod \"test-model-d6291d-kserve-57fd894cc7-lnllx\" (UID: \"f8356480-0e3a-4748-aa8b-af089ce729ef\") " pod="ai-tenant-e2e-inf-a-d6291d/test-model-d6291d-kserve-57fd894cc7-lnllx"
Sep 18 19:56:24.568148 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:56:24.568113    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f8356480-0e3a-4748-aa8b-af089ce729ef-home\") pod \"test-model-d6291d-kserve-57fd894cc7-lnllx\" (UID: \"f8356480-0e3a-4748-aa8b-af089ce729ef\") " pod="ai-tenant-e2e-inf-a-d6291d/test-model-d6291d-kserve-57fd894cc7-lnllx"
Sep 18 19:56:24.568148 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:56:24.568129    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f8356480-0e3a-4748-aa8b-af089ce729ef-tmp-dir\") pod \"test-model-d6291d-kserve-57fd894cc7-lnllx\" (UID: \"f8356480-0e3a-4748-aa8b-af089ce729ef\") " pod="ai-tenant-e2e-inf-a-d6291d/test-model-d6291d-kserve-57fd894cc7-lnllx"
Sep 18 19:56:24.568253 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:56:24.568198    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f8356480-0e3a-4748-aa8b-af089ce729ef-model-cache\") pod \"test-model-d6291d-kserve-57fd894cc7-lnllx\" (UID: \"f8356480-0e3a-4748-aa8b-af089ce729ef\") " pod="ai-tenant-e2e-inf-a-d6291d/test-model-d6291d-kserve-57fd894cc7-lnllx"
Sep 18 19:56:24.570340 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:56:24.570314    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f8356480-0e3a-4748-aa8b-af089ce729ef-dshm\") pod \"test-model-d6291d-kserve-57fd894cc7-lnllx\" (UID: \"f8356480-0e3a-4748-aa8b-af089ce729ef\") " pod="ai-tenant-e2e-inf-a-d6291d/test-model-d6291d-kserve-57fd894cc7-lnllx"
Sep 18 19:56:24.570482 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:56:24.570388    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f8356480-0e3a-4748-aa8b-af089ce729ef-tls-certs\") pod \"test-model-d6291d-kserve-57fd894cc7-lnllx\" (UID: \"f8356480-0e3a-4748-aa8b-af089ce729ef\") " pod="ai-tenant-e2e-inf-a-d6291d/test-model-d6291d-kserve-57fd894cc7-lnllx"
Sep 18 19:56:24.576338 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:56:24.576313    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-t62f2\" (UniqueName: \"kubernetes.io/projected/f8356480-0e3a-4748-aa8b-af089ce729ef-kube-api-access-t62f2\") pod \"test-model-d6291d-kserve-57fd894cc7-lnllx\" (UID: \"f8356480-0e3a-4748-aa8b-af089ce729ef\") " pod="ai-tenant-e2e-inf-a-d6291d/test-model-d6291d-kserve-57fd894cc7-lnllx"
Sep 18 19:56:24.686377 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:56:24.686340    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-d6291d/test-model-d6291d-kserve-57fd894cc7-lnllx"
Sep 18 19:56:24.958335 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:56:24.958308    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-d6291d/test-model-d6291d-kserve-57fd894cc7-lnllx"]
Sep 18 19:56:24.960712 ip-10-0-136-185 kubenswrapper[2585]: W0918 19:56:24.960686    2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf8356480_0e3a_4748_aa8b_af089ce729ef.slice/crio-a9013906413ddef8644e1948a249ec33edb349f72192ed08aecd633e5f563206 WatchSource:0}: Error finding container a9013906413ddef8644e1948a249ec33edb349f72192ed08aecd633e5f563206: Status 404 returned error can't find the container with id a9013906413ddef8644e1948a249ec33edb349f72192ed08aecd633e5f563206
Sep 18 19:56:25.001070 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:56:25.001037    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-d6291d/test-model-d6291d-kserve-57fd894cc7-lnllx" event={"ID":"f8356480-0e3a-4748-aa8b-af089ce729ef","Type":"ContainerStarted","Data":"a9013906413ddef8644e1948a249ec33edb349f72192ed08aecd633e5f563206"}
Sep 18 19:56:25.769344 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:56:25.769301    2585 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-autores-1d073e/autores-model-1d073e-kserve-685c979499-mb8sc"
Sep 18 19:56:25.769743 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:56:25.769356    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-autores-1d073e/autores-model-1d073e-kserve-685c979499-mb8sc"
Sep 18 19:56:25.782253 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:56:25.782227    2585 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-autores-1d073e/autores-model-1d073e-kserve-685c979499-mb8sc"
Sep 18 19:56:26.007981 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:56:26.007946    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-d6291d/test-model-d6291d-kserve-57fd894cc7-lnllx" event={"ID":"f8356480-0e3a-4748-aa8b-af089ce729ef","Type":"ContainerStarted","Data":"94cfc271d2a4d1495c14d6efd35a314ffe6b8bbafc2656c36db0a6d98cc1d13b"}
Sep 18 19:56:26.019542 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:56:26.019460    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-autores-1d073e/autores-model-1d073e-kserve-685c979499-mb8sc"
Sep 18 19:56:26.024203 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:56:26.024020    2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-inf-a-d6291d/test-model-d6291d-kserve-57fd894cc7-lnllx" podStartSLOduration=2.024000315 podStartE2EDuration="2.024000315s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 19:56:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 19:56:26.023079823 +0000 UTC m=+1027.547685247" watchObservedRunningTime="2026-09-18 19:56:26.024000315 +0000 UTC m=+1027.548605739"
Sep 18 19:56:31.711367 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:56:31.711319    2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-5b1253e5-74b6b7c4c8-f8nmn"]
Sep 18 19:56:31.711857 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:56:31.711578    2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-5b1253e5-74b6b7c4c8-f8nmn" podUID="326a12a2-68f5-45f8-b175-392f97f5e3c2" containerName="maas-api" containerID="cri-o://4a35eab48b73560ba0d5ed12ca921ee867009baa9363e004656eac9f89b26433" gracePeriod=30
Sep 18 19:56:31.959394 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:56:31.959365    2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-5b1253e5-74b6b7c4c8-f8nmn"
Sep 18 19:56:32.039047 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:56:32.038962    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9bgs2\" (UniqueName: \"kubernetes.io/projected/326a12a2-68f5-45f8-b175-392f97f5e3c2-kube-api-access-9bgs2\") pod \"326a12a2-68f5-45f8-b175-392f97f5e3c2\" (UID: \"326a12a2-68f5-45f8-b175-392f97f5e3c2\") "
Sep 18 19:56:32.039047 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:56:32.039011    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/326a12a2-68f5-45f8-b175-392f97f5e3c2-metrics-certs\") pod \"326a12a2-68f5-45f8-b175-392f97f5e3c2\" (UID: \"326a12a2-68f5-45f8-b175-392f97f5e3c2\") "
Sep 18 19:56:32.039265 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:56:32.039052    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/326a12a2-68f5-45f8-b175-392f97f5e3c2-openshift-service-ca\") pod \"326a12a2-68f5-45f8-b175-392f97f5e3c2\" (UID: \"326a12a2-68f5-45f8-b175-392f97f5e3c2\") "
Sep 18 19:56:32.039265 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:56:32.039079    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/326a12a2-68f5-45f8-b175-392f97f5e3c2-maas-api-tls\") pod \"326a12a2-68f5-45f8-b175-392f97f5e3c2\" (UID: \"326a12a2-68f5-45f8-b175-392f97f5e3c2\") "
Sep 18 19:56:32.039535 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:56:32.039494    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/326a12a2-68f5-45f8-b175-392f97f5e3c2-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "326a12a2-68f5-45f8-b175-392f97f5e3c2" (UID: "326a12a2-68f5-45f8-b175-392f97f5e3c2"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 19:56:32.040548 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:56:32.040499    2585 generic.go:356] "Generic (PLEG): container finished" podID="326a12a2-68f5-45f8-b175-392f97f5e3c2" containerID="4a35eab48b73560ba0d5ed12ca921ee867009baa9363e004656eac9f89b26433" exitCode=0
Sep 18 19:56:32.040548 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:56:32.040542    2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-5b1253e5-74b6b7c4c8-f8nmn"
Sep 18 19:56:32.040740 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:56:32.040591    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-5b1253e5-74b6b7c4c8-f8nmn" event={"ID":"326a12a2-68f5-45f8-b175-392f97f5e3c2","Type":"ContainerDied","Data":"4a35eab48b73560ba0d5ed12ca921ee867009baa9363e004656eac9f89b26433"}
Sep 18 19:56:32.040740 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:56:32.040656    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-5b1253e5-74b6b7c4c8-f8nmn" event={"ID":"326a12a2-68f5-45f8-b175-392f97f5e3c2","Type":"ContainerDied","Data":"aa4302d735c400e43f57cc716d509bbee1a920df0a69503a36de98139955d089"}
Sep 18 19:56:32.040740 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:56:32.040689    2585 scope.go:117] "RemoveContainer" containerID="4a35eab48b73560ba0d5ed12ca921ee867009baa9363e004656eac9f89b26433"
Sep 18 19:56:32.041580 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:56:32.041550    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/326a12a2-68f5-45f8-b175-392f97f5e3c2-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "326a12a2-68f5-45f8-b175-392f97f5e3c2" (UID: "326a12a2-68f5-45f8-b175-392f97f5e3c2"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 19:56:32.041693 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:56:32.041673    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/326a12a2-68f5-45f8-b175-392f97f5e3c2-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "326a12a2-68f5-45f8-b175-392f97f5e3c2" (UID: "326a12a2-68f5-45f8-b175-392f97f5e3c2"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 19:56:32.041745 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:56:32.041707    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/326a12a2-68f5-45f8-b175-392f97f5e3c2-kube-api-access-9bgs2" (OuterVolumeSpecName: "kube-api-access-9bgs2") pod "326a12a2-68f5-45f8-b175-392f97f5e3c2" (UID: "326a12a2-68f5-45f8-b175-392f97f5e3c2"). InnerVolumeSpecName "kube-api-access-9bgs2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 19:56:32.059821 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:56:32.059777    2585 scope.go:117] "RemoveContainer" containerID="4a35eab48b73560ba0d5ed12ca921ee867009baa9363e004656eac9f89b26433"
Sep 18 19:56:32.060087 ip-10-0-136-185 kubenswrapper[2585]: E0918 19:56:32.060064    2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4a35eab48b73560ba0d5ed12ca921ee867009baa9363e004656eac9f89b26433\": container with ID starting with 4a35eab48b73560ba0d5ed12ca921ee867009baa9363e004656eac9f89b26433 not found: ID does not exist" containerID="4a35eab48b73560ba0d5ed12ca921ee867009baa9363e004656eac9f89b26433"
Sep 18 19:56:32.060137 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:56:32.060095    2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4a35eab48b73560ba0d5ed12ca921ee867009baa9363e004656eac9f89b26433"} err="failed to get container status \"4a35eab48b73560ba0d5ed12ca921ee867009baa9363e004656eac9f89b26433\": rpc error: code = NotFound desc = could not find container \"4a35eab48b73560ba0d5ed12ca921ee867009baa9363e004656eac9f89b26433\": container with ID starting with 4a35eab48b73560ba0d5ed12ca921ee867009baa9363e004656eac9f89b26433 not found: ID does not exist"
Sep 18 19:56:32.140927 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:56:32.140891    2585 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9bgs2\" (UniqueName: \"kubernetes.io/projected/326a12a2-68f5-45f8-b175-392f97f5e3c2-kube-api-access-9bgs2\") on node \"ip-10-0-136-185.ec2.internal\" DevicePath \"\""
Sep 18 19:56:32.140927 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:56:32.140922    2585 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/326a12a2-68f5-45f8-b175-392f97f5e3c2-metrics-certs\") on node \"ip-10-0-136-185.ec2.internal\" DevicePath \"\""
Sep 18 19:56:32.140927 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:56:32.140932    2585 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/326a12a2-68f5-45f8-b175-392f97f5e3c2-openshift-service-ca\") on node \"ip-10-0-136-185.ec2.internal\" DevicePath \"\""
Sep 18 19:56:32.141154 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:56:32.140940    2585 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/326a12a2-68f5-45f8-b175-392f97f5e3c2-maas-api-tls\") on node \"ip-10-0-136-185.ec2.internal\" DevicePath \"\""
Sep 18 19:56:32.378251 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:56:32.378218    2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-5b1253e5-74b6b7c4c8-f8nmn"]
Sep 18 19:56:32.385689 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:56:32.385659    2585 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-5b1253e5-74b6b7c4c8-f8nmn"]
Sep 18 19:56:32.661951 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:56:32.661921    2585 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="326a12a2-68f5-45f8-b175-392f97f5e3c2" path="/var/lib/kubelet/pods/326a12a2-68f5-45f8-b175-392f97f5e3c2/volumes"
Sep 18 19:56:34.687342 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:56:34.687304    2585 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-a-d6291d/test-model-d6291d-kserve-57fd894cc7-lnllx"
Sep 18 19:56:34.687810 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:56:34.687374    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-d6291d/test-model-d6291d-kserve-57fd894cc7-lnllx"
Sep 18 19:56:34.700197 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:56:34.700174    2585 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-a-d6291d/test-model-d6291d-kserve-57fd894cc7-lnllx"
Sep 18 19:56:35.067621 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:56:35.067533    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-d6291d/test-model-d6291d-kserve-57fd894cc7-lnllx"
Sep 18 19:56:44.877367 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:56:44.877327    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-xmwh7"]
Sep 18 19:57:00.591843 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:57:00.591808    2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-autores-1d073e/autores-model-1d073e-kserve-685c979499-mb8sc"]
Sep 18 19:57:01.295985 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:57:01.295949    2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-1d073e-dfd798d4f-gtgfh"]
Sep 18 19:57:01.296278 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:57:01.296222    2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-autores-1d073e-dfd798d4f-gtgfh" podUID="0e6c1362-1133-4772-86c9-b9a0b4378cb5" containerName="payload-processing" containerID="cri-o://baef9a1d360a0a33607fc99c4fcef2ee628f9ae86a87df812b8c26cb056b99fd" gracePeriod=30
Sep 18 19:57:01.303449 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:57:01.303418    2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-1d073e-86f694cfb6-xnhf9"]
Sep 18 19:57:01.303834 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:57:01.303656    2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-autores-1d073e-86f694cfb6-xnhf9" podUID="522fc879-ad65-4bf9-8018-ead5194d612d" containerName="payload-pre-processing" containerID="cri-o://b954c8c66fd99d2b85df79253194228865631811d840b81def83b75b26690950" gracePeriod=30
Sep 18 19:57:01.584472 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:57:01.584442    2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-autores-1d073e-dfd798d4f-gtgfh"
Sep 18 19:57:01.587196 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:57:01.587174    2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-autores-1d073e-86f694cfb6-xnhf9"
Sep 18 19:57:01.619236 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:57:01.619036    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0e6c1362-1133-4772-86c9-b9a0b4378cb5-plugins-config-volume\") pod \"0e6c1362-1133-4772-86c9-b9a0b4378cb5\" (UID: \"0e6c1362-1133-4772-86c9-b9a0b4378cb5\") "
Sep 18 19:57:01.619236 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:57:01.619089    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-98dfl\" (UniqueName: \"kubernetes.io/projected/0e6c1362-1133-4772-86c9-b9a0b4378cb5-kube-api-access-98dfl\") pod \"0e6c1362-1133-4772-86c9-b9a0b4378cb5\" (UID: \"0e6c1362-1133-4772-86c9-b9a0b4378cb5\") "
Sep 18 19:57:01.619236 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:57:01.619125    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/522fc879-ad65-4bf9-8018-ead5194d612d-plugins-config-volume\") pod \"522fc879-ad65-4bf9-8018-ead5194d612d\" (UID: \"522fc879-ad65-4bf9-8018-ead5194d612d\") "
Sep 18 19:57:01.619236 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:57:01.619170    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gmvsg\" (UniqueName: \"kubernetes.io/projected/522fc879-ad65-4bf9-8018-ead5194d612d-kube-api-access-gmvsg\") pod \"522fc879-ad65-4bf9-8018-ead5194d612d\" (UID: \"522fc879-ad65-4bf9-8018-ead5194d612d\") "
Sep 18 19:57:01.620131 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:57:01.620102    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0e6c1362-1133-4772-86c9-b9a0b4378cb5-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "0e6c1362-1133-4772-86c9-b9a0b4378cb5" (UID: "0e6c1362-1133-4772-86c9-b9a0b4378cb5"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 19:57:01.620731 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:57:01.620576    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/522fc879-ad65-4bf9-8018-ead5194d612d-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "522fc879-ad65-4bf9-8018-ead5194d612d" (UID: "522fc879-ad65-4bf9-8018-ead5194d612d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 19:57:01.622045 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:57:01.622011    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/522fc879-ad65-4bf9-8018-ead5194d612d-kube-api-access-gmvsg" (OuterVolumeSpecName: "kube-api-access-gmvsg") pod "522fc879-ad65-4bf9-8018-ead5194d612d" (UID: "522fc879-ad65-4bf9-8018-ead5194d612d"). InnerVolumeSpecName "kube-api-access-gmvsg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 19:57:01.622517 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:57:01.622493    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0e6c1362-1133-4772-86c9-b9a0b4378cb5-kube-api-access-98dfl" (OuterVolumeSpecName: "kube-api-access-98dfl") pod "0e6c1362-1133-4772-86c9-b9a0b4378cb5" (UID: "0e6c1362-1133-4772-86c9-b9a0b4378cb5"). InnerVolumeSpecName "kube-api-access-98dfl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 19:57:01.720343 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:57:01.720303    2585 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0e6c1362-1133-4772-86c9-b9a0b4378cb5-plugins-config-volume\") on node \"ip-10-0-136-185.ec2.internal\" DevicePath \"\""
Sep 18 19:57:01.720343 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:57:01.720333    2585 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-98dfl\" (UniqueName: \"kubernetes.io/projected/0e6c1362-1133-4772-86c9-b9a0b4378cb5-kube-api-access-98dfl\") on node \"ip-10-0-136-185.ec2.internal\" DevicePath \"\""
Sep 18 19:57:01.720343 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:57:01.720343    2585 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/522fc879-ad65-4bf9-8018-ead5194d612d-plugins-config-volume\") on node \"ip-10-0-136-185.ec2.internal\" DevicePath \"\""
Sep 18 19:57:01.720343 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:57:01.720351    2585 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gmvsg\" (UniqueName: \"kubernetes.io/projected/522fc879-ad65-4bf9-8018-ead5194d612d-kube-api-access-gmvsg\") on node \"ip-10-0-136-185.ec2.internal\" DevicePath \"\""
Sep 18 19:57:02.207548 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:57:02.207508    2585 generic.go:356] "Generic (PLEG): container finished" podID="522fc879-ad65-4bf9-8018-ead5194d612d" containerID="b954c8c66fd99d2b85df79253194228865631811d840b81def83b75b26690950" exitCode=0
Sep 18 19:57:02.207548 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:57:02.207551    2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-autores-1d073e-86f694cfb6-xnhf9"
Sep 18 19:57:02.207776 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:57:02.207589    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-autores-1d073e-86f694cfb6-xnhf9" event={"ID":"522fc879-ad65-4bf9-8018-ead5194d612d","Type":"ContainerDied","Data":"b954c8c66fd99d2b85df79253194228865631811d840b81def83b75b26690950"}
Sep 18 19:57:02.207776 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:57:02.207639    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-autores-1d073e-86f694cfb6-xnhf9" event={"ID":"522fc879-ad65-4bf9-8018-ead5194d612d","Type":"ContainerDied","Data":"b53f95a0284f2052e94a2d2664b46f09af7314256ebf262aeb8e66de3a34870c"}
Sep 18 19:57:02.207776 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:57:02.207662    2585 scope.go:117] "RemoveContainer" containerID="b954c8c66fd99d2b85df79253194228865631811d840b81def83b75b26690950"
Sep 18 19:57:02.208963 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:57:02.208940    2585 generic.go:356] "Generic (PLEG): container finished" podID="0e6c1362-1133-4772-86c9-b9a0b4378cb5" containerID="baef9a1d360a0a33607fc99c4fcef2ee628f9ae86a87df812b8c26cb056b99fd" exitCode=0
Sep 18 19:57:02.209083 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:57:02.208981    2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-autores-1d073e-dfd798d4f-gtgfh"
Sep 18 19:57:02.209083 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:57:02.208989    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-autores-1d073e-dfd798d4f-gtgfh" event={"ID":"0e6c1362-1133-4772-86c9-b9a0b4378cb5","Type":"ContainerDied","Data":"baef9a1d360a0a33607fc99c4fcef2ee628f9ae86a87df812b8c26cb056b99fd"}
Sep 18 19:57:02.209083 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:57:02.209027    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-autores-1d073e-dfd798d4f-gtgfh" event={"ID":"0e6c1362-1133-4772-86c9-b9a0b4378cb5","Type":"ContainerDied","Data":"2783aab9a5947a1005f8fbdf1ba1889107002438a8dbdd39d2994e94df670b23"}
Sep 18 19:57:02.217141 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:57:02.217118    2585 scope.go:117] "RemoveContainer" containerID="b954c8c66fd99d2b85df79253194228865631811d840b81def83b75b26690950"
Sep 18 19:57:02.217429 ip-10-0-136-185 kubenswrapper[2585]: E0918 19:57:02.217411    2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b954c8c66fd99d2b85df79253194228865631811d840b81def83b75b26690950\": container with ID starting with b954c8c66fd99d2b85df79253194228865631811d840b81def83b75b26690950 not found: ID does not exist" containerID="b954c8c66fd99d2b85df79253194228865631811d840b81def83b75b26690950"
Sep 18 19:57:02.217506 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:57:02.217437    2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b954c8c66fd99d2b85df79253194228865631811d840b81def83b75b26690950"} err="failed to get container status \"b954c8c66fd99d2b85df79253194228865631811d840b81def83b75b26690950\": rpc error: code = NotFound desc = could not find container \"b954c8c66fd99d2b85df79253194228865631811d840b81def83b75b26690950\": container with ID starting with b954c8c66fd99d2b85df79253194228865631811d840b81def83b75b26690950 not found: ID does not exist"
Sep 18 19:57:02.217506 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:57:02.217453    2585 scope.go:117] "RemoveContainer" containerID="baef9a1d360a0a33607fc99c4fcef2ee628f9ae86a87df812b8c26cb056b99fd"
Sep 18 19:57:02.225598 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:57:02.225582    2585 scope.go:117] "RemoveContainer" containerID="baef9a1d360a0a33607fc99c4fcef2ee628f9ae86a87df812b8c26cb056b99fd"
Sep 18 19:57:02.225906 ip-10-0-136-185 kubenswrapper[2585]: E0918 19:57:02.225881    2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"baef9a1d360a0a33607fc99c4fcef2ee628f9ae86a87df812b8c26cb056b99fd\": container with ID starting with baef9a1d360a0a33607fc99c4fcef2ee628f9ae86a87df812b8c26cb056b99fd not found: ID does not exist" containerID="baef9a1d360a0a33607fc99c4fcef2ee628f9ae86a87df812b8c26cb056b99fd"
Sep 18 19:57:02.225974 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:57:02.225917    2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"baef9a1d360a0a33607fc99c4fcef2ee628f9ae86a87df812b8c26cb056b99fd"} err="failed to get container status \"baef9a1d360a0a33607fc99c4fcef2ee628f9ae86a87df812b8c26cb056b99fd\": rpc error: code = NotFound desc = could not find container \"baef9a1d360a0a33607fc99c4fcef2ee628f9ae86a87df812b8c26cb056b99fd\": container with ID starting with baef9a1d360a0a33607fc99c4fcef2ee628f9ae86a87df812b8c26cb056b99fd not found: ID does not exist"
Sep 18 19:57:02.232109 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:57:02.232089    2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-1d073e-86f694cfb6-xnhf9"]
Sep 18 19:57:02.237782 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:57:02.237759    2585 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-1d073e-86f694cfb6-xnhf9"]
Sep 18 19:57:02.248329 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:57:02.248306    2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-1d073e-dfd798d4f-gtgfh"]
Sep 18 19:57:02.252479 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:57:02.252457    2585 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-1d073e-dfd798d4f-gtgfh"]
Sep 18 19:57:02.662563 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:57:02.662529    2585 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0e6c1362-1133-4772-86c9-b9a0b4378cb5" path="/var/lib/kubelet/pods/0e6c1362-1133-4772-86c9-b9a0b4378cb5/volumes"
Sep 18 19:57:02.662962 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:57:02.662947    2585 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="522fc879-ad65-4bf9-8018-ead5194d612d" path="/var/lib/kubelet/pods/522fc879-ad65-4bf9-8018-ead5194d612d/volumes"
Sep 18 19:57:07.383515 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:57:07.383478    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-xmwh7"]
Sep 18 19:57:15.611466 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:57:15.611428    2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-autores-1d073e/autores-model-1d073e-kserve-685c979499-mb8sc" podUID="4f5b104a-8f72-4cd1-a292-ac8faa8e6391" containerName="main" containerID="cri-o://e237481b0cec53b690db549852463a9384d5511f89104a4b6f391740d20e20fe" gracePeriod=45
Sep 18 19:57:15.866640 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:57:15.866574    2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-autores-1d073e/autores-model-1d073e-kserve-685c979499-mb8sc"
Sep 18 19:57:15.960487 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:57:15.960448    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4f5b104a-8f72-4cd1-a292-ac8faa8e6391-home\") pod \"4f5b104a-8f72-4cd1-a292-ac8faa8e6391\" (UID: \"4f5b104a-8f72-4cd1-a292-ac8faa8e6391\") "
Sep 18 19:57:15.960487 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:57:15.960495    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4f5b104a-8f72-4cd1-a292-ac8faa8e6391-tmp-dir\") pod \"4f5b104a-8f72-4cd1-a292-ac8faa8e6391\" (UID: \"4f5b104a-8f72-4cd1-a292-ac8faa8e6391\") "
Sep 18 19:57:15.960757 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:57:15.960519    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jp6fz\" (UniqueName: \"kubernetes.io/projected/4f5b104a-8f72-4cd1-a292-ac8faa8e6391-kube-api-access-jp6fz\") pod \"4f5b104a-8f72-4cd1-a292-ac8faa8e6391\" (UID: \"4f5b104a-8f72-4cd1-a292-ac8faa8e6391\") "
Sep 18 19:57:15.960757 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:57:15.960567    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4f5b104a-8f72-4cd1-a292-ac8faa8e6391-tls-certs\") pod \"4f5b104a-8f72-4cd1-a292-ac8faa8e6391\" (UID: \"4f5b104a-8f72-4cd1-a292-ac8faa8e6391\") "
Sep 18 19:57:15.960757 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:57:15.960686    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4f5b104a-8f72-4cd1-a292-ac8faa8e6391-model-cache\") pod \"4f5b104a-8f72-4cd1-a292-ac8faa8e6391\" (UID: \"4f5b104a-8f72-4cd1-a292-ac8faa8e6391\") "
Sep 18 19:57:15.960757 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:57:15.960756    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4f5b104a-8f72-4cd1-a292-ac8faa8e6391-dshm\") pod \"4f5b104a-8f72-4cd1-a292-ac8faa8e6391\" (UID: \"4f5b104a-8f72-4cd1-a292-ac8faa8e6391\") "
Sep 18 19:57:15.960990 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:57:15.960827    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4f5b104a-8f72-4cd1-a292-ac8faa8e6391-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "4f5b104a-8f72-4cd1-a292-ac8faa8e6391" (UID: "4f5b104a-8f72-4cd1-a292-ac8faa8e6391"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 19:57:15.960990 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:57:15.960840    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4f5b104a-8f72-4cd1-a292-ac8faa8e6391-home" (OuterVolumeSpecName: "home") pod "4f5b104a-8f72-4cd1-a292-ac8faa8e6391" (UID: "4f5b104a-8f72-4cd1-a292-ac8faa8e6391"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 19:57:15.961067 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:57:15.960995    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4f5b104a-8f72-4cd1-a292-ac8faa8e6391-model-cache" (OuterVolumeSpecName: "model-cache") pod "4f5b104a-8f72-4cd1-a292-ac8faa8e6391" (UID: "4f5b104a-8f72-4cd1-a292-ac8faa8e6391"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 19:57:15.961108 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:57:15.961100    2585 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4f5b104a-8f72-4cd1-a292-ac8faa8e6391-model-cache\") on node \"ip-10-0-136-185.ec2.internal\" DevicePath \"\""
Sep 18 19:57:15.961156 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:57:15.961112    2585 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4f5b104a-8f72-4cd1-a292-ac8faa8e6391-home\") on node \"ip-10-0-136-185.ec2.internal\" DevicePath \"\""
Sep 18 19:57:15.961156 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:57:15.961122    2585 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4f5b104a-8f72-4cd1-a292-ac8faa8e6391-tmp-dir\") on node \"ip-10-0-136-185.ec2.internal\" DevicePath \"\""
Sep 18 19:57:15.962896 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:57:15.962875    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4f5b104a-8f72-4cd1-a292-ac8faa8e6391-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "4f5b104a-8f72-4cd1-a292-ac8faa8e6391" (UID: "4f5b104a-8f72-4cd1-a292-ac8faa8e6391"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 19:57:15.962969 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:57:15.962923    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4f5b104a-8f72-4cd1-a292-ac8faa8e6391-dshm" (OuterVolumeSpecName: "dshm") pod "4f5b104a-8f72-4cd1-a292-ac8faa8e6391" (UID: "4f5b104a-8f72-4cd1-a292-ac8faa8e6391"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 19:57:15.963017 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:57:15.962999    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4f5b104a-8f72-4cd1-a292-ac8faa8e6391-kube-api-access-jp6fz" (OuterVolumeSpecName: "kube-api-access-jp6fz") pod "4f5b104a-8f72-4cd1-a292-ac8faa8e6391" (UID: "4f5b104a-8f72-4cd1-a292-ac8faa8e6391"). InnerVolumeSpecName "kube-api-access-jp6fz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 19:57:16.062270 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:57:16.062217    2585 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jp6fz\" (UniqueName: \"kubernetes.io/projected/4f5b104a-8f72-4cd1-a292-ac8faa8e6391-kube-api-access-jp6fz\") on node \"ip-10-0-136-185.ec2.internal\" DevicePath \"\""
Sep 18 19:57:16.062270 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:57:16.062265    2585 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4f5b104a-8f72-4cd1-a292-ac8faa8e6391-tls-certs\") on node \"ip-10-0-136-185.ec2.internal\" DevicePath \"\""
Sep 18 19:57:16.062270 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:57:16.062277    2585 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4f5b104a-8f72-4cd1-a292-ac8faa8e6391-dshm\") on node \"ip-10-0-136-185.ec2.internal\" DevicePath \"\""
Sep 18 19:57:16.281256 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:57:16.281219    2585 generic.go:356] "Generic (PLEG): container finished" podID="4f5b104a-8f72-4cd1-a292-ac8faa8e6391" containerID="e237481b0cec53b690db549852463a9384d5511f89104a4b6f391740d20e20fe" exitCode=0
Sep 18 19:57:16.281442 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:57:16.281275    2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-autores-1d073e/autores-model-1d073e-kserve-685c979499-mb8sc"
Sep 18 19:57:16.281442 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:57:16.281317    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-autores-1d073e/autores-model-1d073e-kserve-685c979499-mb8sc" event={"ID":"4f5b104a-8f72-4cd1-a292-ac8faa8e6391","Type":"ContainerDied","Data":"e237481b0cec53b690db549852463a9384d5511f89104a4b6f391740d20e20fe"}
Sep 18 19:57:16.281442 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:57:16.281357    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-autores-1d073e/autores-model-1d073e-kserve-685c979499-mb8sc" event={"ID":"4f5b104a-8f72-4cd1-a292-ac8faa8e6391","Type":"ContainerDied","Data":"c218889067496a0dd3d0c94a4e13dd813d99a8e4495b5257655166958f46b722"}
Sep 18 19:57:16.281442 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:57:16.281378    2585 scope.go:117] "RemoveContainer" containerID="e237481b0cec53b690db549852463a9384d5511f89104a4b6f391740d20e20fe"
Sep 18 19:57:16.290889 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:57:16.290866    2585 scope.go:117] "RemoveContainer" containerID="e237481b0cec53b690db549852463a9384d5511f89104a4b6f391740d20e20fe"
Sep 18 19:57:16.291166 ip-10-0-136-185 kubenswrapper[2585]: E0918 19:57:16.291149    2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e237481b0cec53b690db549852463a9384d5511f89104a4b6f391740d20e20fe\": container with ID starting with e237481b0cec53b690db549852463a9384d5511f89104a4b6f391740d20e20fe not found: ID does not exist" containerID="e237481b0cec53b690db549852463a9384d5511f89104a4b6f391740d20e20fe"
Sep 18 19:57:16.291208 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:57:16.291176    2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e237481b0cec53b690db549852463a9384d5511f89104a4b6f391740d20e20fe"} err="failed to get container status \"e237481b0cec53b690db549852463a9384d5511f89104a4b6f391740d20e20fe\": rpc error: code = NotFound desc = could not find container \"e237481b0cec53b690db549852463a9384d5511f89104a4b6f391740d20e20fe\": container with ID starting with e237481b0cec53b690db549852463a9384d5511f89104a4b6f391740d20e20fe not found: ID does not exist"
Sep 18 19:57:16.302222 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:57:16.302195    2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-autores-1d073e/autores-model-1d073e-kserve-685c979499-mb8sc"]
Sep 18 19:57:16.307673 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:57:16.307649    2585 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-autores-1d073e/autores-model-1d073e-kserve-685c979499-mb8sc"]
Sep 18 19:57:16.663213 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:57:16.663182    2585 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4f5b104a-8f72-4cd1-a292-ac8faa8e6391" path="/var/lib/kubelet/pods/4f5b104a-8f72-4cd1-a292-ac8faa8e6391/volumes"
Sep 18 19:57:25.282930 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:57:25.282894    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-xmwh7"]
Sep 18 19:57:26.578973 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:57:26.578940    2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-674263/premium-w4-674263-kserve-b95548674-rdx2h"]
Sep 18 19:57:26.579340 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:57:26.579316    2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0e6c1362-1133-4772-86c9-b9a0b4378cb5" containerName="payload-processing"
Sep 18 19:57:26.579340 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:57:26.579327    2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="0e6c1362-1133-4772-86c9-b9a0b4378cb5" containerName="payload-processing"
Sep 18 19:57:26.579340 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:57:26.579340    2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4f5b104a-8f72-4cd1-a292-ac8faa8e6391" containerName="main"
Sep 18 19:57:26.579438 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:57:26.579346    2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="4f5b104a-8f72-4cd1-a292-ac8faa8e6391" containerName="main"
Sep 18 19:57:26.579438 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:57:26.579353    2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="326a12a2-68f5-45f8-b175-392f97f5e3c2" containerName="maas-api"
Sep 18 19:57:26.579438 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:57:26.579359    2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="326a12a2-68f5-45f8-b175-392f97f5e3c2" containerName="maas-api"
Sep 18 19:57:26.579438 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:57:26.579367    2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="522fc879-ad65-4bf9-8018-ead5194d612d" containerName="payload-pre-processing"
Sep 18 19:57:26.579438 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:57:26.579372    2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="522fc879-ad65-4bf9-8018-ead5194d612d" containerName="payload-pre-processing"
Sep 18 19:57:26.579438 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:57:26.579426    2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="0e6c1362-1133-4772-86c9-b9a0b4378cb5" containerName="payload-processing"
Sep 18 19:57:26.579438 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:57:26.579434    2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="4f5b104a-8f72-4cd1-a292-ac8faa8e6391" containerName="main"
Sep 18 19:57:26.579641 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:57:26.579442    2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="522fc879-ad65-4bf9-8018-ead5194d612d" containerName="payload-pre-processing"
Sep 18 19:57:26.579641 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:57:26.579448    2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="326a12a2-68f5-45f8-b175-392f97f5e3c2" containerName="maas-api"
Sep 18 19:57:26.584332 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:57:26.584312    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-674263/premium-w4-674263-kserve-b95548674-rdx2h"
Sep 18 19:57:26.590001 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:57:26.589977    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-674263\"/\"premium-w4-674263-kserve-self-signed-certs\""
Sep 18 19:57:26.593737 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:57:26.593716    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-674263/premium-w4-674263-kserve-b95548674-rdx2h"]
Sep 18 19:57:26.662950 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:57:26.662919    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1d6d42ad-9f65-4d73-af4d-5d9e9202e531-home\") pod \"premium-w4-674263-kserve-b95548674-rdx2h\" (UID: \"1d6d42ad-9f65-4d73-af4d-5d9e9202e531\") " pod="e2e-models-e2e-worker-w4-674263/premium-w4-674263-kserve-b95548674-rdx2h"
Sep 18 19:57:26.663126 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:57:26.662958    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1d6d42ad-9f65-4d73-af4d-5d9e9202e531-tls-certs\") pod \"premium-w4-674263-kserve-b95548674-rdx2h\" (UID: \"1d6d42ad-9f65-4d73-af4d-5d9e9202e531\") " pod="e2e-models-e2e-worker-w4-674263/premium-w4-674263-kserve-b95548674-rdx2h"
Sep 18 19:57:26.663126 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:57:26.662982    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1d6d42ad-9f65-4d73-af4d-5d9e9202e531-dshm\") pod \"premium-w4-674263-kserve-b95548674-rdx2h\" (UID: \"1d6d42ad-9f65-4d73-af4d-5d9e9202e531\") " pod="e2e-models-e2e-worker-w4-674263/premium-w4-674263-kserve-b95548674-rdx2h"
Sep 18 19:57:26.663126 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:57:26.663068    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1d6d42ad-9f65-4d73-af4d-5d9e9202e531-model-cache\") pod \"premium-w4-674263-kserve-b95548674-rdx2h\" (UID: \"1d6d42ad-9f65-4d73-af4d-5d9e9202e531\") " pod="e2e-models-e2e-worker-w4-674263/premium-w4-674263-kserve-b95548674-rdx2h"
Sep 18 19:57:26.663126 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:57:26.663094    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1d6d42ad-9f65-4d73-af4d-5d9e9202e531-tmp-dir\") pod \"premium-w4-674263-kserve-b95548674-rdx2h\" (UID: \"1d6d42ad-9f65-4d73-af4d-5d9e9202e531\") " pod="e2e-models-e2e-worker-w4-674263/premium-w4-674263-kserve-b95548674-rdx2h"
Sep 18 19:57:26.663310 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:57:26.663127    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lpwtg\" (UniqueName: \"kubernetes.io/projected/1d6d42ad-9f65-4d73-af4d-5d9e9202e531-kube-api-access-lpwtg\") pod \"premium-w4-674263-kserve-b95548674-rdx2h\" (UID: \"1d6d42ad-9f65-4d73-af4d-5d9e9202e531\") " pod="e2e-models-e2e-worker-w4-674263/premium-w4-674263-kserve-b95548674-rdx2h"
Sep 18 19:57:26.763835 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:57:26.763778    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1d6d42ad-9f65-4d73-af4d-5d9e9202e531-model-cache\") pod \"premium-w4-674263-kserve-b95548674-rdx2h\" (UID: \"1d6d42ad-9f65-4d73-af4d-5d9e9202e531\") " pod="e2e-models-e2e-worker-w4-674263/premium-w4-674263-kserve-b95548674-rdx2h"
Sep 18 19:57:26.763835 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:57:26.763836    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1d6d42ad-9f65-4d73-af4d-5d9e9202e531-tmp-dir\") pod \"premium-w4-674263-kserve-b95548674-rdx2h\" (UID: \"1d6d42ad-9f65-4d73-af4d-5d9e9202e531\") " pod="e2e-models-e2e-worker-w4-674263/premium-w4-674263-kserve-b95548674-rdx2h"
Sep 18 19:57:26.764059 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:57:26.763864    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lpwtg\" (UniqueName: \"kubernetes.io/projected/1d6d42ad-9f65-4d73-af4d-5d9e9202e531-kube-api-access-lpwtg\") pod \"premium-w4-674263-kserve-b95548674-rdx2h\" (UID: \"1d6d42ad-9f65-4d73-af4d-5d9e9202e531\") " pod="e2e-models-e2e-worker-w4-674263/premium-w4-674263-kserve-b95548674-rdx2h"
Sep 18 19:57:26.764059 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:57:26.763899    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1d6d42ad-9f65-4d73-af4d-5d9e9202e531-home\") pod \"premium-w4-674263-kserve-b95548674-rdx2h\" (UID: \"1d6d42ad-9f65-4d73-af4d-5d9e9202e531\") " pod="e2e-models-e2e-worker-w4-674263/premium-w4-674263-kserve-b95548674-rdx2h"
Sep 18 19:57:26.764059 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:57:26.764030    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1d6d42ad-9f65-4d73-af4d-5d9e9202e531-tls-certs\") pod \"premium-w4-674263-kserve-b95548674-rdx2h\" (UID: \"1d6d42ad-9f65-4d73-af4d-5d9e9202e531\") " pod="e2e-models-e2e-worker-w4-674263/premium-w4-674263-kserve-b95548674-rdx2h"
Sep 18 19:57:26.764206 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:57:26.764095    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1d6d42ad-9f65-4d73-af4d-5d9e9202e531-dshm\") pod \"premium-w4-674263-kserve-b95548674-rdx2h\" (UID: \"1d6d42ad-9f65-4d73-af4d-5d9e9202e531\") " pod="e2e-models-e2e-worker-w4-674263/premium-w4-674263-kserve-b95548674-rdx2h"
Sep 18 19:57:26.764264 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:57:26.764164    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1d6d42ad-9f65-4d73-af4d-5d9e9202e531-model-cache\") pod \"premium-w4-674263-kserve-b95548674-rdx2h\" (UID: \"1d6d42ad-9f65-4d73-af4d-5d9e9202e531\") " pod="e2e-models-e2e-worker-w4-674263/premium-w4-674263-kserve-b95548674-rdx2h"
Sep 18 19:57:26.764264 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:57:26.764222    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1d6d42ad-9f65-4d73-af4d-5d9e9202e531-tmp-dir\") pod \"premium-w4-674263-kserve-b95548674-rdx2h\" (UID: \"1d6d42ad-9f65-4d73-af4d-5d9e9202e531\") " pod="e2e-models-e2e-worker-w4-674263/premium-w4-674263-kserve-b95548674-rdx2h"
Sep 18 19:57:26.764366 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:57:26.764265    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1d6d42ad-9f65-4d73-af4d-5d9e9202e531-home\") pod \"premium-w4-674263-kserve-b95548674-rdx2h\" (UID: \"1d6d42ad-9f65-4d73-af4d-5d9e9202e531\") " pod="e2e-models-e2e-worker-w4-674263/premium-w4-674263-kserve-b95548674-rdx2h"
Sep 18 19:57:26.766349 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:57:26.766320    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1d6d42ad-9f65-4d73-af4d-5d9e9202e531-dshm\") pod \"premium-w4-674263-kserve-b95548674-rdx2h\" (UID: \"1d6d42ad-9f65-4d73-af4d-5d9e9202e531\") " pod="e2e-models-e2e-worker-w4-674263/premium-w4-674263-kserve-b95548674-rdx2h"
Sep 18 19:57:26.766587 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:57:26.766571    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1d6d42ad-9f65-4d73-af4d-5d9e9202e531-tls-certs\") pod \"premium-w4-674263-kserve-b95548674-rdx2h\" (UID: \"1d6d42ad-9f65-4d73-af4d-5d9e9202e531\") " pod="e2e-models-e2e-worker-w4-674263/premium-w4-674263-kserve-b95548674-rdx2h"
Sep 18 19:57:26.771966 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:57:26.771933    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lpwtg\" (UniqueName: \"kubernetes.io/projected/1d6d42ad-9f65-4d73-af4d-5d9e9202e531-kube-api-access-lpwtg\") pod \"premium-w4-674263-kserve-b95548674-rdx2h\" (UID: \"1d6d42ad-9f65-4d73-af4d-5d9e9202e531\") " pod="e2e-models-e2e-worker-w4-674263/premium-w4-674263-kserve-b95548674-rdx2h"
Sep 18 19:57:26.894053 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:57:26.894022    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-674263/premium-w4-674263-kserve-b95548674-rdx2h"
Sep 18 19:57:27.163555 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:57:27.163529    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-674263/premium-w4-674263-kserve-b95548674-rdx2h"]
Sep 18 19:57:27.165443 ip-10-0-136-185 kubenswrapper[2585]: W0918 19:57:27.165414    2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod1d6d42ad_9f65_4d73_af4d_5d9e9202e531.slice/crio-77cdc2c37a9f4ceffa2a4860b5537386ecc745720f5a275b38f52f9a8bdd4507 WatchSource:0}: Error finding container 77cdc2c37a9f4ceffa2a4860b5537386ecc745720f5a275b38f52f9a8bdd4507: Status 404 returned error can't find the container with id 77cdc2c37a9f4ceffa2a4860b5537386ecc745720f5a275b38f52f9a8bdd4507
Sep 18 19:57:27.338832 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:57:27.338781    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-674263/premium-w4-674263-kserve-b95548674-rdx2h" event={"ID":"1d6d42ad-9f65-4d73-af4d-5d9e9202e531","Type":"ContainerStarted","Data":"6570ce320bb6a2f24e7534c5bc0d4ee7dc97883b5877919795b6e89a2cfff29f"}
Sep 18 19:57:27.339023 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:57:27.338842    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-674263/premium-w4-674263-kserve-b95548674-rdx2h" event={"ID":"1d6d42ad-9f65-4d73-af4d-5d9e9202e531","Type":"ContainerStarted","Data":"77cdc2c37a9f4ceffa2a4860b5537386ecc745720f5a275b38f52f9a8bdd4507"}
Sep 18 19:57:27.355962 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:57:27.355916    2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-674263/premium-w4-674263-kserve-b95548674-rdx2h" podStartSLOduration=1.355900804 podStartE2EDuration="1.355900804s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 19:57:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 19:57:27.353602069 +0000 UTC m=+1088.878207493" watchObservedRunningTime="2026-09-18 19:57:27.355900804 +0000 UTC m=+1088.880506227"
Sep 18 19:57:36.894875 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:57:36.894829    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-674263/premium-w4-674263-kserve-b95548674-rdx2h"
Sep 18 19:57:36.895468 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:57:36.894892    2585 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-674263/premium-w4-674263-kserve-b95548674-rdx2h"
Sep 18 19:57:36.907643 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:57:36.907613    2585 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-674263/premium-w4-674263-kserve-b95548674-rdx2h"
Sep 18 19:57:37.400714 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:57:37.400685    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-674263/premium-w4-674263-kserve-b95548674-rdx2h"
Sep 18 19:57:41.767303 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:57:41.767271    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-xmwh7"]
Sep 18 19:57:59.272337 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:57:59.272300    2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-b-d5f7da/test-model-d5f7da-kserve-67f5967d47-sxvzr"]
Sep 18 19:57:59.276032 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:57:59.276011    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-d5f7da/test-model-d5f7da-kserve-67f5967d47-sxvzr"
Sep 18 19:57:59.279111 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:57:59.279085    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-d5f7da\"/\"default-dockercfg-m2gd8\""
Sep 18 19:57:59.279250 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:57:59.279087    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-d5f7da\"/\"test-model-d5f7da-kserve-self-signed-certs\""
Sep 18 19:57:59.279250 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:57:59.279131    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-d5f7da\"/\"kube-root-ca.crt\""
Sep 18 19:57:59.279250 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:57:59.279211    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-d5f7da\"/\"openshift-service-ca.crt\""
Sep 18 19:57:59.286071 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:57:59.286041    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-d5f7da/test-model-d5f7da-kserve-67f5967d47-sxvzr"]
Sep 18 19:57:59.357972 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:57:59.357939    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ab57d15d-6af5-49db-8446-7df8a5a15a14-tls-certs\") pod \"test-model-d5f7da-kserve-67f5967d47-sxvzr\" (UID: \"ab57d15d-6af5-49db-8446-7df8a5a15a14\") " pod="ai-tenant-e2e-inf-b-d5f7da/test-model-d5f7da-kserve-67f5967d47-sxvzr"
Sep 18 19:57:59.357972 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:57:59.357973    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ab57d15d-6af5-49db-8446-7df8a5a15a14-dshm\") pod \"test-model-d5f7da-kserve-67f5967d47-sxvzr\" (UID: \"ab57d15d-6af5-49db-8446-7df8a5a15a14\") " pod="ai-tenant-e2e-inf-b-d5f7da/test-model-d5f7da-kserve-67f5967d47-sxvzr"
Sep 18 19:57:59.358177 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:57:59.358048    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ab57d15d-6af5-49db-8446-7df8a5a15a14-home\") pod \"test-model-d5f7da-kserve-67f5967d47-sxvzr\" (UID: \"ab57d15d-6af5-49db-8446-7df8a5a15a14\") " pod="ai-tenant-e2e-inf-b-d5f7da/test-model-d5f7da-kserve-67f5967d47-sxvzr"
Sep 18 19:57:59.358177 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:57:59.358098    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ab57d15d-6af5-49db-8446-7df8a5a15a14-tmp-dir\") pod \"test-model-d5f7da-kserve-67f5967d47-sxvzr\" (UID: \"ab57d15d-6af5-49db-8446-7df8a5a15a14\") " pod="ai-tenant-e2e-inf-b-d5f7da/test-model-d5f7da-kserve-67f5967d47-sxvzr"
Sep 18 19:57:59.358251 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:57:59.358178    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ab57d15d-6af5-49db-8446-7df8a5a15a14-model-cache\") pod \"test-model-d5f7da-kserve-67f5967d47-sxvzr\" (UID: \"ab57d15d-6af5-49db-8446-7df8a5a15a14\") " pod="ai-tenant-e2e-inf-b-d5f7da/test-model-d5f7da-kserve-67f5967d47-sxvzr"
Sep 18 19:57:59.358251 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:57:59.358205    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5d66n\" (UniqueName: \"kubernetes.io/projected/ab57d15d-6af5-49db-8446-7df8a5a15a14-kube-api-access-5d66n\") pod \"test-model-d5f7da-kserve-67f5967d47-sxvzr\" (UID: \"ab57d15d-6af5-49db-8446-7df8a5a15a14\") " pod="ai-tenant-e2e-inf-b-d5f7da/test-model-d5f7da-kserve-67f5967d47-sxvzr"
Sep 18 19:57:59.459117 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:57:59.459075    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ab57d15d-6af5-49db-8446-7df8a5a15a14-tmp-dir\") pod \"test-model-d5f7da-kserve-67f5967d47-sxvzr\" (UID: \"ab57d15d-6af5-49db-8446-7df8a5a15a14\") " pod="ai-tenant-e2e-inf-b-d5f7da/test-model-d5f7da-kserve-67f5967d47-sxvzr"
Sep 18 19:57:59.459304 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:57:59.459158    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ab57d15d-6af5-49db-8446-7df8a5a15a14-model-cache\") pod \"test-model-d5f7da-kserve-67f5967d47-sxvzr\" (UID: \"ab57d15d-6af5-49db-8446-7df8a5a15a14\") " pod="ai-tenant-e2e-inf-b-d5f7da/test-model-d5f7da-kserve-67f5967d47-sxvzr"
Sep 18 19:57:59.459304 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:57:59.459188    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5d66n\" (UniqueName: \"kubernetes.io/projected/ab57d15d-6af5-49db-8446-7df8a5a15a14-kube-api-access-5d66n\") pod \"test-model-d5f7da-kserve-67f5967d47-sxvzr\" (UID: \"ab57d15d-6af5-49db-8446-7df8a5a15a14\") " pod="ai-tenant-e2e-inf-b-d5f7da/test-model-d5f7da-kserve-67f5967d47-sxvzr"
Sep 18 19:57:59.459304 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:57:59.459223    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ab57d15d-6af5-49db-8446-7df8a5a15a14-tls-certs\") pod \"test-model-d5f7da-kserve-67f5967d47-sxvzr\" (UID: \"ab57d15d-6af5-49db-8446-7df8a5a15a14\") " pod="ai-tenant-e2e-inf-b-d5f7da/test-model-d5f7da-kserve-67f5967d47-sxvzr"
Sep 18 19:57:59.459304 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:57:59.459246    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ab57d15d-6af5-49db-8446-7df8a5a15a14-dshm\") pod \"test-model-d5f7da-kserve-67f5967d47-sxvzr\" (UID: \"ab57d15d-6af5-49db-8446-7df8a5a15a14\") " pod="ai-tenant-e2e-inf-b-d5f7da/test-model-d5f7da-kserve-67f5967d47-sxvzr"
Sep 18 19:57:59.459518 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:57:59.459310    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ab57d15d-6af5-49db-8446-7df8a5a15a14-home\") pod \"test-model-d5f7da-kserve-67f5967d47-sxvzr\" (UID: \"ab57d15d-6af5-49db-8446-7df8a5a15a14\") " pod="ai-tenant-e2e-inf-b-d5f7da/test-model-d5f7da-kserve-67f5967d47-sxvzr"
Sep 18 19:57:59.459584 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:57:59.459514    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ab57d15d-6af5-49db-8446-7df8a5a15a14-tmp-dir\") pod \"test-model-d5f7da-kserve-67f5967d47-sxvzr\" (UID: \"ab57d15d-6af5-49db-8446-7df8a5a15a14\") " pod="ai-tenant-e2e-inf-b-d5f7da/test-model-d5f7da-kserve-67f5967d47-sxvzr"
Sep 18 19:57:59.459584 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:57:59.459555    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ab57d15d-6af5-49db-8446-7df8a5a15a14-model-cache\") pod \"test-model-d5f7da-kserve-67f5967d47-sxvzr\" (UID: \"ab57d15d-6af5-49db-8446-7df8a5a15a14\") " pod="ai-tenant-e2e-inf-b-d5f7da/test-model-d5f7da-kserve-67f5967d47-sxvzr"
Sep 18 19:57:59.459694 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:57:59.459681    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ab57d15d-6af5-49db-8446-7df8a5a15a14-home\") pod \"test-model-d5f7da-kserve-67f5967d47-sxvzr\" (UID: \"ab57d15d-6af5-49db-8446-7df8a5a15a14\") " pod="ai-tenant-e2e-inf-b-d5f7da/test-model-d5f7da-kserve-67f5967d47-sxvzr"
Sep 18 19:57:59.461588 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:57:59.461561    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ab57d15d-6af5-49db-8446-7df8a5a15a14-dshm\") pod \"test-model-d5f7da-kserve-67f5967d47-sxvzr\" (UID: \"ab57d15d-6af5-49db-8446-7df8a5a15a14\") " pod="ai-tenant-e2e-inf-b-d5f7da/test-model-d5f7da-kserve-67f5967d47-sxvzr"
Sep 18 19:57:59.461718 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:57:59.461694    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ab57d15d-6af5-49db-8446-7df8a5a15a14-tls-certs\") pod \"test-model-d5f7da-kserve-67f5967d47-sxvzr\" (UID: \"ab57d15d-6af5-49db-8446-7df8a5a15a14\") " pod="ai-tenant-e2e-inf-b-d5f7da/test-model-d5f7da-kserve-67f5967d47-sxvzr"
Sep 18 19:57:59.467477 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:57:59.467455    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5d66n\" (UniqueName: \"kubernetes.io/projected/ab57d15d-6af5-49db-8446-7df8a5a15a14-kube-api-access-5d66n\") pod \"test-model-d5f7da-kserve-67f5967d47-sxvzr\" (UID: \"ab57d15d-6af5-49db-8446-7df8a5a15a14\") " pod="ai-tenant-e2e-inf-b-d5f7da/test-model-d5f7da-kserve-67f5967d47-sxvzr"
Sep 18 19:57:59.587040 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:57:59.586949    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-d5f7da/test-model-d5f7da-kserve-67f5967d47-sxvzr"
Sep 18 19:57:59.857377 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:57:59.857339    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-d5f7da/test-model-d5f7da-kserve-67f5967d47-sxvzr"]
Sep 18 19:57:59.859328 ip-10-0-136-185 kubenswrapper[2585]: W0918 19:57:59.859301    2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podab57d15d_6af5_49db_8446_7df8a5a15a14.slice/crio-c46ed84807eba0a6946c10dbd443bed14f650bb0becd00494dd55376dc7723fb WatchSource:0}: Error finding container c46ed84807eba0a6946c10dbd443bed14f650bb0becd00494dd55376dc7723fb: Status 404 returned error can't find the container with id c46ed84807eba0a6946c10dbd443bed14f650bb0becd00494dd55376dc7723fb
Sep 18 19:58:00.510616 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:58:00.510577    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-d5f7da/test-model-d5f7da-kserve-67f5967d47-sxvzr" event={"ID":"ab57d15d-6af5-49db-8446-7df8a5a15a14","Type":"ContainerStarted","Data":"d5e8ef574bc8781b42c6f268247b937ded2a1288f0527b06a8d660978a2dc896"}
Sep 18 19:58:00.510616 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:58:00.510625    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-d5f7da/test-model-d5f7da-kserve-67f5967d47-sxvzr" event={"ID":"ab57d15d-6af5-49db-8446-7df8a5a15a14","Type":"ContainerStarted","Data":"c46ed84807eba0a6946c10dbd443bed14f650bb0becd00494dd55376dc7723fb"}
Sep 18 19:58:04.971161 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:58:04.971108    2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-inf-b-d5f7da/test-model-d5f7da-kserve-67f5967d47-sxvzr" podStartSLOduration=5.971091101 podStartE2EDuration="5.971091101s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 19:57:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 19:58:00.546945829 +0000 UTC m=+1122.071551252" watchObservedRunningTime="2026-09-18 19:58:04.971091101 +0000 UTC m=+1126.495696523"
Sep 18 19:58:04.972493 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:58:04.972474    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-xmwh7"]
Sep 18 19:58:09.587256 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:58:09.587221    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-d5f7da/test-model-d5f7da-kserve-67f5967d47-sxvzr"
Sep 18 19:58:09.587733 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:58:09.587272    2585 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-b-d5f7da/test-model-d5f7da-kserve-67f5967d47-sxvzr"
Sep 18 19:58:09.600004 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:58:09.599981    2585 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-b-d5f7da/test-model-d5f7da-kserve-67f5967d47-sxvzr"
Sep 18 19:58:10.571600 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:58:10.571568    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-d5f7da/test-model-d5f7da-kserve-67f5967d47-sxvzr"
Sep 18 19:58:36.372913 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:58:36.372831    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-xmwh7"]
Sep 18 19:58:55.093840 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:58:55.093804    2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-870f71c0-8649575654-226c4"]
Sep 18 19:58:55.097281 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:58:55.097261    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-870f71c0-8649575654-226c4"
Sep 18 19:58:55.100370 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:58:55.100340    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-870f71c0\""
Sep 18 19:58:55.100493 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:58:55.100413    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-870f71c0-dockercfg-8jtww\""
Sep 18 19:58:55.108022 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:58:55.107995    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-870f71c0-8649575654-226c4"]
Sep 18 19:58:55.208783 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:58:55.208743    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j42x4\" (UniqueName: \"kubernetes.io/projected/aba2d9b1-7fa2-453d-a879-deaf0a26ffaf-kube-api-access-j42x4\") pod \"payload-pre-processing-e2e-mt-870f71c0-8649575654-226c4\" (UID: \"aba2d9b1-7fa2-453d-a879-deaf0a26ffaf\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-870f71c0-8649575654-226c4"
Sep 18 19:58:55.208783 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:58:55.208821    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/aba2d9b1-7fa2-453d-a879-deaf0a26ffaf-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-870f71c0-8649575654-226c4\" (UID: \"aba2d9b1-7fa2-453d-a879-deaf0a26ffaf\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-870f71c0-8649575654-226c4"
Sep 18 19:58:55.310035 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:58:55.309990    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j42x4\" (UniqueName: \"kubernetes.io/projected/aba2d9b1-7fa2-453d-a879-deaf0a26ffaf-kube-api-access-j42x4\") pod \"payload-pre-processing-e2e-mt-870f71c0-8649575654-226c4\" (UID: \"aba2d9b1-7fa2-453d-a879-deaf0a26ffaf\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-870f71c0-8649575654-226c4"
Sep 18 19:58:55.310212 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:58:55.310055    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/aba2d9b1-7fa2-453d-a879-deaf0a26ffaf-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-870f71c0-8649575654-226c4\" (UID: \"aba2d9b1-7fa2-453d-a879-deaf0a26ffaf\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-870f71c0-8649575654-226c4"
Sep 18 19:58:55.310704 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:58:55.310683    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/aba2d9b1-7fa2-453d-a879-deaf0a26ffaf-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-870f71c0-8649575654-226c4\" (UID: \"aba2d9b1-7fa2-453d-a879-deaf0a26ffaf\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-870f71c0-8649575654-226c4"
Sep 18 19:58:55.321755 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:58:55.321729    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-j42x4\" (UniqueName: \"kubernetes.io/projected/aba2d9b1-7fa2-453d-a879-deaf0a26ffaf-kube-api-access-j42x4\") pod \"payload-pre-processing-e2e-mt-870f71c0-8649575654-226c4\" (UID: \"aba2d9b1-7fa2-453d-a879-deaf0a26ffaf\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-870f71c0-8649575654-226c4"
Sep 18 19:58:55.409264 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:58:55.409235    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-870f71c0-8649575654-226c4"
Sep 18 19:58:55.680438 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:58:55.680407    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-870f71c0-8649575654-226c4"]
Sep 18 19:58:55.682901 ip-10-0-136-185 kubenswrapper[2585]: W0918 19:58:55.682863    2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaba2d9b1_7fa2_453d_a879_deaf0a26ffaf.slice/crio-e51bf06943f39db6c1615129e60abd8ac5f5de7135938665f092cd4b9d78076b WatchSource:0}: Error finding container e51bf06943f39db6c1615129e60abd8ac5f5de7135938665f092cd4b9d78076b: Status 404 returned error can't find the container with id e51bf06943f39db6c1615129e60abd8ac5f5de7135938665f092cd4b9d78076b
Sep 18 19:58:55.787719 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:58:55.787683    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-870f71c0-8649575654-226c4" event={"ID":"aba2d9b1-7fa2-453d-a879-deaf0a26ffaf","Type":"ContainerStarted","Data":"4e3cf69c78a5b73fcbbbce1e5bc0c58f4e613366b5c154e0859a0efe68dfbf7a"}
Sep 18 19:58:55.787926 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:58:55.787735    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-870f71c0-8649575654-226c4" event={"ID":"aba2d9b1-7fa2-453d-a879-deaf0a26ffaf","Type":"ContainerStarted","Data":"e51bf06943f39db6c1615129e60abd8ac5f5de7135938665f092cd4b9d78076b"}
Sep 18 19:58:55.787926 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:58:55.787759    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-870f71c0-8649575654-226c4"
Sep 18 19:58:55.805653 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:58:55.805608    2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-mt-870f71c0-8649575654-226c4" podStartSLOduration=0.805591937 podStartE2EDuration="805.591937ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 19:58:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 19:58:55.805000066 +0000 UTC m=+1177.329605489" watchObservedRunningTime="2026-09-18 19:58:55.805591937 +0000 UTC m=+1177.330197359"
Sep 18 19:59:05.574962 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:59:05.574930    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-xmwh7"]
Sep 18 19:59:06.794958 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:59:06.794931    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-870f71c0-8649575654-226c4"
Sep 18 19:59:09.468919 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:59:09.468888    2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-explicit-be2171/explicit-model-be2171-kserve-544f9f7657-j45gg"]
Sep 18 19:59:09.472748 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:59:09.472327    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-explicit-be2171/explicit-model-be2171-kserve-544f9f7657-j45gg"
Sep 18 19:59:09.480910 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:59:09.480882    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-explicit-be2171\"/\"default-dockercfg-s75gn\""
Sep 18 19:59:09.483023 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:59:09.482997    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-explicit-be2171\"/\"explicit-model-be2171-kserve-self-signed-certs\""
Sep 18 19:59:09.483260 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:59:09.483241    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-explicit-be2171\"/\"openshift-service-ca.crt\""
Sep 18 19:59:09.483644 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:59:09.483623    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-explicit-be2171\"/\"kube-root-ca.crt\""
Sep 18 19:59:09.485669 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:59:09.485646    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-explicit-be2171/explicit-model-be2171-kserve-544f9f7657-j45gg"]
Sep 18 19:59:09.544329 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:59:09.544297    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3eacd4e7-191c-4bb7-8735-d0a3db7813eb-tmp-dir\") pod \"explicit-model-be2171-kserve-544f9f7657-j45gg\" (UID: \"3eacd4e7-191c-4bb7-8735-d0a3db7813eb\") " pod="ai-tenant-e2e-explicit-be2171/explicit-model-be2171-kserve-544f9f7657-j45gg"
Sep 18 19:59:09.544501 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:59:09.544345    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3eacd4e7-191c-4bb7-8735-d0a3db7813eb-model-cache\") pod \"explicit-model-be2171-kserve-544f9f7657-j45gg\" (UID: \"3eacd4e7-191c-4bb7-8735-d0a3db7813eb\") " pod="ai-tenant-e2e-explicit-be2171/explicit-model-be2171-kserve-544f9f7657-j45gg"
Sep 18 19:59:09.544501 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:59:09.544384    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rd8j4\" (UniqueName: \"kubernetes.io/projected/3eacd4e7-191c-4bb7-8735-d0a3db7813eb-kube-api-access-rd8j4\") pod \"explicit-model-be2171-kserve-544f9f7657-j45gg\" (UID: \"3eacd4e7-191c-4bb7-8735-d0a3db7813eb\") " pod="ai-tenant-e2e-explicit-be2171/explicit-model-be2171-kserve-544f9f7657-j45gg"
Sep 18 19:59:09.544501 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:59:09.544446    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3eacd4e7-191c-4bb7-8735-d0a3db7813eb-home\") pod \"explicit-model-be2171-kserve-544f9f7657-j45gg\" (UID: \"3eacd4e7-191c-4bb7-8735-d0a3db7813eb\") " pod="ai-tenant-e2e-explicit-be2171/explicit-model-be2171-kserve-544f9f7657-j45gg"
Sep 18 19:59:09.544647 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:59:09.544568    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3eacd4e7-191c-4bb7-8735-d0a3db7813eb-dshm\") pod \"explicit-model-be2171-kserve-544f9f7657-j45gg\" (UID: \"3eacd4e7-191c-4bb7-8735-d0a3db7813eb\") " pod="ai-tenant-e2e-explicit-be2171/explicit-model-be2171-kserve-544f9f7657-j45gg"
Sep 18 19:59:09.544647 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:59:09.544627    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3eacd4e7-191c-4bb7-8735-d0a3db7813eb-tls-certs\") pod \"explicit-model-be2171-kserve-544f9f7657-j45gg\" (UID: \"3eacd4e7-191c-4bb7-8735-d0a3db7813eb\") " pod="ai-tenant-e2e-explicit-be2171/explicit-model-be2171-kserve-544f9f7657-j45gg"
Sep 18 19:59:09.645608 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:59:09.645567    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3eacd4e7-191c-4bb7-8735-d0a3db7813eb-dshm\") pod \"explicit-model-be2171-kserve-544f9f7657-j45gg\" (UID: \"3eacd4e7-191c-4bb7-8735-d0a3db7813eb\") " pod="ai-tenant-e2e-explicit-be2171/explicit-model-be2171-kserve-544f9f7657-j45gg"
Sep 18 19:59:09.645854 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:59:09.645617    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3eacd4e7-191c-4bb7-8735-d0a3db7813eb-tls-certs\") pod \"explicit-model-be2171-kserve-544f9f7657-j45gg\" (UID: \"3eacd4e7-191c-4bb7-8735-d0a3db7813eb\") " pod="ai-tenant-e2e-explicit-be2171/explicit-model-be2171-kserve-544f9f7657-j45gg"
Sep 18 19:59:09.645854 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:59:09.645662    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3eacd4e7-191c-4bb7-8735-d0a3db7813eb-tmp-dir\") pod \"explicit-model-be2171-kserve-544f9f7657-j45gg\" (UID: \"3eacd4e7-191c-4bb7-8735-d0a3db7813eb\") " pod="ai-tenant-e2e-explicit-be2171/explicit-model-be2171-kserve-544f9f7657-j45gg"
Sep 18 19:59:09.645854 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:59:09.645704    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3eacd4e7-191c-4bb7-8735-d0a3db7813eb-model-cache\") pod \"explicit-model-be2171-kserve-544f9f7657-j45gg\" (UID: \"3eacd4e7-191c-4bb7-8735-d0a3db7813eb\") " pod="ai-tenant-e2e-explicit-be2171/explicit-model-be2171-kserve-544f9f7657-j45gg"
Sep 18 19:59:09.645854 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:59:09.645754    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rd8j4\" (UniqueName: \"kubernetes.io/projected/3eacd4e7-191c-4bb7-8735-d0a3db7813eb-kube-api-access-rd8j4\") pod \"explicit-model-be2171-kserve-544f9f7657-j45gg\" (UID: \"3eacd4e7-191c-4bb7-8735-d0a3db7813eb\") " pod="ai-tenant-e2e-explicit-be2171/explicit-model-be2171-kserve-544f9f7657-j45gg"
Sep 18 19:59:09.645854 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:59:09.645811    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3eacd4e7-191c-4bb7-8735-d0a3db7813eb-home\") pod \"explicit-model-be2171-kserve-544f9f7657-j45gg\" (UID: \"3eacd4e7-191c-4bb7-8735-d0a3db7813eb\") " pod="ai-tenant-e2e-explicit-be2171/explicit-model-be2171-kserve-544f9f7657-j45gg"
Sep 18 19:59:09.646169 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:59:09.646105    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3eacd4e7-191c-4bb7-8735-d0a3db7813eb-tmp-dir\") pod \"explicit-model-be2171-kserve-544f9f7657-j45gg\" (UID: \"3eacd4e7-191c-4bb7-8735-d0a3db7813eb\") " pod="ai-tenant-e2e-explicit-be2171/explicit-model-be2171-kserve-544f9f7657-j45gg"
Sep 18 19:59:09.646169 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:59:09.646150    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3eacd4e7-191c-4bb7-8735-d0a3db7813eb-home\") pod \"explicit-model-be2171-kserve-544f9f7657-j45gg\" (UID: \"3eacd4e7-191c-4bb7-8735-d0a3db7813eb\") " pod="ai-tenant-e2e-explicit-be2171/explicit-model-be2171-kserve-544f9f7657-j45gg"
Sep 18 19:59:09.646282 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:59:09.646221    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3eacd4e7-191c-4bb7-8735-d0a3db7813eb-model-cache\") pod \"explicit-model-be2171-kserve-544f9f7657-j45gg\" (UID: \"3eacd4e7-191c-4bb7-8735-d0a3db7813eb\") " pod="ai-tenant-e2e-explicit-be2171/explicit-model-be2171-kserve-544f9f7657-j45gg"
Sep 18 19:59:09.647979 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:59:09.647956    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3eacd4e7-191c-4bb7-8735-d0a3db7813eb-dshm\") pod \"explicit-model-be2171-kserve-544f9f7657-j45gg\" (UID: \"3eacd4e7-191c-4bb7-8735-d0a3db7813eb\") " pod="ai-tenant-e2e-explicit-be2171/explicit-model-be2171-kserve-544f9f7657-j45gg"
Sep 18 19:59:09.648399 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:59:09.648378    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3eacd4e7-191c-4bb7-8735-d0a3db7813eb-tls-certs\") pod \"explicit-model-be2171-kserve-544f9f7657-j45gg\" (UID: \"3eacd4e7-191c-4bb7-8735-d0a3db7813eb\") " pod="ai-tenant-e2e-explicit-be2171/explicit-model-be2171-kserve-544f9f7657-j45gg"
Sep 18 19:59:09.653379 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:59:09.653357    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rd8j4\" (UniqueName: \"kubernetes.io/projected/3eacd4e7-191c-4bb7-8735-d0a3db7813eb-kube-api-access-rd8j4\") pod \"explicit-model-be2171-kserve-544f9f7657-j45gg\" (UID: \"3eacd4e7-191c-4bb7-8735-d0a3db7813eb\") " pod="ai-tenant-e2e-explicit-be2171/explicit-model-be2171-kserve-544f9f7657-j45gg"
Sep 18 19:59:09.785417 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:59:09.785322    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-explicit-be2171/explicit-model-be2171-kserve-544f9f7657-j45gg"
Sep 18 19:59:10.062799 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:59:10.062758    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-explicit-be2171/explicit-model-be2171-kserve-544f9f7657-j45gg"]
Sep 18 19:59:10.064538 ip-10-0-136-185 kubenswrapper[2585]: W0918 19:59:10.064506    2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod3eacd4e7_191c_4bb7_8735_d0a3db7813eb.slice/crio-13cbd3bf8301475a395ecc2e5d5a7da878f3efb661ab62c004b3f053de1180a5 WatchSource:0}: Error finding container 13cbd3bf8301475a395ecc2e5d5a7da878f3efb661ab62c004b3f053de1180a5: Status 404 returned error can't find the container with id 13cbd3bf8301475a395ecc2e5d5a7da878f3efb661ab62c004b3f053de1180a5
Sep 18 19:59:10.869096 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:59:10.869063    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-explicit-be2171/explicit-model-be2171-kserve-544f9f7657-j45gg" event={"ID":"3eacd4e7-191c-4bb7-8735-d0a3db7813eb","Type":"ContainerStarted","Data":"c82117ca45ab64de3b7206723c1393bd24f42308c2bae3d0e16871e8e4dd5d3a"}
Sep 18 19:59:10.869492 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:59:10.869106    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-explicit-be2171/explicit-model-be2171-kserve-544f9f7657-j45gg" event={"ID":"3eacd4e7-191c-4bb7-8735-d0a3db7813eb","Type":"ContainerStarted","Data":"13cbd3bf8301475a395ecc2e5d5a7da878f3efb661ab62c004b3f053de1180a5"}
Sep 18 19:59:10.886500 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:59:10.886449    2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-explicit-be2171/explicit-model-be2171-kserve-544f9f7657-j45gg" podStartSLOduration=1.886434071 podStartE2EDuration="1.886434071s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 19:59:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 19:59:10.883620878 +0000 UTC m=+1192.408226299" watchObservedRunningTime="2026-09-18 19:59:10.886434071 +0000 UTC m=+1192.411039493"
Sep 18 19:59:17.256959 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:59:17.256926    2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-674263/e2e-conflict-a-1b1b9f14-kserve-69899bd87f-stx9d"]
Sep 18 19:59:17.261083 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:59:17.261058    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-674263/e2e-conflict-a-1b1b9f14-kserve-69899bd87f-stx9d"
Sep 18 19:59:17.263387 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:59:17.263365    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-674263\"/\"e2e-conflict-a-1b1b9f14-kserve-self-signed-certs\""
Sep 18 19:59:17.272466 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:59:17.272438    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-674263/e2e-conflict-a-1b1b9f14-kserve-69899bd87f-stx9d"]
Sep 18 19:59:17.317979 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:59:17.317942    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6c48fbd5-b360-49ab-9df2-5e5183112082-model-cache\") pod \"e2e-conflict-a-1b1b9f14-kserve-69899bd87f-stx9d\" (UID: \"6c48fbd5-b360-49ab-9df2-5e5183112082\") " pod="e2e-models-e2e-worker-w4-674263/e2e-conflict-a-1b1b9f14-kserve-69899bd87f-stx9d"
Sep 18 19:59:17.318155 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:59:17.318041    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6c48fbd5-b360-49ab-9df2-5e5183112082-tls-certs\") pod \"e2e-conflict-a-1b1b9f14-kserve-69899bd87f-stx9d\" (UID: \"6c48fbd5-b360-49ab-9df2-5e5183112082\") " pod="e2e-models-e2e-worker-w4-674263/e2e-conflict-a-1b1b9f14-kserve-69899bd87f-stx9d"
Sep 18 19:59:17.318155 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:59:17.318107    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6c48fbd5-b360-49ab-9df2-5e5183112082-tmp-dir\") pod \"e2e-conflict-a-1b1b9f14-kserve-69899bd87f-stx9d\" (UID: \"6c48fbd5-b360-49ab-9df2-5e5183112082\") " pod="e2e-models-e2e-worker-w4-674263/e2e-conflict-a-1b1b9f14-kserve-69899bd87f-stx9d"
Sep 18 19:59:17.318155 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:59:17.318125    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6c48fbd5-b360-49ab-9df2-5e5183112082-home\") pod \"e2e-conflict-a-1b1b9f14-kserve-69899bd87f-stx9d\" (UID: \"6c48fbd5-b360-49ab-9df2-5e5183112082\") " pod="e2e-models-e2e-worker-w4-674263/e2e-conflict-a-1b1b9f14-kserve-69899bd87f-stx9d"
Sep 18 19:59:17.318155 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:59:17.318142    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4phnt\" (UniqueName: \"kubernetes.io/projected/6c48fbd5-b360-49ab-9df2-5e5183112082-kube-api-access-4phnt\") pod \"e2e-conflict-a-1b1b9f14-kserve-69899bd87f-stx9d\" (UID: \"6c48fbd5-b360-49ab-9df2-5e5183112082\") " pod="e2e-models-e2e-worker-w4-674263/e2e-conflict-a-1b1b9f14-kserve-69899bd87f-stx9d"
Sep 18 19:59:17.318305 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:59:17.318174    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6c48fbd5-b360-49ab-9df2-5e5183112082-dshm\") pod \"e2e-conflict-a-1b1b9f14-kserve-69899bd87f-stx9d\" (UID: \"6c48fbd5-b360-49ab-9df2-5e5183112082\") " pod="e2e-models-e2e-worker-w4-674263/e2e-conflict-a-1b1b9f14-kserve-69899bd87f-stx9d"
Sep 18 19:59:17.419324 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:59:17.419290    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6c48fbd5-b360-49ab-9df2-5e5183112082-model-cache\") pod \"e2e-conflict-a-1b1b9f14-kserve-69899bd87f-stx9d\" (UID: \"6c48fbd5-b360-49ab-9df2-5e5183112082\") " pod="e2e-models-e2e-worker-w4-674263/e2e-conflict-a-1b1b9f14-kserve-69899bd87f-stx9d"
Sep 18 19:59:17.419525 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:59:17.419362    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6c48fbd5-b360-49ab-9df2-5e5183112082-tls-certs\") pod \"e2e-conflict-a-1b1b9f14-kserve-69899bd87f-stx9d\" (UID: \"6c48fbd5-b360-49ab-9df2-5e5183112082\") " pod="e2e-models-e2e-worker-w4-674263/e2e-conflict-a-1b1b9f14-kserve-69899bd87f-stx9d"
Sep 18 19:59:17.419525 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:59:17.419411    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6c48fbd5-b360-49ab-9df2-5e5183112082-tmp-dir\") pod \"e2e-conflict-a-1b1b9f14-kserve-69899bd87f-stx9d\" (UID: \"6c48fbd5-b360-49ab-9df2-5e5183112082\") " pod="e2e-models-e2e-worker-w4-674263/e2e-conflict-a-1b1b9f14-kserve-69899bd87f-stx9d"
Sep 18 19:59:17.419525 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:59:17.419428    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6c48fbd5-b360-49ab-9df2-5e5183112082-home\") pod \"e2e-conflict-a-1b1b9f14-kserve-69899bd87f-stx9d\" (UID: \"6c48fbd5-b360-49ab-9df2-5e5183112082\") " pod="e2e-models-e2e-worker-w4-674263/e2e-conflict-a-1b1b9f14-kserve-69899bd87f-stx9d"
Sep 18 19:59:17.419525 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:59:17.419448    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4phnt\" (UniqueName: \"kubernetes.io/projected/6c48fbd5-b360-49ab-9df2-5e5183112082-kube-api-access-4phnt\") pod \"e2e-conflict-a-1b1b9f14-kserve-69899bd87f-stx9d\" (UID: \"6c48fbd5-b360-49ab-9df2-5e5183112082\") " pod="e2e-models-e2e-worker-w4-674263/e2e-conflict-a-1b1b9f14-kserve-69899bd87f-stx9d"
Sep 18 19:59:17.419810 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:59:17.419567    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6c48fbd5-b360-49ab-9df2-5e5183112082-dshm\") pod \"e2e-conflict-a-1b1b9f14-kserve-69899bd87f-stx9d\" (UID: \"6c48fbd5-b360-49ab-9df2-5e5183112082\") " pod="e2e-models-e2e-worker-w4-674263/e2e-conflict-a-1b1b9f14-kserve-69899bd87f-stx9d"
Sep 18 19:59:17.419810 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:59:17.419729    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6c48fbd5-b360-49ab-9df2-5e5183112082-model-cache\") pod \"e2e-conflict-a-1b1b9f14-kserve-69899bd87f-stx9d\" (UID: \"6c48fbd5-b360-49ab-9df2-5e5183112082\") " pod="e2e-models-e2e-worker-w4-674263/e2e-conflict-a-1b1b9f14-kserve-69899bd87f-stx9d"
Sep 18 19:59:17.419920 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:59:17.419843    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6c48fbd5-b360-49ab-9df2-5e5183112082-home\") pod \"e2e-conflict-a-1b1b9f14-kserve-69899bd87f-stx9d\" (UID: \"6c48fbd5-b360-49ab-9df2-5e5183112082\") " pod="e2e-models-e2e-worker-w4-674263/e2e-conflict-a-1b1b9f14-kserve-69899bd87f-stx9d"
Sep 18 19:59:17.419920 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:59:17.419883    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6c48fbd5-b360-49ab-9df2-5e5183112082-tmp-dir\") pod \"e2e-conflict-a-1b1b9f14-kserve-69899bd87f-stx9d\" (UID: \"6c48fbd5-b360-49ab-9df2-5e5183112082\") " pod="e2e-models-e2e-worker-w4-674263/e2e-conflict-a-1b1b9f14-kserve-69899bd87f-stx9d"
Sep 18 19:59:17.421762 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:59:17.421734    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6c48fbd5-b360-49ab-9df2-5e5183112082-dshm\") pod \"e2e-conflict-a-1b1b9f14-kserve-69899bd87f-stx9d\" (UID: \"6c48fbd5-b360-49ab-9df2-5e5183112082\") " pod="e2e-models-e2e-worker-w4-674263/e2e-conflict-a-1b1b9f14-kserve-69899bd87f-stx9d"
Sep 18 19:59:17.422048 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:59:17.422028    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6c48fbd5-b360-49ab-9df2-5e5183112082-tls-certs\") pod \"e2e-conflict-a-1b1b9f14-kserve-69899bd87f-stx9d\" (UID: \"6c48fbd5-b360-49ab-9df2-5e5183112082\") " pod="e2e-models-e2e-worker-w4-674263/e2e-conflict-a-1b1b9f14-kserve-69899bd87f-stx9d"
Sep 18 19:59:17.427988 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:59:17.427963    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4phnt\" (UniqueName: \"kubernetes.io/projected/6c48fbd5-b360-49ab-9df2-5e5183112082-kube-api-access-4phnt\") pod \"e2e-conflict-a-1b1b9f14-kserve-69899bd87f-stx9d\" (UID: \"6c48fbd5-b360-49ab-9df2-5e5183112082\") " pod="e2e-models-e2e-worker-w4-674263/e2e-conflict-a-1b1b9f14-kserve-69899bd87f-stx9d"
Sep 18 19:59:17.573777 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:59:17.573678    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-674263/e2e-conflict-a-1b1b9f14-kserve-69899bd87f-stx9d"
Sep 18 19:59:17.845079 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:59:17.845054    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-674263/e2e-conflict-a-1b1b9f14-kserve-69899bd87f-stx9d"]
Sep 18 19:59:17.846783 ip-10-0-136-185 kubenswrapper[2585]: W0918 19:59:17.846754    2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod6c48fbd5_b360_49ab_9df2_5e5183112082.slice/crio-fa662d05319aa75fc15045485d810d4d7e6e9b39e7f824ba8b4fbc218edd0073 WatchSource:0}: Error finding container fa662d05319aa75fc15045485d810d4d7e6e9b39e7f824ba8b4fbc218edd0073: Status 404 returned error can't find the container with id fa662d05319aa75fc15045485d810d4d7e6e9b39e7f824ba8b4fbc218edd0073
Sep 18 19:59:17.905357 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:59:17.905326    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-674263/e2e-conflict-a-1b1b9f14-kserve-69899bd87f-stx9d" event={"ID":"6c48fbd5-b360-49ab-9df2-5e5183112082","Type":"ContainerStarted","Data":"fa662d05319aa75fc15045485d810d4d7e6e9b39e7f824ba8b4fbc218edd0073"}
Sep 18 19:59:18.912113 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:59:18.912073    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-674263/e2e-conflict-a-1b1b9f14-kserve-69899bd87f-stx9d" event={"ID":"6c48fbd5-b360-49ab-9df2-5e5183112082","Type":"ContainerStarted","Data":"16ea1a9fb4e1ac800155b99190d83f22d6c8e1c6bec66a071c9140422fba5bcf"}
Sep 18 19:59:18.929441 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:59:18.929384    2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-674263/e2e-conflict-a-1b1b9f14-kserve-69899bd87f-stx9d" podStartSLOduration=1.929367816 podStartE2EDuration="1.929367816s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 19:59:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 19:59:18.928347967 +0000 UTC m=+1200.452953399" watchObservedRunningTime="2026-09-18 19:59:18.929367816 +0000 UTC m=+1200.453973239"
Sep 18 19:59:19.786335 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:59:19.786300    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-explicit-be2171/explicit-model-be2171-kserve-544f9f7657-j45gg"
Sep 18 19:59:19.786620 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:59:19.786598    2585 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-explicit-be2171/explicit-model-be2171-kserve-544f9f7657-j45gg"
Sep 18 19:59:19.799525 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:59:19.799495    2585 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-explicit-be2171/explicit-model-be2171-kserve-544f9f7657-j45gg"
Sep 18 19:59:19.928364 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:59:19.928331    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-explicit-be2171/explicit-model-be2171-kserve-544f9f7657-j45gg"
Sep 18 19:59:20.625897 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:59:20.625872    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-rctld_cc0e0de2-75d5-43a1-b64b-b071427cd8ce/ovn-acl-logging/0.log"
Sep 18 19:59:20.633017 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:59:20.632986    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-rctld_cc0e0de2-75d5-43a1-b64b-b071427cd8ce/ovn-acl-logging/0.log"
Sep 18 19:59:20.968347 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:59:20.968312    2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-explicit-be2171/explicit-model-be2171-kserve-544f9f7657-j45gg"]
Sep 18 19:59:21.056313 ip-10-0-136-185 kubenswrapper[2585]: E0918 19:59:21.056271    2585 secret.go:189] Couldn't get secret ai-tenant-e2e-explicit-be2171/explicit-model-be2171-kserve-self-signed-certs: secret "explicit-model-be2171-kserve-self-signed-certs" not found
Sep 18 19:59:21.056492 ip-10-0-136-185 kubenswrapper[2585]: E0918 19:59:21.056360    2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3eacd4e7-191c-4bb7-8735-d0a3db7813eb-tls-certs podName:3eacd4e7-191c-4bb7-8735-d0a3db7813eb nodeName:}" failed. No retries permitted until 2026-09-18 19:59:21.556344996 +0000 UTC m=+1203.080950396 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/3eacd4e7-191c-4bb7-8735-d0a3db7813eb-tls-certs") pod "explicit-model-be2171-kserve-544f9f7657-j45gg" (UID: "3eacd4e7-191c-4bb7-8735-d0a3db7813eb") : secret "explicit-model-be2171-kserve-self-signed-certs" not found
Sep 18 19:59:21.561673 ip-10-0-136-185 kubenswrapper[2585]: E0918 19:59:21.561637    2585 secret.go:189] Couldn't get secret ai-tenant-e2e-explicit-be2171/explicit-model-be2171-kserve-self-signed-certs: secret "explicit-model-be2171-kserve-self-signed-certs" not found
Sep 18 19:59:21.561891 ip-10-0-136-185 kubenswrapper[2585]: E0918 19:59:21.561716    2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3eacd4e7-191c-4bb7-8735-d0a3db7813eb-tls-certs podName:3eacd4e7-191c-4bb7-8735-d0a3db7813eb nodeName:}" failed. No retries permitted until 2026-09-18 19:59:22.561698749 +0000 UTC m=+1204.086304150 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/3eacd4e7-191c-4bb7-8735-d0a3db7813eb-tls-certs") pod "explicit-model-be2171-kserve-544f9f7657-j45gg" (UID: "3eacd4e7-191c-4bb7-8735-d0a3db7813eb") : secret "explicit-model-be2171-kserve-self-signed-certs" not found
Sep 18 19:59:22.572094 ip-10-0-136-185 kubenswrapper[2585]: E0918 19:59:22.572053    2585 secret.go:189] Couldn't get secret ai-tenant-e2e-explicit-be2171/explicit-model-be2171-kserve-self-signed-certs: secret "explicit-model-be2171-kserve-self-signed-certs" not found
Sep 18 19:59:22.572548 ip-10-0-136-185 kubenswrapper[2585]: E0918 19:59:22.572141    2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3eacd4e7-191c-4bb7-8735-d0a3db7813eb-tls-certs podName:3eacd4e7-191c-4bb7-8735-d0a3db7813eb nodeName:}" failed. No retries permitted until 2026-09-18 19:59:24.572121488 +0000 UTC m=+1206.096726889 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/3eacd4e7-191c-4bb7-8735-d0a3db7813eb-tls-certs") pod "explicit-model-be2171-kserve-544f9f7657-j45gg" (UID: "3eacd4e7-191c-4bb7-8735-d0a3db7813eb") : secret "explicit-model-be2171-kserve-self-signed-certs" not found
Sep 18 19:59:24.591932 ip-10-0-136-185 kubenswrapper[2585]: E0918 19:59:24.591893    2585 secret.go:189] Couldn't get secret ai-tenant-e2e-explicit-be2171/explicit-model-be2171-kserve-self-signed-certs: secret "explicit-model-be2171-kserve-self-signed-certs" not found
Sep 18 19:59:24.592337 ip-10-0-136-185 kubenswrapper[2585]: E0918 19:59:24.591968    2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3eacd4e7-191c-4bb7-8735-d0a3db7813eb-tls-certs podName:3eacd4e7-191c-4bb7-8735-d0a3db7813eb nodeName:}" failed. No retries permitted until 2026-09-18 19:59:28.591952402 +0000 UTC m=+1210.116557819 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/3eacd4e7-191c-4bb7-8735-d0a3db7813eb-tls-certs") pod "explicit-model-be2171-kserve-544f9f7657-j45gg" (UID: "3eacd4e7-191c-4bb7-8735-d0a3db7813eb") : secret "explicit-model-be2171-kserve-self-signed-certs" not found
Sep 18 19:59:27.574848 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:59:27.574780    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-674263/e2e-conflict-a-1b1b9f14-kserve-69899bd87f-stx9d"
Sep 18 19:59:27.577425 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:59:27.574870    2585 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-674263/e2e-conflict-a-1b1b9f14-kserve-69899bd87f-stx9d"
Sep 18 19:59:27.587368 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:59:27.587345    2585 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-674263/e2e-conflict-a-1b1b9f14-kserve-69899bd87f-stx9d"
Sep 18 19:59:27.968946 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:59:27.968914    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-674263/e2e-conflict-a-1b1b9f14-kserve-69899bd87f-stx9d"
Sep 18 19:59:28.635828 ip-10-0-136-185 kubenswrapper[2585]: E0918 19:59:28.635767    2585 secret.go:189] Couldn't get secret ai-tenant-e2e-explicit-be2171/explicit-model-be2171-kserve-self-signed-certs: secret "explicit-model-be2171-kserve-self-signed-certs" not found
Sep 18 19:59:28.636222 ip-10-0-136-185 kubenswrapper[2585]: E0918 19:59:28.635876    2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3eacd4e7-191c-4bb7-8735-d0a3db7813eb-tls-certs podName:3eacd4e7-191c-4bb7-8735-d0a3db7813eb nodeName:}" failed. No retries permitted until 2026-09-18 19:59:36.635859115 +0000 UTC m=+1218.160464525 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/3eacd4e7-191c-4bb7-8735-d0a3db7813eb-tls-certs") pod "explicit-model-be2171-kserve-544f9f7657-j45gg" (UID: "3eacd4e7-191c-4bb7-8735-d0a3db7813eb") : secret "explicit-model-be2171-kserve-self-signed-certs" not found
Sep 18 19:59:31.477508 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:59:31.477467    2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-d94b9e/distinct-w0-d94b9e-kserve-654c9b789b-w7qlc"]
Sep 18 19:59:31.481147 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:59:31.481125    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-d94b9e/distinct-w0-d94b9e-kserve-654c9b789b-w7qlc"
Sep 18 19:59:31.483821 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:59:31.483775    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-d94b9e\"/\"distinct-w0-d94b9e-kserve-self-signed-certs\""
Sep 18 19:59:31.492474 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:59:31.492452    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-d94b9e/distinct-w0-d94b9e-kserve-654c9b789b-w7qlc"]
Sep 18 19:59:31.562736 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:59:31.562707    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/706cd513-2d52-43a6-aeed-4382e477fea2-dshm\") pod \"distinct-w0-d94b9e-kserve-654c9b789b-w7qlc\" (UID: \"706cd513-2d52-43a6-aeed-4382e477fea2\") " pod="e2e-models-e2e-worker-w0-d94b9e/distinct-w0-d94b9e-kserve-654c9b789b-w7qlc"
Sep 18 19:59:31.562929 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:59:31.562767    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/706cd513-2d52-43a6-aeed-4382e477fea2-model-cache\") pod \"distinct-w0-d94b9e-kserve-654c9b789b-w7qlc\" (UID: \"706cd513-2d52-43a6-aeed-4382e477fea2\") " pod="e2e-models-e2e-worker-w0-d94b9e/distinct-w0-d94b9e-kserve-654c9b789b-w7qlc"
Sep 18 19:59:31.562929 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:59:31.562820    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b9qtk\" (UniqueName: \"kubernetes.io/projected/706cd513-2d52-43a6-aeed-4382e477fea2-kube-api-access-b9qtk\") pod \"distinct-w0-d94b9e-kserve-654c9b789b-w7qlc\" (UID: \"706cd513-2d52-43a6-aeed-4382e477fea2\") " pod="e2e-models-e2e-worker-w0-d94b9e/distinct-w0-d94b9e-kserve-654c9b789b-w7qlc"
Sep 18 19:59:31.562929 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:59:31.562859    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/706cd513-2d52-43a6-aeed-4382e477fea2-tls-certs\") pod \"distinct-w0-d94b9e-kserve-654c9b789b-w7qlc\" (UID: \"706cd513-2d52-43a6-aeed-4382e477fea2\") " pod="e2e-models-e2e-worker-w0-d94b9e/distinct-w0-d94b9e-kserve-654c9b789b-w7qlc"
Sep 18 19:59:31.562929 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:59:31.562891    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/706cd513-2d52-43a6-aeed-4382e477fea2-home\") pod \"distinct-w0-d94b9e-kserve-654c9b789b-w7qlc\" (UID: \"706cd513-2d52-43a6-aeed-4382e477fea2\") " pod="e2e-models-e2e-worker-w0-d94b9e/distinct-w0-d94b9e-kserve-654c9b789b-w7qlc"
Sep 18 19:59:31.563071 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:59:31.562973    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/706cd513-2d52-43a6-aeed-4382e477fea2-tmp-dir\") pod \"distinct-w0-d94b9e-kserve-654c9b789b-w7qlc\" (UID: \"706cd513-2d52-43a6-aeed-4382e477fea2\") " pod="e2e-models-e2e-worker-w0-d94b9e/distinct-w0-d94b9e-kserve-654c9b789b-w7qlc"
Sep 18 19:59:31.664229 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:59:31.664188    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/706cd513-2d52-43a6-aeed-4382e477fea2-dshm\") pod \"distinct-w0-d94b9e-kserve-654c9b789b-w7qlc\" (UID: \"706cd513-2d52-43a6-aeed-4382e477fea2\") " pod="e2e-models-e2e-worker-w0-d94b9e/distinct-w0-d94b9e-kserve-654c9b789b-w7qlc"
Sep 18 19:59:31.664413 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:59:31.664251    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/706cd513-2d52-43a6-aeed-4382e477fea2-model-cache\") pod \"distinct-w0-d94b9e-kserve-654c9b789b-w7qlc\" (UID: \"706cd513-2d52-43a6-aeed-4382e477fea2\") " pod="e2e-models-e2e-worker-w0-d94b9e/distinct-w0-d94b9e-kserve-654c9b789b-w7qlc"
Sep 18 19:59:31.664413 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:59:31.664269    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b9qtk\" (UniqueName: \"kubernetes.io/projected/706cd513-2d52-43a6-aeed-4382e477fea2-kube-api-access-b9qtk\") pod \"distinct-w0-d94b9e-kserve-654c9b789b-w7qlc\" (UID: \"706cd513-2d52-43a6-aeed-4382e477fea2\") " pod="e2e-models-e2e-worker-w0-d94b9e/distinct-w0-d94b9e-kserve-654c9b789b-w7qlc"
Sep 18 19:59:31.664413 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:59:31.664292    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/706cd513-2d52-43a6-aeed-4382e477fea2-tls-certs\") pod \"distinct-w0-d94b9e-kserve-654c9b789b-w7qlc\" (UID: \"706cd513-2d52-43a6-aeed-4382e477fea2\") " pod="e2e-models-e2e-worker-w0-d94b9e/distinct-w0-d94b9e-kserve-654c9b789b-w7qlc"
Sep 18 19:59:31.664413 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:59:31.664317    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/706cd513-2d52-43a6-aeed-4382e477fea2-home\") pod \"distinct-w0-d94b9e-kserve-654c9b789b-w7qlc\" (UID: \"706cd513-2d52-43a6-aeed-4382e477fea2\") " pod="e2e-models-e2e-worker-w0-d94b9e/distinct-w0-d94b9e-kserve-654c9b789b-w7qlc"
Sep 18 19:59:31.664413 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:59:31.664357    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/706cd513-2d52-43a6-aeed-4382e477fea2-tmp-dir\") pod \"distinct-w0-d94b9e-kserve-654c9b789b-w7qlc\" (UID: \"706cd513-2d52-43a6-aeed-4382e477fea2\") " pod="e2e-models-e2e-worker-w0-d94b9e/distinct-w0-d94b9e-kserve-654c9b789b-w7qlc"
Sep 18 19:59:31.664833 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:59:31.664771    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/706cd513-2d52-43a6-aeed-4382e477fea2-home\") pod \"distinct-w0-d94b9e-kserve-654c9b789b-w7qlc\" (UID: \"706cd513-2d52-43a6-aeed-4382e477fea2\") " pod="e2e-models-e2e-worker-w0-d94b9e/distinct-w0-d94b9e-kserve-654c9b789b-w7qlc"
Sep 18 19:59:31.664833 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:59:31.664822    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/706cd513-2d52-43a6-aeed-4382e477fea2-model-cache\") pod \"distinct-w0-d94b9e-kserve-654c9b789b-w7qlc\" (UID: \"706cd513-2d52-43a6-aeed-4382e477fea2\") " pod="e2e-models-e2e-worker-w0-d94b9e/distinct-w0-d94b9e-kserve-654c9b789b-w7qlc"
Sep 18 19:59:31.665001 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:59:31.664841    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/706cd513-2d52-43a6-aeed-4382e477fea2-tmp-dir\") pod \"distinct-w0-d94b9e-kserve-654c9b789b-w7qlc\" (UID: \"706cd513-2d52-43a6-aeed-4382e477fea2\") " pod="e2e-models-e2e-worker-w0-d94b9e/distinct-w0-d94b9e-kserve-654c9b789b-w7qlc"
Sep 18 19:59:31.666506 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:59:31.666488    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/706cd513-2d52-43a6-aeed-4382e477fea2-dshm\") pod \"distinct-w0-d94b9e-kserve-654c9b789b-w7qlc\" (UID: \"706cd513-2d52-43a6-aeed-4382e477fea2\") " pod="e2e-models-e2e-worker-w0-d94b9e/distinct-w0-d94b9e-kserve-654c9b789b-w7qlc"
Sep 18 19:59:31.666766 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:59:31.666747    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/706cd513-2d52-43a6-aeed-4382e477fea2-tls-certs\") pod \"distinct-w0-d94b9e-kserve-654c9b789b-w7qlc\" (UID: \"706cd513-2d52-43a6-aeed-4382e477fea2\") " pod="e2e-models-e2e-worker-w0-d94b9e/distinct-w0-d94b9e-kserve-654c9b789b-w7qlc"
Sep 18 19:59:31.673688 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:59:31.673665    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-b9qtk\" (UniqueName: \"kubernetes.io/projected/706cd513-2d52-43a6-aeed-4382e477fea2-kube-api-access-b9qtk\") pod \"distinct-w0-d94b9e-kserve-654c9b789b-w7qlc\" (UID: \"706cd513-2d52-43a6-aeed-4382e477fea2\") " pod="e2e-models-e2e-worker-w0-d94b9e/distinct-w0-d94b9e-kserve-654c9b789b-w7qlc"
Sep 18 19:59:31.789950 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:59:31.789853    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-d94b9e/distinct-w0-d94b9e-kserve-654c9b789b-w7qlc"
Sep 18 19:59:32.063001 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:59:32.062973    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-d94b9e/distinct-w0-d94b9e-kserve-654c9b789b-w7qlc"]
Sep 18 19:59:32.065073 ip-10-0-136-185 kubenswrapper[2585]: W0918 19:59:32.065045    2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod706cd513_2d52_43a6_aeed_4382e477fea2.slice/crio-dfc42ce14efbe7b147befd469eae3ff76e03144d03f4073731b5a3fcc0a6e5ec WatchSource:0}: Error finding container dfc42ce14efbe7b147befd469eae3ff76e03144d03f4073731b5a3fcc0a6e5ec: Status 404 returned error can't find the container with id dfc42ce14efbe7b147befd469eae3ff76e03144d03f4073731b5a3fcc0a6e5ec
Sep 18 19:59:32.985497 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:59:32.985456    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-d94b9e/distinct-w0-d94b9e-kserve-654c9b789b-w7qlc" event={"ID":"706cd513-2d52-43a6-aeed-4382e477fea2","Type":"ContainerStarted","Data":"c1f67ed6ca8e921f7e48b4c63e1f2f6b5c1ff741b933fbbb69a9d53bd0cca91d"}
Sep 18 19:59:32.985933 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:59:32.985506    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-d94b9e/distinct-w0-d94b9e-kserve-654c9b789b-w7qlc" event={"ID":"706cd513-2d52-43a6-aeed-4382e477fea2","Type":"ContainerStarted","Data":"dfc42ce14efbe7b147befd469eae3ff76e03144d03f4073731b5a3fcc0a6e5ec"}
Sep 18 19:59:33.002042 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:59:33.001992    2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-d94b9e/distinct-w0-d94b9e-kserve-654c9b789b-w7qlc" podStartSLOduration=2.001977049 podStartE2EDuration="2.001977049s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 19:59:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 19:59:33.000760488 +0000 UTC m=+1214.525365930" watchObservedRunningTime="2026-09-18 19:59:33.001977049 +0000 UTC m=+1214.526582525"
Sep 18 19:59:36.720966 ip-10-0-136-185 kubenswrapper[2585]: E0918 19:59:36.720931    2585 secret.go:189] Couldn't get secret ai-tenant-e2e-explicit-be2171/explicit-model-be2171-kserve-self-signed-certs: secret "explicit-model-be2171-kserve-self-signed-certs" not found
Sep 18 19:59:36.721370 ip-10-0-136-185 kubenswrapper[2585]: E0918 19:59:36.720999    2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3eacd4e7-191c-4bb7-8735-d0a3db7813eb-tls-certs podName:3eacd4e7-191c-4bb7-8735-d0a3db7813eb nodeName:}" failed. No retries permitted until 2026-09-18 19:59:52.720984023 +0000 UTC m=+1234.245589425 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/3eacd4e7-191c-4bb7-8735-d0a3db7813eb-tls-certs") pod "explicit-model-be2171-kserve-544f9f7657-j45gg" (UID: "3eacd4e7-191c-4bb7-8735-d0a3db7813eb") : secret "explicit-model-be2171-kserve-self-signed-certs" not found
Sep 18 19:59:37.950686 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:59:37.950639    2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-explicit-be2171/explicit-model-be2171-kserve-544f9f7657-j45gg" podUID="3eacd4e7-191c-4bb7-8735-d0a3db7813eb" containerName="main" containerID="cri-o://c82117ca45ab64de3b7206723c1393bd24f42308c2bae3d0e16871e8e4dd5d3a" gracePeriod=45
Sep 18 19:59:38.210770 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:59:38.210704    2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-explicit-be2171/explicit-model-be2171-kserve-544f9f7657-j45gg"
Sep 18 19:59:38.236070 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:59:38.236036    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3eacd4e7-191c-4bb7-8735-d0a3db7813eb-dshm\") pod \"3eacd4e7-191c-4bb7-8735-d0a3db7813eb\" (UID: \"3eacd4e7-191c-4bb7-8735-d0a3db7813eb\") "
Sep 18 19:59:38.236320 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:59:38.236111    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3eacd4e7-191c-4bb7-8735-d0a3db7813eb-tmp-dir\") pod \"3eacd4e7-191c-4bb7-8735-d0a3db7813eb\" (UID: \"3eacd4e7-191c-4bb7-8735-d0a3db7813eb\") "
Sep 18 19:59:38.236320 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:59:38.236155    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rd8j4\" (UniqueName: \"kubernetes.io/projected/3eacd4e7-191c-4bb7-8735-d0a3db7813eb-kube-api-access-rd8j4\") pod \"3eacd4e7-191c-4bb7-8735-d0a3db7813eb\" (UID: \"3eacd4e7-191c-4bb7-8735-d0a3db7813eb\") "
Sep 18 19:59:38.236320 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:59:38.236173    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3eacd4e7-191c-4bb7-8735-d0a3db7813eb-home\") pod \"3eacd4e7-191c-4bb7-8735-d0a3db7813eb\" (UID: \"3eacd4e7-191c-4bb7-8735-d0a3db7813eb\") "
Sep 18 19:59:38.236320 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:59:38.236204    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3eacd4e7-191c-4bb7-8735-d0a3db7813eb-model-cache\") pod \"3eacd4e7-191c-4bb7-8735-d0a3db7813eb\" (UID: \"3eacd4e7-191c-4bb7-8735-d0a3db7813eb\") "
Sep 18 19:59:38.236320 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:59:38.236240    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3eacd4e7-191c-4bb7-8735-d0a3db7813eb-tls-certs\") pod \"3eacd4e7-191c-4bb7-8735-d0a3db7813eb\" (UID: \"3eacd4e7-191c-4bb7-8735-d0a3db7813eb\") "
Sep 18 19:59:38.236609 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:59:38.236384    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3eacd4e7-191c-4bb7-8735-d0a3db7813eb-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "3eacd4e7-191c-4bb7-8735-d0a3db7813eb" (UID: "3eacd4e7-191c-4bb7-8735-d0a3db7813eb"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 19:59:38.236609 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:59:38.236458    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3eacd4e7-191c-4bb7-8735-d0a3db7813eb-home" (OuterVolumeSpecName: "home") pod "3eacd4e7-191c-4bb7-8735-d0a3db7813eb" (UID: "3eacd4e7-191c-4bb7-8735-d0a3db7813eb"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 19:59:38.236728 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:59:38.236685    2585 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3eacd4e7-191c-4bb7-8735-d0a3db7813eb-tmp-dir\") on node \"ip-10-0-136-185.ec2.internal\" DevicePath \"\""
Sep 18 19:59:38.236783 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:59:38.236734    2585 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3eacd4e7-191c-4bb7-8735-d0a3db7813eb-home\") on node \"ip-10-0-136-185.ec2.internal\" DevicePath \"\""
Sep 18 19:59:38.236783 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:59:38.236749    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3eacd4e7-191c-4bb7-8735-d0a3db7813eb-model-cache" (OuterVolumeSpecName: "model-cache") pod "3eacd4e7-191c-4bb7-8735-d0a3db7813eb" (UID: "3eacd4e7-191c-4bb7-8735-d0a3db7813eb"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 19:59:38.238510 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:59:38.238481    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3eacd4e7-191c-4bb7-8735-d0a3db7813eb-dshm" (OuterVolumeSpecName: "dshm") pod "3eacd4e7-191c-4bb7-8735-d0a3db7813eb" (UID: "3eacd4e7-191c-4bb7-8735-d0a3db7813eb"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 19:59:38.238709 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:59:38.238689    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3eacd4e7-191c-4bb7-8735-d0a3db7813eb-kube-api-access-rd8j4" (OuterVolumeSpecName: "kube-api-access-rd8j4") pod "3eacd4e7-191c-4bb7-8735-d0a3db7813eb" (UID: "3eacd4e7-191c-4bb7-8735-d0a3db7813eb"). InnerVolumeSpecName "kube-api-access-rd8j4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 19:59:38.239040 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:59:38.239017    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3eacd4e7-191c-4bb7-8735-d0a3db7813eb-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "3eacd4e7-191c-4bb7-8735-d0a3db7813eb" (UID: "3eacd4e7-191c-4bb7-8735-d0a3db7813eb"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 19:59:38.337274 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:59:38.337233    2585 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3eacd4e7-191c-4bb7-8735-d0a3db7813eb-dshm\") on node \"ip-10-0-136-185.ec2.internal\" DevicePath \"\""
Sep 18 19:59:38.337274 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:59:38.337273    2585 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rd8j4\" (UniqueName: \"kubernetes.io/projected/3eacd4e7-191c-4bb7-8735-d0a3db7813eb-kube-api-access-rd8j4\") on node \"ip-10-0-136-185.ec2.internal\" DevicePath \"\""
Sep 18 19:59:38.337504 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:59:38.337288    2585 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3eacd4e7-191c-4bb7-8735-d0a3db7813eb-model-cache\") on node \"ip-10-0-136-185.ec2.internal\" DevicePath \"\""
Sep 18 19:59:38.337504 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:59:38.337301    2585 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3eacd4e7-191c-4bb7-8735-d0a3db7813eb-tls-certs\") on node \"ip-10-0-136-185.ec2.internal\" DevicePath \"\""
Sep 18 19:59:39.018767 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:59:39.018673    2585 generic.go:356] "Generic (PLEG): container finished" podID="3eacd4e7-191c-4bb7-8735-d0a3db7813eb" containerID="c82117ca45ab64de3b7206723c1393bd24f42308c2bae3d0e16871e8e4dd5d3a" exitCode=0
Sep 18 19:59:39.018767 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:59:39.018736    2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-explicit-be2171/explicit-model-be2171-kserve-544f9f7657-j45gg"
Sep 18 19:59:39.018767 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:59:39.018741    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-explicit-be2171/explicit-model-be2171-kserve-544f9f7657-j45gg" event={"ID":"3eacd4e7-191c-4bb7-8735-d0a3db7813eb","Type":"ContainerDied","Data":"c82117ca45ab64de3b7206723c1393bd24f42308c2bae3d0e16871e8e4dd5d3a"}
Sep 18 19:59:39.019395 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:59:39.018780    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-explicit-be2171/explicit-model-be2171-kserve-544f9f7657-j45gg" event={"ID":"3eacd4e7-191c-4bb7-8735-d0a3db7813eb","Type":"ContainerDied","Data":"13cbd3bf8301475a395ecc2e5d5a7da878f3efb661ab62c004b3f053de1180a5"}
Sep 18 19:59:39.019395 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:59:39.018835    2585 scope.go:117] "RemoveContainer" containerID="c82117ca45ab64de3b7206723c1393bd24f42308c2bae3d0e16871e8e4dd5d3a"
Sep 18 19:59:39.028126 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:59:39.028102    2585 scope.go:117] "RemoveContainer" containerID="c82117ca45ab64de3b7206723c1393bd24f42308c2bae3d0e16871e8e4dd5d3a"
Sep 18 19:59:39.028413 ip-10-0-136-185 kubenswrapper[2585]: E0918 19:59:39.028392    2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c82117ca45ab64de3b7206723c1393bd24f42308c2bae3d0e16871e8e4dd5d3a\": container with ID starting with c82117ca45ab64de3b7206723c1393bd24f42308c2bae3d0e16871e8e4dd5d3a not found: ID does not exist" containerID="c82117ca45ab64de3b7206723c1393bd24f42308c2bae3d0e16871e8e4dd5d3a"
Sep 18 19:59:39.028488 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:59:39.028422    2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c82117ca45ab64de3b7206723c1393bd24f42308c2bae3d0e16871e8e4dd5d3a"} err="failed to get container status \"c82117ca45ab64de3b7206723c1393bd24f42308c2bae3d0e16871e8e4dd5d3a\": rpc error: code = NotFound desc = could not find container \"c82117ca45ab64de3b7206723c1393bd24f42308c2bae3d0e16871e8e4dd5d3a\": container with ID starting with c82117ca45ab64de3b7206723c1393bd24f42308c2bae3d0e16871e8e4dd5d3a not found: ID does not exist"
Sep 18 19:59:39.035477 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:59:39.035453    2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-explicit-be2171/explicit-model-be2171-kserve-544f9f7657-j45gg"]
Sep 18 19:59:39.039243 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:59:39.039220    2585 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-explicit-be2171/explicit-model-be2171-kserve-544f9f7657-j45gg"]
Sep 18 19:59:39.428890 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:59:39.428847    2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-870f71c0-8649575654-226c4"]
Sep 18 19:59:39.429189 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:59:39.429146    2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-mt-870f71c0-8649575654-226c4" podUID="aba2d9b1-7fa2-453d-a879-deaf0a26ffaf" containerName="payload-pre-processing" containerID="cri-o://4e3cf69c78a5b73fcbbbce1e5bc0c58f4e613366b5c154e0859a0efe68dfbf7a" gracePeriod=30
Sep 18 19:59:39.688264 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:59:39.688195    2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-870f71c0-8649575654-226c4"
Sep 18 19:59:39.749655 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:59:39.749621    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-j42x4\" (UniqueName: \"kubernetes.io/projected/aba2d9b1-7fa2-453d-a879-deaf0a26ffaf-kube-api-access-j42x4\") pod \"aba2d9b1-7fa2-453d-a879-deaf0a26ffaf\" (UID: \"aba2d9b1-7fa2-453d-a879-deaf0a26ffaf\") "
Sep 18 19:59:39.749836 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:59:39.749712    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/aba2d9b1-7fa2-453d-a879-deaf0a26ffaf-plugins-config-volume\") pod \"aba2d9b1-7fa2-453d-a879-deaf0a26ffaf\" (UID: \"aba2d9b1-7fa2-453d-a879-deaf0a26ffaf\") "
Sep 18 19:59:39.750127 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:59:39.750106    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/aba2d9b1-7fa2-453d-a879-deaf0a26ffaf-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "aba2d9b1-7fa2-453d-a879-deaf0a26ffaf" (UID: "aba2d9b1-7fa2-453d-a879-deaf0a26ffaf"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 19:59:39.751779 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:59:39.751745    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aba2d9b1-7fa2-453d-a879-deaf0a26ffaf-kube-api-access-j42x4" (OuterVolumeSpecName: "kube-api-access-j42x4") pod "aba2d9b1-7fa2-453d-a879-deaf0a26ffaf" (UID: "aba2d9b1-7fa2-453d-a879-deaf0a26ffaf"). InnerVolumeSpecName "kube-api-access-j42x4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 19:59:39.851189 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:59:39.851150    2585 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/aba2d9b1-7fa2-453d-a879-deaf0a26ffaf-plugins-config-volume\") on node \"ip-10-0-136-185.ec2.internal\" DevicePath \"\""
Sep 18 19:59:39.851189 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:59:39.851182    2585 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j42x4\" (UniqueName: \"kubernetes.io/projected/aba2d9b1-7fa2-453d-a879-deaf0a26ffaf-kube-api-access-j42x4\") on node \"ip-10-0-136-185.ec2.internal\" DevicePath \"\""
Sep 18 19:59:40.025720 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:59:40.025634    2585 generic.go:356] "Generic (PLEG): container finished" podID="aba2d9b1-7fa2-453d-a879-deaf0a26ffaf" containerID="4e3cf69c78a5b73fcbbbce1e5bc0c58f4e613366b5c154e0859a0efe68dfbf7a" exitCode=0
Sep 18 19:59:40.025720 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:59:40.025681    2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-870f71c0-8649575654-226c4"
Sep 18 19:59:40.026220 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:59:40.025724    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-870f71c0-8649575654-226c4" event={"ID":"aba2d9b1-7fa2-453d-a879-deaf0a26ffaf","Type":"ContainerDied","Data":"4e3cf69c78a5b73fcbbbce1e5bc0c58f4e613366b5c154e0859a0efe68dfbf7a"}
Sep 18 19:59:40.026220 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:59:40.025776    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-870f71c0-8649575654-226c4" event={"ID":"aba2d9b1-7fa2-453d-a879-deaf0a26ffaf","Type":"ContainerDied","Data":"e51bf06943f39db6c1615129e60abd8ac5f5de7135938665f092cd4b9d78076b"}
Sep 18 19:59:40.026220 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:59:40.025816    2585 scope.go:117] "RemoveContainer" containerID="4e3cf69c78a5b73fcbbbce1e5bc0c58f4e613366b5c154e0859a0efe68dfbf7a"
Sep 18 19:59:40.035188 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:59:40.035173    2585 scope.go:117] "RemoveContainer" containerID="4e3cf69c78a5b73fcbbbce1e5bc0c58f4e613366b5c154e0859a0efe68dfbf7a"
Sep 18 19:59:40.035437 ip-10-0-136-185 kubenswrapper[2585]: E0918 19:59:40.035418    2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4e3cf69c78a5b73fcbbbce1e5bc0c58f4e613366b5c154e0859a0efe68dfbf7a\": container with ID starting with 4e3cf69c78a5b73fcbbbce1e5bc0c58f4e613366b5c154e0859a0efe68dfbf7a not found: ID does not exist" containerID="4e3cf69c78a5b73fcbbbce1e5bc0c58f4e613366b5c154e0859a0efe68dfbf7a"
Sep 18 19:59:40.035482 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:59:40.035445    2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4e3cf69c78a5b73fcbbbce1e5bc0c58f4e613366b5c154e0859a0efe68dfbf7a"} err="failed to get container status \"4e3cf69c78a5b73fcbbbce1e5bc0c58f4e613366b5c154e0859a0efe68dfbf7a\": rpc error: code = NotFound desc = could not find container \"4e3cf69c78a5b73fcbbbce1e5bc0c58f4e613366b5c154e0859a0efe68dfbf7a\": container with ID starting with 4e3cf69c78a5b73fcbbbce1e5bc0c58f4e613366b5c154e0859a0efe68dfbf7a not found: ID does not exist"
Sep 18 19:59:40.053919 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:59:40.053894    2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-870f71c0-8649575654-226c4"]
Sep 18 19:59:40.058835 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:59:40.058812    2585 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-870f71c0-8649575654-226c4"]
Sep 18 19:59:40.662628 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:59:40.662592    2585 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3eacd4e7-191c-4bb7-8735-d0a3db7813eb" path="/var/lib/kubelet/pods/3eacd4e7-191c-4bb7-8735-d0a3db7813eb/volumes"
Sep 18 19:59:40.663071 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:59:40.663048    2585 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="aba2d9b1-7fa2-453d-a879-deaf0a26ffaf" path="/var/lib/kubelet/pods/aba2d9b1-7fa2-453d-a879-deaf0a26ffaf/volumes"
Sep 18 19:59:41.790994 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:59:41.790965    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-d94b9e/distinct-w0-d94b9e-kserve-654c9b789b-w7qlc"
Sep 18 19:59:41.791434 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:59:41.791250    2585 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-d94b9e/distinct-w0-d94b9e-kserve-654c9b789b-w7qlc"
Sep 18 19:59:41.803544 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:59:41.803519    2585 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-d94b9e/distinct-w0-d94b9e-kserve-654c9b789b-w7qlc"
Sep 18 19:59:42.051176 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:59:42.051097    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-d94b9e/distinct-w0-d94b9e-kserve-654c9b789b-w7qlc"
Sep 18 19:59:42.691429 ip-10-0-136-185 kubenswrapper[2585]: I0918 19:59:42.691393    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-xmwh7"]
Sep 18 20:00:00.149657 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:00:00.149618    2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-d94b9e-29829360-psf6f"]
Sep 18 20:00:00.150274 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:00:00.150073    2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3eacd4e7-191c-4bb7-8735-d0a3db7813eb" containerName="main"
Sep 18 20:00:00.150274 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:00:00.150094    2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="3eacd4e7-191c-4bb7-8735-d0a3db7813eb" containerName="main"
Sep 18 20:00:00.150274 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:00:00.150138    2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="aba2d9b1-7fa2-453d-a879-deaf0a26ffaf" containerName="payload-pre-processing"
Sep 18 20:00:00.150274 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:00:00.150147    2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="aba2d9b1-7fa2-453d-a879-deaf0a26ffaf" containerName="payload-pre-processing"
Sep 18 20:00:00.150274 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:00:00.150262    2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="3eacd4e7-191c-4bb7-8735-d0a3db7813eb" containerName="main"
Sep 18 20:00:00.150274 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:00:00.150275    2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="aba2d9b1-7fa2-453d-a879-deaf0a26ffaf" containerName="payload-pre-processing"
Sep 18 20:00:00.155166 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:00:00.155145    2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-d5f7da-29829360-vxqct"]
Sep 18 20:00:00.155285 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:00:00.155265    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-d94b9e-29829360-psf6f"
Sep 18 20:00:00.158849 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:00:00.158817    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-5ttnn\""
Sep 18 20:00:00.165706 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:00:00.165675    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-d5f7da-29829360-vxqct"
Sep 18 20:00:00.165884 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:00:00.165847    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-d94b9e-29829360-psf6f"]
Sep 18 20:00:00.169970 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:00:00.169944    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-d5f7da-29829360-vxqct"]
Sep 18 20:00:00.189863 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:00:00.189836    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-d94b9e-29829360-psf6f"
Sep 18 20:00:00.197019 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:00:00.196997    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-d5f7da-29829360-vxqct"
Sep 18 20:00:00.475655 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:00:00.475628    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-d94b9e-29829360-psf6f"]
Sep 18 20:00:00.476970 ip-10-0-136-185 kubenswrapper[2585]: W0918 20:00:00.476943    2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod915a04ad_037d_4a25_92f6_5ff7da295ac0.slice/crio-1043ef54dc78afcc90e05f92f648d26053b9b33f8fe7590ece0d0b0be0588997 WatchSource:0}: Error finding container 1043ef54dc78afcc90e05f92f648d26053b9b33f8fe7590ece0d0b0be0588997: Status 404 returned error can't find the container with id 1043ef54dc78afcc90e05f92f648d26053b9b33f8fe7590ece0d0b0be0588997
Sep 18 20:00:00.487523 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:00:00.487494    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-d5f7da-29829360-vxqct"]
Sep 18 20:00:00.490635 ip-10-0-136-185 kubenswrapper[2585]: W0918 20:00:00.490610    2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod39a05d80_8e7b_4071_84c3_981703405b45.slice/crio-bd67d8690362576d71665061db5302cc7c44a6fc94e5b23ecb4290a73cc4f045 WatchSource:0}: Error finding container bd67d8690362576d71665061db5302cc7c44a6fc94e5b23ecb4290a73cc4f045: Status 404 returned error can't find the container with id bd67d8690362576d71665061db5302cc7c44a6fc94e5b23ecb4290a73cc4f045
Sep 18 20:00:01.137319 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:00:01.137223    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-d5f7da-29829360-vxqct" event={"ID":"39a05d80-8e7b-4071-84c3-981703405b45","Type":"ContainerStarted","Data":"c52b8d643e3e8e95ed9460f2405c0ae900046e78b42603cf639c65d0b4dad206"}
Sep 18 20:00:01.137319 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:00:01.137287    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-d5f7da-29829360-vxqct" event={"ID":"39a05d80-8e7b-4071-84c3-981703405b45","Type":"ContainerStarted","Data":"bd67d8690362576d71665061db5302cc7c44a6fc94e5b23ecb4290a73cc4f045"}
Sep 18 20:00:01.138832 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:00:01.138780    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-d94b9e-29829360-psf6f" event={"ID":"915a04ad-037d-4a25-92f6-5ff7da295ac0","Type":"ContainerStarted","Data":"c791e0f23bd07884d085d8d0221fff3ee65e1153813100493e42a62761824bdc"}
Sep 18 20:00:01.138953 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:00:01.138849    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-d94b9e-29829360-psf6f" event={"ID":"915a04ad-037d-4a25-92f6-5ff7da295ac0","Type":"ContainerStarted","Data":"1043ef54dc78afcc90e05f92f648d26053b9b33f8fe7590ece0d0b0be0588997"}
Sep 18 20:00:01.154502 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:00:01.154460    2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-d5f7da-29829360-vxqct" podStartSLOduration=1.15444807 podStartE2EDuration="1.15444807s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 20:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 20:00:01.152058879 +0000 UTC m=+1242.676664302" watchObservedRunningTime="2026-09-18 20:00:01.15444807 +0000 UTC m=+1242.679053492"
Sep 18 20:00:01.168085 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:00:01.168038    2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-d94b9e-29829360-psf6f" podStartSLOduration=1.168025147 podStartE2EDuration="1.168025147s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 20:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 20:00:01.166116591 +0000 UTC m=+1242.690722013" watchObservedRunningTime="2026-09-18 20:00:01.168025147 +0000 UTC m=+1242.692630570"
Sep 18 20:00:02.145701 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:00:02.145669    2585 generic.go:356] "Generic (PLEG): container finished" podID="39a05d80-8e7b-4071-84c3-981703405b45" containerID="c52b8d643e3e8e95ed9460f2405c0ae900046e78b42603cf639c65d0b4dad206" exitCode=0
Sep 18 20:00:02.145908 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:00:02.145751    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-d5f7da-29829360-vxqct" event={"ID":"39a05d80-8e7b-4071-84c3-981703405b45","Type":"ContainerDied","Data":"c52b8d643e3e8e95ed9460f2405c0ae900046e78b42603cf639c65d0b4dad206"}
Sep 18 20:00:02.147211 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:00:02.147187    2585 generic.go:356] "Generic (PLEG): container finished" podID="915a04ad-037d-4a25-92f6-5ff7da295ac0" containerID="c791e0f23bd07884d085d8d0221fff3ee65e1153813100493e42a62761824bdc" exitCode=0
Sep 18 20:00:02.147326 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:00:02.147247    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-d94b9e-29829360-psf6f" event={"ID":"915a04ad-037d-4a25-92f6-5ff7da295ac0","Type":"ContainerDied","Data":"c791e0f23bd07884d085d8d0221fff3ee65e1153813100493e42a62761824bdc"}
Sep 18 20:00:03.305611 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:00:03.305588    2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-d94b9e-29829360-psf6f"
Sep 18 20:00:03.308302 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:00:03.308283    2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-d5f7da-29829360-vxqct"
Sep 18 20:00:04.159052 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:00:04.159018    2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-d5f7da-29829360-vxqct"
Sep 18 20:00:04.159052 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:00:04.159046    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-d5f7da-29829360-vxqct" event={"ID":"39a05d80-8e7b-4071-84c3-981703405b45","Type":"ContainerDied","Data":"bd67d8690362576d71665061db5302cc7c44a6fc94e5b23ecb4290a73cc4f045"}
Sep 18 20:00:04.159273 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:00:04.159104    2585 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="bd67d8690362576d71665061db5302cc7c44a6fc94e5b23ecb4290a73cc4f045"
Sep 18 20:00:04.160305 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:00:04.160287    2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-d94b9e-29829360-psf6f"
Sep 18 20:00:04.160380 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:00:04.160325    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-d94b9e-29829360-psf6f" event={"ID":"915a04ad-037d-4a25-92f6-5ff7da295ac0","Type":"ContainerDied","Data":"1043ef54dc78afcc90e05f92f648d26053b9b33f8fe7590ece0d0b0be0588997"}
Sep 18 20:00:04.160380 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:00:04.160359    2585 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1043ef54dc78afcc90e05f92f648d26053b9b33f8fe7590ece0d0b0be0588997"
Sep 18 20:00:13.981755 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:00:13.981713    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-xmwh7"]
Sep 18 20:00:42.091950 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:00:42.091909    2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-9838c2-5dc444f4cd-hnphk"]
Sep 18 20:00:42.092422 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:00:42.092326    2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="915a04ad-037d-4a25-92f6-5ff7da295ac0" containerName="cleanup"
Sep 18 20:00:42.092422 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:00:42.092337    2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="915a04ad-037d-4a25-92f6-5ff7da295ac0" containerName="cleanup"
Sep 18 20:00:42.092422 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:00:42.092356    2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="39a05d80-8e7b-4071-84c3-981703405b45" containerName="cleanup"
Sep 18 20:00:42.092422 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:00:42.092362    2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="39a05d80-8e7b-4071-84c3-981703405b45" containerName="cleanup"
Sep 18 20:00:42.092555 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:00:42.092427    2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="915a04ad-037d-4a25-92f6-5ff7da295ac0" containerName="cleanup"
Sep 18 20:00:42.092555 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:00:42.092438    2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="39a05d80-8e7b-4071-84c3-981703405b45" containerName="cleanup"
Sep 18 20:00:42.095452 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:00:42.095435    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-9838c2-5dc444f4cd-hnphk"
Sep 18 20:00:42.097670 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:00:42.097642    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w2-9838c2\""
Sep 18 20:00:42.097814 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:00:42.097685    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w2-9838c2-dockercfg-mv885\""
Sep 18 20:00:42.104689 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:00:42.104665    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-9838c2-5dc444f4cd-hnphk"]
Sep 18 20:00:42.157745 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:00:42.157710    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hzj2t\" (UniqueName: \"kubernetes.io/projected/e5246851-f8aa-4055-9440-376bc57c630f-kube-api-access-hzj2t\") pod \"payload-pre-processing-e2e-shared-b-w2-9838c2-5dc444f4cd-hnphk\" (UID: \"e5246851-f8aa-4055-9440-376bc57c630f\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-9838c2-5dc444f4cd-hnphk"
Sep 18 20:00:42.157920 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:00:42.157760    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e5246851-f8aa-4055-9440-376bc57c630f-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w2-9838c2-5dc444f4cd-hnphk\" (UID: \"e5246851-f8aa-4055-9440-376bc57c630f\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-9838c2-5dc444f4cd-hnphk"
Sep 18 20:00:42.259059 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:00:42.259020    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hzj2t\" (UniqueName: \"kubernetes.io/projected/e5246851-f8aa-4055-9440-376bc57c630f-kube-api-access-hzj2t\") pod \"payload-pre-processing-e2e-shared-b-w2-9838c2-5dc444f4cd-hnphk\" (UID: \"e5246851-f8aa-4055-9440-376bc57c630f\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-9838c2-5dc444f4cd-hnphk"
Sep 18 20:00:42.259227 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:00:42.259068    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e5246851-f8aa-4055-9440-376bc57c630f-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w2-9838c2-5dc444f4cd-hnphk\" (UID: \"e5246851-f8aa-4055-9440-376bc57c630f\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-9838c2-5dc444f4cd-hnphk"
Sep 18 20:00:42.259767 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:00:42.259748    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e5246851-f8aa-4055-9440-376bc57c630f-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w2-9838c2-5dc444f4cd-hnphk\" (UID: \"e5246851-f8aa-4055-9440-376bc57c630f\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-9838c2-5dc444f4cd-hnphk"
Sep 18 20:00:42.269338 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:00:42.269307    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hzj2t\" (UniqueName: \"kubernetes.io/projected/e5246851-f8aa-4055-9440-376bc57c630f-kube-api-access-hzj2t\") pod \"payload-pre-processing-e2e-shared-b-w2-9838c2-5dc444f4cd-hnphk\" (UID: \"e5246851-f8aa-4055-9440-376bc57c630f\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-9838c2-5dc444f4cd-hnphk"
Sep 18 20:00:42.406557 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:00:42.406522    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-9838c2-5dc444f4cd-hnphk"
Sep 18 20:00:42.682031 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:00:42.682001    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-9838c2-5dc444f4cd-hnphk"]
Sep 18 20:00:42.684221 ip-10-0-136-185 kubenswrapper[2585]: W0918 20:00:42.684192    2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode5246851_f8aa_4055_9440_376bc57c630f.slice/crio-5e054a3daef756b6f9217d53aff3e0d6c629e6a5aeda61ccd00bb1887a19cf03 WatchSource:0}: Error finding container 5e054a3daef756b6f9217d53aff3e0d6c629e6a5aeda61ccd00bb1887a19cf03: Status 404 returned error can't find the container with id 5e054a3daef756b6f9217d53aff3e0d6c629e6a5aeda61ccd00bb1887a19cf03
Sep 18 20:00:43.369471 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:00:43.369435    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-9838c2-5dc444f4cd-hnphk" event={"ID":"e5246851-f8aa-4055-9440-376bc57c630f","Type":"ContainerStarted","Data":"2995479d2c1b67bce98790872c234f1d4ca70ee1b71a8f5d10e47fa787ee83dd"}
Sep 18 20:00:43.369911 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:00:43.369483    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-9838c2-5dc444f4cd-hnphk" event={"ID":"e5246851-f8aa-4055-9440-376bc57c630f","Type":"ContainerStarted","Data":"5e054a3daef756b6f9217d53aff3e0d6c629e6a5aeda61ccd00bb1887a19cf03"}
Sep 18 20:00:43.369911 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:00:43.369540    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-9838c2-5dc444f4cd-hnphk"
Sep 18 20:00:43.385663 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:00:43.385609    2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-9838c2-5dc444f4cd-hnphk" podStartSLOduration=1.385593088 podStartE2EDuration="1.385593088s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 20:00:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 20:00:43.384326108 +0000 UTC m=+1284.908931531" watchObservedRunningTime="2026-09-18 20:00:43.385593088 +0000 UTC m=+1284.910198510"
Sep 18 20:00:48.642478 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:00:48.642440    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-xmwh7"]
Sep 18 20:00:54.377576 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:00:54.377540    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-9838c2-5dc444f4cd-hnphk"
Sep 18 20:00:58.182443 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:00:58.182408    2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-674263/e2e-conflict-b-1b1b9f14-kserve-6dc4d67945-bg7x9"]
Sep 18 20:00:58.185898 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:00:58.185880    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-674263/e2e-conflict-b-1b1b9f14-kserve-6dc4d67945-bg7x9"
Sep 18 20:00:58.188168 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:00:58.188149    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-674263\"/\"e2e-conflict-b-1b1b9f14-kserve-self-signed-certs\""
Sep 18 20:00:58.194225 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:00:58.194201    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-674263/e2e-conflict-b-1b1b9f14-kserve-6dc4d67945-bg7x9"]
Sep 18 20:00:58.217181 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:00:58.217144    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/413d2e4e-1595-4b21-898c-baff083776ae-tmp-dir\") pod \"e2e-conflict-b-1b1b9f14-kserve-6dc4d67945-bg7x9\" (UID: \"413d2e4e-1595-4b21-898c-baff083776ae\") " pod="e2e-models-e2e-worker-w4-674263/e2e-conflict-b-1b1b9f14-kserve-6dc4d67945-bg7x9"
Sep 18 20:00:58.217354 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:00:58.217190    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/413d2e4e-1595-4b21-898c-baff083776ae-home\") pod \"e2e-conflict-b-1b1b9f14-kserve-6dc4d67945-bg7x9\" (UID: \"413d2e4e-1595-4b21-898c-baff083776ae\") " pod="e2e-models-e2e-worker-w4-674263/e2e-conflict-b-1b1b9f14-kserve-6dc4d67945-bg7x9"
Sep 18 20:00:58.217354 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:00:58.217218    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/413d2e4e-1595-4b21-898c-baff083776ae-model-cache\") pod \"e2e-conflict-b-1b1b9f14-kserve-6dc4d67945-bg7x9\" (UID: \"413d2e4e-1595-4b21-898c-baff083776ae\") " pod="e2e-models-e2e-worker-w4-674263/e2e-conflict-b-1b1b9f14-kserve-6dc4d67945-bg7x9"
Sep 18 20:00:58.217354 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:00:58.217341    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/413d2e4e-1595-4b21-898c-baff083776ae-tls-certs\") pod \"e2e-conflict-b-1b1b9f14-kserve-6dc4d67945-bg7x9\" (UID: \"413d2e4e-1595-4b21-898c-baff083776ae\") " pod="e2e-models-e2e-worker-w4-674263/e2e-conflict-b-1b1b9f14-kserve-6dc4d67945-bg7x9"
Sep 18 20:00:58.217461 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:00:58.217367    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-btslz\" (UniqueName: \"kubernetes.io/projected/413d2e4e-1595-4b21-898c-baff083776ae-kube-api-access-btslz\") pod \"e2e-conflict-b-1b1b9f14-kserve-6dc4d67945-bg7x9\" (UID: \"413d2e4e-1595-4b21-898c-baff083776ae\") " pod="e2e-models-e2e-worker-w4-674263/e2e-conflict-b-1b1b9f14-kserve-6dc4d67945-bg7x9"
Sep 18 20:00:58.217461 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:00:58.217399    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/413d2e4e-1595-4b21-898c-baff083776ae-dshm\") pod \"e2e-conflict-b-1b1b9f14-kserve-6dc4d67945-bg7x9\" (UID: \"413d2e4e-1595-4b21-898c-baff083776ae\") " pod="e2e-models-e2e-worker-w4-674263/e2e-conflict-b-1b1b9f14-kserve-6dc4d67945-bg7x9"
Sep 18 20:00:58.318126 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:00:58.318083    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/413d2e4e-1595-4b21-898c-baff083776ae-tls-certs\") pod \"e2e-conflict-b-1b1b9f14-kserve-6dc4d67945-bg7x9\" (UID: \"413d2e4e-1595-4b21-898c-baff083776ae\") " pod="e2e-models-e2e-worker-w4-674263/e2e-conflict-b-1b1b9f14-kserve-6dc4d67945-bg7x9"
Sep 18 20:00:58.318126 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:00:58.318130    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-btslz\" (UniqueName: \"kubernetes.io/projected/413d2e4e-1595-4b21-898c-baff083776ae-kube-api-access-btslz\") pod \"e2e-conflict-b-1b1b9f14-kserve-6dc4d67945-bg7x9\" (UID: \"413d2e4e-1595-4b21-898c-baff083776ae\") " pod="e2e-models-e2e-worker-w4-674263/e2e-conflict-b-1b1b9f14-kserve-6dc4d67945-bg7x9"
Sep 18 20:00:58.318377 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:00:58.318170    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/413d2e4e-1595-4b21-898c-baff083776ae-dshm\") pod \"e2e-conflict-b-1b1b9f14-kserve-6dc4d67945-bg7x9\" (UID: \"413d2e4e-1595-4b21-898c-baff083776ae\") " pod="e2e-models-e2e-worker-w4-674263/e2e-conflict-b-1b1b9f14-kserve-6dc4d67945-bg7x9"
Sep 18 20:00:58.318377 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:00:58.318193    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/413d2e4e-1595-4b21-898c-baff083776ae-tmp-dir\") pod \"e2e-conflict-b-1b1b9f14-kserve-6dc4d67945-bg7x9\" (UID: \"413d2e4e-1595-4b21-898c-baff083776ae\") " pod="e2e-models-e2e-worker-w4-674263/e2e-conflict-b-1b1b9f14-kserve-6dc4d67945-bg7x9"
Sep 18 20:00:58.318377 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:00:58.318222    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/413d2e4e-1595-4b21-898c-baff083776ae-home\") pod \"e2e-conflict-b-1b1b9f14-kserve-6dc4d67945-bg7x9\" (UID: \"413d2e4e-1595-4b21-898c-baff083776ae\") " pod="e2e-models-e2e-worker-w4-674263/e2e-conflict-b-1b1b9f14-kserve-6dc4d67945-bg7x9"
Sep 18 20:00:58.318377 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:00:58.318271    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/413d2e4e-1595-4b21-898c-baff083776ae-model-cache\") pod \"e2e-conflict-b-1b1b9f14-kserve-6dc4d67945-bg7x9\" (UID: \"413d2e4e-1595-4b21-898c-baff083776ae\") " pod="e2e-models-e2e-worker-w4-674263/e2e-conflict-b-1b1b9f14-kserve-6dc4d67945-bg7x9"
Sep 18 20:00:58.318675 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:00:58.318653    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/413d2e4e-1595-4b21-898c-baff083776ae-tmp-dir\") pod \"e2e-conflict-b-1b1b9f14-kserve-6dc4d67945-bg7x9\" (UID: \"413d2e4e-1595-4b21-898c-baff083776ae\") " pod="e2e-models-e2e-worker-w4-674263/e2e-conflict-b-1b1b9f14-kserve-6dc4d67945-bg7x9"
Sep 18 20:00:58.318748 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:00:58.318706    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/413d2e4e-1595-4b21-898c-baff083776ae-home\") pod \"e2e-conflict-b-1b1b9f14-kserve-6dc4d67945-bg7x9\" (UID: \"413d2e4e-1595-4b21-898c-baff083776ae\") " pod="e2e-models-e2e-worker-w4-674263/e2e-conflict-b-1b1b9f14-kserve-6dc4d67945-bg7x9"
Sep 18 20:00:58.318843 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:00:58.318766    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/413d2e4e-1595-4b21-898c-baff083776ae-model-cache\") pod \"e2e-conflict-b-1b1b9f14-kserve-6dc4d67945-bg7x9\" (UID: \"413d2e4e-1595-4b21-898c-baff083776ae\") " pod="e2e-models-e2e-worker-w4-674263/e2e-conflict-b-1b1b9f14-kserve-6dc4d67945-bg7x9"
Sep 18 20:00:58.320621 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:00:58.320591    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/413d2e4e-1595-4b21-898c-baff083776ae-dshm\") pod \"e2e-conflict-b-1b1b9f14-kserve-6dc4d67945-bg7x9\" (UID: \"413d2e4e-1595-4b21-898c-baff083776ae\") " pod="e2e-models-e2e-worker-w4-674263/e2e-conflict-b-1b1b9f14-kserve-6dc4d67945-bg7x9"
Sep 18 20:00:58.320742 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:00:58.320723    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/413d2e4e-1595-4b21-898c-baff083776ae-tls-certs\") pod \"e2e-conflict-b-1b1b9f14-kserve-6dc4d67945-bg7x9\" (UID: \"413d2e4e-1595-4b21-898c-baff083776ae\") " pod="e2e-models-e2e-worker-w4-674263/e2e-conflict-b-1b1b9f14-kserve-6dc4d67945-bg7x9"
Sep 18 20:00:58.325475 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:00:58.325452    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-btslz\" (UniqueName: \"kubernetes.io/projected/413d2e4e-1595-4b21-898c-baff083776ae-kube-api-access-btslz\") pod \"e2e-conflict-b-1b1b9f14-kserve-6dc4d67945-bg7x9\" (UID: \"413d2e4e-1595-4b21-898c-baff083776ae\") " pod="e2e-models-e2e-worker-w4-674263/e2e-conflict-b-1b1b9f14-kserve-6dc4d67945-bg7x9"
Sep 18 20:00:58.495582 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:00:58.495495    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-674263/e2e-conflict-b-1b1b9f14-kserve-6dc4d67945-bg7x9"
Sep 18 20:00:58.768839 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:00:58.768811    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-674263/e2e-conflict-b-1b1b9f14-kserve-6dc4d67945-bg7x9"]
Sep 18 20:00:58.770854 ip-10-0-136-185 kubenswrapper[2585]: W0918 20:00:58.770826    2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod413d2e4e_1595_4b21_898c_baff083776ae.slice/crio-efe3458b298a9353543213811853086c80e33cf241f955ee32e5006d5114512c WatchSource:0}: Error finding container efe3458b298a9353543213811853086c80e33cf241f955ee32e5006d5114512c: Status 404 returned error can't find the container with id efe3458b298a9353543213811853086c80e33cf241f955ee32e5006d5114512c
Sep 18 20:00:59.455275 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:00:59.455238    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-674263/e2e-conflict-b-1b1b9f14-kserve-6dc4d67945-bg7x9" event={"ID":"413d2e4e-1595-4b21-898c-baff083776ae","Type":"ContainerStarted","Data":"e5e832f68724ebd450b64d8304e50818b1769a352fb2a40fd87a8eb47cfe0179"}
Sep 18 20:00:59.455275 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:00:59.455279    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-674263/e2e-conflict-b-1b1b9f14-kserve-6dc4d67945-bg7x9" event={"ID":"413d2e4e-1595-4b21-898c-baff083776ae","Type":"ContainerStarted","Data":"efe3458b298a9353543213811853086c80e33cf241f955ee32e5006d5114512c"}
Sep 18 20:00:59.472783 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:00:59.472738    2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-674263/e2e-conflict-b-1b1b9f14-kserve-6dc4d67945-bg7x9" podStartSLOduration=1.47271772 podStartE2EDuration="1.47271772s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 20:00:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 20:00:59.472339386 +0000 UTC m=+1300.996944810" watchObservedRunningTime="2026-09-18 20:00:59.47271772 +0000 UTC m=+1300.997323233"
Sep 18 20:01:05.204231 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:01:05.204197    2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-d6291d/test-model-d6291d-kserve-57fd894cc7-lnllx"]
Sep 18 20:01:08.496208 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:01:08.496170    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-674263/e2e-conflict-b-1b1b9f14-kserve-6dc4d67945-bg7x9"
Sep 18 20:01:08.496655 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:01:08.496246    2585 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-674263/e2e-conflict-b-1b1b9f14-kserve-6dc4d67945-bg7x9"
Sep 18 20:01:08.509096 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:01:08.509068    2585 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-674263/e2e-conflict-b-1b1b9f14-kserve-6dc4d67945-bg7x9"
Sep 18 20:01:09.520145 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:01:09.520110    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-674263/e2e-conflict-b-1b1b9f14-kserve-6dc4d67945-bg7x9"
Sep 18 20:01:20.224040 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:01:20.223997    2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-inf-a-d6291d/test-model-d6291d-kserve-57fd894cc7-lnllx" podUID="f8356480-0e3a-4748-aa8b-af089ce729ef" containerName="main" containerID="cri-o://94cfc271d2a4d1495c14d6efd35a314ffe6b8bbafc2656c36db0a6d98cc1d13b" gracePeriod=45
Sep 18 20:01:20.478675 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:01:20.478618    2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-d6291d/test-model-d6291d-kserve-57fd894cc7-lnllx"
Sep 18 20:01:20.577868 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:01:20.577827    2585 generic.go:356] "Generic (PLEG): container finished" podID="f8356480-0e3a-4748-aa8b-af089ce729ef" containerID="94cfc271d2a4d1495c14d6efd35a314ffe6b8bbafc2656c36db0a6d98cc1d13b" exitCode=0
Sep 18 20:01:20.578060 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:01:20.577904    2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-d6291d/test-model-d6291d-kserve-57fd894cc7-lnllx"
Sep 18 20:01:20.578060 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:01:20.577915    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-d6291d/test-model-d6291d-kserve-57fd894cc7-lnllx" event={"ID":"f8356480-0e3a-4748-aa8b-af089ce729ef","Type":"ContainerDied","Data":"94cfc271d2a4d1495c14d6efd35a314ffe6b8bbafc2656c36db0a6d98cc1d13b"}
Sep 18 20:01:20.578060 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:01:20.577966    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-d6291d/test-model-d6291d-kserve-57fd894cc7-lnllx" event={"ID":"f8356480-0e3a-4748-aa8b-af089ce729ef","Type":"ContainerDied","Data":"a9013906413ddef8644e1948a249ec33edb349f72192ed08aecd633e5f563206"}
Sep 18 20:01:20.578060 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:01:20.577987    2585 scope.go:117] "RemoveContainer" containerID="94cfc271d2a4d1495c14d6efd35a314ffe6b8bbafc2656c36db0a6d98cc1d13b"
Sep 18 20:01:20.586730 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:01:20.586708    2585 scope.go:117] "RemoveContainer" containerID="94cfc271d2a4d1495c14d6efd35a314ffe6b8bbafc2656c36db0a6d98cc1d13b"
Sep 18 20:01:20.587011 ip-10-0-136-185 kubenswrapper[2585]: E0918 20:01:20.586990    2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"94cfc271d2a4d1495c14d6efd35a314ffe6b8bbafc2656c36db0a6d98cc1d13b\": container with ID starting with 94cfc271d2a4d1495c14d6efd35a314ffe6b8bbafc2656c36db0a6d98cc1d13b not found: ID does not exist" containerID="94cfc271d2a4d1495c14d6efd35a314ffe6b8bbafc2656c36db0a6d98cc1d13b"
Sep 18 20:01:20.587089 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:01:20.587018    2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"94cfc271d2a4d1495c14d6efd35a314ffe6b8bbafc2656c36db0a6d98cc1d13b"} err="failed to get container status \"94cfc271d2a4d1495c14d6efd35a314ffe6b8bbafc2656c36db0a6d98cc1d13b\": rpc error: code = NotFound desc = could not find container \"94cfc271d2a4d1495c14d6efd35a314ffe6b8bbafc2656c36db0a6d98cc1d13b\": container with ID starting with 94cfc271d2a4d1495c14d6efd35a314ffe6b8bbafc2656c36db0a6d98cc1d13b not found: ID does not exist"
Sep 18 20:01:20.650289 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:01:20.650254    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f8356480-0e3a-4748-aa8b-af089ce729ef-dshm\") pod \"f8356480-0e3a-4748-aa8b-af089ce729ef\" (UID: \"f8356480-0e3a-4748-aa8b-af089ce729ef\") "
Sep 18 20:01:20.650289 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:01:20.650293    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f8356480-0e3a-4748-aa8b-af089ce729ef-home\") pod \"f8356480-0e3a-4748-aa8b-af089ce729ef\" (UID: \"f8356480-0e3a-4748-aa8b-af089ce729ef\") "
Sep 18 20:01:20.650527 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:01:20.650324    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f8356480-0e3a-4748-aa8b-af089ce729ef-tls-certs\") pod \"f8356480-0e3a-4748-aa8b-af089ce729ef\" (UID: \"f8356480-0e3a-4748-aa8b-af089ce729ef\") "
Sep 18 20:01:20.650527 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:01:20.650355    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f8356480-0e3a-4748-aa8b-af089ce729ef-tmp-dir\") pod \"f8356480-0e3a-4748-aa8b-af089ce729ef\" (UID: \"f8356480-0e3a-4748-aa8b-af089ce729ef\") "
Sep 18 20:01:20.650527 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:01:20.650389    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f8356480-0e3a-4748-aa8b-af089ce729ef-model-cache\") pod \"f8356480-0e3a-4748-aa8b-af089ce729ef\" (UID: \"f8356480-0e3a-4748-aa8b-af089ce729ef\") "
Sep 18 20:01:20.650527 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:01:20.650451    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-t62f2\" (UniqueName: \"kubernetes.io/projected/f8356480-0e3a-4748-aa8b-af089ce729ef-kube-api-access-t62f2\") pod \"f8356480-0e3a-4748-aa8b-af089ce729ef\" (UID: \"f8356480-0e3a-4748-aa8b-af089ce729ef\") "
Sep 18 20:01:20.650723 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:01:20.650623    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f8356480-0e3a-4748-aa8b-af089ce729ef-home" (OuterVolumeSpecName: "home") pod "f8356480-0e3a-4748-aa8b-af089ce729ef" (UID: "f8356480-0e3a-4748-aa8b-af089ce729ef"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 20:01:20.650723 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:01:20.650674    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f8356480-0e3a-4748-aa8b-af089ce729ef-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "f8356480-0e3a-4748-aa8b-af089ce729ef" (UID: "f8356480-0e3a-4748-aa8b-af089ce729ef"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 20:01:20.650723 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:01:20.650694    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f8356480-0e3a-4748-aa8b-af089ce729ef-model-cache" (OuterVolumeSpecName: "model-cache") pod "f8356480-0e3a-4748-aa8b-af089ce729ef" (UID: "f8356480-0e3a-4748-aa8b-af089ce729ef"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 20:01:20.650887 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:01:20.650862    2585 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f8356480-0e3a-4748-aa8b-af089ce729ef-home\") on node \"ip-10-0-136-185.ec2.internal\" DevicePath \"\""
Sep 18 20:01:20.650887 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:01:20.650876    2585 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f8356480-0e3a-4748-aa8b-af089ce729ef-tmp-dir\") on node \"ip-10-0-136-185.ec2.internal\" DevicePath \"\""
Sep 18 20:01:20.650887 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:01:20.650886    2585 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f8356480-0e3a-4748-aa8b-af089ce729ef-model-cache\") on node \"ip-10-0-136-185.ec2.internal\" DevicePath \"\""
Sep 18 20:01:20.652707 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:01:20.652680    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f8356480-0e3a-4748-aa8b-af089ce729ef-kube-api-access-t62f2" (OuterVolumeSpecName: "kube-api-access-t62f2") pod "f8356480-0e3a-4748-aa8b-af089ce729ef" (UID: "f8356480-0e3a-4748-aa8b-af089ce729ef"). InnerVolumeSpecName "kube-api-access-t62f2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 20:01:20.652993 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:01:20.652965    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f8356480-0e3a-4748-aa8b-af089ce729ef-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "f8356480-0e3a-4748-aa8b-af089ce729ef" (UID: "f8356480-0e3a-4748-aa8b-af089ce729ef"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 20:01:20.652993 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:01:20.652967    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f8356480-0e3a-4748-aa8b-af089ce729ef-dshm" (OuterVolumeSpecName: "dshm") pod "f8356480-0e3a-4748-aa8b-af089ce729ef" (UID: "f8356480-0e3a-4748-aa8b-af089ce729ef"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 20:01:20.691531 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:01:20.691493    2585 scope.go:117] "RemoveContainer" containerID="d3534b450bf252f47ed48e3397a5bbbe1c758344b559b30c5e722f2076ba0092"
Sep 18 20:01:20.752228 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:01:20.752147    2585 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f8356480-0e3a-4748-aa8b-af089ce729ef-dshm\") on node \"ip-10-0-136-185.ec2.internal\" DevicePath \"\""
Sep 18 20:01:20.752228 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:01:20.752175    2585 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f8356480-0e3a-4748-aa8b-af089ce729ef-tls-certs\") on node \"ip-10-0-136-185.ec2.internal\" DevicePath \"\""
Sep 18 20:01:20.752228 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:01:20.752186    2585 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t62f2\" (UniqueName: \"kubernetes.io/projected/f8356480-0e3a-4748-aa8b-af089ce729ef-kube-api-access-t62f2\") on node \"ip-10-0-136-185.ec2.internal\" DevicePath \"\""
Sep 18 20:01:20.893948 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:01:20.893916    2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-d6291d/test-model-d6291d-kserve-57fd894cc7-lnllx"]
Sep 18 20:01:20.897419 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:01:20.897387    2585 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-a-d6291d/test-model-d6291d-kserve-57fd894cc7-lnllx"]
Sep 18 20:01:22.661990 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:01:22.661947    2585 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f8356480-0e3a-4748-aa8b-af089ce729ef" path="/var/lib/kubelet/pods/f8356480-0e3a-4748-aa8b-af089ce729ef/volumes"
Sep 18 20:01:28.686739 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:01:28.686698    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-xmwh7"]
Sep 18 20:01:39.073322 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:01:39.073230    2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-noten-03d45c/noten-model-03d45c-kserve-769bbbd96c-fq9qr"]
Sep 18 20:01:39.073831 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:01:39.073699    2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f8356480-0e3a-4748-aa8b-af089ce729ef" containerName="main"
Sep 18 20:01:39.073831 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:01:39.073714    2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="f8356480-0e3a-4748-aa8b-af089ce729ef" containerName="main"
Sep 18 20:01:39.073831 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:01:39.073813    2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="f8356480-0e3a-4748-aa8b-af089ce729ef" containerName="main"
Sep 18 20:01:39.076920 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:01:39.076904    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-noten-03d45c/noten-model-03d45c-kserve-769bbbd96c-fq9qr"
Sep 18 20:01:39.080170 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:01:39.080141    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-noten-03d45c\"/\"noten-model-03d45c-kserve-self-signed-certs\""
Sep 18 20:01:39.080170 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:01:39.080162    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-noten-03d45c\"/\"kube-root-ca.crt\""
Sep 18 20:01:39.080378 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:01:39.080145    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-noten-03d45c\"/\"default-dockercfg-xd4kq\""
Sep 18 20:01:39.080378 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:01:39.080141    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-noten-03d45c\"/\"openshift-service-ca.crt\""
Sep 18 20:01:39.089688 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:01:39.089664    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-noten-03d45c/noten-model-03d45c-kserve-769bbbd96c-fq9qr"]
Sep 18 20:01:39.132932 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:01:39.132895    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ht45v\" (UniqueName: \"kubernetes.io/projected/12fda611-11c3-4df9-8de0-6898ea0c4ba9-kube-api-access-ht45v\") pod \"noten-model-03d45c-kserve-769bbbd96c-fq9qr\" (UID: \"12fda611-11c3-4df9-8de0-6898ea0c4ba9\") " pod="ai-tenant-e2e-noten-03d45c/noten-model-03d45c-kserve-769bbbd96c-fq9qr"
Sep 18 20:01:39.133155 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:01:39.132938    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/12fda611-11c3-4df9-8de0-6898ea0c4ba9-model-cache\") pod \"noten-model-03d45c-kserve-769bbbd96c-fq9qr\" (UID: \"12fda611-11c3-4df9-8de0-6898ea0c4ba9\") " pod="ai-tenant-e2e-noten-03d45c/noten-model-03d45c-kserve-769bbbd96c-fq9qr"
Sep 18 20:01:39.133155 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:01:39.133009    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/12fda611-11c3-4df9-8de0-6898ea0c4ba9-home\") pod \"noten-model-03d45c-kserve-769bbbd96c-fq9qr\" (UID: \"12fda611-11c3-4df9-8de0-6898ea0c4ba9\") " pod="ai-tenant-e2e-noten-03d45c/noten-model-03d45c-kserve-769bbbd96c-fq9qr"
Sep 18 20:01:39.133155 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:01:39.133073    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/12fda611-11c3-4df9-8de0-6898ea0c4ba9-dshm\") pod \"noten-model-03d45c-kserve-769bbbd96c-fq9qr\" (UID: \"12fda611-11c3-4df9-8de0-6898ea0c4ba9\") " pod="ai-tenant-e2e-noten-03d45c/noten-model-03d45c-kserve-769bbbd96c-fq9qr"
Sep 18 20:01:39.133155 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:01:39.133104    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/12fda611-11c3-4df9-8de0-6898ea0c4ba9-tmp-dir\") pod \"noten-model-03d45c-kserve-769bbbd96c-fq9qr\" (UID: \"12fda611-11c3-4df9-8de0-6898ea0c4ba9\") " pod="ai-tenant-e2e-noten-03d45c/noten-model-03d45c-kserve-769bbbd96c-fq9qr"
Sep 18 20:01:39.133155 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:01:39.133142    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/12fda611-11c3-4df9-8de0-6898ea0c4ba9-tls-certs\") pod \"noten-model-03d45c-kserve-769bbbd96c-fq9qr\" (UID: \"12fda611-11c3-4df9-8de0-6898ea0c4ba9\") " pod="ai-tenant-e2e-noten-03d45c/noten-model-03d45c-kserve-769bbbd96c-fq9qr"
Sep 18 20:01:39.234269 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:01:39.234233    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ht45v\" (UniqueName: \"kubernetes.io/projected/12fda611-11c3-4df9-8de0-6898ea0c4ba9-kube-api-access-ht45v\") pod \"noten-model-03d45c-kserve-769bbbd96c-fq9qr\" (UID: \"12fda611-11c3-4df9-8de0-6898ea0c4ba9\") " pod="ai-tenant-e2e-noten-03d45c/noten-model-03d45c-kserve-769bbbd96c-fq9qr"
Sep 18 20:01:39.234471 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:01:39.234277    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/12fda611-11c3-4df9-8de0-6898ea0c4ba9-model-cache\") pod \"noten-model-03d45c-kserve-769bbbd96c-fq9qr\" (UID: \"12fda611-11c3-4df9-8de0-6898ea0c4ba9\") " pod="ai-tenant-e2e-noten-03d45c/noten-model-03d45c-kserve-769bbbd96c-fq9qr"
Sep 18 20:01:39.234471 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:01:39.234340    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/12fda611-11c3-4df9-8de0-6898ea0c4ba9-home\") pod \"noten-model-03d45c-kserve-769bbbd96c-fq9qr\" (UID: \"12fda611-11c3-4df9-8de0-6898ea0c4ba9\") " pod="ai-tenant-e2e-noten-03d45c/noten-model-03d45c-kserve-769bbbd96c-fq9qr"
Sep 18 20:01:39.234471 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:01:39.234409    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/12fda611-11c3-4df9-8de0-6898ea0c4ba9-dshm\") pod \"noten-model-03d45c-kserve-769bbbd96c-fq9qr\" (UID: \"12fda611-11c3-4df9-8de0-6898ea0c4ba9\") " pod="ai-tenant-e2e-noten-03d45c/noten-model-03d45c-kserve-769bbbd96c-fq9qr"
Sep 18 20:01:39.234471 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:01:39.234441    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/12fda611-11c3-4df9-8de0-6898ea0c4ba9-tmp-dir\") pod \"noten-model-03d45c-kserve-769bbbd96c-fq9qr\" (UID: \"12fda611-11c3-4df9-8de0-6898ea0c4ba9\") " pod="ai-tenant-e2e-noten-03d45c/noten-model-03d45c-kserve-769bbbd96c-fq9qr"
Sep 18 20:01:39.234697 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:01:39.234497    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/12fda611-11c3-4df9-8de0-6898ea0c4ba9-tls-certs\") pod \"noten-model-03d45c-kserve-769bbbd96c-fq9qr\" (UID: \"12fda611-11c3-4df9-8de0-6898ea0c4ba9\") " pod="ai-tenant-e2e-noten-03d45c/noten-model-03d45c-kserve-769bbbd96c-fq9qr"
Sep 18 20:01:39.234855 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:01:39.234820    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/12fda611-11c3-4df9-8de0-6898ea0c4ba9-model-cache\") pod \"noten-model-03d45c-kserve-769bbbd96c-fq9qr\" (UID: \"12fda611-11c3-4df9-8de0-6898ea0c4ba9\") " pod="ai-tenant-e2e-noten-03d45c/noten-model-03d45c-kserve-769bbbd96c-fq9qr"
Sep 18 20:01:39.234855 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:01:39.234834    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/12fda611-11c3-4df9-8de0-6898ea0c4ba9-home\") pod \"noten-model-03d45c-kserve-769bbbd96c-fq9qr\" (UID: \"12fda611-11c3-4df9-8de0-6898ea0c4ba9\") " pod="ai-tenant-e2e-noten-03d45c/noten-model-03d45c-kserve-769bbbd96c-fq9qr"
Sep 18 20:01:39.235001 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:01:39.234943    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/12fda611-11c3-4df9-8de0-6898ea0c4ba9-tmp-dir\") pod \"noten-model-03d45c-kserve-769bbbd96c-fq9qr\" (UID: \"12fda611-11c3-4df9-8de0-6898ea0c4ba9\") " pod="ai-tenant-e2e-noten-03d45c/noten-model-03d45c-kserve-769bbbd96c-fq9qr"
Sep 18 20:01:39.236697 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:01:39.236676    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/12fda611-11c3-4df9-8de0-6898ea0c4ba9-dshm\") pod \"noten-model-03d45c-kserve-769bbbd96c-fq9qr\" (UID: \"12fda611-11c3-4df9-8de0-6898ea0c4ba9\") " pod="ai-tenant-e2e-noten-03d45c/noten-model-03d45c-kserve-769bbbd96c-fq9qr"
Sep 18 20:01:39.237087 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:01:39.237069    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/12fda611-11c3-4df9-8de0-6898ea0c4ba9-tls-certs\") pod \"noten-model-03d45c-kserve-769bbbd96c-fq9qr\" (UID: \"12fda611-11c3-4df9-8de0-6898ea0c4ba9\") " pod="ai-tenant-e2e-noten-03d45c/noten-model-03d45c-kserve-769bbbd96c-fq9qr"
Sep 18 20:01:39.246618 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:01:39.246594    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ht45v\" (UniqueName: \"kubernetes.io/projected/12fda611-11c3-4df9-8de0-6898ea0c4ba9-kube-api-access-ht45v\") pod \"noten-model-03d45c-kserve-769bbbd96c-fq9qr\" (UID: \"12fda611-11c3-4df9-8de0-6898ea0c4ba9\") " pod="ai-tenant-e2e-noten-03d45c/noten-model-03d45c-kserve-769bbbd96c-fq9qr"
Sep 18 20:01:39.387098 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:01:39.387007    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-noten-03d45c/noten-model-03d45c-kserve-769bbbd96c-fq9qr"
Sep 18 20:01:39.658693 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:01:39.658667    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-noten-03d45c/noten-model-03d45c-kserve-769bbbd96c-fq9qr"]
Sep 18 20:01:39.661169 ip-10-0-136-185 kubenswrapper[2585]: W0918 20:01:39.661141    2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod12fda611_11c3_4df9_8de0_6898ea0c4ba9.slice/crio-9feeea8ddbdd5e982a2790e82f6095faaca5700f8c25e60ea46482e7f9f4d23e WatchSource:0}: Error finding container 9feeea8ddbdd5e982a2790e82f6095faaca5700f8c25e60ea46482e7f9f4d23e: Status 404 returned error can't find the container with id 9feeea8ddbdd5e982a2790e82f6095faaca5700f8c25e60ea46482e7f9f4d23e
Sep 18 20:01:39.680606 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:01:39.680579    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-noten-03d45c/noten-model-03d45c-kserve-769bbbd96c-fq9qr" event={"ID":"12fda611-11c3-4df9-8de0-6898ea0c4ba9","Type":"ContainerStarted","Data":"9feeea8ddbdd5e982a2790e82f6095faaca5700f8c25e60ea46482e7f9f4d23e"}
Sep 18 20:01:40.686880 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:01:40.686846    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-noten-03d45c/noten-model-03d45c-kserve-769bbbd96c-fq9qr" event={"ID":"12fda611-11c3-4df9-8de0-6898ea0c4ba9","Type":"ContainerStarted","Data":"a91f448c64603e87267c1fe5060608ea3a9eb9ebc5803aa7d5131b39bde0d3cb"}
Sep 18 20:01:40.704840 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:01:40.704775    2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-noten-03d45c/noten-model-03d45c-kserve-769bbbd96c-fq9qr" podStartSLOduration=1.7047584599999999 podStartE2EDuration="1.70475846s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 20:01:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 20:01:40.703231987 +0000 UTC m=+1342.227837411" watchObservedRunningTime="2026-09-18 20:01:40.70475846 +0000 UTC m=+1342.229363882"
Sep 18 20:01:49.032494 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:01:49.032454    2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-d5f7da-fln9s"]
Sep 18 20:01:49.036243 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:01:49.036222    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-d5f7da-fln9s"
Sep 18 20:01:49.038814 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:01:49.038763    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 18 20:01:49.038942 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:01:49.038884    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-5ttnn\""
Sep 18 20:01:49.045229 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:01:49.045204    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-d5f7da-fln9s"]
Sep 18 20:01:49.136153 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:01:49.136109    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/edee935c-e0d7-456e-96e8-9a45d311264f-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-d5f7da-fln9s\" (UID: \"edee935c-e0d7-456e-96e8-9a45d311264f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-d5f7da-fln9s"
Sep 18 20:01:49.237409 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:01:49.237369    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/edee935c-e0d7-456e-96e8-9a45d311264f-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-d5f7da-fln9s\" (UID: \"edee935c-e0d7-456e-96e8-9a45d311264f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-d5f7da-fln9s"
Sep 18 20:01:49.238050 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:01:49.238029    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/edee935c-e0d7-456e-96e8-9a45d311264f-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-d5f7da-fln9s\" (UID: \"edee935c-e0d7-456e-96e8-9a45d311264f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-d5f7da-fln9s"
Sep 18 20:01:49.350143 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:01:49.350042    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-d5f7da-fln9s"
Sep 18 20:01:49.388205 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:01:49.388164    2585 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-noten-03d45c/noten-model-03d45c-kserve-769bbbd96c-fq9qr"
Sep 18 20:01:49.388380 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:01:49.388226    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-noten-03d45c/noten-model-03d45c-kserve-769bbbd96c-fq9qr"
Sep 18 20:01:49.403489 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:01:49.403094    2585 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-noten-03d45c/noten-model-03d45c-kserve-769bbbd96c-fq9qr"
Sep 18 20:01:49.623290 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:01:49.623266    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-d5f7da-fln9s"]
Sep 18 20:01:49.625598 ip-10-0-136-185 kubenswrapper[2585]: W0918 20:01:49.625571    2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podedee935c_e0d7_456e_96e8_9a45d311264f.slice/crio-f8a4ec2df298794a915d57650ad02e5729bf5761047b92e698b209602babaf40 WatchSource:0}: Error finding container f8a4ec2df298794a915d57650ad02e5729bf5761047b92e698b209602babaf40: Status 404 returned error can't find the container with id f8a4ec2df298794a915d57650ad02e5729bf5761047b92e698b209602babaf40
Sep 18 20:01:49.734398 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:01:49.734357    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-d5f7da-fln9s" event={"ID":"edee935c-e0d7-456e-96e8-9a45d311264f","Type":"ContainerStarted","Data":"f8a4ec2df298794a915d57650ad02e5729bf5761047b92e698b209602babaf40"}
Sep 18 20:01:49.745682 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:01:49.745657    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-noten-03d45c/noten-model-03d45c-kserve-769bbbd96c-fq9qr"
Sep 18 20:01:50.741577 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:01:50.741474    2585 generic.go:356] "Generic (PLEG): container finished" podID="edee935c-e0d7-456e-96e8-9a45d311264f" containerID="611758758b16ed4fdc0feedf15d3230f29ec62172d7d23a826bd1aa096a253fd" exitCode=0
Sep 18 20:01:50.742073 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:01:50.741564    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-d5f7da-fln9s" event={"ID":"edee935c-e0d7-456e-96e8-9a45d311264f","Type":"ContainerDied","Data":"611758758b16ed4fdc0feedf15d3230f29ec62172d7d23a826bd1aa096a253fd"}
Sep 18 20:01:51.886575 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:01:51.886548    2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-d5f7da-fln9s"
Sep 18 20:01:51.965340 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:01:51.965303    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/edee935c-e0d7-456e-96e8-9a45d311264f-maas-api-service-ca\") pod \"edee935c-e0d7-456e-96e8-9a45d311264f\" (UID: \"edee935c-e0d7-456e-96e8-9a45d311264f\") "
Sep 18 20:01:51.965719 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:01:51.965692    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/edee935c-e0d7-456e-96e8-9a45d311264f-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "edee935c-e0d7-456e-96e8-9a45d311264f" (UID: "edee935c-e0d7-456e-96e8-9a45d311264f"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 20:01:52.066813 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:01:52.066707    2585 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/edee935c-e0d7-456e-96e8-9a45d311264f-maas-api-service-ca\") on node \"ip-10-0-136-185.ec2.internal\" DevicePath \"\""
Sep 18 20:01:52.759666 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:01:52.759636    2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-d5f7da-fln9s"
Sep 18 20:01:52.759666 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:01:52.759658    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-d5f7da-fln9s" event={"ID":"edee935c-e0d7-456e-96e8-9a45d311264f","Type":"ContainerDied","Data":"f8a4ec2df298794a915d57650ad02e5729bf5761047b92e698b209602babaf40"}
Sep 18 20:01:52.759910 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:01:52.759727    2585 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f8a4ec2df298794a915d57650ad02e5729bf5761047b92e698b209602babaf40"
Sep 18 20:01:59.508801 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:01:59.508750    2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-d5f7da-29829360-vxqct"]
Sep 18 20:01:59.511039 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:01:59.511016    2585 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-d5f7da-29829360-vxqct"]
Sep 18 20:01:59.539560 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:01:59.539529    2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-d5f7da-cd6fff666-5fkmd"]
Sep 18 20:01:59.539854 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:01:59.539776    2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-d5f7da-cd6fff666-5fkmd" podUID="8b34fcc9-59f3-4642-aa96-2cbc3790cda5" containerName="payload-pre-processing" containerID="cri-o://aad4dc39dc415da8d4800ca9e9e57e3b7eec818216bbe7b2968068083846a89a" gracePeriod=30
Sep 18 20:01:59.724284 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:01:59.724243    2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-d5f7da-fln9s"]
Sep 18 20:01:59.727633 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:01:59.727582    2585 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-d5f7da-fln9s"]
Sep 18 20:01:59.797783 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:01:59.797758    2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-d5f7da-cd6fff666-5fkmd"
Sep 18 20:01:59.798305 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:01:59.798282    2585 generic.go:356] "Generic (PLEG): container finished" podID="8b34fcc9-59f3-4642-aa96-2cbc3790cda5" containerID="aad4dc39dc415da8d4800ca9e9e57e3b7eec818216bbe7b2968068083846a89a" exitCode=0
Sep 18 20:01:59.798385 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:01:59.798357    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-d5f7da-cd6fff666-5fkmd" event={"ID":"8b34fcc9-59f3-4642-aa96-2cbc3790cda5","Type":"ContainerDied","Data":"aad4dc39dc415da8d4800ca9e9e57e3b7eec818216bbe7b2968068083846a89a"}
Sep 18 20:01:59.798447 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:01:59.798425    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-d5f7da-cd6fff666-5fkmd" event={"ID":"8b34fcc9-59f3-4642-aa96-2cbc3790cda5","Type":"ContainerDied","Data":"3510e87116481567581ecac99da34b6fb8934231f6f84c8010815cadd5683a62"}
Sep 18 20:01:59.798503 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:01:59.798458    2585 scope.go:117] "RemoveContainer" containerID="aad4dc39dc415da8d4800ca9e9e57e3b7eec818216bbe7b2968068083846a89a"
Sep 18 20:01:59.807137 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:01:59.807106    2585 scope.go:117] "RemoveContainer" containerID="aad4dc39dc415da8d4800ca9e9e57e3b7eec818216bbe7b2968068083846a89a"
Sep 18 20:01:59.807443 ip-10-0-136-185 kubenswrapper[2585]: E0918 20:01:59.807422    2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"aad4dc39dc415da8d4800ca9e9e57e3b7eec818216bbe7b2968068083846a89a\": container with ID starting with aad4dc39dc415da8d4800ca9e9e57e3b7eec818216bbe7b2968068083846a89a not found: ID does not exist" containerID="aad4dc39dc415da8d4800ca9e9e57e3b7eec818216bbe7b2968068083846a89a"
Sep 18 20:01:59.807533 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:01:59.807456    2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"aad4dc39dc415da8d4800ca9e9e57e3b7eec818216bbe7b2968068083846a89a"} err="failed to get container status \"aad4dc39dc415da8d4800ca9e9e57e3b7eec818216bbe7b2968068083846a89a\": rpc error: code = NotFound desc = could not find container \"aad4dc39dc415da8d4800ca9e9e57e3b7eec818216bbe7b2968068083846a89a\": container with ID starting with aad4dc39dc415da8d4800ca9e9e57e3b7eec818216bbe7b2968068083846a89a not found: ID does not exist"
Sep 18 20:01:59.949289 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:01:59.949249    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8b34fcc9-59f3-4642-aa96-2cbc3790cda5-plugins-config-volume\") pod \"8b34fcc9-59f3-4642-aa96-2cbc3790cda5\" (UID: \"8b34fcc9-59f3-4642-aa96-2cbc3790cda5\") "
Sep 18 20:01:59.949289 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:01:59.949291    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-w9mpq\" (UniqueName: \"kubernetes.io/projected/8b34fcc9-59f3-4642-aa96-2cbc3790cda5-kube-api-access-w9mpq\") pod \"8b34fcc9-59f3-4642-aa96-2cbc3790cda5\" (UID: \"8b34fcc9-59f3-4642-aa96-2cbc3790cda5\") "
Sep 18 20:01:59.949722 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:01:59.949694    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8b34fcc9-59f3-4642-aa96-2cbc3790cda5-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "8b34fcc9-59f3-4642-aa96-2cbc3790cda5" (UID: "8b34fcc9-59f3-4642-aa96-2cbc3790cda5"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 20:01:59.951411 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:01:59.951392    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8b34fcc9-59f3-4642-aa96-2cbc3790cda5-kube-api-access-w9mpq" (OuterVolumeSpecName: "kube-api-access-w9mpq") pod "8b34fcc9-59f3-4642-aa96-2cbc3790cda5" (UID: "8b34fcc9-59f3-4642-aa96-2cbc3790cda5"). InnerVolumeSpecName "kube-api-access-w9mpq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 20:02:00.051034 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:02:00.050942    2585 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8b34fcc9-59f3-4642-aa96-2cbc3790cda5-plugins-config-volume\") on node \"ip-10-0-136-185.ec2.internal\" DevicePath \"\""
Sep 18 20:02:00.051034 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:02:00.050979    2585 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w9mpq\" (UniqueName: \"kubernetes.io/projected/8b34fcc9-59f3-4642-aa96-2cbc3790cda5-kube-api-access-w9mpq\") on node \"ip-10-0-136-185.ec2.internal\" DevicePath \"\""
Sep 18 20:02:00.663616 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:02:00.663581    2585 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="39a05d80-8e7b-4071-84c3-981703405b45" path="/var/lib/kubelet/pods/39a05d80-8e7b-4071-84c3-981703405b45/volumes"
Sep 18 20:02:00.664014 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:02:00.663929    2585 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="edee935c-e0d7-456e-96e8-9a45d311264f" path="/var/lib/kubelet/pods/edee935c-e0d7-456e-96e8-9a45d311264f/volumes"
Sep 18 20:02:00.804689 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:02:00.804658    2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-d5f7da-cd6fff666-5fkmd"
Sep 18 20:02:00.832732 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:02:00.832698    2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-d5f7da-cd6fff666-5fkmd"]
Sep 18 20:02:00.834712 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:02:00.834683    2585 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-d5f7da-cd6fff666-5fkmd"]
Sep 18 20:02:02.661964 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:02:02.661929    2585 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8b34fcc9-59f3-4642-aa96-2cbc3790cda5" path="/var/lib/kubelet/pods/8b34fcc9-59f3-4642-aa96-2cbc3790cda5/volumes"
Sep 18 20:02:04.650754 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:02:04.650720    2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-d5f7da/test-model-d5f7da-kserve-67f5967d47-sxvzr"]
Sep 18 20:02:13.648223 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:02:13.648178    2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-noten-03d45c/noten-model-03d45c-kserve-769bbbd96c-fq9qr"]
Sep 18 20:02:15.087000 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:02:15.086965    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-xmwh7"]
Sep 18 20:02:19.670135 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:02:19.670085    2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-inf-b-d5f7da/test-model-d5f7da-kserve-67f5967d47-sxvzr" podUID="ab57d15d-6af5-49db-8446-7df8a5a15a14" containerName="main" containerID="cri-o://d5e8ef574bc8781b42c6f268247b937ded2a1288f0527b06a8d660978a2dc896" gracePeriod=45
Sep 18 20:02:19.907716 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:02:19.907687    2585 generic.go:356] "Generic (PLEG): container finished" podID="ab57d15d-6af5-49db-8446-7df8a5a15a14" containerID="d5e8ef574bc8781b42c6f268247b937ded2a1288f0527b06a8d660978a2dc896" exitCode=0
Sep 18 20:02:19.907848 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:02:19.907763    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-d5f7da/test-model-d5f7da-kserve-67f5967d47-sxvzr" event={"ID":"ab57d15d-6af5-49db-8446-7df8a5a15a14","Type":"ContainerDied","Data":"d5e8ef574bc8781b42c6f268247b937ded2a1288f0527b06a8d660978a2dc896"}
Sep 18 20:02:19.907848 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:02:19.907815    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-d5f7da/test-model-d5f7da-kserve-67f5967d47-sxvzr" event={"ID":"ab57d15d-6af5-49db-8446-7df8a5a15a14","Type":"ContainerDied","Data":"c46ed84807eba0a6946c10dbd443bed14f650bb0becd00494dd55376dc7723fb"}
Sep 18 20:02:19.907848 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:02:19.907827    2585 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c46ed84807eba0a6946c10dbd443bed14f650bb0becd00494dd55376dc7723fb"
Sep 18 20:02:19.919161 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:02:19.919138    2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-d5f7da/test-model-d5f7da-kserve-67f5967d47-sxvzr"
Sep 18 20:02:20.053227 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:02:20.053141    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ab57d15d-6af5-49db-8446-7df8a5a15a14-tmp-dir\") pod \"ab57d15d-6af5-49db-8446-7df8a5a15a14\" (UID: \"ab57d15d-6af5-49db-8446-7df8a5a15a14\") "
Sep 18 20:02:20.053409 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:02:20.053233    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5d66n\" (UniqueName: \"kubernetes.io/projected/ab57d15d-6af5-49db-8446-7df8a5a15a14-kube-api-access-5d66n\") pod \"ab57d15d-6af5-49db-8446-7df8a5a15a14\" (UID: \"ab57d15d-6af5-49db-8446-7df8a5a15a14\") "
Sep 18 20:02:20.053409 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:02:20.053266    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ab57d15d-6af5-49db-8446-7df8a5a15a14-home\") pod \"ab57d15d-6af5-49db-8446-7df8a5a15a14\" (UID: \"ab57d15d-6af5-49db-8446-7df8a5a15a14\") "
Sep 18 20:02:20.053409 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:02:20.053300    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ab57d15d-6af5-49db-8446-7df8a5a15a14-dshm\") pod \"ab57d15d-6af5-49db-8446-7df8a5a15a14\" (UID: \"ab57d15d-6af5-49db-8446-7df8a5a15a14\") "
Sep 18 20:02:20.053409 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:02:20.053352    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ab57d15d-6af5-49db-8446-7df8a5a15a14-tls-certs\") pod \"ab57d15d-6af5-49db-8446-7df8a5a15a14\" (UID: \"ab57d15d-6af5-49db-8446-7df8a5a15a14\") "
Sep 18 20:02:20.053409 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:02:20.053381    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ab57d15d-6af5-49db-8446-7df8a5a15a14-model-cache\") pod \"ab57d15d-6af5-49db-8446-7df8a5a15a14\" (UID: \"ab57d15d-6af5-49db-8446-7df8a5a15a14\") "
Sep 18 20:02:20.053669 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:02:20.053460    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ab57d15d-6af5-49db-8446-7df8a5a15a14-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "ab57d15d-6af5-49db-8446-7df8a5a15a14" (UID: "ab57d15d-6af5-49db-8446-7df8a5a15a14"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 20:02:20.053669 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:02:20.053543    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ab57d15d-6af5-49db-8446-7df8a5a15a14-home" (OuterVolumeSpecName: "home") pod "ab57d15d-6af5-49db-8446-7df8a5a15a14" (UID: "ab57d15d-6af5-49db-8446-7df8a5a15a14"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 20:02:20.053806 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:02:20.053729    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ab57d15d-6af5-49db-8446-7df8a5a15a14-model-cache" (OuterVolumeSpecName: "model-cache") pod "ab57d15d-6af5-49db-8446-7df8a5a15a14" (UID: "ab57d15d-6af5-49db-8446-7df8a5a15a14"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 20:02:20.053806 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:02:20.053761    2585 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ab57d15d-6af5-49db-8446-7df8a5a15a14-tmp-dir\") on node \"ip-10-0-136-185.ec2.internal\" DevicePath \"\""
Sep 18 20:02:20.053898 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:02:20.053780    2585 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ab57d15d-6af5-49db-8446-7df8a5a15a14-home\") on node \"ip-10-0-136-185.ec2.internal\" DevicePath \"\""
Sep 18 20:02:20.055672 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:02:20.055652    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ab57d15d-6af5-49db-8446-7df8a5a15a14-dshm" (OuterVolumeSpecName: "dshm") pod "ab57d15d-6af5-49db-8446-7df8a5a15a14" (UID: "ab57d15d-6af5-49db-8446-7df8a5a15a14"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 20:02:20.055672 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:02:20.055661    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ab57d15d-6af5-49db-8446-7df8a5a15a14-kube-api-access-5d66n" (OuterVolumeSpecName: "kube-api-access-5d66n") pod "ab57d15d-6af5-49db-8446-7df8a5a15a14" (UID: "ab57d15d-6af5-49db-8446-7df8a5a15a14"). InnerVolumeSpecName "kube-api-access-5d66n". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 20:02:20.055810 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:02:20.055669    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ab57d15d-6af5-49db-8446-7df8a5a15a14-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "ab57d15d-6af5-49db-8446-7df8a5a15a14" (UID: "ab57d15d-6af5-49db-8446-7df8a5a15a14"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 20:02:20.155121 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:02:20.155070    2585 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5d66n\" (UniqueName: \"kubernetes.io/projected/ab57d15d-6af5-49db-8446-7df8a5a15a14-kube-api-access-5d66n\") on node \"ip-10-0-136-185.ec2.internal\" DevicePath \"\""
Sep 18 20:02:20.155121 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:02:20.155113    2585 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ab57d15d-6af5-49db-8446-7df8a5a15a14-dshm\") on node \"ip-10-0-136-185.ec2.internal\" DevicePath \"\""
Sep 18 20:02:20.155121 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:02:20.155123    2585 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ab57d15d-6af5-49db-8446-7df8a5a15a14-tls-certs\") on node \"ip-10-0-136-185.ec2.internal\" DevicePath \"\""
Sep 18 20:02:20.155121 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:02:20.155133    2585 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ab57d15d-6af5-49db-8446-7df8a5a15a14-model-cache\") on node \"ip-10-0-136-185.ec2.internal\" DevicePath \"\""
Sep 18 20:02:20.913321 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:02:20.913291    2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-d5f7da/test-model-d5f7da-kserve-67f5967d47-sxvzr"
Sep 18 20:02:20.931178 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:02:20.931108    2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-d5f7da/test-model-d5f7da-kserve-67f5967d47-sxvzr"]
Sep 18 20:02:20.933233 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:02:20.933209    2585 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-b-d5f7da/test-model-d5f7da-kserve-67f5967d47-sxvzr"]
Sep 18 20:02:22.661912 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:02:22.661869    2585 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ab57d15d-6af5-49db-8446-7df8a5a15a14" path="/var/lib/kubelet/pods/ab57d15d-6af5-49db-8446-7df8a5a15a14/volumes"
Sep 18 20:02:26.289422 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:02:26.289385    2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-mt-166dc9bb/e2e-lifecycle-model-166dc9bb-kserve-54cd4c4579-6xzm4"]
Sep 18 20:02:26.289970 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:02:26.289951    2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="edee935c-e0d7-456e-96e8-9a45d311264f" containerName="revoke-keys"
Sep 18 20:02:26.290054 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:02:26.289973    2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="edee935c-e0d7-456e-96e8-9a45d311264f" containerName="revoke-keys"
Sep 18 20:02:26.290054 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:02:26.290010    2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8b34fcc9-59f3-4642-aa96-2cbc3790cda5" containerName="payload-pre-processing"
Sep 18 20:02:26.290054 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:02:26.290021    2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="8b34fcc9-59f3-4642-aa96-2cbc3790cda5" containerName="payload-pre-processing"
Sep 18 20:02:26.290054 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:02:26.290045    2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ab57d15d-6af5-49db-8446-7df8a5a15a14" containerName="main"
Sep 18 20:02:26.290054 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:02:26.290055    2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="ab57d15d-6af5-49db-8446-7df8a5a15a14" containerName="main"
Sep 18 20:02:26.290303 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:02:26.290154    2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="ab57d15d-6af5-49db-8446-7df8a5a15a14" containerName="main"
Sep 18 20:02:26.290303 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:02:26.290166    2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="edee935c-e0d7-456e-96e8-9a45d311264f" containerName="revoke-keys"
Sep 18 20:02:26.290303 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:02:26.290183    2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="8b34fcc9-59f3-4642-aa96-2cbc3790cda5" containerName="payload-pre-processing"
Sep 18 20:02:26.294608 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:02:26.294583    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-166dc9bb/e2e-lifecycle-model-166dc9bb-kserve-54cd4c4579-6xzm4"
Sep 18 20:02:26.297476 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:02:26.297457    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-166dc9bb\"/\"kube-root-ca.crt\""
Sep 18 20:02:26.297927 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:02:26.297900    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-166dc9bb\"/\"openshift-service-ca.crt\""
Sep 18 20:02:26.298383 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:02:26.298354    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-166dc9bb\"/\"e2e-lifecycle-model-166dc9bb-kserve-self-signed-certs\""
Sep 18 20:02:26.298479 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:02:26.298418    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-166dc9bb\"/\"default-dockercfg-qmdnl\""
Sep 18 20:02:26.304227 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:02:26.304201    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-166dc9bb/e2e-lifecycle-model-166dc9bb-kserve-54cd4c4579-6xzm4"]
Sep 18 20:02:26.417331 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:02:26.417290    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cdd98a98-da96-4013-8bea-a3c721815940-home\") pod \"e2e-lifecycle-model-166dc9bb-kserve-54cd4c4579-6xzm4\" (UID: \"cdd98a98-da96-4013-8bea-a3c721815940\") " pod="ai-tenant-e2e-mt-166dc9bb/e2e-lifecycle-model-166dc9bb-kserve-54cd4c4579-6xzm4"
Sep 18 20:02:26.417331 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:02:26.417336    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cdd98a98-da96-4013-8bea-a3c721815940-tmp-dir\") pod \"e2e-lifecycle-model-166dc9bb-kserve-54cd4c4579-6xzm4\" (UID: \"cdd98a98-da96-4013-8bea-a3c721815940\") " pod="ai-tenant-e2e-mt-166dc9bb/e2e-lifecycle-model-166dc9bb-kserve-54cd4c4579-6xzm4"
Sep 18 20:02:26.417577 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:02:26.417368    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cdd98a98-da96-4013-8bea-a3c721815940-dshm\") pod \"e2e-lifecycle-model-166dc9bb-kserve-54cd4c4579-6xzm4\" (UID: \"cdd98a98-da96-4013-8bea-a3c721815940\") " pod="ai-tenant-e2e-mt-166dc9bb/e2e-lifecycle-model-166dc9bb-kserve-54cd4c4579-6xzm4"
Sep 18 20:02:26.417577 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:02:26.417462    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-82v5c\" (UniqueName: \"kubernetes.io/projected/cdd98a98-da96-4013-8bea-a3c721815940-kube-api-access-82v5c\") pod \"e2e-lifecycle-model-166dc9bb-kserve-54cd4c4579-6xzm4\" (UID: \"cdd98a98-da96-4013-8bea-a3c721815940\") " pod="ai-tenant-e2e-mt-166dc9bb/e2e-lifecycle-model-166dc9bb-kserve-54cd4c4579-6xzm4"
Sep 18 20:02:26.417577 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:02:26.417502    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cdd98a98-da96-4013-8bea-a3c721815940-tls-certs\") pod \"e2e-lifecycle-model-166dc9bb-kserve-54cd4c4579-6xzm4\" (UID: \"cdd98a98-da96-4013-8bea-a3c721815940\") " pod="ai-tenant-e2e-mt-166dc9bb/e2e-lifecycle-model-166dc9bb-kserve-54cd4c4579-6xzm4"
Sep 18 20:02:26.417690 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:02:26.417572    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cdd98a98-da96-4013-8bea-a3c721815940-model-cache\") pod \"e2e-lifecycle-model-166dc9bb-kserve-54cd4c4579-6xzm4\" (UID: \"cdd98a98-da96-4013-8bea-a3c721815940\") " pod="ai-tenant-e2e-mt-166dc9bb/e2e-lifecycle-model-166dc9bb-kserve-54cd4c4579-6xzm4"
Sep 18 20:02:26.518502 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:02:26.518463    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cdd98a98-da96-4013-8bea-a3c721815940-dshm\") pod \"e2e-lifecycle-model-166dc9bb-kserve-54cd4c4579-6xzm4\" (UID: \"cdd98a98-da96-4013-8bea-a3c721815940\") " pod="ai-tenant-e2e-mt-166dc9bb/e2e-lifecycle-model-166dc9bb-kserve-54cd4c4579-6xzm4"
Sep 18 20:02:26.518672 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:02:26.518524    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-82v5c\" (UniqueName: \"kubernetes.io/projected/cdd98a98-da96-4013-8bea-a3c721815940-kube-api-access-82v5c\") pod \"e2e-lifecycle-model-166dc9bb-kserve-54cd4c4579-6xzm4\" (UID: \"cdd98a98-da96-4013-8bea-a3c721815940\") " pod="ai-tenant-e2e-mt-166dc9bb/e2e-lifecycle-model-166dc9bb-kserve-54cd4c4579-6xzm4"
Sep 18 20:02:26.518672 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:02:26.518543    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cdd98a98-da96-4013-8bea-a3c721815940-tls-certs\") pod \"e2e-lifecycle-model-166dc9bb-kserve-54cd4c4579-6xzm4\" (UID: \"cdd98a98-da96-4013-8bea-a3c721815940\") " pod="ai-tenant-e2e-mt-166dc9bb/e2e-lifecycle-model-166dc9bb-kserve-54cd4c4579-6xzm4"
Sep 18 20:02:26.518672 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:02:26.518568    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cdd98a98-da96-4013-8bea-a3c721815940-model-cache\") pod \"e2e-lifecycle-model-166dc9bb-kserve-54cd4c4579-6xzm4\" (UID: \"cdd98a98-da96-4013-8bea-a3c721815940\") " pod="ai-tenant-e2e-mt-166dc9bb/e2e-lifecycle-model-166dc9bb-kserve-54cd4c4579-6xzm4"
Sep 18 20:02:26.518672 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:02:26.518637    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cdd98a98-da96-4013-8bea-a3c721815940-home\") pod \"e2e-lifecycle-model-166dc9bb-kserve-54cd4c4579-6xzm4\" (UID: \"cdd98a98-da96-4013-8bea-a3c721815940\") " pod="ai-tenant-e2e-mt-166dc9bb/e2e-lifecycle-model-166dc9bb-kserve-54cd4c4579-6xzm4"
Sep 18 20:02:26.518672 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:02:26.518666    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cdd98a98-da96-4013-8bea-a3c721815940-tmp-dir\") pod \"e2e-lifecycle-model-166dc9bb-kserve-54cd4c4579-6xzm4\" (UID: \"cdd98a98-da96-4013-8bea-a3c721815940\") " pod="ai-tenant-e2e-mt-166dc9bb/e2e-lifecycle-model-166dc9bb-kserve-54cd4c4579-6xzm4"
Sep 18 20:02:26.519083 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:02:26.519053    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cdd98a98-da96-4013-8bea-a3c721815940-model-cache\") pod \"e2e-lifecycle-model-166dc9bb-kserve-54cd4c4579-6xzm4\" (UID: \"cdd98a98-da96-4013-8bea-a3c721815940\") " pod="ai-tenant-e2e-mt-166dc9bb/e2e-lifecycle-model-166dc9bb-kserve-54cd4c4579-6xzm4"
Sep 18 20:02:26.519169 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:02:26.519108    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cdd98a98-da96-4013-8bea-a3c721815940-tmp-dir\") pod \"e2e-lifecycle-model-166dc9bb-kserve-54cd4c4579-6xzm4\" (UID: \"cdd98a98-da96-4013-8bea-a3c721815940\") " pod="ai-tenant-e2e-mt-166dc9bb/e2e-lifecycle-model-166dc9bb-kserve-54cd4c4579-6xzm4"
Sep 18 20:02:26.519169 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:02:26.519151    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cdd98a98-da96-4013-8bea-a3c721815940-home\") pod \"e2e-lifecycle-model-166dc9bb-kserve-54cd4c4579-6xzm4\" (UID: \"cdd98a98-da96-4013-8bea-a3c721815940\") " pod="ai-tenant-e2e-mt-166dc9bb/e2e-lifecycle-model-166dc9bb-kserve-54cd4c4579-6xzm4"
Sep 18 20:02:26.520961 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:02:26.520943    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cdd98a98-da96-4013-8bea-a3c721815940-dshm\") pod \"e2e-lifecycle-model-166dc9bb-kserve-54cd4c4579-6xzm4\" (UID: \"cdd98a98-da96-4013-8bea-a3c721815940\") " pod="ai-tenant-e2e-mt-166dc9bb/e2e-lifecycle-model-166dc9bb-kserve-54cd4c4579-6xzm4"
Sep 18 20:02:26.521286 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:02:26.521258    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cdd98a98-da96-4013-8bea-a3c721815940-tls-certs\") pod \"e2e-lifecycle-model-166dc9bb-kserve-54cd4c4579-6xzm4\" (UID: \"cdd98a98-da96-4013-8bea-a3c721815940\") " pod="ai-tenant-e2e-mt-166dc9bb/e2e-lifecycle-model-166dc9bb-kserve-54cd4c4579-6xzm4"
Sep 18 20:02:26.526484 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:02:26.526459    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-82v5c\" (UniqueName: \"kubernetes.io/projected/cdd98a98-da96-4013-8bea-a3c721815940-kube-api-access-82v5c\") pod \"e2e-lifecycle-model-166dc9bb-kserve-54cd4c4579-6xzm4\" (UID: \"cdd98a98-da96-4013-8bea-a3c721815940\") " pod="ai-tenant-e2e-mt-166dc9bb/e2e-lifecycle-model-166dc9bb-kserve-54cd4c4579-6xzm4"
Sep 18 20:02:26.607340 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:02:26.607238    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-166dc9bb/e2e-lifecycle-model-166dc9bb-kserve-54cd4c4579-6xzm4"
Sep 18 20:02:26.880431 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:02:26.880407    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-166dc9bb/e2e-lifecycle-model-166dc9bb-kserve-54cd4c4579-6xzm4"]
Sep 18 20:02:26.881965 ip-10-0-136-185 kubenswrapper[2585]: W0918 20:02:26.881929    2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podcdd98a98_da96_4013_8bea_a3c721815940.slice/crio-5c301538be81a634d5fa8a21dd8b781b45567fdcd696e5bcafe4626c7508ba3a WatchSource:0}: Error finding container 5c301538be81a634d5fa8a21dd8b781b45567fdcd696e5bcafe4626c7508ba3a: Status 404 returned error can't find the container with id 5c301538be81a634d5fa8a21dd8b781b45567fdcd696e5bcafe4626c7508ba3a
Sep 18 20:02:26.950049 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:02:26.950021    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-166dc9bb/e2e-lifecycle-model-166dc9bb-kserve-54cd4c4579-6xzm4" event={"ID":"cdd98a98-da96-4013-8bea-a3c721815940","Type":"ContainerStarted","Data":"a8f64c8adadc434bcbf3f57f57f701947f9caa9033a3623a29e991fe254414b3"}
Sep 18 20:02:26.950171 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:02:26.950060    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-166dc9bb/e2e-lifecycle-model-166dc9bb-kserve-54cd4c4579-6xzm4" event={"ID":"cdd98a98-da96-4013-8bea-a3c721815940","Type":"ContainerStarted","Data":"5c301538be81a634d5fa8a21dd8b781b45567fdcd696e5bcafe4626c7508ba3a"}
Sep 18 20:02:26.966926 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:02:26.966874    2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-mt-166dc9bb/e2e-lifecycle-model-166dc9bb-kserve-54cd4c4579-6xzm4" podStartSLOduration=0.966854075 podStartE2EDuration="966.854075ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 20:02:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 20:02:26.965896282 +0000 UTC m=+1388.490501708" watchObservedRunningTime="2026-09-18 20:02:26.966854075 +0000 UTC m=+1388.491459499"
Sep 18 20:02:28.667804 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:02:28.667761    2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-noten-03d45c/noten-model-03d45c-kserve-769bbbd96c-fq9qr" podUID="12fda611-11c3-4df9-8de0-6898ea0c4ba9" containerName="main" containerID="cri-o://a91f448c64603e87267c1fe5060608ea3a9eb9ebc5803aa7d5131b39bde0d3cb" gracePeriod=45
Sep 18 20:02:28.921141 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:02:28.921076    2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-noten-03d45c/noten-model-03d45c-kserve-769bbbd96c-fq9qr"
Sep 18 20:02:28.962181 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:02:28.962148    2585 generic.go:356] "Generic (PLEG): container finished" podID="12fda611-11c3-4df9-8de0-6898ea0c4ba9" containerID="a91f448c64603e87267c1fe5060608ea3a9eb9ebc5803aa7d5131b39bde0d3cb" exitCode=0
Sep 18 20:02:28.962356 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:02:28.962187    2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-noten-03d45c/noten-model-03d45c-kserve-769bbbd96c-fq9qr"
Sep 18 20:02:28.962356 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:02:28.962242    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-noten-03d45c/noten-model-03d45c-kserve-769bbbd96c-fq9qr" event={"ID":"12fda611-11c3-4df9-8de0-6898ea0c4ba9","Type":"ContainerDied","Data":"a91f448c64603e87267c1fe5060608ea3a9eb9ebc5803aa7d5131b39bde0d3cb"}
Sep 18 20:02:28.962356 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:02:28.962291    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-noten-03d45c/noten-model-03d45c-kserve-769bbbd96c-fq9qr" event={"ID":"12fda611-11c3-4df9-8de0-6898ea0c4ba9","Type":"ContainerDied","Data":"9feeea8ddbdd5e982a2790e82f6095faaca5700f8c25e60ea46482e7f9f4d23e"}
Sep 18 20:02:28.962356 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:02:28.962317    2585 scope.go:117] "RemoveContainer" containerID="a91f448c64603e87267c1fe5060608ea3a9eb9ebc5803aa7d5131b39bde0d3cb"
Sep 18 20:02:28.973838 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:02:28.973816    2585 scope.go:117] "RemoveContainer" containerID="a91f448c64603e87267c1fe5060608ea3a9eb9ebc5803aa7d5131b39bde0d3cb"
Sep 18 20:02:28.974166 ip-10-0-136-185 kubenswrapper[2585]: E0918 20:02:28.974146    2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a91f448c64603e87267c1fe5060608ea3a9eb9ebc5803aa7d5131b39bde0d3cb\": container with ID starting with a91f448c64603e87267c1fe5060608ea3a9eb9ebc5803aa7d5131b39bde0d3cb not found: ID does not exist" containerID="a91f448c64603e87267c1fe5060608ea3a9eb9ebc5803aa7d5131b39bde0d3cb"
Sep 18 20:02:28.974216 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:02:28.974179    2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a91f448c64603e87267c1fe5060608ea3a9eb9ebc5803aa7d5131b39bde0d3cb"} err="failed to get container status \"a91f448c64603e87267c1fe5060608ea3a9eb9ebc5803aa7d5131b39bde0d3cb\": rpc error: code = NotFound desc = could not find container \"a91f448c64603e87267c1fe5060608ea3a9eb9ebc5803aa7d5131b39bde0d3cb\": container with ID starting with a91f448c64603e87267c1fe5060608ea3a9eb9ebc5803aa7d5131b39bde0d3cb not found: ID does not exist"
Sep 18 20:02:29.043108 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:02:29.043074    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ht45v\" (UniqueName: \"kubernetes.io/projected/12fda611-11c3-4df9-8de0-6898ea0c4ba9-kube-api-access-ht45v\") pod \"12fda611-11c3-4df9-8de0-6898ea0c4ba9\" (UID: \"12fda611-11c3-4df9-8de0-6898ea0c4ba9\") "
Sep 18 20:02:29.043108 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:02:29.043112    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/12fda611-11c3-4df9-8de0-6898ea0c4ba9-tmp-dir\") pod \"12fda611-11c3-4df9-8de0-6898ea0c4ba9\" (UID: \"12fda611-11c3-4df9-8de0-6898ea0c4ba9\") "
Sep 18 20:02:29.043358 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:02:29.043152    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/12fda611-11c3-4df9-8de0-6898ea0c4ba9-tls-certs\") pod \"12fda611-11c3-4df9-8de0-6898ea0c4ba9\" (UID: \"12fda611-11c3-4df9-8de0-6898ea0c4ba9\") "
Sep 18 20:02:29.043358 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:02:29.043184    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/12fda611-11c3-4df9-8de0-6898ea0c4ba9-home\") pod \"12fda611-11c3-4df9-8de0-6898ea0c4ba9\" (UID: \"12fda611-11c3-4df9-8de0-6898ea0c4ba9\") "
Sep 18 20:02:29.043358 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:02:29.043263    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/12fda611-11c3-4df9-8de0-6898ea0c4ba9-model-cache\") pod \"12fda611-11c3-4df9-8de0-6898ea0c4ba9\" (UID: \"12fda611-11c3-4df9-8de0-6898ea0c4ba9\") "
Sep 18 20:02:29.043358 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:02:29.043320    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/12fda611-11c3-4df9-8de0-6898ea0c4ba9-dshm\") pod \"12fda611-11c3-4df9-8de0-6898ea0c4ba9\" (UID: \"12fda611-11c3-4df9-8de0-6898ea0c4ba9\") "
Sep 18 20:02:29.043567 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:02:29.043472    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/12fda611-11c3-4df9-8de0-6898ea0c4ba9-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "12fda611-11c3-4df9-8de0-6898ea0c4ba9" (UID: "12fda611-11c3-4df9-8de0-6898ea0c4ba9"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 20:02:29.043567 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:02:29.043509    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/12fda611-11c3-4df9-8de0-6898ea0c4ba9-home" (OuterVolumeSpecName: "home") pod "12fda611-11c3-4df9-8de0-6898ea0c4ba9" (UID: "12fda611-11c3-4df9-8de0-6898ea0c4ba9"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 20:02:29.043682 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:02:29.043562    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/12fda611-11c3-4df9-8de0-6898ea0c4ba9-model-cache" (OuterVolumeSpecName: "model-cache") pod "12fda611-11c3-4df9-8de0-6898ea0c4ba9" (UID: "12fda611-11c3-4df9-8de0-6898ea0c4ba9"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 20:02:29.043764 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:02:29.043746    2585 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/12fda611-11c3-4df9-8de0-6898ea0c4ba9-tmp-dir\") on node \"ip-10-0-136-185.ec2.internal\" DevicePath \"\""
Sep 18 20:02:29.043838 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:02:29.043766    2585 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/12fda611-11c3-4df9-8de0-6898ea0c4ba9-home\") on node \"ip-10-0-136-185.ec2.internal\" DevicePath \"\""
Sep 18 20:02:29.043838 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:02:29.043776    2585 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/12fda611-11c3-4df9-8de0-6898ea0c4ba9-model-cache\") on node \"ip-10-0-136-185.ec2.internal\" DevicePath \"\""
Sep 18 20:02:29.045450 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:02:29.045424    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/12fda611-11c3-4df9-8de0-6898ea0c4ba9-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "12fda611-11c3-4df9-8de0-6898ea0c4ba9" (UID: "12fda611-11c3-4df9-8de0-6898ea0c4ba9"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 20:02:29.045719 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:02:29.045700    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/12fda611-11c3-4df9-8de0-6898ea0c4ba9-dshm" (OuterVolumeSpecName: "dshm") pod "12fda611-11c3-4df9-8de0-6898ea0c4ba9" (UID: "12fda611-11c3-4df9-8de0-6898ea0c4ba9"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 20:02:29.045719 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:02:29.045706    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/12fda611-11c3-4df9-8de0-6898ea0c4ba9-kube-api-access-ht45v" (OuterVolumeSpecName: "kube-api-access-ht45v") pod "12fda611-11c3-4df9-8de0-6898ea0c4ba9" (UID: "12fda611-11c3-4df9-8de0-6898ea0c4ba9"). InnerVolumeSpecName "kube-api-access-ht45v". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 20:02:29.144556 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:02:29.144511    2585 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ht45v\" (UniqueName: \"kubernetes.io/projected/12fda611-11c3-4df9-8de0-6898ea0c4ba9-kube-api-access-ht45v\") on node \"ip-10-0-136-185.ec2.internal\" DevicePath \"\""
Sep 18 20:02:29.144556 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:02:29.144546    2585 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/12fda611-11c3-4df9-8de0-6898ea0c4ba9-tls-certs\") on node \"ip-10-0-136-185.ec2.internal\" DevicePath \"\""
Sep 18 20:02:29.144556 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:02:29.144556    2585 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/12fda611-11c3-4df9-8de0-6898ea0c4ba9-dshm\") on node \"ip-10-0-136-185.ec2.internal\" DevicePath \"\""
Sep 18 20:02:29.284902 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:02:29.284870    2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-noten-03d45c/noten-model-03d45c-kserve-769bbbd96c-fq9qr"]
Sep 18 20:02:29.288294 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:02:29.288267    2585 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-noten-03d45c/noten-model-03d45c-kserve-769bbbd96c-fq9qr"]
Sep 18 20:02:30.663942 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:02:30.663910    2585 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="12fda611-11c3-4df9-8de0-6898ea0c4ba9" path="/var/lib/kubelet/pods/12fda611-11c3-4df9-8de0-6898ea0c4ba9/volumes"
Sep 18 20:02:36.079021 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:02:36.078984    2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-674263/e2e-conflict-b-1b1b9f14-kserve-6dc4d67945-bg7x9"]
Sep 18 20:02:36.608281 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:02:36.608220    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-mt-166dc9bb/e2e-lifecycle-model-166dc9bb-kserve-54cd4c4579-6xzm4"
Sep 18 20:02:36.608281 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:02:36.608294    2585 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-mt-166dc9bb/e2e-lifecycle-model-166dc9bb-kserve-54cd4c4579-6xzm4"
Sep 18 20:02:36.621352 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:02:36.621326    2585 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-mt-166dc9bb/e2e-lifecycle-model-166dc9bb-kserve-54cd4c4579-6xzm4"
Sep 18 20:02:37.014191 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:02:37.014160    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-mt-166dc9bb/e2e-lifecycle-model-166dc9bb-kserve-54cd4c4579-6xzm4"
Sep 18 20:02:38.180740 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:02:38.180706    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-xmwh7"]
Sep 18 20:02:48.695331 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:02:48.695297    2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-674263/e2e-conflict-a-1b1b9f14-kserve-69899bd87f-stx9d"]
Sep 18 20:02:51.100521 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:02:51.100466    2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-674263/e2e-conflict-b-1b1b9f14-kserve-6dc4d67945-bg7x9" podUID="413d2e4e-1595-4b21-898c-baff083776ae" containerName="main" containerID="cri-o://e5e832f68724ebd450b64d8304e50818b1769a352fb2a40fd87a8eb47cfe0179" gracePeriod=45
Sep 18 20:02:51.354344 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:02:51.354282    2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-674263/e2e-conflict-b-1b1b9f14-kserve-6dc4d67945-bg7x9"
Sep 18 20:02:51.466306 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:02:51.466272    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/413d2e4e-1595-4b21-898c-baff083776ae-tmp-dir\") pod \"413d2e4e-1595-4b21-898c-baff083776ae\" (UID: \"413d2e4e-1595-4b21-898c-baff083776ae\") "
Sep 18 20:02:51.466306 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:02:51.466309    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/413d2e4e-1595-4b21-898c-baff083776ae-tls-certs\") pod \"413d2e4e-1595-4b21-898c-baff083776ae\" (UID: \"413d2e4e-1595-4b21-898c-baff083776ae\") "
Sep 18 20:02:51.466567 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:02:51.466383    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/413d2e4e-1595-4b21-898c-baff083776ae-model-cache\") pod \"413d2e4e-1595-4b21-898c-baff083776ae\" (UID: \"413d2e4e-1595-4b21-898c-baff083776ae\") "
Sep 18 20:02:51.466567 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:02:51.466425    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/413d2e4e-1595-4b21-898c-baff083776ae-home\") pod \"413d2e4e-1595-4b21-898c-baff083776ae\" (UID: \"413d2e4e-1595-4b21-898c-baff083776ae\") "
Sep 18 20:02:51.466567 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:02:51.466447    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-btslz\" (UniqueName: \"kubernetes.io/projected/413d2e4e-1595-4b21-898c-baff083776ae-kube-api-access-btslz\") pod \"413d2e4e-1595-4b21-898c-baff083776ae\" (UID: \"413d2e4e-1595-4b21-898c-baff083776ae\") "
Sep 18 20:02:51.466567 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:02:51.466475    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/413d2e4e-1595-4b21-898c-baff083776ae-dshm\") pod \"413d2e4e-1595-4b21-898c-baff083776ae\" (UID: \"413d2e4e-1595-4b21-898c-baff083776ae\") "
Sep 18 20:02:51.466778 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:02:51.466631    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/413d2e4e-1595-4b21-898c-baff083776ae-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "413d2e4e-1595-4b21-898c-baff083776ae" (UID: "413d2e4e-1595-4b21-898c-baff083776ae"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 20:02:51.466778 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:02:51.466712    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/413d2e4e-1595-4b21-898c-baff083776ae-home" (OuterVolumeSpecName: "home") pod "413d2e4e-1595-4b21-898c-baff083776ae" (UID: "413d2e4e-1595-4b21-898c-baff083776ae"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 20:02:51.466778 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:02:51.466729    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/413d2e4e-1595-4b21-898c-baff083776ae-model-cache" (OuterVolumeSpecName: "model-cache") pod "413d2e4e-1595-4b21-898c-baff083776ae" (UID: "413d2e4e-1595-4b21-898c-baff083776ae"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 20:02:51.466930 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:02:51.466896    2585 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/413d2e4e-1595-4b21-898c-baff083776ae-tmp-dir\") on node \"ip-10-0-136-185.ec2.internal\" DevicePath \"\""
Sep 18 20:02:51.466930 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:02:51.466917    2585 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/413d2e4e-1595-4b21-898c-baff083776ae-model-cache\") on node \"ip-10-0-136-185.ec2.internal\" DevicePath \"\""
Sep 18 20:02:51.467002 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:02:51.466931    2585 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/413d2e4e-1595-4b21-898c-baff083776ae-home\") on node \"ip-10-0-136-185.ec2.internal\" DevicePath \"\""
Sep 18 20:02:51.468805 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:02:51.468767    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/413d2e4e-1595-4b21-898c-baff083776ae-kube-api-access-btslz" (OuterVolumeSpecName: "kube-api-access-btslz") pod "413d2e4e-1595-4b21-898c-baff083776ae" (UID: "413d2e4e-1595-4b21-898c-baff083776ae"). InnerVolumeSpecName "kube-api-access-btslz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 20:02:51.468805 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:02:51.468782    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/413d2e4e-1595-4b21-898c-baff083776ae-dshm" (OuterVolumeSpecName: "dshm") pod "413d2e4e-1595-4b21-898c-baff083776ae" (UID: "413d2e4e-1595-4b21-898c-baff083776ae"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 20:02:51.468969 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:02:51.468813    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/413d2e4e-1595-4b21-898c-baff083776ae-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "413d2e4e-1595-4b21-898c-baff083776ae" (UID: "413d2e4e-1595-4b21-898c-baff083776ae"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 20:02:51.568447 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:02:51.568411    2585 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-btslz\" (UniqueName: \"kubernetes.io/projected/413d2e4e-1595-4b21-898c-baff083776ae-kube-api-access-btslz\") on node \"ip-10-0-136-185.ec2.internal\" DevicePath \"\""
Sep 18 20:02:51.568447 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:02:51.568445    2585 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/413d2e4e-1595-4b21-898c-baff083776ae-dshm\") on node \"ip-10-0-136-185.ec2.internal\" DevicePath \"\""
Sep 18 20:02:51.568676 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:02:51.568459    2585 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/413d2e4e-1595-4b21-898c-baff083776ae-tls-certs\") on node \"ip-10-0-136-185.ec2.internal\" DevicePath \"\""
Sep 18 20:02:52.083253 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:02:52.083219    2585 generic.go:356] "Generic (PLEG): container finished" podID="413d2e4e-1595-4b21-898c-baff083776ae" containerID="e5e832f68724ebd450b64d8304e50818b1769a352fb2a40fd87a8eb47cfe0179" exitCode=0
Sep 18 20:02:52.083438 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:02:52.083263    2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-674263/e2e-conflict-b-1b1b9f14-kserve-6dc4d67945-bg7x9"
Sep 18 20:02:52.083438 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:02:52.083301    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-674263/e2e-conflict-b-1b1b9f14-kserve-6dc4d67945-bg7x9" event={"ID":"413d2e4e-1595-4b21-898c-baff083776ae","Type":"ContainerDied","Data":"e5e832f68724ebd450b64d8304e50818b1769a352fb2a40fd87a8eb47cfe0179"}
Sep 18 20:02:52.083438 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:02:52.083346    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-674263/e2e-conflict-b-1b1b9f14-kserve-6dc4d67945-bg7x9" event={"ID":"413d2e4e-1595-4b21-898c-baff083776ae","Type":"ContainerDied","Data":"efe3458b298a9353543213811853086c80e33cf241f955ee32e5006d5114512c"}
Sep 18 20:02:52.083438 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:02:52.083363    2585 scope.go:117] "RemoveContainer" containerID="e5e832f68724ebd450b64d8304e50818b1769a352fb2a40fd87a8eb47cfe0179"
Sep 18 20:02:52.099279 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:02:52.099256    2585 scope.go:117] "RemoveContainer" containerID="e5e832f68724ebd450b64d8304e50818b1769a352fb2a40fd87a8eb47cfe0179"
Sep 18 20:02:52.099607 ip-10-0-136-185 kubenswrapper[2585]: E0918 20:02:52.099581    2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e5e832f68724ebd450b64d8304e50818b1769a352fb2a40fd87a8eb47cfe0179\": container with ID starting with e5e832f68724ebd450b64d8304e50818b1769a352fb2a40fd87a8eb47cfe0179 not found: ID does not exist" containerID="e5e832f68724ebd450b64d8304e50818b1769a352fb2a40fd87a8eb47cfe0179"
Sep 18 20:02:52.099714 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:02:52.099613    2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e5e832f68724ebd450b64d8304e50818b1769a352fb2a40fd87a8eb47cfe0179"} err="failed to get container status \"e5e832f68724ebd450b64d8304e50818b1769a352fb2a40fd87a8eb47cfe0179\": rpc error: code = NotFound desc = could not find container \"e5e832f68724ebd450b64d8304e50818b1769a352fb2a40fd87a8eb47cfe0179\": container with ID starting with e5e832f68724ebd450b64d8304e50818b1769a352fb2a40fd87a8eb47cfe0179 not found: ID does not exist"
Sep 18 20:02:52.113856 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:02:52.113829    2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-674263/e2e-conflict-b-1b1b9f14-kserve-6dc4d67945-bg7x9"]
Sep 18 20:02:52.118537 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:02:52.118514    2585 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-674263/e2e-conflict-b-1b1b9f14-kserve-6dc4d67945-bg7x9"]
Sep 18 20:02:52.351480 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:02:52.351402    2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-57b6ce-5cd84f68c5-n6t8k"]
Sep 18 20:02:52.351836 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:02:52.351823    2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="413d2e4e-1595-4b21-898c-baff083776ae" containerName="main"
Sep 18 20:02:52.351884 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:02:52.351838    2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="413d2e4e-1595-4b21-898c-baff083776ae" containerName="main"
Sep 18 20:02:52.351884 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:02:52.351856    2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="12fda611-11c3-4df9-8de0-6898ea0c4ba9" containerName="main"
Sep 18 20:02:52.351884 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:02:52.351863    2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="12fda611-11c3-4df9-8de0-6898ea0c4ba9" containerName="main"
Sep 18 20:02:52.351979 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:02:52.351920    2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="12fda611-11c3-4df9-8de0-6898ea0c4ba9" containerName="main"
Sep 18 20:02:52.351979 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:02:52.351932    2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="413d2e4e-1595-4b21-898c-baff083776ae" containerName="main"
Sep 18 20:02:52.356324 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:02:52.356307    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-57b6ce-5cd84f68c5-n6t8k"
Sep 18 20:02:52.358680 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:02:52.358658    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w3-57b6ce\""
Sep 18 20:02:52.358872 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:02:52.358846    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w3-57b6ce-dockercfg-r5rx4\""
Sep 18 20:02:52.363745 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:02:52.363564    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-57b6ce-5cd84f68c5-n6t8k"]
Sep 18 20:02:52.477420 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:02:52.477373    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/66260672-3320-4a75-a853-ba1df8d1e872-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w3-57b6ce-5cd84f68c5-n6t8k\" (UID: \"66260672-3320-4a75-a853-ba1df8d1e872\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-57b6ce-5cd84f68c5-n6t8k"
Sep 18 20:02:52.477420 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:02:52.477422    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xxrjc\" (UniqueName: \"kubernetes.io/projected/66260672-3320-4a75-a853-ba1df8d1e872-kube-api-access-xxrjc\") pod \"payload-processing-e2e-shared-a-w3-57b6ce-5cd84f68c5-n6t8k\" (UID: \"66260672-3320-4a75-a853-ba1df8d1e872\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-57b6ce-5cd84f68c5-n6t8k"
Sep 18 20:02:52.578655 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:02:52.578611    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/66260672-3320-4a75-a853-ba1df8d1e872-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w3-57b6ce-5cd84f68c5-n6t8k\" (UID: \"66260672-3320-4a75-a853-ba1df8d1e872\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-57b6ce-5cd84f68c5-n6t8k"
Sep 18 20:02:52.578909 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:02:52.578674    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xxrjc\" (UniqueName: \"kubernetes.io/projected/66260672-3320-4a75-a853-ba1df8d1e872-kube-api-access-xxrjc\") pod \"payload-processing-e2e-shared-a-w3-57b6ce-5cd84f68c5-n6t8k\" (UID: \"66260672-3320-4a75-a853-ba1df8d1e872\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-57b6ce-5cd84f68c5-n6t8k"
Sep 18 20:02:52.579392 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:02:52.579371    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/66260672-3320-4a75-a853-ba1df8d1e872-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w3-57b6ce-5cd84f68c5-n6t8k\" (UID: \"66260672-3320-4a75-a853-ba1df8d1e872\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-57b6ce-5cd84f68c5-n6t8k"
Sep 18 20:02:52.587092 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:02:52.587060    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xxrjc\" (UniqueName: \"kubernetes.io/projected/66260672-3320-4a75-a853-ba1df8d1e872-kube-api-access-xxrjc\") pod \"payload-processing-e2e-shared-a-w3-57b6ce-5cd84f68c5-n6t8k\" (UID: \"66260672-3320-4a75-a853-ba1df8d1e872\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-57b6ce-5cd84f68c5-n6t8k"
Sep 18 20:02:52.662325 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:02:52.662299    2585 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="413d2e4e-1595-4b21-898c-baff083776ae" path="/var/lib/kubelet/pods/413d2e4e-1595-4b21-898c-baff083776ae/volumes"
Sep 18 20:02:52.668239 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:02:52.668220    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-57b6ce-5cd84f68c5-n6t8k"
Sep 18 20:02:52.941907 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:02:52.941882    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-57b6ce-5cd84f68c5-n6t8k"]
Sep 18 20:02:52.944774 ip-10-0-136-185 kubenswrapper[2585]: W0918 20:02:52.944739    2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod66260672_3320_4a75_a853_ba1df8d1e872.slice/crio-6b7a38960b32737a7cda49dbd7b19d0ae94fe97c5bfe6a5e7dd3399a2e736936 WatchSource:0}: Error finding container 6b7a38960b32737a7cda49dbd7b19d0ae94fe97c5bfe6a5e7dd3399a2e736936: Status 404 returned error can't find the container with id 6b7a38960b32737a7cda49dbd7b19d0ae94fe97c5bfe6a5e7dd3399a2e736936
Sep 18 20:02:53.091280 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:02:53.091246    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-57b6ce-5cd84f68c5-n6t8k" event={"ID":"66260672-3320-4a75-a853-ba1df8d1e872","Type":"ContainerStarted","Data":"9d2b223a9929ac6f6dffbf993bfa61a85e24ca16729b05ca70e225c15406737a"}
Sep 18 20:02:53.091452 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:02:53.091303    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-57b6ce-5cd84f68c5-n6t8k" event={"ID":"66260672-3320-4a75-a853-ba1df8d1e872","Type":"ContainerStarted","Data":"6b7a38960b32737a7cda49dbd7b19d0ae94fe97c5bfe6a5e7dd3399a2e736936"}
Sep 18 20:02:53.091505 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:02:53.091456    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-57b6ce-5cd84f68c5-n6t8k"
Sep 18 20:02:53.108848 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:02:53.108796    2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-57b6ce-5cd84f68c5-n6t8k" podStartSLOduration=1.108764818 podStartE2EDuration="1.108764818s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 20:02:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 20:02:53.106577342 +0000 UTC m=+1414.631182765" watchObservedRunningTime="2026-09-18 20:02:53.108764818 +0000 UTC m=+1414.633370244"
Sep 18 20:03:03.714041 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:03:03.713989    2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-674263/e2e-conflict-a-1b1b9f14-kserve-69899bd87f-stx9d" podUID="6c48fbd5-b360-49ab-9df2-5e5183112082" containerName="main" containerID="cri-o://16ea1a9fb4e1ac800155b99190d83f22d6c8e1c6bec66a071c9140422fba5bcf" gracePeriod=45
Sep 18 20:03:03.967897 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:03:03.967841    2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-674263/e2e-conflict-a-1b1b9f14-kserve-69899bd87f-stx9d"
Sep 18 20:03:04.092750 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:03:04.092706    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6c48fbd5-b360-49ab-9df2-5e5183112082-tls-certs\") pod \"6c48fbd5-b360-49ab-9df2-5e5183112082\" (UID: \"6c48fbd5-b360-49ab-9df2-5e5183112082\") "
Sep 18 20:03:04.092969 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:03:04.092834    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4phnt\" (UniqueName: \"kubernetes.io/projected/6c48fbd5-b360-49ab-9df2-5e5183112082-kube-api-access-4phnt\") pod \"6c48fbd5-b360-49ab-9df2-5e5183112082\" (UID: \"6c48fbd5-b360-49ab-9df2-5e5183112082\") "
Sep 18 20:03:04.092969 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:03:04.092879    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6c48fbd5-b360-49ab-9df2-5e5183112082-tmp-dir\") pod \"6c48fbd5-b360-49ab-9df2-5e5183112082\" (UID: \"6c48fbd5-b360-49ab-9df2-5e5183112082\") "
Sep 18 20:03:04.092969 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:03:04.092907    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6c48fbd5-b360-49ab-9df2-5e5183112082-home\") pod \"6c48fbd5-b360-49ab-9df2-5e5183112082\" (UID: \"6c48fbd5-b360-49ab-9df2-5e5183112082\") "
Sep 18 20:03:04.093204 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:03:04.093032    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6c48fbd5-b360-49ab-9df2-5e5183112082-dshm\") pod \"6c48fbd5-b360-49ab-9df2-5e5183112082\" (UID: \"6c48fbd5-b360-49ab-9df2-5e5183112082\") "
Sep 18 20:03:04.093204 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:03:04.093085    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6c48fbd5-b360-49ab-9df2-5e5183112082-model-cache\") pod \"6c48fbd5-b360-49ab-9df2-5e5183112082\" (UID: \"6c48fbd5-b360-49ab-9df2-5e5183112082\") "
Sep 18 20:03:04.093204 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:03:04.093158    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6c48fbd5-b360-49ab-9df2-5e5183112082-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "6c48fbd5-b360-49ab-9df2-5e5183112082" (UID: "6c48fbd5-b360-49ab-9df2-5e5183112082"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 20:03:04.093204 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:03:04.093175    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6c48fbd5-b360-49ab-9df2-5e5183112082-home" (OuterVolumeSpecName: "home") pod "6c48fbd5-b360-49ab-9df2-5e5183112082" (UID: "6c48fbd5-b360-49ab-9df2-5e5183112082"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 20:03:04.093488 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:03:04.093464    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6c48fbd5-b360-49ab-9df2-5e5183112082-model-cache" (OuterVolumeSpecName: "model-cache") pod "6c48fbd5-b360-49ab-9df2-5e5183112082" (UID: "6c48fbd5-b360-49ab-9df2-5e5183112082"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 20:03:04.093571 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:03:04.093476    2585 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6c48fbd5-b360-49ab-9df2-5e5183112082-tmp-dir\") on node \"ip-10-0-136-185.ec2.internal\" DevicePath \"\""
Sep 18 20:03:04.093571 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:03:04.093511    2585 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6c48fbd5-b360-49ab-9df2-5e5183112082-home\") on node \"ip-10-0-136-185.ec2.internal\" DevicePath \"\""
Sep 18 20:03:04.094879 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:03:04.094855    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6c48fbd5-b360-49ab-9df2-5e5183112082-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "6c48fbd5-b360-49ab-9df2-5e5183112082" (UID: "6c48fbd5-b360-49ab-9df2-5e5183112082"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 20:03:04.095045 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:03:04.095016    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6c48fbd5-b360-49ab-9df2-5e5183112082-dshm" (OuterVolumeSpecName: "dshm") pod "6c48fbd5-b360-49ab-9df2-5e5183112082" (UID: "6c48fbd5-b360-49ab-9df2-5e5183112082"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 20:03:04.095089 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:03:04.095064    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6c48fbd5-b360-49ab-9df2-5e5183112082-kube-api-access-4phnt" (OuterVolumeSpecName: "kube-api-access-4phnt") pod "6c48fbd5-b360-49ab-9df2-5e5183112082" (UID: "6c48fbd5-b360-49ab-9df2-5e5183112082"). InnerVolumeSpecName "kube-api-access-4phnt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 20:03:04.099247 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:03:04.099229    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-57b6ce-5cd84f68c5-n6t8k"
Sep 18 20:03:04.152708 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:03:04.152676    2585 generic.go:356] "Generic (PLEG): container finished" podID="6c48fbd5-b360-49ab-9df2-5e5183112082" containerID="16ea1a9fb4e1ac800155b99190d83f22d6c8e1c6bec66a071c9140422fba5bcf" exitCode=0
Sep 18 20:03:04.152906 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:03:04.152713    2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-674263/e2e-conflict-a-1b1b9f14-kserve-69899bd87f-stx9d"
Sep 18 20:03:04.152906 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:03:04.152752    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-674263/e2e-conflict-a-1b1b9f14-kserve-69899bd87f-stx9d" event={"ID":"6c48fbd5-b360-49ab-9df2-5e5183112082","Type":"ContainerDied","Data":"16ea1a9fb4e1ac800155b99190d83f22d6c8e1c6bec66a071c9140422fba5bcf"}
Sep 18 20:03:04.152906 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:03:04.152811    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-674263/e2e-conflict-a-1b1b9f14-kserve-69899bd87f-stx9d" event={"ID":"6c48fbd5-b360-49ab-9df2-5e5183112082","Type":"ContainerDied","Data":"fa662d05319aa75fc15045485d810d4d7e6e9b39e7f824ba8b4fbc218edd0073"}
Sep 18 20:03:04.152906 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:03:04.152834    2585 scope.go:117] "RemoveContainer" containerID="16ea1a9fb4e1ac800155b99190d83f22d6c8e1c6bec66a071c9140422fba5bcf"
Sep 18 20:03:04.162866 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:03:04.162850    2585 scope.go:117] "RemoveContainer" containerID="16ea1a9fb4e1ac800155b99190d83f22d6c8e1c6bec66a071c9140422fba5bcf"
Sep 18 20:03:04.163134 ip-10-0-136-185 kubenswrapper[2585]: E0918 20:03:04.163117    2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"16ea1a9fb4e1ac800155b99190d83f22d6c8e1c6bec66a071c9140422fba5bcf\": container with ID starting with 16ea1a9fb4e1ac800155b99190d83f22d6c8e1c6bec66a071c9140422fba5bcf not found: ID does not exist" containerID="16ea1a9fb4e1ac800155b99190d83f22d6c8e1c6bec66a071c9140422fba5bcf"
Sep 18 20:03:04.163202 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:03:04.163149    2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"16ea1a9fb4e1ac800155b99190d83f22d6c8e1c6bec66a071c9140422fba5bcf"} err="failed to get container status \"16ea1a9fb4e1ac800155b99190d83f22d6c8e1c6bec66a071c9140422fba5bcf\": rpc error: code = NotFound desc = could not find container \"16ea1a9fb4e1ac800155b99190d83f22d6c8e1c6bec66a071c9140422fba5bcf\": container with ID starting with 16ea1a9fb4e1ac800155b99190d83f22d6c8e1c6bec66a071c9140422fba5bcf not found: ID does not exist"
Sep 18 20:03:04.176574 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:03:04.176548    2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-674263/e2e-conflict-a-1b1b9f14-kserve-69899bd87f-stx9d"]
Sep 18 20:03:04.181270 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:03:04.181245    2585 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-674263/e2e-conflict-a-1b1b9f14-kserve-69899bd87f-stx9d"]
Sep 18 20:03:04.194232 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:03:04.194213    2585 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6c48fbd5-b360-49ab-9df2-5e5183112082-tls-certs\") on node \"ip-10-0-136-185.ec2.internal\" DevicePath \"\""
Sep 18 20:03:04.194232 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:03:04.194233    2585 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4phnt\" (UniqueName: \"kubernetes.io/projected/6c48fbd5-b360-49ab-9df2-5e5183112082-kube-api-access-4phnt\") on node \"ip-10-0-136-185.ec2.internal\" DevicePath \"\""
Sep 18 20:03:04.194380 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:03:04.194242    2585 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6c48fbd5-b360-49ab-9df2-5e5183112082-dshm\") on node \"ip-10-0-136-185.ec2.internal\" DevicePath \"\""
Sep 18 20:03:04.194380 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:03:04.194252    2585 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6c48fbd5-b360-49ab-9df2-5e5183112082-model-cache\") on node \"ip-10-0-136-185.ec2.internal\" DevicePath \"\""
Sep 18 20:03:04.662431 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:03:04.662396    2585 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6c48fbd5-b360-49ab-9df2-5e5183112082" path="/var/lib/kubelet/pods/6c48fbd5-b360-49ab-9df2-5e5183112082/volumes"
Sep 18 20:03:09.382617 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:03:09.382578    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-xmwh7"]
Sep 18 20:03:24.736513 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:03:24.736477    2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-5547b6-576c844cb6-jz58p"]
Sep 18 20:03:24.736986 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:03:24.736875    2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6c48fbd5-b360-49ab-9df2-5e5183112082" containerName="main"
Sep 18 20:03:24.736986 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:03:24.736887    2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="6c48fbd5-b360-49ab-9df2-5e5183112082" containerName="main"
Sep 18 20:03:24.736986 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:03:24.736965    2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="6c48fbd5-b360-49ab-9df2-5e5183112082" containerName="main"
Sep 18 20:03:24.741462 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:03:24.741439    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-5547b6-576c844cb6-jz58p"
Sep 18 20:03:24.744505 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:03:24.744377    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w3-5547b6\""
Sep 18 20:03:24.751914 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:03:24.751888    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-5547b6-576c844cb6-jz58p"]
Sep 18 20:03:24.790615 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:03:24.790581    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/56b475f5-65d7-4b6d-a8ba-89654ec9adaa-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w3-5547b6-576c844cb6-jz58p\" (UID: \"56b475f5-65d7-4b6d-a8ba-89654ec9adaa\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-5547b6-576c844cb6-jz58p"
Sep 18 20:03:24.790769 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:03:24.790647    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tnqld\" (UniqueName: \"kubernetes.io/projected/56b475f5-65d7-4b6d-a8ba-89654ec9adaa-kube-api-access-tnqld\") pod \"payload-pre-processing-e2e-shared-b-w3-5547b6-576c844cb6-jz58p\" (UID: \"56b475f5-65d7-4b6d-a8ba-89654ec9adaa\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-5547b6-576c844cb6-jz58p"
Sep 18 20:03:24.892297 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:03:24.892258    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/56b475f5-65d7-4b6d-a8ba-89654ec9adaa-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w3-5547b6-576c844cb6-jz58p\" (UID: \"56b475f5-65d7-4b6d-a8ba-89654ec9adaa\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-5547b6-576c844cb6-jz58p"
Sep 18 20:03:24.892458 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:03:24.892342    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tnqld\" (UniqueName: \"kubernetes.io/projected/56b475f5-65d7-4b6d-a8ba-89654ec9adaa-kube-api-access-tnqld\") pod \"payload-pre-processing-e2e-shared-b-w3-5547b6-576c844cb6-jz58p\" (UID: \"56b475f5-65d7-4b6d-a8ba-89654ec9adaa\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-5547b6-576c844cb6-jz58p"
Sep 18 20:03:24.893061 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:03:24.893042    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/56b475f5-65d7-4b6d-a8ba-89654ec9adaa-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w3-5547b6-576c844cb6-jz58p\" (UID: \"56b475f5-65d7-4b6d-a8ba-89654ec9adaa\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-5547b6-576c844cb6-jz58p"
Sep 18 20:03:24.902038 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:03:24.902014    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tnqld\" (UniqueName: \"kubernetes.io/projected/56b475f5-65d7-4b6d-a8ba-89654ec9adaa-kube-api-access-tnqld\") pod \"payload-pre-processing-e2e-shared-b-w3-5547b6-576c844cb6-jz58p\" (UID: \"56b475f5-65d7-4b6d-a8ba-89654ec9adaa\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-5547b6-576c844cb6-jz58p"
Sep 18 20:03:25.057606 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:03:25.057507    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-5547b6-576c844cb6-jz58p"
Sep 18 20:03:25.336059 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:03:25.335836    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-5547b6-576c844cb6-jz58p"]
Sep 18 20:03:25.338319 ip-10-0-136-185 kubenswrapper[2585]: W0918 20:03:25.338288    2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod56b475f5_65d7_4b6d_a8ba_89654ec9adaa.slice/crio-e1043238b5be88bb33bc07ca4558c4c5179ee49d6b1fda17ba9aebc9beb409ea WatchSource:0}: Error finding container e1043238b5be88bb33bc07ca4558c4c5179ee49d6b1fda17ba9aebc9beb409ea: Status 404 returned error can't find the container with id e1043238b5be88bb33bc07ca4558c4c5179ee49d6b1fda17ba9aebc9beb409ea
Sep 18 20:03:26.269872 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:03:26.269835    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-5547b6-576c844cb6-jz58p" event={"ID":"56b475f5-65d7-4b6d-a8ba-89654ec9adaa","Type":"ContainerStarted","Data":"9be2061167406c237d7862973bfdc207551b0cc8418ff5b307568652fc5341d8"}
Sep 18 20:03:26.270282 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:03:26.269900    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-5547b6-576c844cb6-jz58p" event={"ID":"56b475f5-65d7-4b6d-a8ba-89654ec9adaa","Type":"ContainerStarted","Data":"e1043238b5be88bb33bc07ca4558c4c5179ee49d6b1fda17ba9aebc9beb409ea"}
Sep 18 20:03:26.270282 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:03:26.269962    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-5547b6-576c844cb6-jz58p"
Sep 18 20:03:26.286372 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:03:26.286321    2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-5547b6-576c844cb6-jz58p" podStartSLOduration=2.286303451 podStartE2EDuration="2.286303451s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 20:03:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 20:03:26.285243183 +0000 UTC m=+1447.809848616" watchObservedRunningTime="2026-09-18 20:03:26.286303451 +0000 UTC m=+1447.810908874"
Sep 18 20:03:37.278211 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:03:37.278159    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-5547b6-576c844cb6-jz58p"
Sep 18 20:03:43.409777 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:03:43.409738    2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-166dc9bb/e2e-lifecycle-model-166dc9bb-kserve-54cd4c4579-6xzm4"]
Sep 18 20:03:44.778710 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:03:44.778666    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-xmwh7"]
Sep 18 20:03:54.873538 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:03:54.873506    2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-674263/distinct-w4-674263-kserve-69895788f-9wp56"]
Sep 18 20:03:54.877493 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:03:54.877473    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-674263/distinct-w4-674263-kserve-69895788f-9wp56"
Sep 18 20:03:54.879755 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:03:54.879731    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-674263\"/\"distinct-w4-674263-kserve-self-signed-certs\""
Sep 18 20:03:54.886055 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:03:54.886025    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-674263/distinct-w4-674263-kserve-69895788f-9wp56"]
Sep 18 20:03:54.992775 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:03:54.992736    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4ffc064a-d322-43ef-b803-f51096f08bc8-home\") pod \"distinct-w4-674263-kserve-69895788f-9wp56\" (UID: \"4ffc064a-d322-43ef-b803-f51096f08bc8\") " pod="e2e-models-e2e-worker-w4-674263/distinct-w4-674263-kserve-69895788f-9wp56"
Sep 18 20:03:54.992775 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:03:54.992774    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mrvj5\" (UniqueName: \"kubernetes.io/projected/4ffc064a-d322-43ef-b803-f51096f08bc8-kube-api-access-mrvj5\") pod \"distinct-w4-674263-kserve-69895788f-9wp56\" (UID: \"4ffc064a-d322-43ef-b803-f51096f08bc8\") " pod="e2e-models-e2e-worker-w4-674263/distinct-w4-674263-kserve-69895788f-9wp56"
Sep 18 20:03:54.993042 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:03:54.992846    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4ffc064a-d322-43ef-b803-f51096f08bc8-tls-certs\") pod \"distinct-w4-674263-kserve-69895788f-9wp56\" (UID: \"4ffc064a-d322-43ef-b803-f51096f08bc8\") " pod="e2e-models-e2e-worker-w4-674263/distinct-w4-674263-kserve-69895788f-9wp56"
Sep 18 20:03:54.993042 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:03:54.992881    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4ffc064a-d322-43ef-b803-f51096f08bc8-model-cache\") pod \"distinct-w4-674263-kserve-69895788f-9wp56\" (UID: \"4ffc064a-d322-43ef-b803-f51096f08bc8\") " pod="e2e-models-e2e-worker-w4-674263/distinct-w4-674263-kserve-69895788f-9wp56"
Sep 18 20:03:54.993042 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:03:54.992950    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4ffc064a-d322-43ef-b803-f51096f08bc8-dshm\") pod \"distinct-w4-674263-kserve-69895788f-9wp56\" (UID: \"4ffc064a-d322-43ef-b803-f51096f08bc8\") " pod="e2e-models-e2e-worker-w4-674263/distinct-w4-674263-kserve-69895788f-9wp56"
Sep 18 20:03:54.993042 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:03:54.993013    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4ffc064a-d322-43ef-b803-f51096f08bc8-tmp-dir\") pod \"distinct-w4-674263-kserve-69895788f-9wp56\" (UID: \"4ffc064a-d322-43ef-b803-f51096f08bc8\") " pod="e2e-models-e2e-worker-w4-674263/distinct-w4-674263-kserve-69895788f-9wp56"
Sep 18 20:03:55.093556 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:03:55.093517    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4ffc064a-d322-43ef-b803-f51096f08bc8-tmp-dir\") pod \"distinct-w4-674263-kserve-69895788f-9wp56\" (UID: \"4ffc064a-d322-43ef-b803-f51096f08bc8\") " pod="e2e-models-e2e-worker-w4-674263/distinct-w4-674263-kserve-69895788f-9wp56"
Sep 18 20:03:55.093757 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:03:55.093591    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4ffc064a-d322-43ef-b803-f51096f08bc8-home\") pod \"distinct-w4-674263-kserve-69895788f-9wp56\" (UID: \"4ffc064a-d322-43ef-b803-f51096f08bc8\") " pod="e2e-models-e2e-worker-w4-674263/distinct-w4-674263-kserve-69895788f-9wp56"
Sep 18 20:03:55.093757 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:03:55.093611    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mrvj5\" (UniqueName: \"kubernetes.io/projected/4ffc064a-d322-43ef-b803-f51096f08bc8-kube-api-access-mrvj5\") pod \"distinct-w4-674263-kserve-69895788f-9wp56\" (UID: \"4ffc064a-d322-43ef-b803-f51096f08bc8\") " pod="e2e-models-e2e-worker-w4-674263/distinct-w4-674263-kserve-69895788f-9wp56"
Sep 18 20:03:55.093757 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:03:55.093630    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4ffc064a-d322-43ef-b803-f51096f08bc8-tls-certs\") pod \"distinct-w4-674263-kserve-69895788f-9wp56\" (UID: \"4ffc064a-d322-43ef-b803-f51096f08bc8\") " pod="e2e-models-e2e-worker-w4-674263/distinct-w4-674263-kserve-69895788f-9wp56"
Sep 18 20:03:55.093757 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:03:55.093652    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4ffc064a-d322-43ef-b803-f51096f08bc8-model-cache\") pod \"distinct-w4-674263-kserve-69895788f-9wp56\" (UID: \"4ffc064a-d322-43ef-b803-f51096f08bc8\") " pod="e2e-models-e2e-worker-w4-674263/distinct-w4-674263-kserve-69895788f-9wp56"
Sep 18 20:03:55.093757 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:03:55.093684    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4ffc064a-d322-43ef-b803-f51096f08bc8-dshm\") pod \"distinct-w4-674263-kserve-69895788f-9wp56\" (UID: \"4ffc064a-d322-43ef-b803-f51096f08bc8\") " pod="e2e-models-e2e-worker-w4-674263/distinct-w4-674263-kserve-69895788f-9wp56"
Sep 18 20:03:55.094058 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:03:55.094039    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4ffc064a-d322-43ef-b803-f51096f08bc8-tmp-dir\") pod \"distinct-w4-674263-kserve-69895788f-9wp56\" (UID: \"4ffc064a-d322-43ef-b803-f51096f08bc8\") " pod="e2e-models-e2e-worker-w4-674263/distinct-w4-674263-kserve-69895788f-9wp56"
Sep 18 20:03:55.094109 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:03:55.094070    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4ffc064a-d322-43ef-b803-f51096f08bc8-model-cache\") pod \"distinct-w4-674263-kserve-69895788f-9wp56\" (UID: \"4ffc064a-d322-43ef-b803-f51096f08bc8\") " pod="e2e-models-e2e-worker-w4-674263/distinct-w4-674263-kserve-69895788f-9wp56"
Sep 18 20:03:55.094174 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:03:55.094157    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4ffc064a-d322-43ef-b803-f51096f08bc8-home\") pod \"distinct-w4-674263-kserve-69895788f-9wp56\" (UID: \"4ffc064a-d322-43ef-b803-f51096f08bc8\") " pod="e2e-models-e2e-worker-w4-674263/distinct-w4-674263-kserve-69895788f-9wp56"
Sep 18 20:03:55.095974 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:03:55.095955    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4ffc064a-d322-43ef-b803-f51096f08bc8-dshm\") pod \"distinct-w4-674263-kserve-69895788f-9wp56\" (UID: \"4ffc064a-d322-43ef-b803-f51096f08bc8\") " pod="e2e-models-e2e-worker-w4-674263/distinct-w4-674263-kserve-69895788f-9wp56"
Sep 18 20:03:55.096298 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:03:55.096281    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4ffc064a-d322-43ef-b803-f51096f08bc8-tls-certs\") pod \"distinct-w4-674263-kserve-69895788f-9wp56\" (UID: \"4ffc064a-d322-43ef-b803-f51096f08bc8\") " pod="e2e-models-e2e-worker-w4-674263/distinct-w4-674263-kserve-69895788f-9wp56"
Sep 18 20:03:55.101970 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:03:55.101949    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mrvj5\" (UniqueName: \"kubernetes.io/projected/4ffc064a-d322-43ef-b803-f51096f08bc8-kube-api-access-mrvj5\") pod \"distinct-w4-674263-kserve-69895788f-9wp56\" (UID: \"4ffc064a-d322-43ef-b803-f51096f08bc8\") " pod="e2e-models-e2e-worker-w4-674263/distinct-w4-674263-kserve-69895788f-9wp56"
Sep 18 20:03:55.188383 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:03:55.188334    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-674263/distinct-w4-674263-kserve-69895788f-9wp56"
Sep 18 20:03:55.465610 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:03:55.465585    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-674263/distinct-w4-674263-kserve-69895788f-9wp56"]
Sep 18 20:03:55.467542 ip-10-0-136-185 kubenswrapper[2585]: W0918 20:03:55.467516    2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod4ffc064a_d322_43ef_b803_f51096f08bc8.slice/crio-8005b4759d7b1f1c0fc78c4ebb9003606aa7431658f7244834a9e7792ac4ca65 WatchSource:0}: Error finding container 8005b4759d7b1f1c0fc78c4ebb9003606aa7431658f7244834a9e7792ac4ca65: Status 404 returned error can't find the container with id 8005b4759d7b1f1c0fc78c4ebb9003606aa7431658f7244834a9e7792ac4ca65
Sep 18 20:03:56.431870 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:03:56.431827    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-674263/distinct-w4-674263-kserve-69895788f-9wp56" event={"ID":"4ffc064a-d322-43ef-b803-f51096f08bc8","Type":"ContainerStarted","Data":"ff07c99a809aaf344fcf3bfd0e173a527729032c91749c0c9f9c5227651727e3"}
Sep 18 20:03:56.432259 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:03:56.431879    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-674263/distinct-w4-674263-kserve-69895788f-9wp56" event={"ID":"4ffc064a-d322-43ef-b803-f51096f08bc8","Type":"ContainerStarted","Data":"8005b4759d7b1f1c0fc78c4ebb9003606aa7431658f7244834a9e7792ac4ca65"}
Sep 18 20:03:56.448764 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:03:56.448710    2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-674263/distinct-w4-674263-kserve-69895788f-9wp56" podStartSLOduration=2.44869212 podStartE2EDuration="2.44869212s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 20:03:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 20:03:56.447962812 +0000 UTC m=+1477.972568234" watchObservedRunningTime="2026-09-18 20:03:56.44869212 +0000 UTC m=+1477.973297542"
Sep 18 20:03:58.429470 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:03:58.429425    2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-mt-166dc9bb/e2e-lifecycle-model-166dc9bb-kserve-54cd4c4579-6xzm4" podUID="cdd98a98-da96-4013-8bea-a3c721815940" containerName="main" containerID="cri-o://a8f64c8adadc434bcbf3f57f57f701947f9caa9033a3623a29e991fe254414b3" gracePeriod=45
Sep 18 20:03:58.682286 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:03:58.682224    2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-166dc9bb/e2e-lifecycle-model-166dc9bb-kserve-54cd4c4579-6xzm4"
Sep 18 20:03:58.730097 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:03:58.730053    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cdd98a98-da96-4013-8bea-a3c721815940-model-cache\") pod \"cdd98a98-da96-4013-8bea-a3c721815940\" (UID: \"cdd98a98-da96-4013-8bea-a3c721815940\") "
Sep 18 20:03:58.730305 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:03:58.730150    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cdd98a98-da96-4013-8bea-a3c721815940-dshm\") pod \"cdd98a98-da96-4013-8bea-a3c721815940\" (UID: \"cdd98a98-da96-4013-8bea-a3c721815940\") "
Sep 18 20:03:58.730305 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:03:58.730190    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-82v5c\" (UniqueName: \"kubernetes.io/projected/cdd98a98-da96-4013-8bea-a3c721815940-kube-api-access-82v5c\") pod \"cdd98a98-da96-4013-8bea-a3c721815940\" (UID: \"cdd98a98-da96-4013-8bea-a3c721815940\") "
Sep 18 20:03:58.730305 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:03:58.730219    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cdd98a98-da96-4013-8bea-a3c721815940-tmp-dir\") pod \"cdd98a98-da96-4013-8bea-a3c721815940\" (UID: \"cdd98a98-da96-4013-8bea-a3c721815940\") "
Sep 18 20:03:58.730305 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:03:58.730263    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cdd98a98-da96-4013-8bea-a3c721815940-home\") pod \"cdd98a98-da96-4013-8bea-a3c721815940\" (UID: \"cdd98a98-da96-4013-8bea-a3c721815940\") "
Sep 18 20:03:58.730502 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:03:58.730316    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cdd98a98-da96-4013-8bea-a3c721815940-tls-certs\") pod \"cdd98a98-da96-4013-8bea-a3c721815940\" (UID: \"cdd98a98-da96-4013-8bea-a3c721815940\") "
Sep 18 20:03:58.730901 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:03:58.730851    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cdd98a98-da96-4013-8bea-a3c721815940-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "cdd98a98-da96-4013-8bea-a3c721815940" (UID: "cdd98a98-da96-4013-8bea-a3c721815940"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 20:03:58.731088 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:03:58.730901    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cdd98a98-da96-4013-8bea-a3c721815940-model-cache" (OuterVolumeSpecName: "model-cache") pod "cdd98a98-da96-4013-8bea-a3c721815940" (UID: "cdd98a98-da96-4013-8bea-a3c721815940"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 20:03:58.731444 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:03:58.731407    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cdd98a98-da96-4013-8bea-a3c721815940-home" (OuterVolumeSpecName: "home") pod "cdd98a98-da96-4013-8bea-a3c721815940" (UID: "cdd98a98-da96-4013-8bea-a3c721815940"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 20:03:58.733313 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:03:58.733279    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cdd98a98-da96-4013-8bea-a3c721815940-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "cdd98a98-da96-4013-8bea-a3c721815940" (UID: "cdd98a98-da96-4013-8bea-a3c721815940"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 20:03:58.733554 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:03:58.733526    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cdd98a98-da96-4013-8bea-a3c721815940-kube-api-access-82v5c" (OuterVolumeSpecName: "kube-api-access-82v5c") pod "cdd98a98-da96-4013-8bea-a3c721815940" (UID: "cdd98a98-da96-4013-8bea-a3c721815940"). InnerVolumeSpecName "kube-api-access-82v5c". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 20:03:58.733554 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:03:58.733539    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cdd98a98-da96-4013-8bea-a3c721815940-dshm" (OuterVolumeSpecName: "dshm") pod "cdd98a98-da96-4013-8bea-a3c721815940" (UID: "cdd98a98-da96-4013-8bea-a3c721815940"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 20:03:58.832279 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:03:58.832239    2585 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cdd98a98-da96-4013-8bea-a3c721815940-model-cache\") on node \"ip-10-0-136-185.ec2.internal\" DevicePath \"\""
Sep 18 20:03:58.832279 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:03:58.832271    2585 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cdd98a98-da96-4013-8bea-a3c721815940-dshm\") on node \"ip-10-0-136-185.ec2.internal\" DevicePath \"\""
Sep 18 20:03:58.832279 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:03:58.832282    2585 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-82v5c\" (UniqueName: \"kubernetes.io/projected/cdd98a98-da96-4013-8bea-a3c721815940-kube-api-access-82v5c\") on node \"ip-10-0-136-185.ec2.internal\" DevicePath \"\""
Sep 18 20:03:58.832532 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:03:58.832291    2585 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cdd98a98-da96-4013-8bea-a3c721815940-tmp-dir\") on node \"ip-10-0-136-185.ec2.internal\" DevicePath \"\""
Sep 18 20:03:58.832532 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:03:58.832302    2585 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cdd98a98-da96-4013-8bea-a3c721815940-home\") on node \"ip-10-0-136-185.ec2.internal\" DevicePath \"\""
Sep 18 20:03:58.832532 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:03:58.832310    2585 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cdd98a98-da96-4013-8bea-a3c721815940-tls-certs\") on node \"ip-10-0-136-185.ec2.internal\" DevicePath \"\""
Sep 18 20:03:59.449277 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:03:59.449246    2585 generic.go:356] "Generic (PLEG): container finished" podID="cdd98a98-da96-4013-8bea-a3c721815940" containerID="a8f64c8adadc434bcbf3f57f57f701947f9caa9033a3623a29e991fe254414b3" exitCode=0
Sep 18 20:03:59.449711 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:03:59.449289    2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-166dc9bb/e2e-lifecycle-model-166dc9bb-kserve-54cd4c4579-6xzm4"
Sep 18 20:03:59.449711 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:03:59.449351    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-166dc9bb/e2e-lifecycle-model-166dc9bb-kserve-54cd4c4579-6xzm4" event={"ID":"cdd98a98-da96-4013-8bea-a3c721815940","Type":"ContainerDied","Data":"a8f64c8adadc434bcbf3f57f57f701947f9caa9033a3623a29e991fe254414b3"}
Sep 18 20:03:59.449711 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:03:59.449391    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-166dc9bb/e2e-lifecycle-model-166dc9bb-kserve-54cd4c4579-6xzm4" event={"ID":"cdd98a98-da96-4013-8bea-a3c721815940","Type":"ContainerDied","Data":"5c301538be81a634d5fa8a21dd8b781b45567fdcd696e5bcafe4626c7508ba3a"}
Sep 18 20:03:59.449711 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:03:59.449409    2585 scope.go:117] "RemoveContainer" containerID="a8f64c8adadc434bcbf3f57f57f701947f9caa9033a3623a29e991fe254414b3"
Sep 18 20:03:59.459002 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:03:59.458979    2585 scope.go:117] "RemoveContainer" containerID="a8f64c8adadc434bcbf3f57f57f701947f9caa9033a3623a29e991fe254414b3"
Sep 18 20:03:59.459306 ip-10-0-136-185 kubenswrapper[2585]: E0918 20:03:59.459287    2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a8f64c8adadc434bcbf3f57f57f701947f9caa9033a3623a29e991fe254414b3\": container with ID starting with a8f64c8adadc434bcbf3f57f57f701947f9caa9033a3623a29e991fe254414b3 not found: ID does not exist" containerID="a8f64c8adadc434bcbf3f57f57f701947f9caa9033a3623a29e991fe254414b3"
Sep 18 20:03:59.459377 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:03:59.459315    2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a8f64c8adadc434bcbf3f57f57f701947f9caa9033a3623a29e991fe254414b3"} err="failed to get container status \"a8f64c8adadc434bcbf3f57f57f701947f9caa9033a3623a29e991fe254414b3\": rpc error: code = NotFound desc = could not find container \"a8f64c8adadc434bcbf3f57f57f701947f9caa9033a3623a29e991fe254414b3\": container with ID starting with a8f64c8adadc434bcbf3f57f57f701947f9caa9033a3623a29e991fe254414b3 not found: ID does not exist"
Sep 18 20:03:59.470555 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:03:59.470527    2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-166dc9bb/e2e-lifecycle-model-166dc9bb-kserve-54cd4c4579-6xzm4"]
Sep 18 20:03:59.474821 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:03:59.474779    2585 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-mt-166dc9bb/e2e-lifecycle-model-166dc9bb-kserve-54cd4c4579-6xzm4"]
Sep 18 20:04:00.663367 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:04:00.663331    2585 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="cdd98a98-da96-4013-8bea-a3c721815940" path="/var/lib/kubelet/pods/cdd98a98-da96-4013-8bea-a3c721815940/volumes"
Sep 18 20:04:05.188860 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:04:05.188819    2585 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-674263/distinct-w4-674263-kserve-69895788f-9wp56"
Sep 18 20:04:05.188860 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:04:05.188871    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-674263/distinct-w4-674263-kserve-69895788f-9wp56"
Sep 18 20:04:05.201831 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:04:05.201780    2585 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-674263/distinct-w4-674263-kserve-69895788f-9wp56"
Sep 18 20:04:05.494871 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:04:05.494771    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-674263/distinct-w4-674263-kserve-69895788f-9wp56"
Sep 18 20:04:05.883281 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:04:05.883198    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-xmwh7"]
Sep 18 20:04:15.370208 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:04:15.370172    2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w3-57b6ce/auth-test-model-57b6ce-kserve-66868b47d7-xkkds"]
Sep 18 20:04:15.370846 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:04:15.370824    2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cdd98a98-da96-4013-8bea-a3c721815940" containerName="main"
Sep 18 20:04:15.370942 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:04:15.370847    2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="cdd98a98-da96-4013-8bea-a3c721815940" containerName="main"
Sep 18 20:04:15.371003 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:04:15.370940    2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="cdd98a98-da96-4013-8bea-a3c721815940" containerName="main"
Sep 18 20:04:15.373934 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:04:15.373911    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-57b6ce/auth-test-model-57b6ce-kserve-66868b47d7-xkkds"
Sep 18 20:04:15.377100 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:04:15.377078    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-57b6ce\"/\"default-dockercfg-mrlqv\""
Sep 18 20:04:15.377190 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:04:15.377116    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w3-57b6ce\"/\"kube-root-ca.crt\""
Sep 18 20:04:15.377190 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:04:15.377077    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w3-57b6ce\"/\"openshift-service-ca.crt\""
Sep 18 20:04:15.377278 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:04:15.377146    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-57b6ce\"/\"auth-test-model-57b6ce-kserve-self-signed-certs\""
Sep 18 20:04:15.381983 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:04:15.381961    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-57b6ce/auth-test-model-57b6ce-kserve-66868b47d7-xkkds"]
Sep 18 20:04:15.505435 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:04:15.505393    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dc70d7c6-212a-403f-8204-d18d36daa4dd-model-cache\") pod \"auth-test-model-57b6ce-kserve-66868b47d7-xkkds\" (UID: \"dc70d7c6-212a-403f-8204-d18d36daa4dd\") " pod="ai-tenant-e2e-shared-a-w3-57b6ce/auth-test-model-57b6ce-kserve-66868b47d7-xkkds"
Sep 18 20:04:15.505435 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:04:15.505431    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dc70d7c6-212a-403f-8204-d18d36daa4dd-tls-certs\") pod \"auth-test-model-57b6ce-kserve-66868b47d7-xkkds\" (UID: \"dc70d7c6-212a-403f-8204-d18d36daa4dd\") " pod="ai-tenant-e2e-shared-a-w3-57b6ce/auth-test-model-57b6ce-kserve-66868b47d7-xkkds"
Sep 18 20:04:15.505653 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:04:15.505514    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dc70d7c6-212a-403f-8204-d18d36daa4dd-home\") pod \"auth-test-model-57b6ce-kserve-66868b47d7-xkkds\" (UID: \"dc70d7c6-212a-403f-8204-d18d36daa4dd\") " pod="ai-tenant-e2e-shared-a-w3-57b6ce/auth-test-model-57b6ce-kserve-66868b47d7-xkkds"
Sep 18 20:04:15.505653 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:04:15.505563    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dc70d7c6-212a-403f-8204-d18d36daa4dd-tmp-dir\") pod \"auth-test-model-57b6ce-kserve-66868b47d7-xkkds\" (UID: \"dc70d7c6-212a-403f-8204-d18d36daa4dd\") " pod="ai-tenant-e2e-shared-a-w3-57b6ce/auth-test-model-57b6ce-kserve-66868b47d7-xkkds"
Sep 18 20:04:15.505653 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:04:15.505597    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-498l8\" (UniqueName: \"kubernetes.io/projected/dc70d7c6-212a-403f-8204-d18d36daa4dd-kube-api-access-498l8\") pod \"auth-test-model-57b6ce-kserve-66868b47d7-xkkds\" (UID: \"dc70d7c6-212a-403f-8204-d18d36daa4dd\") " pod="ai-tenant-e2e-shared-a-w3-57b6ce/auth-test-model-57b6ce-kserve-66868b47d7-xkkds"
Sep 18 20:04:15.505653 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:04:15.505621    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dc70d7c6-212a-403f-8204-d18d36daa4dd-dshm\") pod \"auth-test-model-57b6ce-kserve-66868b47d7-xkkds\" (UID: \"dc70d7c6-212a-403f-8204-d18d36daa4dd\") " pod="ai-tenant-e2e-shared-a-w3-57b6ce/auth-test-model-57b6ce-kserve-66868b47d7-xkkds"
Sep 18 20:04:15.606936 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:04:15.606896    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dc70d7c6-212a-403f-8204-d18d36daa4dd-model-cache\") pod \"auth-test-model-57b6ce-kserve-66868b47d7-xkkds\" (UID: \"dc70d7c6-212a-403f-8204-d18d36daa4dd\") " pod="ai-tenant-e2e-shared-a-w3-57b6ce/auth-test-model-57b6ce-kserve-66868b47d7-xkkds"
Sep 18 20:04:15.606936 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:04:15.606936    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dc70d7c6-212a-403f-8204-d18d36daa4dd-tls-certs\") pod \"auth-test-model-57b6ce-kserve-66868b47d7-xkkds\" (UID: \"dc70d7c6-212a-403f-8204-d18d36daa4dd\") " pod="ai-tenant-e2e-shared-a-w3-57b6ce/auth-test-model-57b6ce-kserve-66868b47d7-xkkds"
Sep 18 20:04:15.607198 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:04:15.606971    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dc70d7c6-212a-403f-8204-d18d36daa4dd-home\") pod \"auth-test-model-57b6ce-kserve-66868b47d7-xkkds\" (UID: \"dc70d7c6-212a-403f-8204-d18d36daa4dd\") " pod="ai-tenant-e2e-shared-a-w3-57b6ce/auth-test-model-57b6ce-kserve-66868b47d7-xkkds"
Sep 18 20:04:15.607198 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:04:15.607008    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dc70d7c6-212a-403f-8204-d18d36daa4dd-tmp-dir\") pod \"auth-test-model-57b6ce-kserve-66868b47d7-xkkds\" (UID: \"dc70d7c6-212a-403f-8204-d18d36daa4dd\") " pod="ai-tenant-e2e-shared-a-w3-57b6ce/auth-test-model-57b6ce-kserve-66868b47d7-xkkds"
Sep 18 20:04:15.607198 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:04:15.607036    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-498l8\" (UniqueName: \"kubernetes.io/projected/dc70d7c6-212a-403f-8204-d18d36daa4dd-kube-api-access-498l8\") pod \"auth-test-model-57b6ce-kserve-66868b47d7-xkkds\" (UID: \"dc70d7c6-212a-403f-8204-d18d36daa4dd\") " pod="ai-tenant-e2e-shared-a-w3-57b6ce/auth-test-model-57b6ce-kserve-66868b47d7-xkkds"
Sep 18 20:04:15.607198 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:04:15.607061    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dc70d7c6-212a-403f-8204-d18d36daa4dd-dshm\") pod \"auth-test-model-57b6ce-kserve-66868b47d7-xkkds\" (UID: \"dc70d7c6-212a-403f-8204-d18d36daa4dd\") " pod="ai-tenant-e2e-shared-a-w3-57b6ce/auth-test-model-57b6ce-kserve-66868b47d7-xkkds"
Sep 18 20:04:15.607407 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:04:15.607316    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dc70d7c6-212a-403f-8204-d18d36daa4dd-model-cache\") pod \"auth-test-model-57b6ce-kserve-66868b47d7-xkkds\" (UID: \"dc70d7c6-212a-403f-8204-d18d36daa4dd\") " pod="ai-tenant-e2e-shared-a-w3-57b6ce/auth-test-model-57b6ce-kserve-66868b47d7-xkkds"
Sep 18 20:04:15.607407 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:04:15.607378    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dc70d7c6-212a-403f-8204-d18d36daa4dd-home\") pod \"auth-test-model-57b6ce-kserve-66868b47d7-xkkds\" (UID: \"dc70d7c6-212a-403f-8204-d18d36daa4dd\") " pod="ai-tenant-e2e-shared-a-w3-57b6ce/auth-test-model-57b6ce-kserve-66868b47d7-xkkds"
Sep 18 20:04:15.607509 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:04:15.607489    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dc70d7c6-212a-403f-8204-d18d36daa4dd-tmp-dir\") pod \"auth-test-model-57b6ce-kserve-66868b47d7-xkkds\" (UID: \"dc70d7c6-212a-403f-8204-d18d36daa4dd\") " pod="ai-tenant-e2e-shared-a-w3-57b6ce/auth-test-model-57b6ce-kserve-66868b47d7-xkkds"
Sep 18 20:04:15.609282 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:04:15.609253    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dc70d7c6-212a-403f-8204-d18d36daa4dd-dshm\") pod \"auth-test-model-57b6ce-kserve-66868b47d7-xkkds\" (UID: \"dc70d7c6-212a-403f-8204-d18d36daa4dd\") " pod="ai-tenant-e2e-shared-a-w3-57b6ce/auth-test-model-57b6ce-kserve-66868b47d7-xkkds"
Sep 18 20:04:15.609640 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:04:15.609623    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dc70d7c6-212a-403f-8204-d18d36daa4dd-tls-certs\") pod \"auth-test-model-57b6ce-kserve-66868b47d7-xkkds\" (UID: \"dc70d7c6-212a-403f-8204-d18d36daa4dd\") " pod="ai-tenant-e2e-shared-a-w3-57b6ce/auth-test-model-57b6ce-kserve-66868b47d7-xkkds"
Sep 18 20:04:15.615176 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:04:15.615153    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-498l8\" (UniqueName: \"kubernetes.io/projected/dc70d7c6-212a-403f-8204-d18d36daa4dd-kube-api-access-498l8\") pod \"auth-test-model-57b6ce-kserve-66868b47d7-xkkds\" (UID: \"dc70d7c6-212a-403f-8204-d18d36daa4dd\") " pod="ai-tenant-e2e-shared-a-w3-57b6ce/auth-test-model-57b6ce-kserve-66868b47d7-xkkds"
Sep 18 20:04:15.683759 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:04:15.683726    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-57b6ce/auth-test-model-57b6ce-kserve-66868b47d7-xkkds"
Sep 18 20:04:15.955370 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:04:15.955343    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-57b6ce/auth-test-model-57b6ce-kserve-66868b47d7-xkkds"]
Sep 18 20:04:15.956852 ip-10-0-136-185 kubenswrapper[2585]: W0918 20:04:15.956824    2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poddc70d7c6_212a_403f_8204_d18d36daa4dd.slice/crio-ddfc417fa04a16af1cd9bc4a3e8ffb3d82195d0e973cc9f0431e2f5c1b1e52c4 WatchSource:0}: Error finding container ddfc417fa04a16af1cd9bc4a3e8ffb3d82195d0e973cc9f0431e2f5c1b1e52c4: Status 404 returned error can't find the container with id ddfc417fa04a16af1cd9bc4a3e8ffb3d82195d0e973cc9f0431e2f5c1b1e52c4
Sep 18 20:04:16.543550 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:04:16.543511    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-57b6ce/auth-test-model-57b6ce-kserve-66868b47d7-xkkds" event={"ID":"dc70d7c6-212a-403f-8204-d18d36daa4dd","Type":"ContainerStarted","Data":"bbfc9502ef9c55652574bdab451a058b2a04c342881ad099ab90ef6dadbd22f1"}
Sep 18 20:04:16.543550 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:04:16.543560    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-57b6ce/auth-test-model-57b6ce-kserve-66868b47d7-xkkds" event={"ID":"dc70d7c6-212a-403f-8204-d18d36daa4dd","Type":"ContainerStarted","Data":"ddfc417fa04a16af1cd9bc4a3e8ffb3d82195d0e973cc9f0431e2f5c1b1e52c4"}
Sep 18 20:04:16.560763 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:04:16.560704    2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w3-57b6ce/auth-test-model-57b6ce-kserve-66868b47d7-xkkds" podStartSLOduration=1.5606900449999999 podStartE2EDuration="1.560690045s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 20:04:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 20:04:16.557992021 +0000 UTC m=+1498.082597445" watchObservedRunningTime="2026-09-18 20:04:16.560690045 +0000 UTC m=+1498.085295467"
Sep 18 20:04:20.661230 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:04:20.661204    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-rctld_cc0e0de2-75d5-43a1-b64b-b071427cd8ce/ovn-acl-logging/0.log"
Sep 18 20:04:20.669064 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:04:20.669036    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-rctld_cc0e0de2-75d5-43a1-b64b-b071427cd8ce/ovn-acl-logging/0.log"
Sep 18 20:04:20.750766 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:04:20.750729    2585 scope.go:117] "RemoveContainer" containerID="d5e8ef574bc8781b42c6f268247b937ded2a1288f0527b06a8d660978a2dc896"
Sep 18 20:04:25.684372 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:04:25.684333    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w3-57b6ce/auth-test-model-57b6ce-kserve-66868b47d7-xkkds"
Sep 18 20:04:25.684977 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:04:25.684391    2585 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w3-57b6ce/auth-test-model-57b6ce-kserve-66868b47d7-xkkds"
Sep 18 20:04:25.696512 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:04:25.696490    2585 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w3-57b6ce/auth-test-model-57b6ce-kserve-66868b47d7-xkkds"
Sep 18 20:04:26.608011 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:04:26.607978    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w3-57b6ce/auth-test-model-57b6ce-kserve-66868b47d7-xkkds"
Sep 18 20:04:43.113364 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:04:43.113282    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-xmwh7"]
Sep 18 20:04:48.475955 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:04:48.475917    2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-d94b9e/embedding-w0-d94b9e-kserve-6f556c7b9-5n4l2"]
Sep 18 20:04:48.479738 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:04:48.479719    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-d94b9e/embedding-w0-d94b9e-kserve-6f556c7b9-5n4l2"
Sep 18 20:04:48.482016 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:04:48.481993    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-d94b9e\"/\"embedding-w0-d94b9e-kserve-self-signed-certs\""
Sep 18 20:04:48.487439 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:04:48.487415    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-d94b9e/embedding-w0-d94b9e-kserve-6f556c7b9-5n4l2"]
Sep 18 20:04:48.623358 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:04:48.623311    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/631d6c09-0992-4283-b9ba-ae8c39be8dd1-tmp-dir\") pod \"embedding-w0-d94b9e-kserve-6f556c7b9-5n4l2\" (UID: \"631d6c09-0992-4283-b9ba-ae8c39be8dd1\") " pod="e2e-models-e2e-worker-w0-d94b9e/embedding-w0-d94b9e-kserve-6f556c7b9-5n4l2"
Sep 18 20:04:48.623547 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:04:48.623414    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/631d6c09-0992-4283-b9ba-ae8c39be8dd1-dshm\") pod \"embedding-w0-d94b9e-kserve-6f556c7b9-5n4l2\" (UID: \"631d6c09-0992-4283-b9ba-ae8c39be8dd1\") " pod="e2e-models-e2e-worker-w0-d94b9e/embedding-w0-d94b9e-kserve-6f556c7b9-5n4l2"
Sep 18 20:04:48.623547 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:04:48.623485    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6z7h5\" (UniqueName: \"kubernetes.io/projected/631d6c09-0992-4283-b9ba-ae8c39be8dd1-kube-api-access-6z7h5\") pod \"embedding-w0-d94b9e-kserve-6f556c7b9-5n4l2\" (UID: \"631d6c09-0992-4283-b9ba-ae8c39be8dd1\") " pod="e2e-models-e2e-worker-w0-d94b9e/embedding-w0-d94b9e-kserve-6f556c7b9-5n4l2"
Sep 18 20:04:48.623547 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:04:48.623543    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/631d6c09-0992-4283-b9ba-ae8c39be8dd1-home\") pod \"embedding-w0-d94b9e-kserve-6f556c7b9-5n4l2\" (UID: \"631d6c09-0992-4283-b9ba-ae8c39be8dd1\") " pod="e2e-models-e2e-worker-w0-d94b9e/embedding-w0-d94b9e-kserve-6f556c7b9-5n4l2"
Sep 18 20:04:48.623690 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:04:48.623563    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/631d6c09-0992-4283-b9ba-ae8c39be8dd1-model-cache\") pod \"embedding-w0-d94b9e-kserve-6f556c7b9-5n4l2\" (UID: \"631d6c09-0992-4283-b9ba-ae8c39be8dd1\") " pod="e2e-models-e2e-worker-w0-d94b9e/embedding-w0-d94b9e-kserve-6f556c7b9-5n4l2"
Sep 18 20:04:48.623690 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:04:48.623607    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/631d6c09-0992-4283-b9ba-ae8c39be8dd1-tls-certs\") pod \"embedding-w0-d94b9e-kserve-6f556c7b9-5n4l2\" (UID: \"631d6c09-0992-4283-b9ba-ae8c39be8dd1\") " pod="e2e-models-e2e-worker-w0-d94b9e/embedding-w0-d94b9e-kserve-6f556c7b9-5n4l2"
Sep 18 20:04:48.724421 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:04:48.724381    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6z7h5\" (UniqueName: \"kubernetes.io/projected/631d6c09-0992-4283-b9ba-ae8c39be8dd1-kube-api-access-6z7h5\") pod \"embedding-w0-d94b9e-kserve-6f556c7b9-5n4l2\" (UID: \"631d6c09-0992-4283-b9ba-ae8c39be8dd1\") " pod="e2e-models-e2e-worker-w0-d94b9e/embedding-w0-d94b9e-kserve-6f556c7b9-5n4l2"
Sep 18 20:04:48.724635 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:04:48.724441    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/631d6c09-0992-4283-b9ba-ae8c39be8dd1-home\") pod \"embedding-w0-d94b9e-kserve-6f556c7b9-5n4l2\" (UID: \"631d6c09-0992-4283-b9ba-ae8c39be8dd1\") " pod="e2e-models-e2e-worker-w0-d94b9e/embedding-w0-d94b9e-kserve-6f556c7b9-5n4l2"
Sep 18 20:04:48.724635 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:04:48.724464    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/631d6c09-0992-4283-b9ba-ae8c39be8dd1-model-cache\") pod \"embedding-w0-d94b9e-kserve-6f556c7b9-5n4l2\" (UID: \"631d6c09-0992-4283-b9ba-ae8c39be8dd1\") " pod="e2e-models-e2e-worker-w0-d94b9e/embedding-w0-d94b9e-kserve-6f556c7b9-5n4l2"
Sep 18 20:04:48.724635 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:04:48.724501    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/631d6c09-0992-4283-b9ba-ae8c39be8dd1-tls-certs\") pod \"embedding-w0-d94b9e-kserve-6f556c7b9-5n4l2\" (UID: \"631d6c09-0992-4283-b9ba-ae8c39be8dd1\") " pod="e2e-models-e2e-worker-w0-d94b9e/embedding-w0-d94b9e-kserve-6f556c7b9-5n4l2"
Sep 18 20:04:48.724635 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:04:48.724564    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/631d6c09-0992-4283-b9ba-ae8c39be8dd1-tmp-dir\") pod \"embedding-w0-d94b9e-kserve-6f556c7b9-5n4l2\" (UID: \"631d6c09-0992-4283-b9ba-ae8c39be8dd1\") " pod="e2e-models-e2e-worker-w0-d94b9e/embedding-w0-d94b9e-kserve-6f556c7b9-5n4l2"
Sep 18 20:04:48.724895 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:04:48.724637    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/631d6c09-0992-4283-b9ba-ae8c39be8dd1-dshm\") pod \"embedding-w0-d94b9e-kserve-6f556c7b9-5n4l2\" (UID: \"631d6c09-0992-4283-b9ba-ae8c39be8dd1\") " pod="e2e-models-e2e-worker-w0-d94b9e/embedding-w0-d94b9e-kserve-6f556c7b9-5n4l2"
Sep 18 20:04:48.724956 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:04:48.724890    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/631d6c09-0992-4283-b9ba-ae8c39be8dd1-home\") pod \"embedding-w0-d94b9e-kserve-6f556c7b9-5n4l2\" (UID: \"631d6c09-0992-4283-b9ba-ae8c39be8dd1\") " pod="e2e-models-e2e-worker-w0-d94b9e/embedding-w0-d94b9e-kserve-6f556c7b9-5n4l2"
Sep 18 20:04:48.724956 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:04:48.724935    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/631d6c09-0992-4283-b9ba-ae8c39be8dd1-model-cache\") pod \"embedding-w0-d94b9e-kserve-6f556c7b9-5n4l2\" (UID: \"631d6c09-0992-4283-b9ba-ae8c39be8dd1\") " pod="e2e-models-e2e-worker-w0-d94b9e/embedding-w0-d94b9e-kserve-6f556c7b9-5n4l2"
Sep 18 20:04:48.725061 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:04:48.724953    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/631d6c09-0992-4283-b9ba-ae8c39be8dd1-tmp-dir\") pod \"embedding-w0-d94b9e-kserve-6f556c7b9-5n4l2\" (UID: \"631d6c09-0992-4283-b9ba-ae8c39be8dd1\") " pod="e2e-models-e2e-worker-w0-d94b9e/embedding-w0-d94b9e-kserve-6f556c7b9-5n4l2"
Sep 18 20:04:48.726959 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:04:48.726896    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/631d6c09-0992-4283-b9ba-ae8c39be8dd1-dshm\") pod \"embedding-w0-d94b9e-kserve-6f556c7b9-5n4l2\" (UID: \"631d6c09-0992-4283-b9ba-ae8c39be8dd1\") " pod="e2e-models-e2e-worker-w0-d94b9e/embedding-w0-d94b9e-kserve-6f556c7b9-5n4l2"
Sep 18 20:04:48.727509 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:04:48.727487    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/631d6c09-0992-4283-b9ba-ae8c39be8dd1-tls-certs\") pod \"embedding-w0-d94b9e-kserve-6f556c7b9-5n4l2\" (UID: \"631d6c09-0992-4283-b9ba-ae8c39be8dd1\") " pod="e2e-models-e2e-worker-w0-d94b9e/embedding-w0-d94b9e-kserve-6f556c7b9-5n4l2"
Sep 18 20:04:48.732854 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:04:48.732833    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6z7h5\" (UniqueName: \"kubernetes.io/projected/631d6c09-0992-4283-b9ba-ae8c39be8dd1-kube-api-access-6z7h5\") pod \"embedding-w0-d94b9e-kserve-6f556c7b9-5n4l2\" (UID: \"631d6c09-0992-4283-b9ba-ae8c39be8dd1\") " pod="e2e-models-e2e-worker-w0-d94b9e/embedding-w0-d94b9e-kserve-6f556c7b9-5n4l2"
Sep 18 20:04:48.791012 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:04:48.790976    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-d94b9e/embedding-w0-d94b9e-kserve-6f556c7b9-5n4l2"
Sep 18 20:04:49.064675 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:04:49.064649    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-d94b9e/embedding-w0-d94b9e-kserve-6f556c7b9-5n4l2"]
Sep 18 20:04:49.066217 ip-10-0-136-185 kubenswrapper[2585]: W0918 20:04:49.066189    2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod631d6c09_0992_4283_b9ba_ae8c39be8dd1.slice/crio-551ecf2eb07d51712cfbc1f11d358e1d8439a7fbbdefaefd87ca6faa08664e51 WatchSource:0}: Error finding container 551ecf2eb07d51712cfbc1f11d358e1d8439a7fbbdefaefd87ca6faa08664e51: Status 404 returned error can't find the container with id 551ecf2eb07d51712cfbc1f11d358e1d8439a7fbbdefaefd87ca6faa08664e51
Sep 18 20:04:49.717591 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:04:49.717548    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-d94b9e/embedding-w0-d94b9e-kserve-6f556c7b9-5n4l2" event={"ID":"631d6c09-0992-4283-b9ba-ae8c39be8dd1","Type":"ContainerStarted","Data":"99d5d234ceb9b0087a9ae4eeab3297e89fcb65acaf6a9234d68944d9e4375a97"}
Sep 18 20:04:49.717591 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:04:49.717598    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-d94b9e/embedding-w0-d94b9e-kserve-6f556c7b9-5n4l2" event={"ID":"631d6c09-0992-4283-b9ba-ae8c39be8dd1","Type":"ContainerStarted","Data":"551ecf2eb07d51712cfbc1f11d358e1d8439a7fbbdefaefd87ca6faa08664e51"}
Sep 18 20:04:49.734584 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:04:49.734189    2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-d94b9e/embedding-w0-d94b9e-kserve-6f556c7b9-5n4l2" podStartSLOduration=1.734171392 podStartE2EDuration="1.734171392s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 20:04:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 20:04:49.73337585 +0000 UTC m=+1531.257981285" watchObservedRunningTime="2026-09-18 20:04:49.734171392 +0000 UTC m=+1531.258776816"
Sep 18 20:04:58.791356 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:04:58.791316    2585 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-d94b9e/embedding-w0-d94b9e-kserve-6f556c7b9-5n4l2"
Sep 18 20:04:58.791356 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:04:58.791361    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-d94b9e/embedding-w0-d94b9e-kserve-6f556c7b9-5n4l2"
Sep 18 20:04:58.804201 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:04:58.804170    2585 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-d94b9e/embedding-w0-d94b9e-kserve-6f556c7b9-5n4l2"
Sep 18 20:04:59.783350 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:04:59.783322    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-d94b9e/embedding-w0-d94b9e-kserve-6f556c7b9-5n4l2"
Sep 18 20:05:25.283637 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:05:25.283599    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-xmwh7"]
Sep 18 20:05:35.668733 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:05:35.668688    2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-c3d85f-85c49fff7c-657d8"]
Sep 18 20:05:35.673345 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:05:35.673303    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-c3d85f-85c49fff7c-657d8"
Sep 18 20:05:35.675639 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:05:35.675611    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-a-c3d85f-dockercfg-fkzb6\""
Sep 18 20:05:35.675639 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:05:35.675634    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-a-c3d85f\""
Sep 18 20:05:35.683645 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:05:35.683619    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-c3d85f-85c49fff7c-657d8"]
Sep 18 20:05:35.812136 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:05:35.812100    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mxm9t\" (UniqueName: \"kubernetes.io/projected/ed61d470-33dd-4ed3-a0da-835e282231c2-kube-api-access-mxm9t\") pod \"payload-processing-e2e-api-a-c3d85f-85c49fff7c-657d8\" (UID: \"ed61d470-33dd-4ed3-a0da-835e282231c2\") " pod="openshift-ingress/payload-processing-e2e-api-a-c3d85f-85c49fff7c-657d8"
Sep 18 20:05:35.812136 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:05:35.812144    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ed61d470-33dd-4ed3-a0da-835e282231c2-plugins-config-volume\") pod \"payload-processing-e2e-api-a-c3d85f-85c49fff7c-657d8\" (UID: \"ed61d470-33dd-4ed3-a0da-835e282231c2\") " pod="openshift-ingress/payload-processing-e2e-api-a-c3d85f-85c49fff7c-657d8"
Sep 18 20:05:35.913630 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:05:35.913589    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mxm9t\" (UniqueName: \"kubernetes.io/projected/ed61d470-33dd-4ed3-a0da-835e282231c2-kube-api-access-mxm9t\") pod \"payload-processing-e2e-api-a-c3d85f-85c49fff7c-657d8\" (UID: \"ed61d470-33dd-4ed3-a0da-835e282231c2\") " pod="openshift-ingress/payload-processing-e2e-api-a-c3d85f-85c49fff7c-657d8"
Sep 18 20:05:35.913630 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:05:35.913631    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ed61d470-33dd-4ed3-a0da-835e282231c2-plugins-config-volume\") pod \"payload-processing-e2e-api-a-c3d85f-85c49fff7c-657d8\" (UID: \"ed61d470-33dd-4ed3-a0da-835e282231c2\") " pod="openshift-ingress/payload-processing-e2e-api-a-c3d85f-85c49fff7c-657d8"
Sep 18 20:05:35.914379 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:05:35.914360    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ed61d470-33dd-4ed3-a0da-835e282231c2-plugins-config-volume\") pod \"payload-processing-e2e-api-a-c3d85f-85c49fff7c-657d8\" (UID: \"ed61d470-33dd-4ed3-a0da-835e282231c2\") " pod="openshift-ingress/payload-processing-e2e-api-a-c3d85f-85c49fff7c-657d8"
Sep 18 20:05:35.921688 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:05:35.921626    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mxm9t\" (UniqueName: \"kubernetes.io/projected/ed61d470-33dd-4ed3-a0da-835e282231c2-kube-api-access-mxm9t\") pod \"payload-processing-e2e-api-a-c3d85f-85c49fff7c-657d8\" (UID: \"ed61d470-33dd-4ed3-a0da-835e282231c2\") " pod="openshift-ingress/payload-processing-e2e-api-a-c3d85f-85c49fff7c-657d8"
Sep 18 20:05:35.986108 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:05:35.986077    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-c3d85f-85c49fff7c-657d8"
Sep 18 20:05:36.257133 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:05:36.257106    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-c3d85f-85c49fff7c-657d8"]
Sep 18 20:05:36.259007 ip-10-0-136-185 kubenswrapper[2585]: W0918 20:05:36.258974    2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poded61d470_33dd_4ed3_a0da_835e282231c2.slice/crio-4f76ecad4e26f0333cc1ff9e4fc479912d16078a72159d63222a9af4bf2befb5 WatchSource:0}: Error finding container 4f76ecad4e26f0333cc1ff9e4fc479912d16078a72159d63222a9af4bf2befb5: Status 404 returned error can't find the container with id 4f76ecad4e26f0333cc1ff9e4fc479912d16078a72159d63222a9af4bf2befb5
Sep 18 20:05:36.966899 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:05:36.966864    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-c3d85f-85c49fff7c-657d8" event={"ID":"ed61d470-33dd-4ed3-a0da-835e282231c2","Type":"ContainerStarted","Data":"d375b331c3ed63f10f9e912431e03e3d251ca32f0d7ab0aa18e07fdfe3e53a1c"}
Sep 18 20:05:36.967318 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:05:36.966915    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-c3d85f-85c49fff7c-657d8" event={"ID":"ed61d470-33dd-4ed3-a0da-835e282231c2","Type":"ContainerStarted","Data":"4f76ecad4e26f0333cc1ff9e4fc479912d16078a72159d63222a9af4bf2befb5"}
Sep 18 20:05:36.967318 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:05:36.966947    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-a-c3d85f-85c49fff7c-657d8"
Sep 18 20:05:36.983280 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:05:36.983221    2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-a-c3d85f-85c49fff7c-657d8" podStartSLOduration=1.983179545 podStartE2EDuration="1.983179545s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 20:05:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 20:05:36.982516026 +0000 UTC m=+1578.507121450" watchObservedRunningTime="2026-09-18 20:05:36.983179545 +0000 UTC m=+1578.507784969"
Sep 18 20:05:47.975001 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:05:47.974958    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-a-c3d85f-85c49fff7c-657d8"
Sep 18 20:06:06.587884 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:06:06.587846    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-xmwh7"]
Sep 18 20:06:10.313327 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:06:10.313292    2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-13f9e5-6f594f7fcd-8t98g"]
Sep 18 20:06:10.317432 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:06:10.317408    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-13f9e5-6f594f7fcd-8t98g"
Sep 18 20:06:10.319629 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:06:10.319608    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-b-13f9e5\""
Sep 18 20:06:10.319782 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:06:10.319611    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-b-13f9e5-dockercfg-xkfb9\""
Sep 18 20:06:10.326432 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:06:10.326409    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-13f9e5-6f594f7fcd-8t98g"]
Sep 18 20:06:10.461653 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:06:10.461618    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/47694d12-7008-4989-afc8-f128484ab6c5-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-13f9e5-6f594f7fcd-8t98g\" (UID: \"47694d12-7008-4989-afc8-f128484ab6c5\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-13f9e5-6f594f7fcd-8t98g"
Sep 18 20:06:10.461653 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:06:10.461663    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-95zbw\" (UniqueName: \"kubernetes.io/projected/47694d12-7008-4989-afc8-f128484ab6c5-kube-api-access-95zbw\") pod \"payload-pre-processing-e2e-api-b-13f9e5-6f594f7fcd-8t98g\" (UID: \"47694d12-7008-4989-afc8-f128484ab6c5\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-13f9e5-6f594f7fcd-8t98g"
Sep 18 20:06:10.563203 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:06:10.563154    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/47694d12-7008-4989-afc8-f128484ab6c5-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-13f9e5-6f594f7fcd-8t98g\" (UID: \"47694d12-7008-4989-afc8-f128484ab6c5\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-13f9e5-6f594f7fcd-8t98g"
Sep 18 20:06:10.563203 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:06:10.563206    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-95zbw\" (UniqueName: \"kubernetes.io/projected/47694d12-7008-4989-afc8-f128484ab6c5-kube-api-access-95zbw\") pod \"payload-pre-processing-e2e-api-b-13f9e5-6f594f7fcd-8t98g\" (UID: \"47694d12-7008-4989-afc8-f128484ab6c5\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-13f9e5-6f594f7fcd-8t98g"
Sep 18 20:06:10.563950 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:06:10.563902    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/47694d12-7008-4989-afc8-f128484ab6c5-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-13f9e5-6f594f7fcd-8t98g\" (UID: \"47694d12-7008-4989-afc8-f128484ab6c5\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-13f9e5-6f594f7fcd-8t98g"
Sep 18 20:06:10.570864 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:06:10.570843    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-95zbw\" (UniqueName: \"kubernetes.io/projected/47694d12-7008-4989-afc8-f128484ab6c5-kube-api-access-95zbw\") pod \"payload-pre-processing-e2e-api-b-13f9e5-6f594f7fcd-8t98g\" (UID: \"47694d12-7008-4989-afc8-f128484ab6c5\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-13f9e5-6f594f7fcd-8t98g"
Sep 18 20:06:10.629707 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:06:10.629662    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-13f9e5-6f594f7fcd-8t98g"
Sep 18 20:06:10.903388 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:06:10.903365    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-13f9e5-6f594f7fcd-8t98g"]
Sep 18 20:06:10.905457 ip-10-0-136-185 kubenswrapper[2585]: W0918 20:06:10.905428    2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod47694d12_7008_4989_afc8_f128484ab6c5.slice/crio-f1a0f590d587dfece870a03e9b0702749daa3a868e70d650790195c1770e824c WatchSource:0}: Error finding container f1a0f590d587dfece870a03e9b0702749daa3a868e70d650790195c1770e824c: Status 404 returned error can't find the container with id f1a0f590d587dfece870a03e9b0702749daa3a868e70d650790195c1770e824c
Sep 18 20:06:11.150492 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:06:11.150454    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-13f9e5-6f594f7fcd-8t98g" event={"ID":"47694d12-7008-4989-afc8-f128484ab6c5","Type":"ContainerStarted","Data":"0185a04104a0306a07de4fce1789772b53bc004f687f31d6a23daa835c15dc92"}
Sep 18 20:06:11.150677 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:06:11.150508    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-13f9e5-6f594f7fcd-8t98g" event={"ID":"47694d12-7008-4989-afc8-f128484ab6c5","Type":"ContainerStarted","Data":"f1a0f590d587dfece870a03e9b0702749daa3a868e70d650790195c1770e824c"}
Sep 18 20:06:11.150677 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:06:11.150596    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-13f9e5-6f594f7fcd-8t98g"
Sep 18 20:06:11.167905 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:06:11.167819    2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-b-13f9e5-6f594f7fcd-8t98g" podStartSLOduration=1.167778086 podStartE2EDuration="1.167778086s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 20:06:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 20:06:11.166039775 +0000 UTC m=+1612.690645198" watchObservedRunningTime="2026-09-18 20:06:11.167778086 +0000 UTC m=+1612.692383535"
Sep 18 20:06:20.679481 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:06:20.679443    2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-d94b9e/distinct-w0-d94b9e-kserve-654c9b789b-w7qlc"]
Sep 18 20:06:20.699852 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:06:20.699816    2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-d94b9e/embedding-w0-d94b9e-kserve-6f556c7b9-5n4l2"]
Sep 18 20:06:20.729716 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:06:20.729684    2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-d94b9e/simulator-w0-d94b9e-kserve-7669969b8-6p4m8"]
Sep 18 20:06:20.785769 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:06:20.785740    2585 scope.go:117] "RemoveContainer" containerID="c52b8d643e3e8e95ed9460f2405c0ae900046e78b42603cf639c65d0b4dad206"
Sep 18 20:06:22.158338 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:06:22.158305    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-13f9e5-6f594f7fcd-8t98g"
Sep 18 20:06:26.266966 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:06:26.266929    2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-c3d85f-tpctg"]
Sep 18 20:06:26.272216 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:06:26.272198    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-c3d85f-tpctg"
Sep 18 20:06:26.274404 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:06:26.274386    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 18 20:06:26.274637 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:06:26.274623    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-5ttnn\""
Sep 18 20:06:26.278773 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:06:26.278746    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-c3d85f-tpctg"]
Sep 18 20:06:26.317352 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:06:26.317315    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6bb618be-f82b-414c-9823-88a7f208fcdb-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-a-c3d85f-tpctg\" (UID: \"6bb618be-f82b-414c-9823-88a7f208fcdb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-c3d85f-tpctg"
Sep 18 20:06:26.418830 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:06:26.418783    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6bb618be-f82b-414c-9823-88a7f208fcdb-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-a-c3d85f-tpctg\" (UID: \"6bb618be-f82b-414c-9823-88a7f208fcdb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-c3d85f-tpctg"
Sep 18 20:06:26.419361 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:06:26.419341    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6bb618be-f82b-414c-9823-88a7f208fcdb-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-a-c3d85f-tpctg\" (UID: \"6bb618be-f82b-414c-9823-88a7f208fcdb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-c3d85f-tpctg"
Sep 18 20:06:26.584286 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:06:26.584190    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-c3d85f-tpctg"
Sep 18 20:06:26.855589 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:06:26.855566    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-c3d85f-tpctg"]
Sep 18 20:06:27.242702 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:06:27.242662    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-c3d85f-tpctg" event={"ID":"6bb618be-f82b-414c-9823-88a7f208fcdb","Type":"ContainerStarted","Data":"5ff5ef87e21db1b4143913570ffcd6b3e9fea319ddb45025e7560cf54c3d9644"}
Sep 18 20:06:27.242897 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:06:27.242741    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-c3d85f-tpctg" event={"ID":"6bb618be-f82b-414c-9823-88a7f208fcdb","Type":"ContainerStarted","Data":"11061ed5afe82ac13a006042ebaeadb0ef2a4abe48fe341e278946b1391ced42"}
Sep 18 20:06:27.259543 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:06:27.259484    2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-c3d85f-tpctg" podStartSLOduration=1.2594672949999999 podStartE2EDuration="1.259467295s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 20:06:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 20:06:27.257608914 +0000 UTC m=+1628.782214336" watchObservedRunningTime="2026-09-18 20:06:27.259467295 +0000 UTC m=+1628.784072718"
Sep 18 20:06:28.249972 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:06:28.249937    2585 generic.go:356] "Generic (PLEG): container finished" podID="6bb618be-f82b-414c-9823-88a7f208fcdb" containerID="5ff5ef87e21db1b4143913570ffcd6b3e9fea319ddb45025e7560cf54c3d9644" exitCode=0
Sep 18 20:06:28.250363 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:06:28.250026    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-c3d85f-tpctg" event={"ID":"6bb618be-f82b-414c-9823-88a7f208fcdb","Type":"ContainerDied","Data":"5ff5ef87e21db1b4143913570ffcd6b3e9fea319ddb45025e7560cf54c3d9644"}
Sep 18 20:06:29.389906 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:06:29.389880    2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-c3d85f-tpctg"
Sep 18 20:06:29.452618 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:06:29.452580    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6bb618be-f82b-414c-9823-88a7f208fcdb-maas-api-service-ca\") pod \"6bb618be-f82b-414c-9823-88a7f208fcdb\" (UID: \"6bb618be-f82b-414c-9823-88a7f208fcdb\") "
Sep 18 20:06:29.452992 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:06:29.452969    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6bb618be-f82b-414c-9823-88a7f208fcdb-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "6bb618be-f82b-414c-9823-88a7f208fcdb" (UID: "6bb618be-f82b-414c-9823-88a7f208fcdb"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 20:06:29.553994 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:06:29.553906    2585 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6bb618be-f82b-414c-9823-88a7f208fcdb-maas-api-service-ca\") on node \"ip-10-0-136-185.ec2.internal\" DevicePath \"\""
Sep 18 20:06:30.261543 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:06:30.261516    2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-c3d85f-tpctg"
Sep 18 20:06:30.261721 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:06:30.261550    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-c3d85f-tpctg" event={"ID":"6bb618be-f82b-414c-9823-88a7f208fcdb","Type":"ContainerDied","Data":"11061ed5afe82ac13a006042ebaeadb0ef2a4abe48fe341e278946b1391ced42"}
Sep 18 20:06:30.261721 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:06:30.261595    2585 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="11061ed5afe82ac13a006042ebaeadb0ef2a4abe48fe341e278946b1391ced42"
Sep 18 20:06:34.694364 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:06:34.694322    2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-d94b9e-29829360-psf6f"]
Sep 18 20:06:34.700743 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:06:34.700709    2585 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-d94b9e-29829360-psf6f"]
Sep 18 20:06:34.724961 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:06:34.724925    2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-d94b9e-cd84d77-hd88x"]
Sep 18 20:06:34.725199 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:06:34.725172    2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w0-d94b9e-cd84d77-hd88x" podUID="661ede6f-264a-4fbe-9037-9c61a63697e4" containerName="payload-processing" containerID="cri-o://5abbfc123c29f3e80c8336cc8a9fd2e87b926c1521e41736062608431b5c3718" gracePeriod=30
Sep 18 20:06:34.740466 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:06:34.740432    2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-d94b9e-57d4f777b6-tslhj"]
Sep 18 20:06:34.740697 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:06:34.740675    2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-d94b9e-57d4f777b6-tslhj" podUID="c67b483b-0a36-4990-97a3-3227c41fcca4" containerName="payload-pre-processing" containerID="cri-o://bf0d0c10c5b90a71048153959d776c338ee5ca4f08951b0f8808a70a9a925383" gracePeriod=30
Sep 18 20:06:34.987891 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:06:34.987863    2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-d94b9e-cd84d77-hd88x"
Sep 18 20:06:35.017398 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:06:35.017372    2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-d94b9e-57d4f777b6-tslhj"
Sep 18 20:06:35.108101 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:06:35.108056    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/661ede6f-264a-4fbe-9037-9c61a63697e4-plugins-config-volume\") pod \"661ede6f-264a-4fbe-9037-9c61a63697e4\" (UID: \"661ede6f-264a-4fbe-9037-9c61a63697e4\") "
Sep 18 20:06:35.108290 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:06:35.108117    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-d7k4q\" (UniqueName: \"kubernetes.io/projected/c67b483b-0a36-4990-97a3-3227c41fcca4-kube-api-access-d7k4q\") pod \"c67b483b-0a36-4990-97a3-3227c41fcca4\" (UID: \"c67b483b-0a36-4990-97a3-3227c41fcca4\") "
Sep 18 20:06:35.108290 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:06:35.108237    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4pjkz\" (UniqueName: \"kubernetes.io/projected/661ede6f-264a-4fbe-9037-9c61a63697e4-kube-api-access-4pjkz\") pod \"661ede6f-264a-4fbe-9037-9c61a63697e4\" (UID: \"661ede6f-264a-4fbe-9037-9c61a63697e4\") "
Sep 18 20:06:35.108421 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:06:35.108308    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c67b483b-0a36-4990-97a3-3227c41fcca4-plugins-config-volume\") pod \"c67b483b-0a36-4990-97a3-3227c41fcca4\" (UID: \"c67b483b-0a36-4990-97a3-3227c41fcca4\") "
Sep 18 20:06:35.108520 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:06:35.108477    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/661ede6f-264a-4fbe-9037-9c61a63697e4-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "661ede6f-264a-4fbe-9037-9c61a63697e4" (UID: "661ede6f-264a-4fbe-9037-9c61a63697e4"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 20:06:35.108679 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:06:35.108661    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c67b483b-0a36-4990-97a3-3227c41fcca4-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "c67b483b-0a36-4990-97a3-3227c41fcca4" (UID: "c67b483b-0a36-4990-97a3-3227c41fcca4"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 20:06:35.108743 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:06:35.108716    2585 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/661ede6f-264a-4fbe-9037-9c61a63697e4-plugins-config-volume\") on node \"ip-10-0-136-185.ec2.internal\" DevicePath \"\""
Sep 18 20:06:35.110326 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:06:35.110302    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/661ede6f-264a-4fbe-9037-9c61a63697e4-kube-api-access-4pjkz" (OuterVolumeSpecName: "kube-api-access-4pjkz") pod "661ede6f-264a-4fbe-9037-9c61a63697e4" (UID: "661ede6f-264a-4fbe-9037-9c61a63697e4"). InnerVolumeSpecName "kube-api-access-4pjkz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 20:06:35.110430 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:06:35.110332    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c67b483b-0a36-4990-97a3-3227c41fcca4-kube-api-access-d7k4q" (OuterVolumeSpecName: "kube-api-access-d7k4q") pod "c67b483b-0a36-4990-97a3-3227c41fcca4" (UID: "c67b483b-0a36-4990-97a3-3227c41fcca4"). InnerVolumeSpecName "kube-api-access-d7k4q". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 20:06:35.209846 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:06:35.209806    2585 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c67b483b-0a36-4990-97a3-3227c41fcca4-plugins-config-volume\") on node \"ip-10-0-136-185.ec2.internal\" DevicePath \"\""
Sep 18 20:06:35.209846 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:06:35.209842    2585 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d7k4q\" (UniqueName: \"kubernetes.io/projected/c67b483b-0a36-4990-97a3-3227c41fcca4-kube-api-access-d7k4q\") on node \"ip-10-0-136-185.ec2.internal\" DevicePath \"\""
Sep 18 20:06:35.209846 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:06:35.209854    2585 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4pjkz\" (UniqueName: \"kubernetes.io/projected/661ede6f-264a-4fbe-9037-9c61a63697e4-kube-api-access-4pjkz\") on node \"ip-10-0-136-185.ec2.internal\" DevicePath \"\""
Sep 18 20:06:35.289921 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:06:35.289837    2585 generic.go:356] "Generic (PLEG): container finished" podID="c67b483b-0a36-4990-97a3-3227c41fcca4" containerID="bf0d0c10c5b90a71048153959d776c338ee5ca4f08951b0f8808a70a9a925383" exitCode=0
Sep 18 20:06:35.289921 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:06:35.289885    2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-d94b9e-57d4f777b6-tslhj"
Sep 18 20:06:35.290133 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:06:35.289916    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-d94b9e-57d4f777b6-tslhj" event={"ID":"c67b483b-0a36-4990-97a3-3227c41fcca4","Type":"ContainerDied","Data":"bf0d0c10c5b90a71048153959d776c338ee5ca4f08951b0f8808a70a9a925383"}
Sep 18 20:06:35.290133 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:06:35.289965    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-d94b9e-57d4f777b6-tslhj" event={"ID":"c67b483b-0a36-4990-97a3-3227c41fcca4","Type":"ContainerDied","Data":"9b8e07b76e2f4a4262590adde7317721b53937368d86fa66270346d61a5c625f"}
Sep 18 20:06:35.290133 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:06:35.289988    2585 scope.go:117] "RemoveContainer" containerID="bf0d0c10c5b90a71048153959d776c338ee5ca4f08951b0f8808a70a9a925383"
Sep 18 20:06:35.291465 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:06:35.291432    2585 generic.go:356] "Generic (PLEG): container finished" podID="661ede6f-264a-4fbe-9037-9c61a63697e4" containerID="5abbfc123c29f3e80c8336cc8a9fd2e87b926c1521e41736062608431b5c3718" exitCode=0
Sep 18 20:06:35.291590 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:06:35.291477    2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-d94b9e-cd84d77-hd88x"
Sep 18 20:06:35.291590 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:06:35.291508    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-d94b9e-cd84d77-hd88x" event={"ID":"661ede6f-264a-4fbe-9037-9c61a63697e4","Type":"ContainerDied","Data":"5abbfc123c29f3e80c8336cc8a9fd2e87b926c1521e41736062608431b5c3718"}
Sep 18 20:06:35.291590 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:06:35.291564    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-d94b9e-cd84d77-hd88x" event={"ID":"661ede6f-264a-4fbe-9037-9c61a63697e4","Type":"ContainerDied","Data":"e192f95c7e34f6803c72a059fbac01b8aa86dabada7e415e1d45e3304ff7ae7f"}
Sep 18 20:06:35.300663 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:06:35.300643    2585 scope.go:117] "RemoveContainer" containerID="bf0d0c10c5b90a71048153959d776c338ee5ca4f08951b0f8808a70a9a925383"
Sep 18 20:06:35.301035 ip-10-0-136-185 kubenswrapper[2585]: E0918 20:06:35.301013    2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bf0d0c10c5b90a71048153959d776c338ee5ca4f08951b0f8808a70a9a925383\": container with ID starting with bf0d0c10c5b90a71048153959d776c338ee5ca4f08951b0f8808a70a9a925383 not found: ID does not exist" containerID="bf0d0c10c5b90a71048153959d776c338ee5ca4f08951b0f8808a70a9a925383"
Sep 18 20:06:35.301129 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:06:35.301048    2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bf0d0c10c5b90a71048153959d776c338ee5ca4f08951b0f8808a70a9a925383"} err="failed to get container status \"bf0d0c10c5b90a71048153959d776c338ee5ca4f08951b0f8808a70a9a925383\": rpc error: code = NotFound desc = could not find container \"bf0d0c10c5b90a71048153959d776c338ee5ca4f08951b0f8808a70a9a925383\": container with ID starting with bf0d0c10c5b90a71048153959d776c338ee5ca4f08951b0f8808a70a9a925383 not found: ID does not exist"
Sep 18 20:06:35.301129 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:06:35.301074    2585 scope.go:117] "RemoveContainer" containerID="5abbfc123c29f3e80c8336cc8a9fd2e87b926c1521e41736062608431b5c3718"
Sep 18 20:06:35.309933 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:06:35.309914    2585 scope.go:117] "RemoveContainer" containerID="5abbfc123c29f3e80c8336cc8a9fd2e87b926c1521e41736062608431b5c3718"
Sep 18 20:06:35.310233 ip-10-0-136-185 kubenswrapper[2585]: E0918 20:06:35.310209    2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5abbfc123c29f3e80c8336cc8a9fd2e87b926c1521e41736062608431b5c3718\": container with ID starting with 5abbfc123c29f3e80c8336cc8a9fd2e87b926c1521e41736062608431b5c3718 not found: ID does not exist" containerID="5abbfc123c29f3e80c8336cc8a9fd2e87b926c1521e41736062608431b5c3718"
Sep 18 20:06:35.310285 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:06:35.310244    2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5abbfc123c29f3e80c8336cc8a9fd2e87b926c1521e41736062608431b5c3718"} err="failed to get container status \"5abbfc123c29f3e80c8336cc8a9fd2e87b926c1521e41736062608431b5c3718\": rpc error: code = NotFound desc = could not find container \"5abbfc123c29f3e80c8336cc8a9fd2e87b926c1521e41736062608431b5c3718\": container with ID starting with 5abbfc123c29f3e80c8336cc8a9fd2e87b926c1521e41736062608431b5c3718 not found: ID does not exist"
Sep 18 20:06:35.317498 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:06:35.317469    2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-d94b9e-cd84d77-hd88x"]
Sep 18 20:06:35.321843 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:06:35.321806    2585 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-d94b9e-cd84d77-hd88x"]
Sep 18 20:06:35.332115 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:06:35.332083    2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-d94b9e-57d4f777b6-tslhj"]
Sep 18 20:06:35.336284 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:06:35.336255    2585 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-d94b9e-57d4f777b6-tslhj"]
Sep 18 20:06:35.699019 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:06:35.698982    2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-d94b9e/distinct-w0-d94b9e-kserve-654c9b789b-w7qlc" podUID="706cd513-2d52-43a6-aeed-4382e477fea2" containerName="main" containerID="cri-o://c1f67ed6ca8e921f7e48b4c63e1f2f6b5c1ff741b933fbbb69a9d53bd0cca91d" gracePeriod=45
Sep 18 20:06:35.719660 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:06:35.719606    2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-d94b9e/embedding-w0-d94b9e-kserve-6f556c7b9-5n4l2" podUID="631d6c09-0992-4283-b9ba-ae8c39be8dd1" containerName="main" containerID="cri-o://99d5d234ceb9b0087a9ae4eeab3297e89fcb65acaf6a9234d68944d9e4375a97" gracePeriod=45
Sep 18 20:06:35.749165 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:06:35.749125    2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-d94b9e/simulator-w0-d94b9e-kserve-7669969b8-6p4m8" podUID="cc71b9f8-e3bb-4704-98f7-0d07a7f6fe9b" containerName="main" containerID="cri-o://ec0eb1f0385975278f3cd3cb69fbf9771ca5b678b8806381a50185b95029d696" gracePeriod=45
Sep 18 20:06:36.040382 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:06:36.040355    2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-d94b9e/distinct-w0-d94b9e-kserve-654c9b789b-w7qlc"
Sep 18 20:06:36.043377 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:06:36.043357    2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-d94b9e/embedding-w0-d94b9e-kserve-6f556c7b9-5n4l2"
Sep 18 20:06:36.046557 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:06:36.046535    2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-d94b9e/simulator-w0-d94b9e-kserve-7669969b8-6p4m8"
Sep 18 20:06:36.117759 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:06:36.117724    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6z7h5\" (UniqueName: \"kubernetes.io/projected/631d6c09-0992-4283-b9ba-ae8c39be8dd1-kube-api-access-6z7h5\") pod \"631d6c09-0992-4283-b9ba-ae8c39be8dd1\" (UID: \"631d6c09-0992-4283-b9ba-ae8c39be8dd1\") "
Sep 18 20:06:36.117934 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:06:36.117843    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/706cd513-2d52-43a6-aeed-4382e477fea2-dshm\") pod \"706cd513-2d52-43a6-aeed-4382e477fea2\" (UID: \"706cd513-2d52-43a6-aeed-4382e477fea2\") "
Sep 18 20:06:36.117934 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:06:36.117879    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/706cd513-2d52-43a6-aeed-4382e477fea2-tls-certs\") pod \"706cd513-2d52-43a6-aeed-4382e477fea2\" (UID: \"706cd513-2d52-43a6-aeed-4382e477fea2\") "
Sep 18 20:06:36.117934 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:06:36.117914    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cc71b9f8-e3bb-4704-98f7-0d07a7f6fe9b-home\") pod \"cc71b9f8-e3bb-4704-98f7-0d07a7f6fe9b\" (UID: \"cc71b9f8-e3bb-4704-98f7-0d07a7f6fe9b\") "
Sep 18 20:06:36.118126 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:06:36.117950    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mwwjm\" (UniqueName: \"kubernetes.io/projected/cc71b9f8-e3bb-4704-98f7-0d07a7f6fe9b-kube-api-access-mwwjm\") pod \"cc71b9f8-e3bb-4704-98f7-0d07a7f6fe9b\" (UID: \"cc71b9f8-e3bb-4704-98f7-0d07a7f6fe9b\") "
Sep 18 20:06:36.118126 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:06:36.117978    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/706cd513-2d52-43a6-aeed-4382e477fea2-model-cache\") pod \"706cd513-2d52-43a6-aeed-4382e477fea2\" (UID: \"706cd513-2d52-43a6-aeed-4382e477fea2\") "
Sep 18 20:06:36.118126 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:06:36.118025    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/706cd513-2d52-43a6-aeed-4382e477fea2-tmp-dir\") pod \"706cd513-2d52-43a6-aeed-4382e477fea2\" (UID: \"706cd513-2d52-43a6-aeed-4382e477fea2\") "
Sep 18 20:06:36.118126 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:06:36.118069    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cc71b9f8-e3bb-4704-98f7-0d07a7f6fe9b-dshm\") pod \"cc71b9f8-e3bb-4704-98f7-0d07a7f6fe9b\" (UID: \"cc71b9f8-e3bb-4704-98f7-0d07a7f6fe9b\") "
Sep 18 20:06:36.118126 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:06:36.118097    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/631d6c09-0992-4283-b9ba-ae8c39be8dd1-model-cache\") pod \"631d6c09-0992-4283-b9ba-ae8c39be8dd1\" (UID: \"631d6c09-0992-4283-b9ba-ae8c39be8dd1\") "
Sep 18 20:06:36.118384 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:06:36.118139    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/631d6c09-0992-4283-b9ba-ae8c39be8dd1-tmp-dir\") pod \"631d6c09-0992-4283-b9ba-ae8c39be8dd1\" (UID: \"631d6c09-0992-4283-b9ba-ae8c39be8dd1\") "
Sep 18 20:06:36.118384 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:06:36.118176    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cc71b9f8-e3bb-4704-98f7-0d07a7f6fe9b-tmp-dir\") pod \"cc71b9f8-e3bb-4704-98f7-0d07a7f6fe9b\" (UID: \"cc71b9f8-e3bb-4704-98f7-0d07a7f6fe9b\") "
Sep 18 20:06:36.118384 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:06:36.118207    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cc71b9f8-e3bb-4704-98f7-0d07a7f6fe9b-tls-certs\") pod \"cc71b9f8-e3bb-4704-98f7-0d07a7f6fe9b\" (UID: \"cc71b9f8-e3bb-4704-98f7-0d07a7f6fe9b\") "
Sep 18 20:06:36.118384 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:06:36.118231    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/706cd513-2d52-43a6-aeed-4382e477fea2-home\") pod \"706cd513-2d52-43a6-aeed-4382e477fea2\" (UID: \"706cd513-2d52-43a6-aeed-4382e477fea2\") "
Sep 18 20:06:36.118384 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:06:36.118258    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-b9qtk\" (UniqueName: \"kubernetes.io/projected/706cd513-2d52-43a6-aeed-4382e477fea2-kube-api-access-b9qtk\") pod \"706cd513-2d52-43a6-aeed-4382e477fea2\" (UID: \"706cd513-2d52-43a6-aeed-4382e477fea2\") "
Sep 18 20:06:36.118384 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:06:36.118306    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cc71b9f8-e3bb-4704-98f7-0d07a7f6fe9b-model-cache\") pod \"cc71b9f8-e3bb-4704-98f7-0d07a7f6fe9b\" (UID: \"cc71b9f8-e3bb-4704-98f7-0d07a7f6fe9b\") "
Sep 18 20:06:36.118384 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:06:36.118335    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/631d6c09-0992-4283-b9ba-ae8c39be8dd1-tls-certs\") pod \"631d6c09-0992-4283-b9ba-ae8c39be8dd1\" (UID: \"631d6c09-0992-4283-b9ba-ae8c39be8dd1\") "
Sep 18 20:06:36.118384 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:06:36.118363    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/631d6c09-0992-4283-b9ba-ae8c39be8dd1-dshm\") pod \"631d6c09-0992-4283-b9ba-ae8c39be8dd1\" (UID: \"631d6c09-0992-4283-b9ba-ae8c39be8dd1\") "
Sep 18 20:06:36.118828 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:06:36.118402    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/631d6c09-0992-4283-b9ba-ae8c39be8dd1-home\") pod \"631d6c09-0992-4283-b9ba-ae8c39be8dd1\" (UID: \"631d6c09-0992-4283-b9ba-ae8c39be8dd1\") "
Sep 18 20:06:36.118828 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:06:36.118395    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/706cd513-2d52-43a6-aeed-4382e477fea2-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "706cd513-2d52-43a6-aeed-4382e477fea2" (UID: "706cd513-2d52-43a6-aeed-4382e477fea2"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 20:06:36.118828 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:06:36.118613    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/706cd513-2d52-43a6-aeed-4382e477fea2-model-cache" (OuterVolumeSpecName: "model-cache") pod "706cd513-2d52-43a6-aeed-4382e477fea2" (UID: "706cd513-2d52-43a6-aeed-4382e477fea2"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 20:06:36.118993 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:06:36.118830    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cc71b9f8-e3bb-4704-98f7-0d07a7f6fe9b-home" (OuterVolumeSpecName: "home") pod "cc71b9f8-e3bb-4704-98f7-0d07a7f6fe9b" (UID: "cc71b9f8-e3bb-4704-98f7-0d07a7f6fe9b"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 20:06:36.118993 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:06:36.118898    2585 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/706cd513-2d52-43a6-aeed-4382e477fea2-tmp-dir\") on node \"ip-10-0-136-185.ec2.internal\" DevicePath \"\""
Sep 18 20:06:36.118993 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:06:36.118912    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/631d6c09-0992-4283-b9ba-ae8c39be8dd1-home" (OuterVolumeSpecName: "home") pod "631d6c09-0992-4283-b9ba-ae8c39be8dd1" (UID: "631d6c09-0992-4283-b9ba-ae8c39be8dd1"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 20:06:36.118993 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:06:36.118921    2585 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/706cd513-2d52-43a6-aeed-4382e477fea2-model-cache\") on node \"ip-10-0-136-185.ec2.internal\" DevicePath \"\""
Sep 18 20:06:36.119187 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:06:36.119121    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cc71b9f8-e3bb-4704-98f7-0d07a7f6fe9b-model-cache" (OuterVolumeSpecName: "model-cache") pod "cc71b9f8-e3bb-4704-98f7-0d07a7f6fe9b" (UID: "cc71b9f8-e3bb-4704-98f7-0d07a7f6fe9b"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 20:06:36.119327 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:06:36.119286    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/631d6c09-0992-4283-b9ba-ae8c39be8dd1-model-cache" (OuterVolumeSpecName: "model-cache") pod "631d6c09-0992-4283-b9ba-ae8c39be8dd1" (UID: "631d6c09-0992-4283-b9ba-ae8c39be8dd1"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 20:06:36.119327 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:06:36.119317    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/706cd513-2d52-43a6-aeed-4382e477fea2-home" (OuterVolumeSpecName: "home") pod "706cd513-2d52-43a6-aeed-4382e477fea2" (UID: "706cd513-2d52-43a6-aeed-4382e477fea2"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 20:06:36.121756 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:06:36.121481    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cc71b9f8-e3bb-4704-98f7-0d07a7f6fe9b-kube-api-access-mwwjm" (OuterVolumeSpecName: "kube-api-access-mwwjm") pod "cc71b9f8-e3bb-4704-98f7-0d07a7f6fe9b" (UID: "cc71b9f8-e3bb-4704-98f7-0d07a7f6fe9b"). InnerVolumeSpecName "kube-api-access-mwwjm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 20:06:36.121929 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:06:36.121782    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/631d6c09-0992-4283-b9ba-ae8c39be8dd1-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "631d6c09-0992-4283-b9ba-ae8c39be8dd1" (UID: "631d6c09-0992-4283-b9ba-ae8c39be8dd1"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 20:06:36.121929 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:06:36.121837    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/706cd513-2d52-43a6-aeed-4382e477fea2-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "706cd513-2d52-43a6-aeed-4382e477fea2" (UID: "706cd513-2d52-43a6-aeed-4382e477fea2"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 20:06:36.122117 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:06:36.122021    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/631d6c09-0992-4283-b9ba-ae8c39be8dd1-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "631d6c09-0992-4283-b9ba-ae8c39be8dd1" (UID: "631d6c09-0992-4283-b9ba-ae8c39be8dd1"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 20:06:36.122117 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:06:36.122049    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cc71b9f8-e3bb-4704-98f7-0d07a7f6fe9b-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "cc71b9f8-e3bb-4704-98f7-0d07a7f6fe9b" (UID: "cc71b9f8-e3bb-4704-98f7-0d07a7f6fe9b"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 20:06:36.122506 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:06:36.122477    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cc71b9f8-e3bb-4704-98f7-0d07a7f6fe9b-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "cc71b9f8-e3bb-4704-98f7-0d07a7f6fe9b" (UID: "cc71b9f8-e3bb-4704-98f7-0d07a7f6fe9b"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 20:06:36.122626 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:06:36.122558    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/706cd513-2d52-43a6-aeed-4382e477fea2-dshm" (OuterVolumeSpecName: "dshm") pod "706cd513-2d52-43a6-aeed-4382e477fea2" (UID: "706cd513-2d52-43a6-aeed-4382e477fea2"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 20:06:36.122716 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:06:36.122695    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cc71b9f8-e3bb-4704-98f7-0d07a7f6fe9b-dshm" (OuterVolumeSpecName: "dshm") pod "cc71b9f8-e3bb-4704-98f7-0d07a7f6fe9b" (UID: "cc71b9f8-e3bb-4704-98f7-0d07a7f6fe9b"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 20:06:36.123035 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:06:36.123016    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/706cd513-2d52-43a6-aeed-4382e477fea2-kube-api-access-b9qtk" (OuterVolumeSpecName: "kube-api-access-b9qtk") pod "706cd513-2d52-43a6-aeed-4382e477fea2" (UID: "706cd513-2d52-43a6-aeed-4382e477fea2"). InnerVolumeSpecName "kube-api-access-b9qtk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 20:06:36.123284 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:06:36.123268    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/631d6c09-0992-4283-b9ba-ae8c39be8dd1-kube-api-access-6z7h5" (OuterVolumeSpecName: "kube-api-access-6z7h5") pod "631d6c09-0992-4283-b9ba-ae8c39be8dd1" (UID: "631d6c09-0992-4283-b9ba-ae8c39be8dd1"). InnerVolumeSpecName "kube-api-access-6z7h5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 20:06:36.123961 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:06:36.123936    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/631d6c09-0992-4283-b9ba-ae8c39be8dd1-dshm" (OuterVolumeSpecName: "dshm") pod "631d6c09-0992-4283-b9ba-ae8c39be8dd1" (UID: "631d6c09-0992-4283-b9ba-ae8c39be8dd1"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 20:06:36.219709 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:06:36.219668    2585 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6z7h5\" (UniqueName: \"kubernetes.io/projected/631d6c09-0992-4283-b9ba-ae8c39be8dd1-kube-api-access-6z7h5\") on node \"ip-10-0-136-185.ec2.internal\" DevicePath \"\""
Sep 18 20:06:36.219709 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:06:36.219700    2585 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/706cd513-2d52-43a6-aeed-4382e477fea2-dshm\") on node \"ip-10-0-136-185.ec2.internal\" DevicePath \"\""
Sep 18 20:06:36.219709 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:06:36.219709    2585 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/706cd513-2d52-43a6-aeed-4382e477fea2-tls-certs\") on node \"ip-10-0-136-185.ec2.internal\" DevicePath \"\""
Sep 18 20:06:36.219709 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:06:36.219717    2585 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cc71b9f8-e3bb-4704-98f7-0d07a7f6fe9b-home\") on node \"ip-10-0-136-185.ec2.internal\" DevicePath \"\""
Sep 18 20:06:36.220005 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:06:36.219726    2585 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mwwjm\" (UniqueName: \"kubernetes.io/projected/cc71b9f8-e3bb-4704-98f7-0d07a7f6fe9b-kube-api-access-mwwjm\") on node \"ip-10-0-136-185.ec2.internal\" DevicePath \"\""
Sep 18 20:06:36.220005 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:06:36.219733    2585 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cc71b9f8-e3bb-4704-98f7-0d07a7f6fe9b-dshm\") on node \"ip-10-0-136-185.ec2.internal\" DevicePath \"\""
Sep 18 20:06:36.220005 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:06:36.219741    2585 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/631d6c09-0992-4283-b9ba-ae8c39be8dd1-model-cache\") on node \"ip-10-0-136-185.ec2.internal\" DevicePath \"\""
Sep 18 20:06:36.220005 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:06:36.219750    2585 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/631d6c09-0992-4283-b9ba-ae8c39be8dd1-tmp-dir\") on node \"ip-10-0-136-185.ec2.internal\" DevicePath \"\""
Sep 18 20:06:36.220005 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:06:36.219757    2585 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cc71b9f8-e3bb-4704-98f7-0d07a7f6fe9b-tmp-dir\") on node \"ip-10-0-136-185.ec2.internal\" DevicePath \"\""
Sep 18 20:06:36.220005 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:06:36.219764    2585 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cc71b9f8-e3bb-4704-98f7-0d07a7f6fe9b-tls-certs\") on node \"ip-10-0-136-185.ec2.internal\" DevicePath \"\""
Sep 18 20:06:36.220005 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:06:36.219771    2585 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/706cd513-2d52-43a6-aeed-4382e477fea2-home\") on node \"ip-10-0-136-185.ec2.internal\" DevicePath \"\""
Sep 18 20:06:36.220005 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:06:36.219778    2585 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b9qtk\" (UniqueName: \"kubernetes.io/projected/706cd513-2d52-43a6-aeed-4382e477fea2-kube-api-access-b9qtk\") on node \"ip-10-0-136-185.ec2.internal\" DevicePath \"\""
Sep 18 20:06:36.220005 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:06:36.219808    2585 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cc71b9f8-e3bb-4704-98f7-0d07a7f6fe9b-model-cache\") on node \"ip-10-0-136-185.ec2.internal\" DevicePath \"\""
Sep 18 20:06:36.220005 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:06:36.219816    2585 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/631d6c09-0992-4283-b9ba-ae8c39be8dd1-tls-certs\") on node \"ip-10-0-136-185.ec2.internal\" DevicePath \"\""
Sep 18 20:06:36.220005 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:06:36.219823    2585 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/631d6c09-0992-4283-b9ba-ae8c39be8dd1-dshm\") on node \"ip-10-0-136-185.ec2.internal\" DevicePath \"\""
Sep 18 20:06:36.220005 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:06:36.219831    2585 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/631d6c09-0992-4283-b9ba-ae8c39be8dd1-home\") on node \"ip-10-0-136-185.ec2.internal\" DevicePath \"\""
Sep 18 20:06:36.298638 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:06:36.298553    2585 generic.go:356] "Generic (PLEG): container finished" podID="cc71b9f8-e3bb-4704-98f7-0d07a7f6fe9b" containerID="ec0eb1f0385975278f3cd3cb69fbf9771ca5b678b8806381a50185b95029d696" exitCode=0
Sep 18 20:06:36.298638 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:06:36.298604    2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-d94b9e/simulator-w0-d94b9e-kserve-7669969b8-6p4m8"
Sep 18 20:06:36.298638 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:06:36.298633    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-d94b9e/simulator-w0-d94b9e-kserve-7669969b8-6p4m8" event={"ID":"cc71b9f8-e3bb-4704-98f7-0d07a7f6fe9b","Type":"ContainerDied","Data":"ec0eb1f0385975278f3cd3cb69fbf9771ca5b678b8806381a50185b95029d696"}
Sep 18 20:06:36.298952 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:06:36.298672    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-d94b9e/simulator-w0-d94b9e-kserve-7669969b8-6p4m8" event={"ID":"cc71b9f8-e3bb-4704-98f7-0d07a7f6fe9b","Type":"ContainerDied","Data":"1017d7175b6c07535113e05ccc8e1c3ae46033ef879c4accb2772549a2c80106"}
Sep 18 20:06:36.298952 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:06:36.298693    2585 scope.go:117] "RemoveContainer" containerID="ec0eb1f0385975278f3cd3cb69fbf9771ca5b678b8806381a50185b95029d696"
Sep 18 20:06:36.302566 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:06:36.302537    2585 generic.go:356] "Generic (PLEG): container finished" podID="706cd513-2d52-43a6-aeed-4382e477fea2" containerID="c1f67ed6ca8e921f7e48b4c63e1f2f6b5c1ff741b933fbbb69a9d53bd0cca91d" exitCode=0
Sep 18 20:06:36.302700 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:06:36.302582    2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-d94b9e/distinct-w0-d94b9e-kserve-654c9b789b-w7qlc"
Sep 18 20:06:36.302700 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:06:36.302629    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-d94b9e/distinct-w0-d94b9e-kserve-654c9b789b-w7qlc" event={"ID":"706cd513-2d52-43a6-aeed-4382e477fea2","Type":"ContainerDied","Data":"c1f67ed6ca8e921f7e48b4c63e1f2f6b5c1ff741b933fbbb69a9d53bd0cca91d"}
Sep 18 20:06:36.302700 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:06:36.302669    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-d94b9e/distinct-w0-d94b9e-kserve-654c9b789b-w7qlc" event={"ID":"706cd513-2d52-43a6-aeed-4382e477fea2","Type":"ContainerDied","Data":"dfc42ce14efbe7b147befd469eae3ff76e03144d03f4073731b5a3fcc0a6e5ec"}
Sep 18 20:06:36.304325 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:06:36.304299    2585 generic.go:356] "Generic (PLEG): container finished" podID="631d6c09-0992-4283-b9ba-ae8c39be8dd1" containerID="99d5d234ceb9b0087a9ae4eeab3297e89fcb65acaf6a9234d68944d9e4375a97" exitCode=0
Sep 18 20:06:36.304436 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:06:36.304339    2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-d94b9e/embedding-w0-d94b9e-kserve-6f556c7b9-5n4l2"
Sep 18 20:06:36.304436 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:06:36.304388    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-d94b9e/embedding-w0-d94b9e-kserve-6f556c7b9-5n4l2" event={"ID":"631d6c09-0992-4283-b9ba-ae8c39be8dd1","Type":"ContainerDied","Data":"99d5d234ceb9b0087a9ae4eeab3297e89fcb65acaf6a9234d68944d9e4375a97"}
Sep 18 20:06:36.304436 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:06:36.304426    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-d94b9e/embedding-w0-d94b9e-kserve-6f556c7b9-5n4l2" event={"ID":"631d6c09-0992-4283-b9ba-ae8c39be8dd1","Type":"ContainerDied","Data":"551ecf2eb07d51712cfbc1f11d358e1d8439a7fbbdefaefd87ca6faa08664e51"}
Sep 18 20:06:36.309933 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:06:36.309895    2585 scope.go:117] "RemoveContainer" containerID="ec0eb1f0385975278f3cd3cb69fbf9771ca5b678b8806381a50185b95029d696"
Sep 18 20:06:36.310274 ip-10-0-136-185 kubenswrapper[2585]: E0918 20:06:36.310252    2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ec0eb1f0385975278f3cd3cb69fbf9771ca5b678b8806381a50185b95029d696\": container with ID starting with ec0eb1f0385975278f3cd3cb69fbf9771ca5b678b8806381a50185b95029d696 not found: ID does not exist" containerID="ec0eb1f0385975278f3cd3cb69fbf9771ca5b678b8806381a50185b95029d696"
Sep 18 20:06:36.310363 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:06:36.310288    2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ec0eb1f0385975278f3cd3cb69fbf9771ca5b678b8806381a50185b95029d696"} err="failed to get container status \"ec0eb1f0385975278f3cd3cb69fbf9771ca5b678b8806381a50185b95029d696\": rpc error: code = NotFound desc = could not find container \"ec0eb1f0385975278f3cd3cb69fbf9771ca5b678b8806381a50185b95029d696\": container with ID starting with ec0eb1f0385975278f3cd3cb69fbf9771ca5b678b8806381a50185b95029d696 not found: ID does not exist"
Sep 18 20:06:36.310363 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:06:36.310313    2585 scope.go:117] "RemoveContainer" containerID="c1f67ed6ca8e921f7e48b4c63e1f2f6b5c1ff741b933fbbb69a9d53bd0cca91d"
Sep 18 20:06:36.319359 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:06:36.319341    2585 scope.go:117] "RemoveContainer" containerID="c1f67ed6ca8e921f7e48b4c63e1f2f6b5c1ff741b933fbbb69a9d53bd0cca91d"
Sep 18 20:06:36.319660 ip-10-0-136-185 kubenswrapper[2585]: E0918 20:06:36.319641    2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c1f67ed6ca8e921f7e48b4c63e1f2f6b5c1ff741b933fbbb69a9d53bd0cca91d\": container with ID starting with c1f67ed6ca8e921f7e48b4c63e1f2f6b5c1ff741b933fbbb69a9d53bd0cca91d not found: ID does not exist" containerID="c1f67ed6ca8e921f7e48b4c63e1f2f6b5c1ff741b933fbbb69a9d53bd0cca91d"
Sep 18 20:06:36.319709 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:06:36.319668    2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c1f67ed6ca8e921f7e48b4c63e1f2f6b5c1ff741b933fbbb69a9d53bd0cca91d"} err="failed to get container status \"c1f67ed6ca8e921f7e48b4c63e1f2f6b5c1ff741b933fbbb69a9d53bd0cca91d\": rpc error: code = NotFound desc = could not find container \"c1f67ed6ca8e921f7e48b4c63e1f2f6b5c1ff741b933fbbb69a9d53bd0cca91d\": container with ID starting with c1f67ed6ca8e921f7e48b4c63e1f2f6b5c1ff741b933fbbb69a9d53bd0cca91d not found: ID does not exist"
Sep 18 20:06:36.319709 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:06:36.319687    2585 scope.go:117] "RemoveContainer" containerID="99d5d234ceb9b0087a9ae4eeab3297e89fcb65acaf6a9234d68944d9e4375a97"
Sep 18 20:06:36.323559 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:06:36.323536    2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-d94b9e/simulator-w0-d94b9e-kserve-7669969b8-6p4m8"]
Sep 18 20:06:36.335354 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:06:36.329596    2585 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-d94b9e/simulator-w0-d94b9e-kserve-7669969b8-6p4m8"]
Sep 18 20:06:36.335641 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:06:36.335615    2585 scope.go:117] "RemoveContainer" containerID="99d5d234ceb9b0087a9ae4eeab3297e89fcb65acaf6a9234d68944d9e4375a97"
Sep 18 20:06:36.336211 ip-10-0-136-185 kubenswrapper[2585]: E0918 20:06:36.336182    2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"99d5d234ceb9b0087a9ae4eeab3297e89fcb65acaf6a9234d68944d9e4375a97\": container with ID starting with 99d5d234ceb9b0087a9ae4eeab3297e89fcb65acaf6a9234d68944d9e4375a97 not found: ID does not exist" containerID="99d5d234ceb9b0087a9ae4eeab3297e89fcb65acaf6a9234d68944d9e4375a97"
Sep 18 20:06:36.336299 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:06:36.336219    2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"99d5d234ceb9b0087a9ae4eeab3297e89fcb65acaf6a9234d68944d9e4375a97"} err="failed to get container status \"99d5d234ceb9b0087a9ae4eeab3297e89fcb65acaf6a9234d68944d9e4375a97\": rpc error: code = NotFound desc = could not find container \"99d5d234ceb9b0087a9ae4eeab3297e89fcb65acaf6a9234d68944d9e4375a97\": container with ID starting with 99d5d234ceb9b0087a9ae4eeab3297e89fcb65acaf6a9234d68944d9e4375a97 not found: ID does not exist"
Sep 18 20:06:36.346208 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:06:36.346174    2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-d94b9e/embedding-w0-d94b9e-kserve-6f556c7b9-5n4l2"]
Sep 18 20:06:36.352489 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:06:36.352460    2585 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-d94b9e/embedding-w0-d94b9e-kserve-6f556c7b9-5n4l2"]
Sep 18 20:06:36.366829 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:06:36.366770    2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-d94b9e/distinct-w0-d94b9e-kserve-654c9b789b-w7qlc"]
Sep 18 20:06:36.370676 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:06:36.370651    2585 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-d94b9e/distinct-w0-d94b9e-kserve-654c9b789b-w7qlc"]
Sep 18 20:06:36.663517 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:06:36.663485    2585 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="631d6c09-0992-4283-b9ba-ae8c39be8dd1" path="/var/lib/kubelet/pods/631d6c09-0992-4283-b9ba-ae8c39be8dd1/volumes"
Sep 18 20:06:36.663947 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:06:36.663933    2585 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="661ede6f-264a-4fbe-9037-9c61a63697e4" path="/var/lib/kubelet/pods/661ede6f-264a-4fbe-9037-9c61a63697e4/volumes"
Sep 18 20:06:36.664297 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:06:36.664284    2585 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="706cd513-2d52-43a6-aeed-4382e477fea2" path="/var/lib/kubelet/pods/706cd513-2d52-43a6-aeed-4382e477fea2/volumes"
Sep 18 20:06:36.664524 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:06:36.664514    2585 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="915a04ad-037d-4a25-92f6-5ff7da295ac0" path="/var/lib/kubelet/pods/915a04ad-037d-4a25-92f6-5ff7da295ac0/volumes"
Sep 18 20:06:36.665126 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:06:36.665109    2585 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c67b483b-0a36-4990-97a3-3227c41fcca4" path="/var/lib/kubelet/pods/c67b483b-0a36-4990-97a3-3227c41fcca4/volumes"
Sep 18 20:06:36.665477 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:06:36.665463    2585 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="cc71b9f8-e3bb-4704-98f7-0d07a7f6fe9b" path="/var/lib/kubelet/pods/cc71b9f8-e3bb-4704-98f7-0d07a7f6fe9b/volumes"
Sep 18 20:06:46.667253 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:06:46.667219    2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-c3d85f-85c49fff7c-657d8"]
Sep 18 20:06:46.667842 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:06:46.667473    2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-a-c3d85f-85c49fff7c-657d8" podUID="ed61d470-33dd-4ed3-a0da-835e282231c2" containerName="payload-processing" containerID="cri-o://d375b331c3ed63f10f9e912431e03e3d251ca32f0d7ab0aa18e07fdfe3e53a1c" gracePeriod=30
Sep 18 20:06:46.842589 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:06:46.842543    2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-c3d85f-tpctg"]
Sep 18 20:06:46.844814 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:06:46.844770    2585 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-c3d85f-tpctg"]
Sep 18 20:06:46.923772 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:06:46.923704    2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-c3d85f-85c49fff7c-657d8"
Sep 18 20:06:47.024388 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:06:47.024338    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mxm9t\" (UniqueName: \"kubernetes.io/projected/ed61d470-33dd-4ed3-a0da-835e282231c2-kube-api-access-mxm9t\") pod \"ed61d470-33dd-4ed3-a0da-835e282231c2\" (UID: \"ed61d470-33dd-4ed3-a0da-835e282231c2\") "
Sep 18 20:06:47.024573 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:06:47.024480    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ed61d470-33dd-4ed3-a0da-835e282231c2-plugins-config-volume\") pod \"ed61d470-33dd-4ed3-a0da-835e282231c2\" (UID: \"ed61d470-33dd-4ed3-a0da-835e282231c2\") "
Sep 18 20:06:47.024878 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:06:47.024856    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ed61d470-33dd-4ed3-a0da-835e282231c2-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "ed61d470-33dd-4ed3-a0da-835e282231c2" (UID: "ed61d470-33dd-4ed3-a0da-835e282231c2"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 20:06:47.026510 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:06:47.026488    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ed61d470-33dd-4ed3-a0da-835e282231c2-kube-api-access-mxm9t" (OuterVolumeSpecName: "kube-api-access-mxm9t") pod "ed61d470-33dd-4ed3-a0da-835e282231c2" (UID: "ed61d470-33dd-4ed3-a0da-835e282231c2"). InnerVolumeSpecName "kube-api-access-mxm9t". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 20:06:47.125810 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:06:47.125749    2585 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ed61d470-33dd-4ed3-a0da-835e282231c2-plugins-config-volume\") on node \"ip-10-0-136-185.ec2.internal\" DevicePath \"\""
Sep 18 20:06:47.125810 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:06:47.125802    2585 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mxm9t\" (UniqueName: \"kubernetes.io/projected/ed61d470-33dd-4ed3-a0da-835e282231c2-kube-api-access-mxm9t\") on node \"ip-10-0-136-185.ec2.internal\" DevicePath \"\""
Sep 18 20:06:47.366983 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:06:47.366907    2585 generic.go:356] "Generic (PLEG): container finished" podID="ed61d470-33dd-4ed3-a0da-835e282231c2" containerID="d375b331c3ed63f10f9e912431e03e3d251ca32f0d7ab0aa18e07fdfe3e53a1c" exitCode=0
Sep 18 20:06:47.366983 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:06:47.366947    2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-c3d85f-85c49fff7c-657d8"
Sep 18 20:06:47.367170 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:06:47.366988    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-c3d85f-85c49fff7c-657d8" event={"ID":"ed61d470-33dd-4ed3-a0da-835e282231c2","Type":"ContainerDied","Data":"d375b331c3ed63f10f9e912431e03e3d251ca32f0d7ab0aa18e07fdfe3e53a1c"}
Sep 18 20:06:47.367170 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:06:47.367042    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-c3d85f-85c49fff7c-657d8" event={"ID":"ed61d470-33dd-4ed3-a0da-835e282231c2","Type":"ContainerDied","Data":"4f76ecad4e26f0333cc1ff9e4fc479912d16078a72159d63222a9af4bf2befb5"}
Sep 18 20:06:47.367170 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:06:47.367068    2585 scope.go:117] "RemoveContainer" containerID="d375b331c3ed63f10f9e912431e03e3d251ca32f0d7ab0aa18e07fdfe3e53a1c"
Sep 18 20:06:47.376160 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:06:47.376145    2585 scope.go:117] "RemoveContainer" containerID="d375b331c3ed63f10f9e912431e03e3d251ca32f0d7ab0aa18e07fdfe3e53a1c"
Sep 18 20:06:47.376411 ip-10-0-136-185 kubenswrapper[2585]: E0918 20:06:47.376393    2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d375b331c3ed63f10f9e912431e03e3d251ca32f0d7ab0aa18e07fdfe3e53a1c\": container with ID starting with d375b331c3ed63f10f9e912431e03e3d251ca32f0d7ab0aa18e07fdfe3e53a1c not found: ID does not exist" containerID="d375b331c3ed63f10f9e912431e03e3d251ca32f0d7ab0aa18e07fdfe3e53a1c"
Sep 18 20:06:47.376496 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:06:47.376420    2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d375b331c3ed63f10f9e912431e03e3d251ca32f0d7ab0aa18e07fdfe3e53a1c"} err="failed to get container status \"d375b331c3ed63f10f9e912431e03e3d251ca32f0d7ab0aa18e07fdfe3e53a1c\": rpc error: code = NotFound desc = could not find container \"d375b331c3ed63f10f9e912431e03e3d251ca32f0d7ab0aa18e07fdfe3e53a1c\": container with ID starting with d375b331c3ed63f10f9e912431e03e3d251ca32f0d7ab0aa18e07fdfe3e53a1c not found: ID does not exist"
Sep 18 20:06:47.394036 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:06:47.394014    2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-c3d85f-85c49fff7c-657d8"]
Sep 18 20:06:47.399193 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:06:47.399171    2585 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-c3d85f-85c49fff7c-657d8"]
Sep 18 20:06:48.661662 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:06:48.661626    2585 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6bb618be-f82b-414c-9823-88a7f208fcdb" path="/var/lib/kubelet/pods/6bb618be-f82b-414c-9823-88a7f208fcdb/volumes"
Sep 18 20:06:48.662072 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:06:48.662033    2585 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ed61d470-33dd-4ed3-a0da-835e282231c2" path="/var/lib/kubelet/pods/ed61d470-33dd-4ed3-a0da-835e282231c2/volumes"
Sep 18 20:06:53.808834 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:06:53.808778    2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-13f9e5-pzzqp"]
Sep 18 20:06:53.810271 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:06:53.810241    2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ed61d470-33dd-4ed3-a0da-835e282231c2" containerName="payload-processing"
Sep 18 20:06:53.810435 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:06:53.810424    2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="ed61d470-33dd-4ed3-a0da-835e282231c2" containerName="payload-processing"
Sep 18 20:06:53.810529 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:06:53.810519    2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cc71b9f8-e3bb-4704-98f7-0d07a7f6fe9b" containerName="main"
Sep 18 20:06:53.810599 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:06:53.810590    2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="cc71b9f8-e3bb-4704-98f7-0d07a7f6fe9b" containerName="main"
Sep 18 20:06:53.810693 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:06:53.810684    2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c67b483b-0a36-4990-97a3-3227c41fcca4" containerName="payload-pre-processing"
Sep 18 20:06:53.810759 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:06:53.810751    2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="c67b483b-0a36-4990-97a3-3227c41fcca4" containerName="payload-pre-processing"
Sep 18 20:06:53.810868 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:06:53.810857    2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="661ede6f-264a-4fbe-9037-9c61a63697e4" containerName="payload-processing"
Sep 18 20:06:53.810933 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:06:53.810925    2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="661ede6f-264a-4fbe-9037-9c61a63697e4" containerName="payload-processing"
Sep 18 20:06:53.811037 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:06:53.811028    2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6bb618be-f82b-414c-9823-88a7f208fcdb" containerName="revoke-keys"
Sep 18 20:06:53.811105 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:06:53.811096    2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="6bb618be-f82b-414c-9823-88a7f208fcdb" containerName="revoke-keys"
Sep 18 20:06:53.811184 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:06:53.811175    2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="631d6c09-0992-4283-b9ba-ae8c39be8dd1" containerName="main"
Sep 18 20:06:53.811252 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:06:53.811244    2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="631d6c09-0992-4283-b9ba-ae8c39be8dd1" containerName="main"
Sep 18 20:06:53.811321 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:06:53.811312    2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="706cd513-2d52-43a6-aeed-4382e477fea2" containerName="main"
Sep 18 20:06:53.811388 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:06:53.811380    2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="706cd513-2d52-43a6-aeed-4382e477fea2" containerName="main"
Sep 18 20:06:53.811646 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:06:53.811632    2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="661ede6f-264a-4fbe-9037-9c61a63697e4" containerName="payload-processing"
Sep 18 20:06:53.811742 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:06:53.811731    2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="c67b483b-0a36-4990-97a3-3227c41fcca4" containerName="payload-pre-processing"
Sep 18 20:06:53.811841 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:06:53.811830    2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="706cd513-2d52-43a6-aeed-4382e477fea2" containerName="main"
Sep 18 20:06:53.811924 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:06:53.811916    2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="ed61d470-33dd-4ed3-a0da-835e282231c2" containerName="payload-processing"
Sep 18 20:06:53.812003 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:06:53.811995    2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="631d6c09-0992-4283-b9ba-ae8c39be8dd1" containerName="main"
Sep 18 20:06:53.812091 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:06:53.812083    2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="cc71b9f8-e3bb-4704-98f7-0d07a7f6fe9b" containerName="main"
Sep 18 20:06:53.812174 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:06:53.812165    2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="6bb618be-f82b-414c-9823-88a7f208fcdb" containerName="revoke-keys"
Sep 18 20:06:53.817938 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:06:53.817909    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-13f9e5-pzzqp"
Sep 18 20:06:53.819833 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:06:53.819808    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-13f9e5-pzzqp"]
Sep 18 20:06:53.820255 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:06:53.820234    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-5ttnn\""
Sep 18 20:06:53.820362 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:06:53.820234    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 18 20:06:53.884319 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:06:53.884275    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/63e548c5-71e0-4edf-b845-fd2b124353de-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-b-13f9e5-pzzqp\" (UID: \"63e548c5-71e0-4edf-b845-fd2b124353de\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-13f9e5-pzzqp"
Sep 18 20:06:53.985302 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:06:53.985261    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/63e548c5-71e0-4edf-b845-fd2b124353de-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-b-13f9e5-pzzqp\" (UID: \"63e548c5-71e0-4edf-b845-fd2b124353de\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-13f9e5-pzzqp"
Sep 18 20:06:53.985967 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:06:53.985945    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/63e548c5-71e0-4edf-b845-fd2b124353de-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-b-13f9e5-pzzqp\" (UID: \"63e548c5-71e0-4edf-b845-fd2b124353de\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-13f9e5-pzzqp"
Sep 18 20:06:54.129832 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:06:54.129713    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-13f9e5-pzzqp"
Sep 18 20:06:54.400556 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:06:54.400526    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-13f9e5-pzzqp"]
Sep 18 20:06:54.402459 ip-10-0-136-185 kubenswrapper[2585]: W0918 20:06:54.402426    2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod63e548c5_71e0_4edf_b845_fd2b124353de.slice/crio-1cfad5a632460b9ac80484cd2f0c7cf4377b0ad7ccabea226ee197b19102f548 WatchSource:0}: Error finding container 1cfad5a632460b9ac80484cd2f0c7cf4377b0ad7ccabea226ee197b19102f548: Status 404 returned error can't find the container with id 1cfad5a632460b9ac80484cd2f0c7cf4377b0ad7ccabea226ee197b19102f548
Sep 18 20:06:55.411731 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:06:55.411693    2585 generic.go:356] "Generic (PLEG): container finished" podID="63e548c5-71e0-4edf-b845-fd2b124353de" containerID="040b95888eae6c3c85c3cf0195147c0df74e0f6f26c560655430274797b5ca3e" exitCode=0
Sep 18 20:06:55.412148 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:06:55.411812    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-13f9e5-pzzqp" event={"ID":"63e548c5-71e0-4edf-b845-fd2b124353de","Type":"ContainerDied","Data":"040b95888eae6c3c85c3cf0195147c0df74e0f6f26c560655430274797b5ca3e"}
Sep 18 20:06:55.412148 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:06:55.411859    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-13f9e5-pzzqp" event={"ID":"63e548c5-71e0-4edf-b845-fd2b124353de","Type":"ContainerStarted","Data":"1cfad5a632460b9ac80484cd2f0c7cf4377b0ad7ccabea226ee197b19102f548"}
Sep 18 20:06:56.559105 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:06:56.559073    2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-13f9e5-pzzqp"
Sep 18 20:06:56.613671 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:06:56.613640    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/63e548c5-71e0-4edf-b845-fd2b124353de-maas-api-service-ca\") pod \"63e548c5-71e0-4edf-b845-fd2b124353de\" (UID: \"63e548c5-71e0-4edf-b845-fd2b124353de\") "
Sep 18 20:06:56.614022 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:06:56.614001    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/63e548c5-71e0-4edf-b845-fd2b124353de-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "63e548c5-71e0-4edf-b845-fd2b124353de" (UID: "63e548c5-71e0-4edf-b845-fd2b124353de"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 20:06:56.715183 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:06:56.715146    2585 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/63e548c5-71e0-4edf-b845-fd2b124353de-maas-api-service-ca\") on node \"ip-10-0-136-185.ec2.internal\" DevicePath \"\""
Sep 18 20:06:57.423953 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:06:57.423926    2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-13f9e5-pzzqp"
Sep 18 20:06:57.423953 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:06:57.423943    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-13f9e5-pzzqp" event={"ID":"63e548c5-71e0-4edf-b845-fd2b124353de","Type":"ContainerDied","Data":"1cfad5a632460b9ac80484cd2f0c7cf4377b0ad7ccabea226ee197b19102f548"}
Sep 18 20:06:57.424210 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:06:57.423989    2585 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1cfad5a632460b9ac80484cd2f0c7cf4377b0ad7ccabea226ee197b19102f548"
Sep 18 20:07:04.270688 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:07:04.270646    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-xmwh7"]
Sep 18 20:07:09.386185 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:07:09.386151    2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-13f9e5-6f594f7fcd-8t98g"]
Sep 18 20:07:09.386679 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:07:09.386463    2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-b-13f9e5-6f594f7fcd-8t98g" podUID="47694d12-7008-4989-afc8-f128484ab6c5" containerName="payload-pre-processing" containerID="cri-o://0185a04104a0306a07de4fce1789772b53bc004f687f31d6a23daa835c15dc92" gracePeriod=30
Sep 18 20:07:09.547380 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:07:09.547339    2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-13f9e5-pzzqp"]
Sep 18 20:07:09.549893 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:07:09.549863    2585 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-13f9e5-pzzqp"]
Sep 18 20:07:09.640558 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:07:09.640539    2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-13f9e5-6f594f7fcd-8t98g"
Sep 18 20:07:09.739625 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:07:09.739595    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-95zbw\" (UniqueName: \"kubernetes.io/projected/47694d12-7008-4989-afc8-f128484ab6c5-kube-api-access-95zbw\") pod \"47694d12-7008-4989-afc8-f128484ab6c5\" (UID: \"47694d12-7008-4989-afc8-f128484ab6c5\") "
Sep 18 20:07:09.739823 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:07:09.739641    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/47694d12-7008-4989-afc8-f128484ab6c5-plugins-config-volume\") pod \"47694d12-7008-4989-afc8-f128484ab6c5\" (UID: \"47694d12-7008-4989-afc8-f128484ab6c5\") "
Sep 18 20:07:09.740124 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:07:09.740092    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/47694d12-7008-4989-afc8-f128484ab6c5-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "47694d12-7008-4989-afc8-f128484ab6c5" (UID: "47694d12-7008-4989-afc8-f128484ab6c5"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 20:07:09.741713 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:07:09.741693    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/47694d12-7008-4989-afc8-f128484ab6c5-kube-api-access-95zbw" (OuterVolumeSpecName: "kube-api-access-95zbw") pod "47694d12-7008-4989-afc8-f128484ab6c5" (UID: "47694d12-7008-4989-afc8-f128484ab6c5"). InnerVolumeSpecName "kube-api-access-95zbw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 20:07:09.841071 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:07:09.841029    2585 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-95zbw\" (UniqueName: \"kubernetes.io/projected/47694d12-7008-4989-afc8-f128484ab6c5-kube-api-access-95zbw\") on node \"ip-10-0-136-185.ec2.internal\" DevicePath \"\""
Sep 18 20:07:09.841071 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:07:09.841066    2585 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/47694d12-7008-4989-afc8-f128484ab6c5-plugins-config-volume\") on node \"ip-10-0-136-185.ec2.internal\" DevicePath \"\""
Sep 18 20:07:10.493455 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:07:10.493421    2585 generic.go:356] "Generic (PLEG): container finished" podID="47694d12-7008-4989-afc8-f128484ab6c5" containerID="0185a04104a0306a07de4fce1789772b53bc004f687f31d6a23daa835c15dc92" exitCode=0
Sep 18 20:07:10.493920 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:07:10.493462    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-13f9e5-6f594f7fcd-8t98g" event={"ID":"47694d12-7008-4989-afc8-f128484ab6c5","Type":"ContainerDied","Data":"0185a04104a0306a07de4fce1789772b53bc004f687f31d6a23daa835c15dc92"}
Sep 18 20:07:10.493920 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:07:10.493503    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-13f9e5-6f594f7fcd-8t98g" event={"ID":"47694d12-7008-4989-afc8-f128484ab6c5","Type":"ContainerDied","Data":"f1a0f590d587dfece870a03e9b0702749daa3a868e70d650790195c1770e824c"}
Sep 18 20:07:10.493920 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:07:10.493522    2585 scope.go:117] "RemoveContainer" containerID="0185a04104a0306a07de4fce1789772b53bc004f687f31d6a23daa835c15dc92"
Sep 18 20:07:10.493920 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:07:10.493464    2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-13f9e5-6f594f7fcd-8t98g"
Sep 18 20:07:10.502811 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:07:10.502774    2585 scope.go:117] "RemoveContainer" containerID="0185a04104a0306a07de4fce1789772b53bc004f687f31d6a23daa835c15dc92"
Sep 18 20:07:10.503102 ip-10-0-136-185 kubenswrapper[2585]: E0918 20:07:10.503085    2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0185a04104a0306a07de4fce1789772b53bc004f687f31d6a23daa835c15dc92\": container with ID starting with 0185a04104a0306a07de4fce1789772b53bc004f687f31d6a23daa835c15dc92 not found: ID does not exist" containerID="0185a04104a0306a07de4fce1789772b53bc004f687f31d6a23daa835c15dc92"
Sep 18 20:07:10.503162 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:07:10.503113    2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0185a04104a0306a07de4fce1789772b53bc004f687f31d6a23daa835c15dc92"} err="failed to get container status \"0185a04104a0306a07de4fce1789772b53bc004f687f31d6a23daa835c15dc92\": rpc error: code = NotFound desc = could not find container \"0185a04104a0306a07de4fce1789772b53bc004f687f31d6a23daa835c15dc92\": container with ID starting with 0185a04104a0306a07de4fce1789772b53bc004f687f31d6a23daa835c15dc92 not found: ID does not exist"
Sep 18 20:07:10.514156 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:07:10.514127    2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-13f9e5-6f594f7fcd-8t98g"]
Sep 18 20:07:10.520880 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:07:10.520847    2585 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-13f9e5-6f594f7fcd-8t98g"]
Sep 18 20:07:10.663156 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:07:10.663123    2585 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="47694d12-7008-4989-afc8-f128484ab6c5" path="/var/lib/kubelet/pods/47694d12-7008-4989-afc8-f128484ab6c5/volumes"
Sep 18 20:07:10.663479 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:07:10.663466    2585 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="63e548c5-71e0-4edf-b845-fd2b124353de" path="/var/lib/kubelet/pods/63e548c5-71e0-4edf-b845-fd2b124353de/volumes"
Sep 18 20:07:16.298705 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:07:16.298666    2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-047bea-nzdkf"]
Sep 18 20:07:16.299156 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:07:16.299129    2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="63e548c5-71e0-4edf-b845-fd2b124353de" containerName="revoke-keys"
Sep 18 20:07:16.299156 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:07:16.299148    2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="63e548c5-71e0-4edf-b845-fd2b124353de" containerName="revoke-keys"
Sep 18 20:07:16.299261 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:07:16.299166    2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="47694d12-7008-4989-afc8-f128484ab6c5" containerName="payload-pre-processing"
Sep 18 20:07:16.299261 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:07:16.299176    2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="47694d12-7008-4989-afc8-f128484ab6c5" containerName="payload-pre-processing"
Sep 18 20:07:16.299330 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:07:16.299303    2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="47694d12-7008-4989-afc8-f128484ab6c5" containerName="payload-pre-processing"
Sep 18 20:07:16.299330 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:07:16.299317    2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="63e548c5-71e0-4edf-b845-fd2b124353de" containerName="revoke-keys"
Sep 18 20:07:16.302392 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:07:16.302372    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-047bea-nzdkf"
Sep 18 20:07:16.304574 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:07:16.304550    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 18 20:07:16.304707 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:07:16.304577    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-5ttnn\""
Sep 18 20:07:16.309254 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:07:16.309229    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-047bea-nzdkf"]
Sep 18 20:07:16.411452 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:07:16.411410    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/744752fd-1b78-4532-ade6-77cf6de4f6b9-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-047bea-nzdkf\" (UID: \"744752fd-1b78-4532-ade6-77cf6de4f6b9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-047bea-nzdkf"
Sep 18 20:07:16.512013 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:07:16.511978    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/744752fd-1b78-4532-ade6-77cf6de4f6b9-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-047bea-nzdkf\" (UID: \"744752fd-1b78-4532-ade6-77cf6de4f6b9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-047bea-nzdkf"
Sep 18 20:07:16.512622 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:07:16.512601    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/744752fd-1b78-4532-ade6-77cf6de4f6b9-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-047bea-nzdkf\" (UID: \"744752fd-1b78-4532-ade6-77cf6de4f6b9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-047bea-nzdkf"
Sep 18 20:07:16.615616 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:07:16.615537    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-047bea-nzdkf"
Sep 18 20:07:16.886134 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:07:16.886105    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-047bea-nzdkf"]
Sep 18 20:07:16.889185 ip-10-0-136-185 kubenswrapper[2585]: W0918 20:07:16.889158    2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod744752fd_1b78_4532_ade6_77cf6de4f6b9.slice/crio-5c715fdaedc320e9e9cd5eb12bdeae577af453da830c8bd5f92ee393fa868371 WatchSource:0}: Error finding container 5c715fdaedc320e9e9cd5eb12bdeae577af453da830c8bd5f92ee393fa868371: Status 404 returned error can't find the container with id 5c715fdaedc320e9e9cd5eb12bdeae577af453da830c8bd5f92ee393fa868371
Sep 18 20:07:17.533318 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:07:17.533221    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-047bea-nzdkf" event={"ID":"744752fd-1b78-4532-ade6-77cf6de4f6b9","Type":"ContainerStarted","Data":"0a6ce61cef2d58e3da0b850db3ca667e0e5f0196deda716dbe67bb9aee152c75"}
Sep 18 20:07:17.533318 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:07:17.533281    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-047bea-nzdkf" event={"ID":"744752fd-1b78-4532-ade6-77cf6de4f6b9","Type":"ContainerStarted","Data":"5c715fdaedc320e9e9cd5eb12bdeae577af453da830c8bd5f92ee393fa868371"}
Sep 18 20:07:17.556176 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:07:17.556111    2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-047bea-nzdkf" podStartSLOduration=1.556089649 podStartE2EDuration="1.556089649s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 20:07:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 20:07:17.555850316 +0000 UTC m=+1679.080455741" watchObservedRunningTime="2026-09-18 20:07:17.556089649 +0000 UTC m=+1679.080695073"
Sep 18 20:07:18.544869 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:07:18.544832    2585 generic.go:356] "Generic (PLEG): container finished" podID="744752fd-1b78-4532-ade6-77cf6de4f6b9" containerID="0a6ce61cef2d58e3da0b850db3ca667e0e5f0196deda716dbe67bb9aee152c75" exitCode=0
Sep 18 20:07:18.545249 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:07:18.544896    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-047bea-nzdkf" event={"ID":"744752fd-1b78-4532-ade6-77cf6de4f6b9","Type":"ContainerDied","Data":"0a6ce61cef2d58e3da0b850db3ca667e0e5f0196deda716dbe67bb9aee152c75"}
Sep 18 20:07:19.685301 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:07:19.685263    2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-047bea-nzdkf"
Sep 18 20:07:19.851672 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:07:19.851577    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/744752fd-1b78-4532-ade6-77cf6de4f6b9-maas-api-service-ca\") pod \"744752fd-1b78-4532-ade6-77cf6de4f6b9\" (UID: \"744752fd-1b78-4532-ade6-77cf6de4f6b9\") "
Sep 18 20:07:19.851986 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:07:19.851962    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/744752fd-1b78-4532-ade6-77cf6de4f6b9-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "744752fd-1b78-4532-ade6-77cf6de4f6b9" (UID: "744752fd-1b78-4532-ade6-77cf6de4f6b9"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 20:07:19.952864 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:07:19.952827    2585 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/744752fd-1b78-4532-ade6-77cf6de4f6b9-maas-api-service-ca\") on node \"ip-10-0-136-185.ec2.internal\" DevicePath \"\""
Sep 18 20:07:20.555713 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:07:20.555684    2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-047bea-nzdkf"
Sep 18 20:07:20.555875 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:07:20.555711    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-047bea-nzdkf" event={"ID":"744752fd-1b78-4532-ade6-77cf6de4f6b9","Type":"ContainerDied","Data":"5c715fdaedc320e9e9cd5eb12bdeae577af453da830c8bd5f92ee393fa868371"}
Sep 18 20:07:20.555875 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:07:20.555759    2585 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5c715fdaedc320e9e9cd5eb12bdeae577af453da830c8bd5f92ee393fa868371"
Sep 18 20:07:20.806683 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:07:20.806587    2585 scope.go:117] "RemoveContainer" containerID="c791e0f23bd07884d085d8d0221fff3ee65e1153813100493e42a62761824bdc"
Sep 18 20:07:27.058147 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:07:27.058109    2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-047bea-nzdkf"]
Sep 18 20:07:27.063001 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:07:27.062963    2585 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-047bea-nzdkf"]
Sep 18 20:07:28.668056 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:07:28.668018    2585 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="744752fd-1b78-4532-ade6-77cf6de4f6b9" path="/var/lib/kubelet/pods/744752fd-1b78-4532-ade6-77cf6de4f6b9/volumes"
Sep 18 20:07:39.687817 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:07:39.687727    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-xmwh7"]
Sep 18 20:07:44.647863 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:07:44.647818    2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-9838c2-5dc444f4cd-hnphk"]
Sep 18 20:07:44.648327 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:07:44.648104    2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-9838c2-5dc444f4cd-hnphk" podUID="e5246851-f8aa-4055-9440-376bc57c630f" containerName="payload-pre-processing" containerID="cri-o://2995479d2c1b67bce98790872c234f1d4ca70ee1b71a8f5d10e47fa787ee83dd" gracePeriod=30
Sep 18 20:07:44.909157 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:07:44.909100    2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-9838c2-5dc444f4cd-hnphk"
Sep 18 20:07:44.980908 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:07:44.980877    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hzj2t\" (UniqueName: \"kubernetes.io/projected/e5246851-f8aa-4055-9440-376bc57c630f-kube-api-access-hzj2t\") pod \"e5246851-f8aa-4055-9440-376bc57c630f\" (UID: \"e5246851-f8aa-4055-9440-376bc57c630f\") "
Sep 18 20:07:44.981081 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:07:44.980951    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e5246851-f8aa-4055-9440-376bc57c630f-plugins-config-volume\") pod \"e5246851-f8aa-4055-9440-376bc57c630f\" (UID: \"e5246851-f8aa-4055-9440-376bc57c630f\") "
Sep 18 20:07:44.981288 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:07:44.981264    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e5246851-f8aa-4055-9440-376bc57c630f-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "e5246851-f8aa-4055-9440-376bc57c630f" (UID: "e5246851-f8aa-4055-9440-376bc57c630f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 20:07:44.983114 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:07:44.983092    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e5246851-f8aa-4055-9440-376bc57c630f-kube-api-access-hzj2t" (OuterVolumeSpecName: "kube-api-access-hzj2t") pod "e5246851-f8aa-4055-9440-376bc57c630f" (UID: "e5246851-f8aa-4055-9440-376bc57c630f"). InnerVolumeSpecName "kube-api-access-hzj2t". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 20:07:45.081984 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:07:45.081952    2585 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hzj2t\" (UniqueName: \"kubernetes.io/projected/e5246851-f8aa-4055-9440-376bc57c630f-kube-api-access-hzj2t\") on node \"ip-10-0-136-185.ec2.internal\" DevicePath \"\""
Sep 18 20:07:45.081984 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:07:45.081978    2585 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e5246851-f8aa-4055-9440-376bc57c630f-plugins-config-volume\") on node \"ip-10-0-136-185.ec2.internal\" DevicePath \"\""
Sep 18 20:07:45.691724 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:07:45.691686    2585 generic.go:356] "Generic (PLEG): container finished" podID="e5246851-f8aa-4055-9440-376bc57c630f" containerID="2995479d2c1b67bce98790872c234f1d4ca70ee1b71a8f5d10e47fa787ee83dd" exitCode=0
Sep 18 20:07:45.691724 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:07:45.691725    2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-9838c2-5dc444f4cd-hnphk"
Sep 18 20:07:45.692342 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:07:45.691765    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-9838c2-5dc444f4cd-hnphk" event={"ID":"e5246851-f8aa-4055-9440-376bc57c630f","Type":"ContainerDied","Data":"2995479d2c1b67bce98790872c234f1d4ca70ee1b71a8f5d10e47fa787ee83dd"}
Sep 18 20:07:45.692342 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:07:45.691836    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-9838c2-5dc444f4cd-hnphk" event={"ID":"e5246851-f8aa-4055-9440-376bc57c630f","Type":"ContainerDied","Data":"5e054a3daef756b6f9217d53aff3e0d6c629e6a5aeda61ccd00bb1887a19cf03"}
Sep 18 20:07:45.692342 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:07:45.691859    2585 scope.go:117] "RemoveContainer" containerID="2995479d2c1b67bce98790872c234f1d4ca70ee1b71a8f5d10e47fa787ee83dd"
Sep 18 20:07:45.701127 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:07:45.701107    2585 scope.go:117] "RemoveContainer" containerID="2995479d2c1b67bce98790872c234f1d4ca70ee1b71a8f5d10e47fa787ee83dd"
Sep 18 20:07:45.701376 ip-10-0-136-185 kubenswrapper[2585]: E0918 20:07:45.701358    2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2995479d2c1b67bce98790872c234f1d4ca70ee1b71a8f5d10e47fa787ee83dd\": container with ID starting with 2995479d2c1b67bce98790872c234f1d4ca70ee1b71a8f5d10e47fa787ee83dd not found: ID does not exist" containerID="2995479d2c1b67bce98790872c234f1d4ca70ee1b71a8f5d10e47fa787ee83dd"
Sep 18 20:07:45.701426 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:07:45.701385    2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2995479d2c1b67bce98790872c234f1d4ca70ee1b71a8f5d10e47fa787ee83dd"} err="failed to get container status \"2995479d2c1b67bce98790872c234f1d4ca70ee1b71a8f5d10e47fa787ee83dd\": rpc error: code = NotFound desc = could not find container \"2995479d2c1b67bce98790872c234f1d4ca70ee1b71a8f5d10e47fa787ee83dd\": container with ID starting with 2995479d2c1b67bce98790872c234f1d4ca70ee1b71a8f5d10e47fa787ee83dd not found: ID does not exist"
Sep 18 20:07:45.712222 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:07:45.712197    2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-9838c2-5dc444f4cd-hnphk"]
Sep 18 20:07:45.716016 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:07:45.715995    2585 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-9838c2-5dc444f4cd-hnphk"]
Sep 18 20:07:46.662288 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:07:46.662256    2585 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e5246851-f8aa-4055-9440-376bc57c630f" path="/var/lib/kubelet/pods/e5246851-f8aa-4055-9440-376bc57c630f/volumes"
Sep 18 20:08:00.475520 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:08:00.475481    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-xmwh7"]
Sep 18 20:08:03.776526 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:08:03.776483    2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-674263/distinct-w4-674263-kserve-69895788f-9wp56"]
Sep 18 20:08:03.782213 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:08:03.782186    2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-674263/premium-w4-674263-kserve-b95548674-rdx2h"]
Sep 18 20:08:03.797143 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:08:03.797107    2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-674263/simulator-w4-674263-kserve-78dd9c6fcf-zq5lx"]
Sep 18 20:08:14.667752 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:08:14.667720    2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-674263-565d4c87c9-wpr88"]
Sep 18 20:08:14.668270 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:08:14.667927    2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w4-674263-565d4c87c9-wpr88" podUID="e8a9b3b2-b551-48de-90fe-e30eddd5f9a4" containerName="payload-processing" containerID="cri-o://dde5bbab0ce5ac2a4ea802d3c73816b8d371f80bedf7751d1ca50f17bc90b219" gracePeriod=30
Sep 18 20:08:14.850504 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:08:14.850469    2585 generic.go:356] "Generic (PLEG): container finished" podID="e8a9b3b2-b551-48de-90fe-e30eddd5f9a4" containerID="dde5bbab0ce5ac2a4ea802d3c73816b8d371f80bedf7751d1ca50f17bc90b219" exitCode=0
Sep 18 20:08:14.850675 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:08:14.850533    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w4-674263-565d4c87c9-wpr88" event={"ID":"e8a9b3b2-b551-48de-90fe-e30eddd5f9a4","Type":"ContainerDied","Data":"dde5bbab0ce5ac2a4ea802d3c73816b8d371f80bedf7751d1ca50f17bc90b219"}
Sep 18 20:08:14.914445 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:08:14.914141    2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w4-674263-565d4c87c9-wpr88"
Sep 18 20:08:14.966670 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:08:14.966587    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9fqqq\" (UniqueName: \"kubernetes.io/projected/e8a9b3b2-b551-48de-90fe-e30eddd5f9a4-kube-api-access-9fqqq\") pod \"e8a9b3b2-b551-48de-90fe-e30eddd5f9a4\" (UID: \"e8a9b3b2-b551-48de-90fe-e30eddd5f9a4\") "
Sep 18 20:08:14.966832 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:08:14.966689    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e8a9b3b2-b551-48de-90fe-e30eddd5f9a4-plugins-config-volume\") pod \"e8a9b3b2-b551-48de-90fe-e30eddd5f9a4\" (UID: \"e8a9b3b2-b551-48de-90fe-e30eddd5f9a4\") "
Sep 18 20:08:14.967069 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:08:14.967040    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e8a9b3b2-b551-48de-90fe-e30eddd5f9a4-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "e8a9b3b2-b551-48de-90fe-e30eddd5f9a4" (UID: "e8a9b3b2-b551-48de-90fe-e30eddd5f9a4"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 20:08:14.967184 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:08:14.967126    2585 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e8a9b3b2-b551-48de-90fe-e30eddd5f9a4-plugins-config-volume\") on node \"ip-10-0-136-185.ec2.internal\" DevicePath \"\""
Sep 18 20:08:14.968653 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:08:14.968635    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e8a9b3b2-b551-48de-90fe-e30eddd5f9a4-kube-api-access-9fqqq" (OuterVolumeSpecName: "kube-api-access-9fqqq") pod "e8a9b3b2-b551-48de-90fe-e30eddd5f9a4" (UID: "e8a9b3b2-b551-48de-90fe-e30eddd5f9a4"). InnerVolumeSpecName "kube-api-access-9fqqq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 20:08:15.068349 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:08:15.068313    2585 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9fqqq\" (UniqueName: \"kubernetes.io/projected/e8a9b3b2-b551-48de-90fe-e30eddd5f9a4-kube-api-access-9fqqq\") on node \"ip-10-0-136-185.ec2.internal\" DevicePath \"\""
Sep 18 20:08:15.856929 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:08:15.856898    2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w4-674263-565d4c87c9-wpr88"
Sep 18 20:08:15.857381 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:08:15.856897    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w4-674263-565d4c87c9-wpr88" event={"ID":"e8a9b3b2-b551-48de-90fe-e30eddd5f9a4","Type":"ContainerDied","Data":"2d848bb7d612901cbff21ce3d4598ecc9b31f81d756d5e3f075e18f4553a098e"}
Sep 18 20:08:15.857381 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:08:15.857031    2585 scope.go:117] "RemoveContainer" containerID="dde5bbab0ce5ac2a4ea802d3c73816b8d371f80bedf7751d1ca50f17bc90b219"
Sep 18 20:08:15.880099 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:08:15.880072    2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-674263-565d4c87c9-wpr88"]
Sep 18 20:08:15.886149 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:08:15.886123    2585 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-674263-565d4c87c9-wpr88"]
Sep 18 20:08:16.575254 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:08:16.575219    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-xmwh7"]
Sep 18 20:08:16.661519 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:08:16.661487    2585 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e8a9b3b2-b551-48de-90fe-e30eddd5f9a4" path="/var/lib/kubelet/pods/e8a9b3b2-b551-48de-90fe-e30eddd5f9a4/volumes"
Sep 18 20:08:18.797434 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:08:18.797366    2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-674263/distinct-w4-674263-kserve-69895788f-9wp56" podUID="4ffc064a-d322-43ef-b803-f51096f08bc8" containerName="main" containerID="cri-o://ff07c99a809aaf344fcf3bfd0e173a527729032c91749c0c9f9c5227651727e3" gracePeriod=45
Sep 18 20:08:18.803669 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:08:18.803630    2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-674263/premium-w4-674263-kserve-b95548674-rdx2h" podUID="1d6d42ad-9f65-4d73-af4d-5d9e9202e531" containerName="main" containerID="cri-o://6570ce320bb6a2f24e7534c5bc0d4ee7dc97883b5877919795b6e89a2cfff29f" gracePeriod=45
Sep 18 20:08:18.816385 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:08:18.816347    2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-674263/simulator-w4-674263-kserve-78dd9c6fcf-zq5lx" podUID="51c71218-f838-4b23-a5bf-c5e274fa9c3d" containerName="main" containerID="cri-o://2c1dd408b960480306c4e5853af2aeb0031b7170579759234de6e0e4f725da5b" gracePeriod=45
Sep 18 20:08:19.075656 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:08:19.075630    2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-674263/distinct-w4-674263-kserve-69895788f-9wp56"
Sep 18 20:08:19.108179 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:08:19.108140    2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-674263/simulator-w4-674263-kserve-78dd9c6fcf-zq5lx"
Sep 18 20:08:19.110660 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:08:19.110638    2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-674263/premium-w4-674263-kserve-b95548674-rdx2h"
Sep 18 20:08:19.207759 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:08:19.207728    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/51c71218-f838-4b23-a5bf-c5e274fa9c3d-model-cache\") pod \"51c71218-f838-4b23-a5bf-c5e274fa9c3d\" (UID: \"51c71218-f838-4b23-a5bf-c5e274fa9c3d\") "
Sep 18 20:08:19.207946 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:08:19.207778    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1d6d42ad-9f65-4d73-af4d-5d9e9202e531-tmp-dir\") pod \"1d6d42ad-9f65-4d73-af4d-5d9e9202e531\" (UID: \"1d6d42ad-9f65-4d73-af4d-5d9e9202e531\") "
Sep 18 20:08:19.207946 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:08:19.207819    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1d6d42ad-9f65-4d73-af4d-5d9e9202e531-dshm\") pod \"1d6d42ad-9f65-4d73-af4d-5d9e9202e531\" (UID: \"1d6d42ad-9f65-4d73-af4d-5d9e9202e531\") "
Sep 18 20:08:19.207946 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:08:19.207839    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lpwtg\" (UniqueName: \"kubernetes.io/projected/1d6d42ad-9f65-4d73-af4d-5d9e9202e531-kube-api-access-lpwtg\") pod \"1d6d42ad-9f65-4d73-af4d-5d9e9202e531\" (UID: \"1d6d42ad-9f65-4d73-af4d-5d9e9202e531\") "
Sep 18 20:08:19.207946 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:08:19.207859    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/51c71218-f838-4b23-a5bf-c5e274fa9c3d-tls-certs\") pod \"51c71218-f838-4b23-a5bf-c5e274fa9c3d\" (UID: \"51c71218-f838-4b23-a5bf-c5e274fa9c3d\") "
Sep 18 20:08:19.207946 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:08:19.207902    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1d6d42ad-9f65-4d73-af4d-5d9e9202e531-tls-certs\") pod \"1d6d42ad-9f65-4d73-af4d-5d9e9202e531\" (UID: \"1d6d42ad-9f65-4d73-af4d-5d9e9202e531\") "
Sep 18 20:08:19.207946 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:08:19.207925    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4ffc064a-d322-43ef-b803-f51096f08bc8-home\") pod \"4ffc064a-d322-43ef-b803-f51096f08bc8\" (UID: \"4ffc064a-d322-43ef-b803-f51096f08bc8\") "
Sep 18 20:08:19.208267 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:08:19.208039    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/51c71218-f838-4b23-a5bf-c5e274fa9c3d-model-cache" (OuterVolumeSpecName: "model-cache") pod "51c71218-f838-4b23-a5bf-c5e274fa9c3d" (UID: "51c71218-f838-4b23-a5bf-c5e274fa9c3d"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 20:08:19.208267 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:08:19.208062    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4ffc064a-d322-43ef-b803-f51096f08bc8-tmp-dir\") pod \"4ffc064a-d322-43ef-b803-f51096f08bc8\" (UID: \"4ffc064a-d322-43ef-b803-f51096f08bc8\") "
Sep 18 20:08:19.208267 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:08:19.208114    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4ffc064a-d322-43ef-b803-f51096f08bc8-model-cache\") pod \"4ffc064a-d322-43ef-b803-f51096f08bc8\" (UID: \"4ffc064a-d322-43ef-b803-f51096f08bc8\") "
Sep 18 20:08:19.208267 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:08:19.208143    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/51c71218-f838-4b23-a5bf-c5e274fa9c3d-tmp-dir\") pod \"51c71218-f838-4b23-a5bf-c5e274fa9c3d\" (UID: \"51c71218-f838-4b23-a5bf-c5e274fa9c3d\") "
Sep 18 20:08:19.208267 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:08:19.208185    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4ffc064a-d322-43ef-b803-f51096f08bc8-tls-certs\") pod \"4ffc064a-d322-43ef-b803-f51096f08bc8\" (UID: \"4ffc064a-d322-43ef-b803-f51096f08bc8\") "
Sep 18 20:08:19.208267 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:08:19.208238    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nn9s9\" (UniqueName: \"kubernetes.io/projected/51c71218-f838-4b23-a5bf-c5e274fa9c3d-kube-api-access-nn9s9\") pod \"51c71218-f838-4b23-a5bf-c5e274fa9c3d\" (UID: \"51c71218-f838-4b23-a5bf-c5e274fa9c3d\") "
Sep 18 20:08:19.208580 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:08:19.208270    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mrvj5\" (UniqueName: \"kubernetes.io/projected/4ffc064a-d322-43ef-b803-f51096f08bc8-kube-api-access-mrvj5\") pod \"4ffc064a-d322-43ef-b803-f51096f08bc8\" (UID: \"4ffc064a-d322-43ef-b803-f51096f08bc8\") "
Sep 18 20:08:19.208580 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:08:19.208303    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4ffc064a-d322-43ef-b803-f51096f08bc8-dshm\") pod \"4ffc064a-d322-43ef-b803-f51096f08bc8\" (UID: \"4ffc064a-d322-43ef-b803-f51096f08bc8\") "
Sep 18 20:08:19.208580 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:08:19.208343    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1d6d42ad-9f65-4d73-af4d-5d9e9202e531-home\") pod \"1d6d42ad-9f65-4d73-af4d-5d9e9202e531\" (UID: \"1d6d42ad-9f65-4d73-af4d-5d9e9202e531\") "
Sep 18 20:08:19.208580 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:08:19.208371    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/51c71218-f838-4b23-a5bf-c5e274fa9c3d-dshm\") pod \"51c71218-f838-4b23-a5bf-c5e274fa9c3d\" (UID: \"51c71218-f838-4b23-a5bf-c5e274fa9c3d\") "
Sep 18 20:08:19.208580 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:08:19.208396    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/51c71218-f838-4b23-a5bf-c5e274fa9c3d-home\") pod \"51c71218-f838-4b23-a5bf-c5e274fa9c3d\" (UID: \"51c71218-f838-4b23-a5bf-c5e274fa9c3d\") "
Sep 18 20:08:19.208580 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:08:19.208419    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1d6d42ad-9f65-4d73-af4d-5d9e9202e531-model-cache\") pod \"1d6d42ad-9f65-4d73-af4d-5d9e9202e531\" (UID: \"1d6d42ad-9f65-4d73-af4d-5d9e9202e531\") "
Sep 18 20:08:19.208935 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:08:19.208800    2585 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/51c71218-f838-4b23-a5bf-c5e274fa9c3d-model-cache\") on node \"ip-10-0-136-185.ec2.internal\" DevicePath \"\""
Sep 18 20:08:19.210500 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:08:19.208141    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4ffc064a-d322-43ef-b803-f51096f08bc8-home" (OuterVolumeSpecName: "home") pod "4ffc064a-d322-43ef-b803-f51096f08bc8" (UID: "4ffc064a-d322-43ef-b803-f51096f08bc8"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 20:08:19.210639 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:08:19.208323    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4ffc064a-d322-43ef-b803-f51096f08bc8-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "4ffc064a-d322-43ef-b803-f51096f08bc8" (UID: "4ffc064a-d322-43ef-b803-f51096f08bc8"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 20:08:19.210639 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:08:19.208346    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4ffc064a-d322-43ef-b803-f51096f08bc8-model-cache" (OuterVolumeSpecName: "model-cache") pod "4ffc064a-d322-43ef-b803-f51096f08bc8" (UID: "4ffc064a-d322-43ef-b803-f51096f08bc8"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 20:08:19.210639 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:08:19.208416    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1d6d42ad-9f65-4d73-af4d-5d9e9202e531-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "1d6d42ad-9f65-4d73-af4d-5d9e9202e531" (UID: "1d6d42ad-9f65-4d73-af4d-5d9e9202e531"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 20:08:19.210639 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:08:19.208607    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1d6d42ad-9f65-4d73-af4d-5d9e9202e531-home" (OuterVolumeSpecName: "home") pod "1d6d42ad-9f65-4d73-af4d-5d9e9202e531" (UID: "1d6d42ad-9f65-4d73-af4d-5d9e9202e531"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 20:08:19.210639 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:08:19.209024    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1d6d42ad-9f65-4d73-af4d-5d9e9202e531-model-cache" (OuterVolumeSpecName: "model-cache") pod "1d6d42ad-9f65-4d73-af4d-5d9e9202e531" (UID: "1d6d42ad-9f65-4d73-af4d-5d9e9202e531"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 20:08:19.210639 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:08:19.210551    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1d6d42ad-9f65-4d73-af4d-5d9e9202e531-kube-api-access-lpwtg" (OuterVolumeSpecName: "kube-api-access-lpwtg") pod "1d6d42ad-9f65-4d73-af4d-5d9e9202e531" (UID: "1d6d42ad-9f65-4d73-af4d-5d9e9202e531"). InnerVolumeSpecName "kube-api-access-lpwtg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 20:08:19.210639 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:08:19.210460    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1d6d42ad-9f65-4d73-af4d-5d9e9202e531-dshm" (OuterVolumeSpecName: "dshm") pod "1d6d42ad-9f65-4d73-af4d-5d9e9202e531" (UID: "1d6d42ad-9f65-4d73-af4d-5d9e9202e531"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 20:08:19.210639 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:08:19.210604    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/51c71218-f838-4b23-a5bf-c5e274fa9c3d-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "51c71218-f838-4b23-a5bf-c5e274fa9c3d" (UID: "51c71218-f838-4b23-a5bf-c5e274fa9c3d"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 20:08:19.211050 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:08:19.210716    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4ffc064a-d322-43ef-b803-f51096f08bc8-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "4ffc064a-d322-43ef-b803-f51096f08bc8" (UID: "4ffc064a-d322-43ef-b803-f51096f08bc8"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 20:08:19.211050 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:08:19.210755    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/51c71218-f838-4b23-a5bf-c5e274fa9c3d-home" (OuterVolumeSpecName: "home") pod "51c71218-f838-4b23-a5bf-c5e274fa9c3d" (UID: "51c71218-f838-4b23-a5bf-c5e274fa9c3d"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 20:08:19.211050 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:08:19.210938    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/51c71218-f838-4b23-a5bf-c5e274fa9c3d-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "51c71218-f838-4b23-a5bf-c5e274fa9c3d" (UID: "51c71218-f838-4b23-a5bf-c5e274fa9c3d"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 20:08:19.211050 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:08:19.211018    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1d6d42ad-9f65-4d73-af4d-5d9e9202e531-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "1d6d42ad-9f65-4d73-af4d-5d9e9202e531" (UID: "1d6d42ad-9f65-4d73-af4d-5d9e9202e531"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 20:08:19.211220 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:08:19.211107    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4ffc064a-d322-43ef-b803-f51096f08bc8-kube-api-access-mrvj5" (OuterVolumeSpecName: "kube-api-access-mrvj5") pod "4ffc064a-d322-43ef-b803-f51096f08bc8" (UID: "4ffc064a-d322-43ef-b803-f51096f08bc8"). InnerVolumeSpecName "kube-api-access-mrvj5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 20:08:19.211405 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:08:19.211387    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4ffc064a-d322-43ef-b803-f51096f08bc8-dshm" (OuterVolumeSpecName: "dshm") pod "4ffc064a-d322-43ef-b803-f51096f08bc8" (UID: "4ffc064a-d322-43ef-b803-f51096f08bc8"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 20:08:19.211460 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:08:19.211438    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/51c71218-f838-4b23-a5bf-c5e274fa9c3d-dshm" (OuterVolumeSpecName: "dshm") pod "51c71218-f838-4b23-a5bf-c5e274fa9c3d" (UID: "51c71218-f838-4b23-a5bf-c5e274fa9c3d"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 20:08:19.212447 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:08:19.212430    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/51c71218-f838-4b23-a5bf-c5e274fa9c3d-kube-api-access-nn9s9" (OuterVolumeSpecName: "kube-api-access-nn9s9") pod "51c71218-f838-4b23-a5bf-c5e274fa9c3d" (UID: "51c71218-f838-4b23-a5bf-c5e274fa9c3d"). InnerVolumeSpecName "kube-api-access-nn9s9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 20:08:19.309544 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:08:19.309458    2585 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1d6d42ad-9f65-4d73-af4d-5d9e9202e531-tmp-dir\") on node \"ip-10-0-136-185.ec2.internal\" DevicePath \"\""
Sep 18 20:08:19.309544 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:08:19.309488    2585 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1d6d42ad-9f65-4d73-af4d-5d9e9202e531-dshm\") on node \"ip-10-0-136-185.ec2.internal\" DevicePath \"\""
Sep 18 20:08:19.309544 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:08:19.309498    2585 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lpwtg\" (UniqueName: \"kubernetes.io/projected/1d6d42ad-9f65-4d73-af4d-5d9e9202e531-kube-api-access-lpwtg\") on node \"ip-10-0-136-185.ec2.internal\" DevicePath \"\""
Sep 18 20:08:19.309544 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:08:19.309507    2585 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/51c71218-f838-4b23-a5bf-c5e274fa9c3d-tls-certs\") on node \"ip-10-0-136-185.ec2.internal\" DevicePath \"\""
Sep 18 20:08:19.309544 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:08:19.309516    2585 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1d6d42ad-9f65-4d73-af4d-5d9e9202e531-tls-certs\") on node \"ip-10-0-136-185.ec2.internal\" DevicePath \"\""
Sep 18 20:08:19.309544 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:08:19.309524    2585 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4ffc064a-d322-43ef-b803-f51096f08bc8-home\") on node \"ip-10-0-136-185.ec2.internal\" DevicePath \"\""
Sep 18 20:08:19.309544 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:08:19.309530    2585 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4ffc064a-d322-43ef-b803-f51096f08bc8-tmp-dir\") on node \"ip-10-0-136-185.ec2.internal\" DevicePath \"\""
Sep 18 20:08:19.309544 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:08:19.309539    2585 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4ffc064a-d322-43ef-b803-f51096f08bc8-model-cache\") on node \"ip-10-0-136-185.ec2.internal\" DevicePath \"\""
Sep 18 20:08:19.309544 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:08:19.309546    2585 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/51c71218-f838-4b23-a5bf-c5e274fa9c3d-tmp-dir\") on node \"ip-10-0-136-185.ec2.internal\" DevicePath \"\""
Sep 18 20:08:19.309544 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:08:19.309553    2585 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4ffc064a-d322-43ef-b803-f51096f08bc8-tls-certs\") on node \"ip-10-0-136-185.ec2.internal\" DevicePath \"\""
Sep 18 20:08:19.309998 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:08:19.309561    2585 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nn9s9\" (UniqueName: \"kubernetes.io/projected/51c71218-f838-4b23-a5bf-c5e274fa9c3d-kube-api-access-nn9s9\") on node \"ip-10-0-136-185.ec2.internal\" DevicePath \"\""
Sep 18 20:08:19.309998 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:08:19.309570    2585 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mrvj5\" (UniqueName: \"kubernetes.io/projected/4ffc064a-d322-43ef-b803-f51096f08bc8-kube-api-access-mrvj5\") on node \"ip-10-0-136-185.ec2.internal\" DevicePath \"\""
Sep 18 20:08:19.309998 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:08:19.309577    2585 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4ffc064a-d322-43ef-b803-f51096f08bc8-dshm\") on node \"ip-10-0-136-185.ec2.internal\" DevicePath \"\""
Sep 18 20:08:19.309998 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:08:19.309585    2585 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1d6d42ad-9f65-4d73-af4d-5d9e9202e531-home\") on node \"ip-10-0-136-185.ec2.internal\" DevicePath \"\""
Sep 18 20:08:19.309998 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:08:19.309592    2585 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/51c71218-f838-4b23-a5bf-c5e274fa9c3d-dshm\") on node \"ip-10-0-136-185.ec2.internal\" DevicePath \"\""
Sep 18 20:08:19.309998 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:08:19.309598    2585 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/51c71218-f838-4b23-a5bf-c5e274fa9c3d-home\") on node \"ip-10-0-136-185.ec2.internal\" DevicePath \"\""
Sep 18 20:08:19.309998 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:08:19.309604    2585 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1d6d42ad-9f65-4d73-af4d-5d9e9202e531-model-cache\") on node \"ip-10-0-136-185.ec2.internal\" DevicePath \"\""
Sep 18 20:08:19.880230 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:08:19.880195    2585 generic.go:356] "Generic (PLEG): container finished" podID="4ffc064a-d322-43ef-b803-f51096f08bc8" containerID="ff07c99a809aaf344fcf3bfd0e173a527729032c91749c0c9f9c5227651727e3" exitCode=0
Sep 18 20:08:19.880714 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:08:19.880255    2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-674263/distinct-w4-674263-kserve-69895788f-9wp56"
Sep 18 20:08:19.880714 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:08:19.880283    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-674263/distinct-w4-674263-kserve-69895788f-9wp56" event={"ID":"4ffc064a-d322-43ef-b803-f51096f08bc8","Type":"ContainerDied","Data":"ff07c99a809aaf344fcf3bfd0e173a527729032c91749c0c9f9c5227651727e3"}
Sep 18 20:08:19.880714 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:08:19.880338    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-674263/distinct-w4-674263-kserve-69895788f-9wp56" event={"ID":"4ffc064a-d322-43ef-b803-f51096f08bc8","Type":"ContainerDied","Data":"8005b4759d7b1f1c0fc78c4ebb9003606aa7431658f7244834a9e7792ac4ca65"}
Sep 18 20:08:19.880714 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:08:19.880362    2585 scope.go:117] "RemoveContainer" containerID="ff07c99a809aaf344fcf3bfd0e173a527729032c91749c0c9f9c5227651727e3"
Sep 18 20:08:19.881714 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:08:19.881620    2585 generic.go:356] "Generic (PLEG): container finished" podID="1d6d42ad-9f65-4d73-af4d-5d9e9202e531" containerID="6570ce320bb6a2f24e7534c5bc0d4ee7dc97883b5877919795b6e89a2cfff29f" exitCode=0
Sep 18 20:08:19.881714 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:08:19.881686    2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-674263/premium-w4-674263-kserve-b95548674-rdx2h"
Sep 18 20:08:19.881714 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:08:19.881702    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-674263/premium-w4-674263-kserve-b95548674-rdx2h" event={"ID":"1d6d42ad-9f65-4d73-af4d-5d9e9202e531","Type":"ContainerDied","Data":"6570ce320bb6a2f24e7534c5bc0d4ee7dc97883b5877919795b6e89a2cfff29f"}
Sep 18 20:08:19.881898 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:08:19.881737    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-674263/premium-w4-674263-kserve-b95548674-rdx2h" event={"ID":"1d6d42ad-9f65-4d73-af4d-5d9e9202e531","Type":"ContainerDied","Data":"77cdc2c37a9f4ceffa2a4860b5537386ecc745720f5a275b38f52f9a8bdd4507"}
Sep 18 20:08:19.883163 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:08:19.883123    2585 generic.go:356] "Generic (PLEG): container finished" podID="51c71218-f838-4b23-a5bf-c5e274fa9c3d" containerID="2c1dd408b960480306c4e5853af2aeb0031b7170579759234de6e0e4f725da5b" exitCode=0
Sep 18 20:08:19.883269 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:08:19.883168    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-674263/simulator-w4-674263-kserve-78dd9c6fcf-zq5lx" event={"ID":"51c71218-f838-4b23-a5bf-c5e274fa9c3d","Type":"ContainerDied","Data":"2c1dd408b960480306c4e5853af2aeb0031b7170579759234de6e0e4f725da5b"}
Sep 18 20:08:19.883269 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:08:19.883193    2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-674263/simulator-w4-674263-kserve-78dd9c6fcf-zq5lx"
Sep 18 20:08:19.883269 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:08:19.883209    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-674263/simulator-w4-674263-kserve-78dd9c6fcf-zq5lx" event={"ID":"51c71218-f838-4b23-a5bf-c5e274fa9c3d","Type":"ContainerDied","Data":"e87f6529b00081ac8561743c4509ad463b10a55999cea884df96c8fe303aecaf"}
Sep 18 20:08:19.891440 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:08:19.891420    2585 scope.go:117] "RemoveContainer" containerID="ff07c99a809aaf344fcf3bfd0e173a527729032c91749c0c9f9c5227651727e3"
Sep 18 20:08:19.891716 ip-10-0-136-185 kubenswrapper[2585]: E0918 20:08:19.891696    2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ff07c99a809aaf344fcf3bfd0e173a527729032c91749c0c9f9c5227651727e3\": container with ID starting with ff07c99a809aaf344fcf3bfd0e173a527729032c91749c0c9f9c5227651727e3 not found: ID does not exist" containerID="ff07c99a809aaf344fcf3bfd0e173a527729032c91749c0c9f9c5227651727e3"
Sep 18 20:08:19.891815 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:08:19.891724    2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ff07c99a809aaf344fcf3bfd0e173a527729032c91749c0c9f9c5227651727e3"} err="failed to get container status \"ff07c99a809aaf344fcf3bfd0e173a527729032c91749c0c9f9c5227651727e3\": rpc error: code = NotFound desc = could not find container \"ff07c99a809aaf344fcf3bfd0e173a527729032c91749c0c9f9c5227651727e3\": container with ID starting with ff07c99a809aaf344fcf3bfd0e173a527729032c91749c0c9f9c5227651727e3 not found: ID does not exist"
Sep 18 20:08:19.891815 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:08:19.891740    2585 scope.go:117] "RemoveContainer" containerID="6570ce320bb6a2f24e7534c5bc0d4ee7dc97883b5877919795b6e89a2cfff29f"
Sep 18 20:08:19.899651 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:08:19.899635    2585 scope.go:117] "RemoveContainer" containerID="6570ce320bb6a2f24e7534c5bc0d4ee7dc97883b5877919795b6e89a2cfff29f"
Sep 18 20:08:19.899925 ip-10-0-136-185 kubenswrapper[2585]: E0918 20:08:19.899909    2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6570ce320bb6a2f24e7534c5bc0d4ee7dc97883b5877919795b6e89a2cfff29f\": container with ID starting with 6570ce320bb6a2f24e7534c5bc0d4ee7dc97883b5877919795b6e89a2cfff29f not found: ID does not exist" containerID="6570ce320bb6a2f24e7534c5bc0d4ee7dc97883b5877919795b6e89a2cfff29f"
Sep 18 20:08:19.899986 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:08:19.899931    2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6570ce320bb6a2f24e7534c5bc0d4ee7dc97883b5877919795b6e89a2cfff29f"} err="failed to get container status \"6570ce320bb6a2f24e7534c5bc0d4ee7dc97883b5877919795b6e89a2cfff29f\": rpc error: code = NotFound desc = could not find container \"6570ce320bb6a2f24e7534c5bc0d4ee7dc97883b5877919795b6e89a2cfff29f\": container with ID starting with 6570ce320bb6a2f24e7534c5bc0d4ee7dc97883b5877919795b6e89a2cfff29f not found: ID does not exist"
Sep 18 20:08:19.899986 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:08:19.899947    2585 scope.go:117] "RemoveContainer" containerID="2c1dd408b960480306c4e5853af2aeb0031b7170579759234de6e0e4f725da5b"
Sep 18 20:08:19.904151 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:08:19.904129    2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-674263/distinct-w4-674263-kserve-69895788f-9wp56"]
Sep 18 20:08:19.907722 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:08:19.907702    2585 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-674263/distinct-w4-674263-kserve-69895788f-9wp56"]
Sep 18 20:08:19.908189 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:08:19.908177    2585 scope.go:117] "RemoveContainer" containerID="2c1dd408b960480306c4e5853af2aeb0031b7170579759234de6e0e4f725da5b"
Sep 18 20:08:19.908439 ip-10-0-136-185 kubenswrapper[2585]: E0918 20:08:19.908420    2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2c1dd408b960480306c4e5853af2aeb0031b7170579759234de6e0e4f725da5b\": container with ID starting with 2c1dd408b960480306c4e5853af2aeb0031b7170579759234de6e0e4f725da5b not found: ID does not exist" containerID="2c1dd408b960480306c4e5853af2aeb0031b7170579759234de6e0e4f725da5b"
Sep 18 20:08:19.908494 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:08:19.908442    2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2c1dd408b960480306c4e5853af2aeb0031b7170579759234de6e0e4f725da5b"} err="failed to get container status \"2c1dd408b960480306c4e5853af2aeb0031b7170579759234de6e0e4f725da5b\": rpc error: code = NotFound desc = could not find container \"2c1dd408b960480306c4e5853af2aeb0031b7170579759234de6e0e4f725da5b\": container with ID starting with 2c1dd408b960480306c4e5853af2aeb0031b7170579759234de6e0e4f725da5b not found: ID does not exist"
Sep 18 20:08:19.918920 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:08:19.918900    2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-674263/simulator-w4-674263-kserve-78dd9c6fcf-zq5lx"]
Sep 18 20:08:19.923182 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:08:19.923160    2585 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-674263/simulator-w4-674263-kserve-78dd9c6fcf-zq5lx"]
Sep 18 20:08:19.933164 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:08:19.933142    2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-674263/premium-w4-674263-kserve-b95548674-rdx2h"]
Sep 18 20:08:19.937682 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:08:19.937662    2585 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-674263/premium-w4-674263-kserve-b95548674-rdx2h"]
Sep 18 20:08:20.663082 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:08:20.663048    2585 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1d6d42ad-9f65-4d73-af4d-5d9e9202e531" path="/var/lib/kubelet/pods/1d6d42ad-9f65-4d73-af4d-5d9e9202e531/volumes"
Sep 18 20:08:20.663454 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:08:20.663442    2585 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4ffc064a-d322-43ef-b803-f51096f08bc8" path="/var/lib/kubelet/pods/4ffc064a-d322-43ef-b803-f51096f08bc8/volumes"
Sep 18 20:08:20.663807 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:08:20.663778    2585 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="51c71218-f838-4b23-a5bf-c5e274fa9c3d" path="/var/lib/kubelet/pods/51c71218-f838-4b23-a5bf-c5e274fa9c3d/volumes"
Sep 18 20:08:20.851350 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:08:20.851311    2585 scope.go:117] "RemoveContainer" containerID="611758758b16ed4fdc0feedf15d3230f29ec62172d7d23a826bd1aa096a253fd"
Sep 18 20:08:35.267412 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:08:35.267371    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-xmwh7"]
Sep 18 20:08:48.976946 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:08:48.976910    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-xmwh7"]
Sep 18 20:09:01.668744 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:09:01.668708    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-xmwh7"]
Sep 18 20:09:03.276637 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:09:03.276604    2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7525-c1534a"]
Sep 18 20:09:03.277048 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:09:03.276972    2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e5246851-f8aa-4055-9440-376bc57c630f" containerName="payload-pre-processing"
Sep 18 20:09:03.277048 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:09:03.276984    2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="e5246851-f8aa-4055-9440-376bc57c630f" containerName="payload-pre-processing"
Sep 18 20:09:03.277048 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:09:03.276997    2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1d6d42ad-9f65-4d73-af4d-5d9e9202e531" containerName="main"
Sep 18 20:09:03.277048 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:09:03.277002    2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="1d6d42ad-9f65-4d73-af4d-5d9e9202e531" containerName="main"
Sep 18 20:09:03.277048 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:09:03.277008    2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="51c71218-f838-4b23-a5bf-c5e274fa9c3d" containerName="main"
Sep 18 20:09:03.277048 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:09:03.277012    2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="51c71218-f838-4b23-a5bf-c5e274fa9c3d" containerName="main"
Sep 18 20:09:03.277048 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:09:03.277020    2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4ffc064a-d322-43ef-b803-f51096f08bc8" containerName="main"
Sep 18 20:09:03.277048 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:09:03.277025    2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="4ffc064a-d322-43ef-b803-f51096f08bc8" containerName="main"
Sep 18 20:09:03.277048 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:09:03.277037    2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="744752fd-1b78-4532-ade6-77cf6de4f6b9" containerName="revoke-keys"
Sep 18 20:09:03.277048 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:09:03.277042    2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="744752fd-1b78-4532-ade6-77cf6de4f6b9" containerName="revoke-keys"
Sep 18 20:09:03.277374 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:09:03.277055    2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e8a9b3b2-b551-48de-90fe-e30eddd5f9a4" containerName="payload-processing"
Sep 18 20:09:03.277374 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:09:03.277060    2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="e8a9b3b2-b551-48de-90fe-e30eddd5f9a4" containerName="payload-processing"
Sep 18 20:09:03.277374 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:09:03.277115    2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="744752fd-1b78-4532-ade6-77cf6de4f6b9" containerName="revoke-keys"
Sep 18 20:09:03.277374 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:09:03.277123    2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="1d6d42ad-9f65-4d73-af4d-5d9e9202e531" containerName="main"
Sep 18 20:09:03.277374 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:09:03.277130    2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="e8a9b3b2-b551-48de-90fe-e30eddd5f9a4" containerName="payload-processing"
Sep 18 20:09:03.277374 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:09:03.277138    2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="e5246851-f8aa-4055-9440-376bc57c630f" containerName="payload-pre-processing"
Sep 18 20:09:03.277374 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:09:03.277144    2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="51c71218-f838-4b23-a5bf-c5e274fa9c3d" containerName="main"
Sep 18 20:09:03.277374 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:09:03.277148    2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="4ffc064a-d322-43ef-b803-f51096f08bc8" containerName="main"
Sep 18 20:09:03.280404 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:09:03.280388    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7525-c1534a"
Sep 18 20:09:03.282600 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:09:03.282571    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-ls85n\""
Sep 18 20:09:03.286212 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:09:03.286185    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7525-c1534a"]
Sep 18 20:09:03.410934 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:09:03.410897    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s5xdr\" (UniqueName: \"kubernetes.io/projected/4f14744f-7e39-4bc7-b97f-15bbd798efe9-kube-api-access-s5xdr\") pod \"test-curl-7525-c1534a\" (UID: \"4f14744f-7e39-4bc7-b97f-15bbd798efe9\") " pod="opendatahub/test-curl-7525-c1534a"
Sep 18 20:09:03.512391 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:09:03.512355    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s5xdr\" (UniqueName: \"kubernetes.io/projected/4f14744f-7e39-4bc7-b97f-15bbd798efe9-kube-api-access-s5xdr\") pod \"test-curl-7525-c1534a\" (UID: \"4f14744f-7e39-4bc7-b97f-15bbd798efe9\") " pod="opendatahub/test-curl-7525-c1534a"
Sep 18 20:09:03.521357 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:09:03.521315    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-s5xdr\" (UniqueName: \"kubernetes.io/projected/4f14744f-7e39-4bc7-b97f-15bbd798efe9-kube-api-access-s5xdr\") pod \"test-curl-7525-c1534a\" (UID: \"4f14744f-7e39-4bc7-b97f-15bbd798efe9\") " pod="opendatahub/test-curl-7525-c1534a"
Sep 18 20:09:03.590844 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:09:03.590744    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7525-c1534a"
Sep 18 20:09:03.859697 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:09:03.859673    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7525-c1534a"]
Sep 18 20:09:03.861981 ip-10-0-136-185 kubenswrapper[2585]: W0918 20:09:03.861951    2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod4f14744f_7e39_4bc7_b97f_15bbd798efe9.slice/crio-fe1e4a565c4cf772d1c044ca5c3ed1865fbb30825f83dffa29a514b51896774c WatchSource:0}: Error finding container fe1e4a565c4cf772d1c044ca5c3ed1865fbb30825f83dffa29a514b51896774c: Status 404 returned error can't find the container with id fe1e4a565c4cf772d1c044ca5c3ed1865fbb30825f83dffa29a514b51896774c
Sep 18 20:09:04.093435 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:09:04.093399    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7525-c1534a" event={"ID":"4f14744f-7e39-4bc7-b97f-15bbd798efe9","Type":"ContainerStarted","Data":"e4f009343fe376cb7f67239a5d8e5b66546b1f4a803297f15f02a1e2da7ffa3b"}
Sep 18 20:09:04.093435 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:09:04.093438    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7525-c1534a" event={"ID":"4f14744f-7e39-4bc7-b97f-15bbd798efe9","Type":"ContainerStarted","Data":"fe1e4a565c4cf772d1c044ca5c3ed1865fbb30825f83dffa29a514b51896774c"}
Sep 18 20:09:04.107291 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:09:04.107230    2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-7525-c1534a" podStartSLOduration=1.107215253 podStartE2EDuration="1.107215253s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 20:09:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 20:09:04.106190206 +0000 UTC m=+1785.630795630" watchObservedRunningTime="2026-09-18 20:09:04.107215253 +0000 UTC m=+1785.631820675"
Sep 18 20:09:04.688025 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:09:04.687989    2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7525-c1534a"]
Sep 18 20:09:04.690574 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:09:04.690542    2585 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7525-c1534a"]
Sep 18 20:09:05.098468 ip-10-0-136-185 kubenswrapper[2585]: E0918 20:09:05.098377    2585 kuberuntime_manager.go:649] "Missing actuated resource record" pod="opendatahub/test-curl-7525-c1534a" container="test-curl-7525-c1534a"
Sep 18 20:09:05.989353 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:09:05.989310    2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-57b6ce/auth-test-model-57b6ce-kserve-66868b47d7-xkkds"]
Sep 18 20:09:06.103033 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:09:06.102988    2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-7525-c1534a" podUID="4f14744f-7e39-4bc7-b97f-15bbd798efe9" containerName="test-curl-7525-c1534a" containerID="cri-o://e4f009343fe376cb7f67239a5d8e5b66546b1f4a803297f15f02a1e2da7ffa3b" gracePeriod=2
Sep 18 20:09:06.104830 ip-10-0-136-185 kubenswrapper[2585]: E0918 20:09:06.104783    2585 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7525-c1534a\" is forbidden: User \"system:node:ip-10-0-136-185.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-136-185.ec2.internal' and this object" podUID="4f14744f-7e39-4bc7-b97f-15bbd798efe9" pod="opendatahub/test-curl-7525-c1534a"
Sep 18 20:09:08.283495 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:09:08.283470    2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7525-c1534a"
Sep 18 20:09:08.359875 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:09:08.359761    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-s5xdr\" (UniqueName: \"kubernetes.io/projected/4f14744f-7e39-4bc7-b97f-15bbd798efe9-kube-api-access-s5xdr\") pod \"4f14744f-7e39-4bc7-b97f-15bbd798efe9\" (UID: \"4f14744f-7e39-4bc7-b97f-15bbd798efe9\") "
Sep 18 20:09:08.361952 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:09:08.361927    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4f14744f-7e39-4bc7-b97f-15bbd798efe9-kube-api-access-s5xdr" (OuterVolumeSpecName: "kube-api-access-s5xdr") pod "4f14744f-7e39-4bc7-b97f-15bbd798efe9" (UID: "4f14744f-7e39-4bc7-b97f-15bbd798efe9"). InnerVolumeSpecName "kube-api-access-s5xdr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 20:09:08.460560 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:09:08.460524    2585 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s5xdr\" (UniqueName: \"kubernetes.io/projected/4f14744f-7e39-4bc7-b97f-15bbd798efe9-kube-api-access-s5xdr\") on node \"ip-10-0-136-185.ec2.internal\" DevicePath \"\""
Sep 18 20:09:08.662244 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:09:08.662211    2585 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4f14744f-7e39-4bc7-b97f-15bbd798efe9" path="/var/lib/kubelet/pods/4f14744f-7e39-4bc7-b97f-15bbd798efe9/volumes"
Sep 18 20:09:09.118036 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:09:09.117945    2585 generic.go:356] "Generic (PLEG): container finished" podID="4f14744f-7e39-4bc7-b97f-15bbd798efe9" containerID="e4f009343fe376cb7f67239a5d8e5b66546b1f4a803297f15f02a1e2da7ffa3b" exitCode=137
Sep 18 20:09:09.118036 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:09:09.117991    2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7525-c1534a"
Sep 18 20:09:09.118228 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:09:09.118049    2585 scope.go:117] "RemoveContainer" containerID="e4f009343fe376cb7f67239a5d8e5b66546b1f4a803297f15f02a1e2da7ffa3b"
Sep 18 20:09:09.126391 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:09:09.126364    2585 scope.go:117] "RemoveContainer" containerID="e4f009343fe376cb7f67239a5d8e5b66546b1f4a803297f15f02a1e2da7ffa3b"
Sep 18 20:09:09.126670 ip-10-0-136-185 kubenswrapper[2585]: E0918 20:09:09.126651    2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e4f009343fe376cb7f67239a5d8e5b66546b1f4a803297f15f02a1e2da7ffa3b\": container with ID starting with e4f009343fe376cb7f67239a5d8e5b66546b1f4a803297f15f02a1e2da7ffa3b not found: ID does not exist" containerID="e4f009343fe376cb7f67239a5d8e5b66546b1f4a803297f15f02a1e2da7ffa3b"
Sep 18 20:09:09.126723 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:09:09.126681    2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e4f009343fe376cb7f67239a5d8e5b66546b1f4a803297f15f02a1e2da7ffa3b"} err="failed to get container status \"e4f009343fe376cb7f67239a5d8e5b66546b1f4a803297f15f02a1e2da7ffa3b\": rpc error: code = NotFound desc = could not find container \"e4f009343fe376cb7f67239a5d8e5b66546b1f4a803297f15f02a1e2da7ffa3b\": container with ID starting with e4f009343fe376cb7f67239a5d8e5b66546b1f4a803297f15f02a1e2da7ffa3b not found: ID does not exist"
Sep 18 20:09:10.066124 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:09:10.066088    2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w3-57b6ce/sub-test-model-57b6ce-kserve-598ddfdfcf-xb6d6"]
Sep 18 20:09:10.066612 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:09:10.066480    2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4f14744f-7e39-4bc7-b97f-15bbd798efe9" containerName="test-curl-7525-c1534a"
Sep 18 20:09:10.066612 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:09:10.066495    2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="4f14744f-7e39-4bc7-b97f-15bbd798efe9" containerName="test-curl-7525-c1534a"
Sep 18 20:09:10.066612 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:09:10.066564    2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="4f14744f-7e39-4bc7-b97f-15bbd798efe9" containerName="test-curl-7525-c1534a"
Sep 18 20:09:10.071064 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:09:10.071040    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-57b6ce/sub-test-model-57b6ce-kserve-598ddfdfcf-xb6d6"
Sep 18 20:09:10.073311 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:09:10.073288    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-57b6ce\"/\"sub-test-model-57b6ce-kserve-self-signed-certs\""
Sep 18 20:09:10.081087 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:09:10.081060    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-57b6ce/sub-test-model-57b6ce-kserve-598ddfdfcf-xb6d6"]
Sep 18 20:09:10.178569 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:09:10.178528    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fb103bd1-453b-40a1-96b6-93c708f6d7d7-tmp-dir\") pod \"sub-test-model-57b6ce-kserve-598ddfdfcf-xb6d6\" (UID: \"fb103bd1-453b-40a1-96b6-93c708f6d7d7\") " pod="ai-tenant-e2e-shared-a-w3-57b6ce/sub-test-model-57b6ce-kserve-598ddfdfcf-xb6d6"
Sep 18 20:09:10.178569 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:09:10.178574    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fb103bd1-453b-40a1-96b6-93c708f6d7d7-dshm\") pod \"sub-test-model-57b6ce-kserve-598ddfdfcf-xb6d6\" (UID: \"fb103bd1-453b-40a1-96b6-93c708f6d7d7\") " pod="ai-tenant-e2e-shared-a-w3-57b6ce/sub-test-model-57b6ce-kserve-598ddfdfcf-xb6d6"
Sep 18 20:09:10.178781 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:09:10.178604    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fb103bd1-453b-40a1-96b6-93c708f6d7d7-home\") pod \"sub-test-model-57b6ce-kserve-598ddfdfcf-xb6d6\" (UID: \"fb103bd1-453b-40a1-96b6-93c708f6d7d7\") " pod="ai-tenant-e2e-shared-a-w3-57b6ce/sub-test-model-57b6ce-kserve-598ddfdfcf-xb6d6"
Sep 18 20:09:10.178781 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:09:10.178648    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fb103bd1-453b-40a1-96b6-93c708f6d7d7-model-cache\") pod \"sub-test-model-57b6ce-kserve-598ddfdfcf-xb6d6\" (UID: \"fb103bd1-453b-40a1-96b6-93c708f6d7d7\") " pod="ai-tenant-e2e-shared-a-w3-57b6ce/sub-test-model-57b6ce-kserve-598ddfdfcf-xb6d6"
Sep 18 20:09:10.178781 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:09:10.178675    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fb103bd1-453b-40a1-96b6-93c708f6d7d7-tls-certs\") pod \"sub-test-model-57b6ce-kserve-598ddfdfcf-xb6d6\" (UID: \"fb103bd1-453b-40a1-96b6-93c708f6d7d7\") " pod="ai-tenant-e2e-shared-a-w3-57b6ce/sub-test-model-57b6ce-kserve-598ddfdfcf-xb6d6"
Sep 18 20:09:10.178781 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:09:10.178699    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ml7q7\" (UniqueName: \"kubernetes.io/projected/fb103bd1-453b-40a1-96b6-93c708f6d7d7-kube-api-access-ml7q7\") pod \"sub-test-model-57b6ce-kserve-598ddfdfcf-xb6d6\" (UID: \"fb103bd1-453b-40a1-96b6-93c708f6d7d7\") " pod="ai-tenant-e2e-shared-a-w3-57b6ce/sub-test-model-57b6ce-kserve-598ddfdfcf-xb6d6"
Sep 18 20:09:10.279443 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:09:10.279403    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fb103bd1-453b-40a1-96b6-93c708f6d7d7-tmp-dir\") pod \"sub-test-model-57b6ce-kserve-598ddfdfcf-xb6d6\" (UID: \"fb103bd1-453b-40a1-96b6-93c708f6d7d7\") " pod="ai-tenant-e2e-shared-a-w3-57b6ce/sub-test-model-57b6ce-kserve-598ddfdfcf-xb6d6"
Sep 18 20:09:10.279443 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:09:10.279443    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fb103bd1-453b-40a1-96b6-93c708f6d7d7-dshm\") pod \"sub-test-model-57b6ce-kserve-598ddfdfcf-xb6d6\" (UID: \"fb103bd1-453b-40a1-96b6-93c708f6d7d7\") " pod="ai-tenant-e2e-shared-a-w3-57b6ce/sub-test-model-57b6ce-kserve-598ddfdfcf-xb6d6"
Sep 18 20:09:10.279696 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:09:10.279470    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fb103bd1-453b-40a1-96b6-93c708f6d7d7-home\") pod \"sub-test-model-57b6ce-kserve-598ddfdfcf-xb6d6\" (UID: \"fb103bd1-453b-40a1-96b6-93c708f6d7d7\") " pod="ai-tenant-e2e-shared-a-w3-57b6ce/sub-test-model-57b6ce-kserve-598ddfdfcf-xb6d6"
Sep 18 20:09:10.279696 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:09:10.279491    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fb103bd1-453b-40a1-96b6-93c708f6d7d7-model-cache\") pod \"sub-test-model-57b6ce-kserve-598ddfdfcf-xb6d6\" (UID: \"fb103bd1-453b-40a1-96b6-93c708f6d7d7\") " pod="ai-tenant-e2e-shared-a-w3-57b6ce/sub-test-model-57b6ce-kserve-598ddfdfcf-xb6d6"
Sep 18 20:09:10.279696 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:09:10.279603    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fb103bd1-453b-40a1-96b6-93c708f6d7d7-tls-certs\") pod \"sub-test-model-57b6ce-kserve-598ddfdfcf-xb6d6\" (UID: \"fb103bd1-453b-40a1-96b6-93c708f6d7d7\") " pod="ai-tenant-e2e-shared-a-w3-57b6ce/sub-test-model-57b6ce-kserve-598ddfdfcf-xb6d6"
Sep 18 20:09:10.279696 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:09:10.279649    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ml7q7\" (UniqueName: \"kubernetes.io/projected/fb103bd1-453b-40a1-96b6-93c708f6d7d7-kube-api-access-ml7q7\") pod \"sub-test-model-57b6ce-kserve-598ddfdfcf-xb6d6\" (UID: \"fb103bd1-453b-40a1-96b6-93c708f6d7d7\") " pod="ai-tenant-e2e-shared-a-w3-57b6ce/sub-test-model-57b6ce-kserve-598ddfdfcf-xb6d6"
Sep 18 20:09:10.279933 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:09:10.279890    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fb103bd1-453b-40a1-96b6-93c708f6d7d7-tmp-dir\") pod \"sub-test-model-57b6ce-kserve-598ddfdfcf-xb6d6\" (UID: \"fb103bd1-453b-40a1-96b6-93c708f6d7d7\") " pod="ai-tenant-e2e-shared-a-w3-57b6ce/sub-test-model-57b6ce-kserve-598ddfdfcf-xb6d6"
Sep 18 20:09:10.279997 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:09:10.279947    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fb103bd1-453b-40a1-96b6-93c708f6d7d7-model-cache\") pod \"sub-test-model-57b6ce-kserve-598ddfdfcf-xb6d6\" (UID: \"fb103bd1-453b-40a1-96b6-93c708f6d7d7\") " pod="ai-tenant-e2e-shared-a-w3-57b6ce/sub-test-model-57b6ce-kserve-598ddfdfcf-xb6d6"
Sep 18 20:09:10.279997 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:09:10.279982    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fb103bd1-453b-40a1-96b6-93c708f6d7d7-home\") pod \"sub-test-model-57b6ce-kserve-598ddfdfcf-xb6d6\" (UID: \"fb103bd1-453b-40a1-96b6-93c708f6d7d7\") " pod="ai-tenant-e2e-shared-a-w3-57b6ce/sub-test-model-57b6ce-kserve-598ddfdfcf-xb6d6"
Sep 18 20:09:10.281753 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:09:10.281711    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fb103bd1-453b-40a1-96b6-93c708f6d7d7-dshm\") pod \"sub-test-model-57b6ce-kserve-598ddfdfcf-xb6d6\" (UID: \"fb103bd1-453b-40a1-96b6-93c708f6d7d7\") " pod="ai-tenant-e2e-shared-a-w3-57b6ce/sub-test-model-57b6ce-kserve-598ddfdfcf-xb6d6"
Sep 18 20:09:10.282098 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:09:10.282080    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fb103bd1-453b-40a1-96b6-93c708f6d7d7-tls-certs\") pod \"sub-test-model-57b6ce-kserve-598ddfdfcf-xb6d6\" (UID: \"fb103bd1-453b-40a1-96b6-93c708f6d7d7\") " pod="ai-tenant-e2e-shared-a-w3-57b6ce/sub-test-model-57b6ce-kserve-598ddfdfcf-xb6d6"
Sep 18 20:09:10.286996 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:09:10.286969    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ml7q7\" (UniqueName: \"kubernetes.io/projected/fb103bd1-453b-40a1-96b6-93c708f6d7d7-kube-api-access-ml7q7\") pod \"sub-test-model-57b6ce-kserve-598ddfdfcf-xb6d6\" (UID: \"fb103bd1-453b-40a1-96b6-93c708f6d7d7\") " pod="ai-tenant-e2e-shared-a-w3-57b6ce/sub-test-model-57b6ce-kserve-598ddfdfcf-xb6d6"
Sep 18 20:09:10.381102 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:09:10.381015    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-57b6ce/sub-test-model-57b6ce-kserve-598ddfdfcf-xb6d6"
Sep 18 20:09:10.653714 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:09:10.653638    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-57b6ce/sub-test-model-57b6ce-kserve-598ddfdfcf-xb6d6"]
Sep 18 20:09:10.656371 ip-10-0-136-185 kubenswrapper[2585]: W0918 20:09:10.656343    2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podfb103bd1_453b_40a1_96b6_93c708f6d7d7.slice/crio-d9b983b4a85acf4c46bb8f0a9247c179fa13ce97f6ad4ef4d06f8ede279d6b07 WatchSource:0}: Error finding container d9b983b4a85acf4c46bb8f0a9247c179fa13ce97f6ad4ef4d06f8ede279d6b07: Status 404 returned error can't find the container with id d9b983b4a85acf4c46bb8f0a9247c179fa13ce97f6ad4ef4d06f8ede279d6b07
Sep 18 20:09:10.672725 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:09:10.672698    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-xmwh7"]
Sep 18 20:09:11.130338 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:09:11.130252    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-57b6ce/sub-test-model-57b6ce-kserve-598ddfdfcf-xb6d6" event={"ID":"fb103bd1-453b-40a1-96b6-93c708f6d7d7","Type":"ContainerStarted","Data":"f3469b58ad5d159064cf778d43751c887072419037c1100fddb6d5a88a324d74"}
Sep 18 20:09:11.130338 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:09:11.130294    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-57b6ce/sub-test-model-57b6ce-kserve-598ddfdfcf-xb6d6" event={"ID":"fb103bd1-453b-40a1-96b6-93c708f6d7d7","Type":"ContainerStarted","Data":"d9b983b4a85acf4c46bb8f0a9247c179fa13ce97f6ad4ef4d06f8ede279d6b07"}
Sep 18 20:09:11.147004 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:09:11.146946    2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w3-57b6ce/sub-test-model-57b6ce-kserve-598ddfdfcf-xb6d6" podStartSLOduration=1.14692856 podStartE2EDuration="1.14692856s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 20:09:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 20:09:11.146596348 +0000 UTC m=+1792.671201772" watchObservedRunningTime="2026-09-18 20:09:11.14692856 +0000 UTC m=+1792.671533983"
Sep 18 20:09:20.381746 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:09:20.381696    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w3-57b6ce/sub-test-model-57b6ce-kserve-598ddfdfcf-xb6d6"
Sep 18 20:09:20.382343 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:09:20.381764    2585 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w3-57b6ce/sub-test-model-57b6ce-kserve-598ddfdfcf-xb6d6"
Sep 18 20:09:20.394419 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:09:20.394390    2585 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w3-57b6ce/sub-test-model-57b6ce-kserve-598ddfdfcf-xb6d6"
Sep 18 20:09:20.689743 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:09:20.689703    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-rctld_cc0e0de2-75d5-43a1-b64b-b071427cd8ce/ovn-acl-logging/0.log"
Sep 18 20:09:20.700067 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:09:20.700042    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-rctld_cc0e0de2-75d5-43a1-b64b-b071427cd8ce/ovn-acl-logging/0.log"
Sep 18 20:09:21.008594 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:09:21.008500    2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w3-57b6ce/auth-test-model-57b6ce-kserve-66868b47d7-xkkds" podUID="dc70d7c6-212a-403f-8204-d18d36daa4dd" containerName="main" containerID="cri-o://bbfc9502ef9c55652574bdab451a058b2a04c342881ad099ab90ef6dadbd22f1" gracePeriod=45
Sep 18 20:09:21.183994 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:09:21.183961    2585 generic.go:356] "Generic (PLEG): container finished" podID="dc70d7c6-212a-403f-8204-d18d36daa4dd" containerID="bbfc9502ef9c55652574bdab451a058b2a04c342881ad099ab90ef6dadbd22f1" exitCode=0
Sep 18 20:09:21.184157 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:09:21.184103    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-57b6ce/auth-test-model-57b6ce-kserve-66868b47d7-xkkds" event={"ID":"dc70d7c6-212a-403f-8204-d18d36daa4dd","Type":"ContainerDied","Data":"bbfc9502ef9c55652574bdab451a058b2a04c342881ad099ab90ef6dadbd22f1"}
Sep 18 20:09:21.197855 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:09:21.197826    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w3-57b6ce/sub-test-model-57b6ce-kserve-598ddfdfcf-xb6d6"
Sep 18 20:09:21.257179 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:09:21.257152    2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-57b6ce/auth-test-model-57b6ce-kserve-66868b47d7-xkkds"
Sep 18 20:09:21.373384 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:09:21.373299    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-xmwh7"]
Sep 18 20:09:21.382556 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:09:21.382531    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-498l8\" (UniqueName: \"kubernetes.io/projected/dc70d7c6-212a-403f-8204-d18d36daa4dd-kube-api-access-498l8\") pod \"dc70d7c6-212a-403f-8204-d18d36daa4dd\" (UID: \"dc70d7c6-212a-403f-8204-d18d36daa4dd\") "
Sep 18 20:09:21.382972 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:09:21.382568    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dc70d7c6-212a-403f-8204-d18d36daa4dd-dshm\") pod \"dc70d7c6-212a-403f-8204-d18d36daa4dd\" (UID: \"dc70d7c6-212a-403f-8204-d18d36daa4dd\") "
Sep 18 20:09:21.382972 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:09:21.382590    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dc70d7c6-212a-403f-8204-d18d36daa4dd-model-cache\") pod \"dc70d7c6-212a-403f-8204-d18d36daa4dd\" (UID: \"dc70d7c6-212a-403f-8204-d18d36daa4dd\") "
Sep 18 20:09:21.382972 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:09:21.382613    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dc70d7c6-212a-403f-8204-d18d36daa4dd-home\") pod \"dc70d7c6-212a-403f-8204-d18d36daa4dd\" (UID: \"dc70d7c6-212a-403f-8204-d18d36daa4dd\") "
Sep 18 20:09:21.382972 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:09:21.382649    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dc70d7c6-212a-403f-8204-d18d36daa4dd-tls-certs\") pod \"dc70d7c6-212a-403f-8204-d18d36daa4dd\" (UID: \"dc70d7c6-212a-403f-8204-d18d36daa4dd\") "
Sep 18 20:09:21.382972 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:09:21.382702    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dc70d7c6-212a-403f-8204-d18d36daa4dd-tmp-dir\") pod \"dc70d7c6-212a-403f-8204-d18d36daa4dd\" (UID: \"dc70d7c6-212a-403f-8204-d18d36daa4dd\") "
Sep 18 20:09:21.382972 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:09:21.382853    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dc70d7c6-212a-403f-8204-d18d36daa4dd-model-cache" (OuterVolumeSpecName: "model-cache") pod "dc70d7c6-212a-403f-8204-d18d36daa4dd" (UID: "dc70d7c6-212a-403f-8204-d18d36daa4dd"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 20:09:21.382972 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:09:21.382930    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dc70d7c6-212a-403f-8204-d18d36daa4dd-home" (OuterVolumeSpecName: "home") pod "dc70d7c6-212a-403f-8204-d18d36daa4dd" (UID: "dc70d7c6-212a-403f-8204-d18d36daa4dd"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 20:09:21.383306 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:09:21.383014    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dc70d7c6-212a-403f-8204-d18d36daa4dd-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "dc70d7c6-212a-403f-8204-d18d36daa4dd" (UID: "dc70d7c6-212a-403f-8204-d18d36daa4dd"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 20:09:21.383306 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:09:21.383100    2585 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dc70d7c6-212a-403f-8204-d18d36daa4dd-model-cache\") on node \"ip-10-0-136-185.ec2.internal\" DevicePath \"\""
Sep 18 20:09:21.383306 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:09:21.383123    2585 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dc70d7c6-212a-403f-8204-d18d36daa4dd-home\") on node \"ip-10-0-136-185.ec2.internal\" DevicePath \"\""
Sep 18 20:09:21.383306 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:09:21.383138    2585 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dc70d7c6-212a-403f-8204-d18d36daa4dd-tmp-dir\") on node \"ip-10-0-136-185.ec2.internal\" DevicePath \"\""
Sep 18 20:09:21.384957 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:09:21.384925    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dc70d7c6-212a-403f-8204-d18d36daa4dd-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "dc70d7c6-212a-403f-8204-d18d36daa4dd" (UID: "dc70d7c6-212a-403f-8204-d18d36daa4dd"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 20:09:21.384957 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:09:21.384937    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dc70d7c6-212a-403f-8204-d18d36daa4dd-kube-api-access-498l8" (OuterVolumeSpecName: "kube-api-access-498l8") pod "dc70d7c6-212a-403f-8204-d18d36daa4dd" (UID: "dc70d7c6-212a-403f-8204-d18d36daa4dd"). InnerVolumeSpecName "kube-api-access-498l8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 20:09:21.385084 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:09:21.384999    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dc70d7c6-212a-403f-8204-d18d36daa4dd-dshm" (OuterVolumeSpecName: "dshm") pod "dc70d7c6-212a-403f-8204-d18d36daa4dd" (UID: "dc70d7c6-212a-403f-8204-d18d36daa4dd"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 20:09:21.484679 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:09:21.484643    2585 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-498l8\" (UniqueName: \"kubernetes.io/projected/dc70d7c6-212a-403f-8204-d18d36daa4dd-kube-api-access-498l8\") on node \"ip-10-0-136-185.ec2.internal\" DevicePath \"\""
Sep 18 20:09:21.484679 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:09:21.484679    2585 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dc70d7c6-212a-403f-8204-d18d36daa4dd-dshm\") on node \"ip-10-0-136-185.ec2.internal\" DevicePath \"\""
Sep 18 20:09:21.484931 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:09:21.484694    2585 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dc70d7c6-212a-403f-8204-d18d36daa4dd-tls-certs\") on node \"ip-10-0-136-185.ec2.internal\" DevicePath \"\""
Sep 18 20:09:22.190552 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:09:22.190466    2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-57b6ce/auth-test-model-57b6ce-kserve-66868b47d7-xkkds"
Sep 18 20:09:22.190552 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:09:22.190509    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-57b6ce/auth-test-model-57b6ce-kserve-66868b47d7-xkkds" event={"ID":"dc70d7c6-212a-403f-8204-d18d36daa4dd","Type":"ContainerDied","Data":"ddfc417fa04a16af1cd9bc4a3e8ffb3d82195d0e973cc9f0431e2f5c1b1e52c4"}
Sep 18 20:09:22.190764 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:09:22.190567    2585 scope.go:117] "RemoveContainer" containerID="bbfc9502ef9c55652574bdab451a058b2a04c342881ad099ab90ef6dadbd22f1"
Sep 18 20:09:22.212358 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:09:22.212332    2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-57b6ce/auth-test-model-57b6ce-kserve-66868b47d7-xkkds"]
Sep 18 20:09:22.217742 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:09:22.217719    2585 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w3-57b6ce/auth-test-model-57b6ce-kserve-66868b47d7-xkkds"]
Sep 18 20:09:22.666111 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:09:22.666077    2585 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="dc70d7c6-212a-403f-8204-d18d36daa4dd" path="/var/lib/kubelet/pods/dc70d7c6-212a-403f-8204-d18d36daa4dd/volumes"
Sep 18 20:09:41.566724 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:09:41.566682    2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w3-5547b6/sub-test-model-5547b6-kserve-575b8546df-q99pl"]
Sep 18 20:09:41.567142 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:09:41.567088    2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dc70d7c6-212a-403f-8204-d18d36daa4dd" containerName="main"
Sep 18 20:09:41.567142 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:09:41.567102    2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="dc70d7c6-212a-403f-8204-d18d36daa4dd" containerName="main"
Sep 18 20:09:41.567222 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:09:41.567183    2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="dc70d7c6-212a-403f-8204-d18d36daa4dd" containerName="main"
Sep 18 20:09:41.571484 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:09:41.571466    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-5547b6/sub-test-model-5547b6-kserve-575b8546df-q99pl"
Sep 18 20:09:41.576421 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:09:41.576397    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-5547b6\"/\"kube-root-ca.crt\""
Sep 18 20:09:41.576421 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:09:41.576412    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-5547b6\"/\"default-dockercfg-fsn7d\""
Sep 18 20:09:41.576825 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:09:41.576803    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-5547b6\"/\"openshift-service-ca.crt\""
Sep 18 20:09:41.576958 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:09:41.576810    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-5547b6\"/\"sub-test-model-5547b6-kserve-self-signed-certs\""
Sep 18 20:09:41.596534 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:09:41.596501    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-5547b6/sub-test-model-5547b6-kserve-575b8546df-q99pl"]
Sep 18 20:09:41.666496 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:09:41.666450    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/62e248cd-e0f0-4ea3-a044-2aa5f1261e4c-tmp-dir\") pod \"sub-test-model-5547b6-kserve-575b8546df-q99pl\" (UID: \"62e248cd-e0f0-4ea3-a044-2aa5f1261e4c\") " pod="ai-tenant-e2e-shared-b-w3-5547b6/sub-test-model-5547b6-kserve-575b8546df-q99pl"
Sep 18 20:09:41.666689 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:09:41.666504    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qjfrz\" (UniqueName: \"kubernetes.io/projected/62e248cd-e0f0-4ea3-a044-2aa5f1261e4c-kube-api-access-qjfrz\") pod \"sub-test-model-5547b6-kserve-575b8546df-q99pl\" (UID: \"62e248cd-e0f0-4ea3-a044-2aa5f1261e4c\") " pod="ai-tenant-e2e-shared-b-w3-5547b6/sub-test-model-5547b6-kserve-575b8546df-q99pl"
Sep 18 20:09:41.666689 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:09:41.666530    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/62e248cd-e0f0-4ea3-a044-2aa5f1261e4c-dshm\") pod \"sub-test-model-5547b6-kserve-575b8546df-q99pl\" (UID: \"62e248cd-e0f0-4ea3-a044-2aa5f1261e4c\") " pod="ai-tenant-e2e-shared-b-w3-5547b6/sub-test-model-5547b6-kserve-575b8546df-q99pl"
Sep 18 20:09:41.666689 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:09:41.666583    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/62e248cd-e0f0-4ea3-a044-2aa5f1261e4c-tls-certs\") pod \"sub-test-model-5547b6-kserve-575b8546df-q99pl\" (UID: \"62e248cd-e0f0-4ea3-a044-2aa5f1261e4c\") " pod="ai-tenant-e2e-shared-b-w3-5547b6/sub-test-model-5547b6-kserve-575b8546df-q99pl"
Sep 18 20:09:41.666689 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:09:41.666645    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/62e248cd-e0f0-4ea3-a044-2aa5f1261e4c-model-cache\") pod \"sub-test-model-5547b6-kserve-575b8546df-q99pl\" (UID: \"62e248cd-e0f0-4ea3-a044-2aa5f1261e4c\") " pod="ai-tenant-e2e-shared-b-w3-5547b6/sub-test-model-5547b6-kserve-575b8546df-q99pl"
Sep 18 20:09:41.666884 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:09:41.666692    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/62e248cd-e0f0-4ea3-a044-2aa5f1261e4c-home\") pod \"sub-test-model-5547b6-kserve-575b8546df-q99pl\" (UID: \"62e248cd-e0f0-4ea3-a044-2aa5f1261e4c\") " pod="ai-tenant-e2e-shared-b-w3-5547b6/sub-test-model-5547b6-kserve-575b8546df-q99pl"
Sep 18 20:09:41.767703 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:09:41.767661    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/62e248cd-e0f0-4ea3-a044-2aa5f1261e4c-tmp-dir\") pod \"sub-test-model-5547b6-kserve-575b8546df-q99pl\" (UID: \"62e248cd-e0f0-4ea3-a044-2aa5f1261e4c\") " pod="ai-tenant-e2e-shared-b-w3-5547b6/sub-test-model-5547b6-kserve-575b8546df-q99pl"
Sep 18 20:09:41.767925 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:09:41.767714    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qjfrz\" (UniqueName: \"kubernetes.io/projected/62e248cd-e0f0-4ea3-a044-2aa5f1261e4c-kube-api-access-qjfrz\") pod \"sub-test-model-5547b6-kserve-575b8546df-q99pl\" (UID: \"62e248cd-e0f0-4ea3-a044-2aa5f1261e4c\") " pod="ai-tenant-e2e-shared-b-w3-5547b6/sub-test-model-5547b6-kserve-575b8546df-q99pl"
Sep 18 20:09:41.767996 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:09:41.767949    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/62e248cd-e0f0-4ea3-a044-2aa5f1261e4c-dshm\") pod \"sub-test-model-5547b6-kserve-575b8546df-q99pl\" (UID: \"62e248cd-e0f0-4ea3-a044-2aa5f1261e4c\") " pod="ai-tenant-e2e-shared-b-w3-5547b6/sub-test-model-5547b6-kserve-575b8546df-q99pl"
Sep 18 20:09:41.768059 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:09:41.768044    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/62e248cd-e0f0-4ea3-a044-2aa5f1261e4c-tls-certs\") pod \"sub-test-model-5547b6-kserve-575b8546df-q99pl\" (UID: \"62e248cd-e0f0-4ea3-a044-2aa5f1261e4c\") " pod="ai-tenant-e2e-shared-b-w3-5547b6/sub-test-model-5547b6-kserve-575b8546df-q99pl"
Sep 18 20:09:41.768119 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:09:41.768091    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/62e248cd-e0f0-4ea3-a044-2aa5f1261e4c-model-cache\") pod \"sub-test-model-5547b6-kserve-575b8546df-q99pl\" (UID: \"62e248cd-e0f0-4ea3-a044-2aa5f1261e4c\") " pod="ai-tenant-e2e-shared-b-w3-5547b6/sub-test-model-5547b6-kserve-575b8546df-q99pl"
Sep 18 20:09:41.768119 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:09:41.768048    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/62e248cd-e0f0-4ea3-a044-2aa5f1261e4c-tmp-dir\") pod \"sub-test-model-5547b6-kserve-575b8546df-q99pl\" (UID: \"62e248cd-e0f0-4ea3-a044-2aa5f1261e4c\") " pod="ai-tenant-e2e-shared-b-w3-5547b6/sub-test-model-5547b6-kserve-575b8546df-q99pl"
Sep 18 20:09:41.768230 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:09:41.768134    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/62e248cd-e0f0-4ea3-a044-2aa5f1261e4c-home\") pod \"sub-test-model-5547b6-kserve-575b8546df-q99pl\" (UID: \"62e248cd-e0f0-4ea3-a044-2aa5f1261e4c\") " pod="ai-tenant-e2e-shared-b-w3-5547b6/sub-test-model-5547b6-kserve-575b8546df-q99pl"
Sep 18 20:09:41.768404 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:09:41.768384    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/62e248cd-e0f0-4ea3-a044-2aa5f1261e4c-model-cache\") pod \"sub-test-model-5547b6-kserve-575b8546df-q99pl\" (UID: \"62e248cd-e0f0-4ea3-a044-2aa5f1261e4c\") " pod="ai-tenant-e2e-shared-b-w3-5547b6/sub-test-model-5547b6-kserve-575b8546df-q99pl"
Sep 18 20:09:41.768487 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:09:41.768470    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/62e248cd-e0f0-4ea3-a044-2aa5f1261e4c-home\") pod \"sub-test-model-5547b6-kserve-575b8546df-q99pl\" (UID: \"62e248cd-e0f0-4ea3-a044-2aa5f1261e4c\") " pod="ai-tenant-e2e-shared-b-w3-5547b6/sub-test-model-5547b6-kserve-575b8546df-q99pl"
Sep 18 20:09:41.770075 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:09:41.770050    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/62e248cd-e0f0-4ea3-a044-2aa5f1261e4c-dshm\") pod \"sub-test-model-5547b6-kserve-575b8546df-q99pl\" (UID: \"62e248cd-e0f0-4ea3-a044-2aa5f1261e4c\") " pod="ai-tenant-e2e-shared-b-w3-5547b6/sub-test-model-5547b6-kserve-575b8546df-q99pl"
Sep 18 20:09:41.770553 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:09:41.770528    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/62e248cd-e0f0-4ea3-a044-2aa5f1261e4c-tls-certs\") pod \"sub-test-model-5547b6-kserve-575b8546df-q99pl\" (UID: \"62e248cd-e0f0-4ea3-a044-2aa5f1261e4c\") " pod="ai-tenant-e2e-shared-b-w3-5547b6/sub-test-model-5547b6-kserve-575b8546df-q99pl"
Sep 18 20:09:41.784303 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:09:41.784274    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qjfrz\" (UniqueName: \"kubernetes.io/projected/62e248cd-e0f0-4ea3-a044-2aa5f1261e4c-kube-api-access-qjfrz\") pod \"sub-test-model-5547b6-kserve-575b8546df-q99pl\" (UID: \"62e248cd-e0f0-4ea3-a044-2aa5f1261e4c\") " pod="ai-tenant-e2e-shared-b-w3-5547b6/sub-test-model-5547b6-kserve-575b8546df-q99pl"
Sep 18 20:09:41.880479 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:09:41.880389    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-5547b6/sub-test-model-5547b6-kserve-575b8546df-q99pl"
Sep 18 20:09:42.151668 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:09:42.151628    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-5547b6/sub-test-model-5547b6-kserve-575b8546df-q99pl"]
Sep 18 20:09:42.155064 ip-10-0-136-185 kubenswrapper[2585]: W0918 20:09:42.155031    2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod62e248cd_e0f0_4ea3_a044_2aa5f1261e4c.slice/crio-e52d4936fe62cfa6691775f135c1d6e1565b15fede4dd36947786668ae3bd6e5 WatchSource:0}: Error finding container e52d4936fe62cfa6691775f135c1d6e1565b15fede4dd36947786668ae3bd6e5: Status 404 returned error can't find the container with id e52d4936fe62cfa6691775f135c1d6e1565b15fede4dd36947786668ae3bd6e5
Sep 18 20:09:42.290066 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:09:42.290027    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-5547b6/sub-test-model-5547b6-kserve-575b8546df-q99pl" event={"ID":"62e248cd-e0f0-4ea3-a044-2aa5f1261e4c","Type":"ContainerStarted","Data":"e1ea7c65416ed2b5b33314ba27ef2413b149f150992d11edf04fe98850ed8fab"}
Sep 18 20:09:42.290262 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:09:42.290076    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-5547b6/sub-test-model-5547b6-kserve-575b8546df-q99pl" event={"ID":"62e248cd-e0f0-4ea3-a044-2aa5f1261e4c","Type":"ContainerStarted","Data":"e52d4936fe62cfa6691775f135c1d6e1565b15fede4dd36947786668ae3bd6e5"}
Sep 18 20:09:42.307840 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:09:42.307767    2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w3-5547b6/sub-test-model-5547b6-kserve-575b8546df-q99pl" podStartSLOduration=1.3077502650000001 podStartE2EDuration="1.307750265s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 20:09:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 20:09:42.305643113 +0000 UTC m=+1823.830248536" watchObservedRunningTime="2026-09-18 20:09:42.307750265 +0000 UTC m=+1823.832355688"
Sep 18 20:09:45.166591 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:09:45.166550    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-xmwh7"]
Sep 18 20:09:51.881144 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:09:51.881099    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w3-5547b6/sub-test-model-5547b6-kserve-575b8546df-q99pl"
Sep 18 20:09:51.881642 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:09:51.881394    2585 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w3-5547b6/sub-test-model-5547b6-kserve-575b8546df-q99pl"
Sep 18 20:09:51.893805 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:09:51.893762    2585 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w3-5547b6/sub-test-model-5547b6-kserve-575b8546df-q99pl"
Sep 18 20:09:52.348219 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:09:52.348190    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w3-5547b6/sub-test-model-5547b6-kserve-575b8546df-q99pl"
Sep 18 20:09:59.782067 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:09:59.782030    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-xmwh7"]
Sep 18 20:10:12.789345 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:10:12.789303    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-xmwh7"]
Sep 18 20:10:24.119191 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:10:24.119156    2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-57b6ce/sub-test-model-57b6ce-kserve-598ddfdfcf-xb6d6"]
Sep 18 20:10:25.181626 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:10:25.181581    2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-5547b6/sub-test-model-5547b6-kserve-575b8546df-q99pl"]
Sep 18 20:10:27.466766 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:10:27.466727    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-xmwh7"]
Sep 18 20:10:39.137847 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:10:39.137754    2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w3-57b6ce/sub-test-model-57b6ce-kserve-598ddfdfcf-xb6d6" podUID="fb103bd1-453b-40a1-96b6-93c708f6d7d7" containerName="main" containerID="cri-o://f3469b58ad5d159064cf778d43751c887072419037c1100fddb6d5a88a324d74" gracePeriod=45
Sep 18 20:10:39.385166 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:10:39.385140    2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-57b6ce/sub-test-model-57b6ce-kserve-598ddfdfcf-xb6d6"
Sep 18 20:10:39.498870 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:10:39.498833    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fb103bd1-453b-40a1-96b6-93c708f6d7d7-home\") pod \"fb103bd1-453b-40a1-96b6-93c708f6d7d7\" (UID: \"fb103bd1-453b-40a1-96b6-93c708f6d7d7\") "
Sep 18 20:10:39.498870 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:10:39.498872    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fb103bd1-453b-40a1-96b6-93c708f6d7d7-model-cache\") pod \"fb103bd1-453b-40a1-96b6-93c708f6d7d7\" (UID: \"fb103bd1-453b-40a1-96b6-93c708f6d7d7\") "
Sep 18 20:10:39.499131 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:10:39.498923    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ml7q7\" (UniqueName: \"kubernetes.io/projected/fb103bd1-453b-40a1-96b6-93c708f6d7d7-kube-api-access-ml7q7\") pod \"fb103bd1-453b-40a1-96b6-93c708f6d7d7\" (UID: \"fb103bd1-453b-40a1-96b6-93c708f6d7d7\") "
Sep 18 20:10:39.499131 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:10:39.498970    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fb103bd1-453b-40a1-96b6-93c708f6d7d7-tls-certs\") pod \"fb103bd1-453b-40a1-96b6-93c708f6d7d7\" (UID: \"fb103bd1-453b-40a1-96b6-93c708f6d7d7\") "
Sep 18 20:10:39.499131 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:10:39.499003    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fb103bd1-453b-40a1-96b6-93c708f6d7d7-tmp-dir\") pod \"fb103bd1-453b-40a1-96b6-93c708f6d7d7\" (UID: \"fb103bd1-453b-40a1-96b6-93c708f6d7d7\") "
Sep 18 20:10:39.499333 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:10:39.499141    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fb103bd1-453b-40a1-96b6-93c708f6d7d7-dshm\") pod \"fb103bd1-453b-40a1-96b6-93c708f6d7d7\" (UID: \"fb103bd1-453b-40a1-96b6-93c708f6d7d7\") "
Sep 18 20:10:39.499333 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:10:39.499190    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fb103bd1-453b-40a1-96b6-93c708f6d7d7-home" (OuterVolumeSpecName: "home") pod "fb103bd1-453b-40a1-96b6-93c708f6d7d7" (UID: "fb103bd1-453b-40a1-96b6-93c708f6d7d7"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 20:10:39.499333 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:10:39.499233    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fb103bd1-453b-40a1-96b6-93c708f6d7d7-model-cache" (OuterVolumeSpecName: "model-cache") pod "fb103bd1-453b-40a1-96b6-93c708f6d7d7" (UID: "fb103bd1-453b-40a1-96b6-93c708f6d7d7"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 20:10:39.499333 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:10:39.499250    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fb103bd1-453b-40a1-96b6-93c708f6d7d7-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "fb103bd1-453b-40a1-96b6-93c708f6d7d7" (UID: "fb103bd1-453b-40a1-96b6-93c708f6d7d7"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 20:10:39.499545 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:10:39.499444    2585 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fb103bd1-453b-40a1-96b6-93c708f6d7d7-tmp-dir\") on node \"ip-10-0-136-185.ec2.internal\" DevicePath \"\""
Sep 18 20:10:39.499545 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:10:39.499458    2585 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fb103bd1-453b-40a1-96b6-93c708f6d7d7-home\") on node \"ip-10-0-136-185.ec2.internal\" DevicePath \"\""
Sep 18 20:10:39.499545 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:10:39.499466    2585 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fb103bd1-453b-40a1-96b6-93c708f6d7d7-model-cache\") on node \"ip-10-0-136-185.ec2.internal\" DevicePath \"\""
Sep 18 20:10:39.501148 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:10:39.501120    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fb103bd1-453b-40a1-96b6-93c708f6d7d7-dshm" (OuterVolumeSpecName: "dshm") pod "fb103bd1-453b-40a1-96b6-93c708f6d7d7" (UID: "fb103bd1-453b-40a1-96b6-93c708f6d7d7"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 20:10:39.501148 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:10:39.501126    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fb103bd1-453b-40a1-96b6-93c708f6d7d7-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "fb103bd1-453b-40a1-96b6-93c708f6d7d7" (UID: "fb103bd1-453b-40a1-96b6-93c708f6d7d7"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 20:10:39.501286 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:10:39.501147    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fb103bd1-453b-40a1-96b6-93c708f6d7d7-kube-api-access-ml7q7" (OuterVolumeSpecName: "kube-api-access-ml7q7") pod "fb103bd1-453b-40a1-96b6-93c708f6d7d7" (UID: "fb103bd1-453b-40a1-96b6-93c708f6d7d7"). InnerVolumeSpecName "kube-api-access-ml7q7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 20:10:39.575939 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:10:39.575905    2585 generic.go:356] "Generic (PLEG): container finished" podID="fb103bd1-453b-40a1-96b6-93c708f6d7d7" containerID="f3469b58ad5d159064cf778d43751c887072419037c1100fddb6d5a88a324d74" exitCode=0
Sep 18 20:10:39.576094 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:10:39.575948    2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-57b6ce/sub-test-model-57b6ce-kserve-598ddfdfcf-xb6d6"
Sep 18 20:10:39.576094 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:10:39.575973    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-57b6ce/sub-test-model-57b6ce-kserve-598ddfdfcf-xb6d6" event={"ID":"fb103bd1-453b-40a1-96b6-93c708f6d7d7","Type":"ContainerDied","Data":"f3469b58ad5d159064cf778d43751c887072419037c1100fddb6d5a88a324d74"}
Sep 18 20:10:39.576094 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:10:39.576007    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-57b6ce/sub-test-model-57b6ce-kserve-598ddfdfcf-xb6d6" event={"ID":"fb103bd1-453b-40a1-96b6-93c708f6d7d7","Type":"ContainerDied","Data":"d9b983b4a85acf4c46bb8f0a9247c179fa13ce97f6ad4ef4d06f8ede279d6b07"}
Sep 18 20:10:39.576094 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:10:39.576024    2585 scope.go:117] "RemoveContainer" containerID="f3469b58ad5d159064cf778d43751c887072419037c1100fddb6d5a88a324d74"
Sep 18 20:10:39.585048 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:10:39.585029    2585 scope.go:117] "RemoveContainer" containerID="f3469b58ad5d159064cf778d43751c887072419037c1100fddb6d5a88a324d74"
Sep 18 20:10:39.585299 ip-10-0-136-185 kubenswrapper[2585]: E0918 20:10:39.585278    2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f3469b58ad5d159064cf778d43751c887072419037c1100fddb6d5a88a324d74\": container with ID starting with f3469b58ad5d159064cf778d43751c887072419037c1100fddb6d5a88a324d74 not found: ID does not exist" containerID="f3469b58ad5d159064cf778d43751c887072419037c1100fddb6d5a88a324d74"
Sep 18 20:10:39.585379 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:10:39.585307    2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f3469b58ad5d159064cf778d43751c887072419037c1100fddb6d5a88a324d74"} err="failed to get container status \"f3469b58ad5d159064cf778d43751c887072419037c1100fddb6d5a88a324d74\": rpc error: code = NotFound desc = could not find container \"f3469b58ad5d159064cf778d43751c887072419037c1100fddb6d5a88a324d74\": container with ID starting with f3469b58ad5d159064cf778d43751c887072419037c1100fddb6d5a88a324d74 not found: ID does not exist"
Sep 18 20:10:39.597088 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:10:39.597065    2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-57b6ce/sub-test-model-57b6ce-kserve-598ddfdfcf-xb6d6"]
Sep 18 20:10:39.600092 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:10:39.600070    2585 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ml7q7\" (UniqueName: \"kubernetes.io/projected/fb103bd1-453b-40a1-96b6-93c708f6d7d7-kube-api-access-ml7q7\") on node \"ip-10-0-136-185.ec2.internal\" DevicePath \"\""
Sep 18 20:10:39.600242 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:10:39.600095    2585 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fb103bd1-453b-40a1-96b6-93c708f6d7d7-tls-certs\") on node \"ip-10-0-136-185.ec2.internal\" DevicePath \"\""
Sep 18 20:10:39.600242 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:10:39.600109    2585 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fb103bd1-453b-40a1-96b6-93c708f6d7d7-dshm\") on node \"ip-10-0-136-185.ec2.internal\" DevicePath \"\""
Sep 18 20:10:39.601880 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:10:39.601861    2585 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w3-57b6ce/sub-test-model-57b6ce-kserve-598ddfdfcf-xb6d6"]
Sep 18 20:10:40.201312 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:10:40.201266    2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w3-5547b6/sub-test-model-5547b6-kserve-575b8546df-q99pl" podUID="62e248cd-e0f0-4ea3-a044-2aa5f1261e4c" containerName="main" containerID="cri-o://e1ea7c65416ed2b5b33314ba27ef2413b149f150992d11edf04fe98850ed8fab" gracePeriod=45
Sep 18 20:10:40.452122 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:10:40.452066    2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-5547b6/sub-test-model-5547b6-kserve-575b8546df-q99pl"
Sep 18 20:10:40.582975 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:10:40.582936    2585 generic.go:356] "Generic (PLEG): container finished" podID="62e248cd-e0f0-4ea3-a044-2aa5f1261e4c" containerID="e1ea7c65416ed2b5b33314ba27ef2413b149f150992d11edf04fe98850ed8fab" exitCode=0
Sep 18 20:10:40.583155 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:10:40.582982    2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-5547b6/sub-test-model-5547b6-kserve-575b8546df-q99pl"
Sep 18 20:10:40.583155 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:10:40.583017    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-5547b6/sub-test-model-5547b6-kserve-575b8546df-q99pl" event={"ID":"62e248cd-e0f0-4ea3-a044-2aa5f1261e4c","Type":"ContainerDied","Data":"e1ea7c65416ed2b5b33314ba27ef2413b149f150992d11edf04fe98850ed8fab"}
Sep 18 20:10:40.583155 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:10:40.583061    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-5547b6/sub-test-model-5547b6-kserve-575b8546df-q99pl" event={"ID":"62e248cd-e0f0-4ea3-a044-2aa5f1261e4c","Type":"ContainerDied","Data":"e52d4936fe62cfa6691775f135c1d6e1565b15fede4dd36947786668ae3bd6e5"}
Sep 18 20:10:40.583155 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:10:40.583083    2585 scope.go:117] "RemoveContainer" containerID="e1ea7c65416ed2b5b33314ba27ef2413b149f150992d11edf04fe98850ed8fab"
Sep 18 20:10:40.591172 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:10:40.591155    2585 scope.go:117] "RemoveContainer" containerID="e1ea7c65416ed2b5b33314ba27ef2413b149f150992d11edf04fe98850ed8fab"
Sep 18 20:10:40.591442 ip-10-0-136-185 kubenswrapper[2585]: E0918 20:10:40.591424    2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e1ea7c65416ed2b5b33314ba27ef2413b149f150992d11edf04fe98850ed8fab\": container with ID starting with e1ea7c65416ed2b5b33314ba27ef2413b149f150992d11edf04fe98850ed8fab not found: ID does not exist" containerID="e1ea7c65416ed2b5b33314ba27ef2413b149f150992d11edf04fe98850ed8fab"
Sep 18 20:10:40.591482 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:10:40.591452    2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e1ea7c65416ed2b5b33314ba27ef2413b149f150992d11edf04fe98850ed8fab"} err="failed to get container status \"e1ea7c65416ed2b5b33314ba27ef2413b149f150992d11edf04fe98850ed8fab\": rpc error: code = NotFound desc = could not find container \"e1ea7c65416ed2b5b33314ba27ef2413b149f150992d11edf04fe98850ed8fab\": container with ID starting with e1ea7c65416ed2b5b33314ba27ef2413b149f150992d11edf04fe98850ed8fab not found: ID does not exist"
Sep 18 20:10:40.609249 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:10:40.609214    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/62e248cd-e0f0-4ea3-a044-2aa5f1261e4c-tls-certs\") pod \"62e248cd-e0f0-4ea3-a044-2aa5f1261e4c\" (UID: \"62e248cd-e0f0-4ea3-a044-2aa5f1261e4c\") "
Sep 18 20:10:40.609379 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:10:40.609301    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/62e248cd-e0f0-4ea3-a044-2aa5f1261e4c-model-cache\") pod \"62e248cd-e0f0-4ea3-a044-2aa5f1261e4c\" (UID: \"62e248cd-e0f0-4ea3-a044-2aa5f1261e4c\") "
Sep 18 20:10:40.609379 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:10:40.609352    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qjfrz\" (UniqueName: \"kubernetes.io/projected/62e248cd-e0f0-4ea3-a044-2aa5f1261e4c-kube-api-access-qjfrz\") pod \"62e248cd-e0f0-4ea3-a044-2aa5f1261e4c\" (UID: \"62e248cd-e0f0-4ea3-a044-2aa5f1261e4c\") "
Sep 18 20:10:40.609487 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:10:40.609384    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/62e248cd-e0f0-4ea3-a044-2aa5f1261e4c-dshm\") pod \"62e248cd-e0f0-4ea3-a044-2aa5f1261e4c\" (UID: \"62e248cd-e0f0-4ea3-a044-2aa5f1261e4c\") "
Sep 18 20:10:40.609487 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:10:40.609446    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/62e248cd-e0f0-4ea3-a044-2aa5f1261e4c-tmp-dir\") pod \"62e248cd-e0f0-4ea3-a044-2aa5f1261e4c\" (UID: \"62e248cd-e0f0-4ea3-a044-2aa5f1261e4c\") "
Sep 18 20:10:40.609487 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:10:40.609477    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/62e248cd-e0f0-4ea3-a044-2aa5f1261e4c-home\") pod \"62e248cd-e0f0-4ea3-a044-2aa5f1261e4c\" (UID: \"62e248cd-e0f0-4ea3-a044-2aa5f1261e4c\") "
Sep 18 20:10:40.609636 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:10:40.609523    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/62e248cd-e0f0-4ea3-a044-2aa5f1261e4c-model-cache" (OuterVolumeSpecName: "model-cache") pod "62e248cd-e0f0-4ea3-a044-2aa5f1261e4c" (UID: "62e248cd-e0f0-4ea3-a044-2aa5f1261e4c"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 20:10:40.609776 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:10:40.609739    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/62e248cd-e0f0-4ea3-a044-2aa5f1261e4c-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "62e248cd-e0f0-4ea3-a044-2aa5f1261e4c" (UID: "62e248cd-e0f0-4ea3-a044-2aa5f1261e4c"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 20:10:40.609940 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:10:40.609824    2585 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/62e248cd-e0f0-4ea3-a044-2aa5f1261e4c-model-cache\") on node \"ip-10-0-136-185.ec2.internal\" DevicePath \"\""
Sep 18 20:10:40.609940 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:10:40.609922    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/62e248cd-e0f0-4ea3-a044-2aa5f1261e4c-home" (OuterVolumeSpecName: "home") pod "62e248cd-e0f0-4ea3-a044-2aa5f1261e4c" (UID: "62e248cd-e0f0-4ea3-a044-2aa5f1261e4c"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 20:10:40.611685 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:10:40.611660    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/62e248cd-e0f0-4ea3-a044-2aa5f1261e4c-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "62e248cd-e0f0-4ea3-a044-2aa5f1261e4c" (UID: "62e248cd-e0f0-4ea3-a044-2aa5f1261e4c"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 20:10:40.611763 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:10:40.611665    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/62e248cd-e0f0-4ea3-a044-2aa5f1261e4c-dshm" (OuterVolumeSpecName: "dshm") pod "62e248cd-e0f0-4ea3-a044-2aa5f1261e4c" (UID: "62e248cd-e0f0-4ea3-a044-2aa5f1261e4c"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 20:10:40.611763 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:10:40.611668    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/62e248cd-e0f0-4ea3-a044-2aa5f1261e4c-kube-api-access-qjfrz" (OuterVolumeSpecName: "kube-api-access-qjfrz") pod "62e248cd-e0f0-4ea3-a044-2aa5f1261e4c" (UID: "62e248cd-e0f0-4ea3-a044-2aa5f1261e4c"). InnerVolumeSpecName "kube-api-access-qjfrz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 20:10:40.662617 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:10:40.662586    2585 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fb103bd1-453b-40a1-96b6-93c708f6d7d7" path="/var/lib/kubelet/pods/fb103bd1-453b-40a1-96b6-93c708f6d7d7/volumes"
Sep 18 20:10:40.711285 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:10:40.711200    2585 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/62e248cd-e0f0-4ea3-a044-2aa5f1261e4c-tmp-dir\") on node \"ip-10-0-136-185.ec2.internal\" DevicePath \"\""
Sep 18 20:10:40.711285 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:10:40.711233    2585 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/62e248cd-e0f0-4ea3-a044-2aa5f1261e4c-home\") on node \"ip-10-0-136-185.ec2.internal\" DevicePath \"\""
Sep 18 20:10:40.711285 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:10:40.711241    2585 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/62e248cd-e0f0-4ea3-a044-2aa5f1261e4c-tls-certs\") on node \"ip-10-0-136-185.ec2.internal\" DevicePath \"\""
Sep 18 20:10:40.711285 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:10:40.711249    2585 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qjfrz\" (UniqueName: \"kubernetes.io/projected/62e248cd-e0f0-4ea3-a044-2aa5f1261e4c-kube-api-access-qjfrz\") on node \"ip-10-0-136-185.ec2.internal\" DevicePath \"\""
Sep 18 20:10:40.711285 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:10:40.711259    2585 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/62e248cd-e0f0-4ea3-a044-2aa5f1261e4c-dshm\") on node \"ip-10-0-136-185.ec2.internal\" DevicePath \"\""
Sep 18 20:10:40.898937 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:10:40.898907    2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-5547b6/sub-test-model-5547b6-kserve-575b8546df-q99pl"]
Sep 18 20:10:40.906069 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:10:40.906038    2585 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w3-5547b6/sub-test-model-5547b6-kserve-575b8546df-q99pl"]
Sep 18 20:10:40.995010 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:10:40.994926    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-xmwh7"]
Sep 18 20:10:42.661081 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:10:42.661047    2585 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="62e248cd-e0f0-4ea3-a044-2aa5f1261e4c" path="/var/lib/kubelet/pods/62e248cd-e0f0-4ea3-a044-2aa5f1261e4c/volumes"
Sep 18 20:11:05.376406 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:11:05.376369    2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w3-5547b6/rate-test-model-5547b6-kserve-645446f8f9-dbbz2"]
Sep 18 20:11:05.376902 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:11:05.376722    2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fb103bd1-453b-40a1-96b6-93c708f6d7d7" containerName="main"
Sep 18 20:11:05.376902 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:11:05.376733    2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="fb103bd1-453b-40a1-96b6-93c708f6d7d7" containerName="main"
Sep 18 20:11:05.376902 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:11:05.376742    2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="62e248cd-e0f0-4ea3-a044-2aa5f1261e4c" containerName="main"
Sep 18 20:11:05.376902 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:11:05.376748    2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="62e248cd-e0f0-4ea3-a044-2aa5f1261e4c" containerName="main"
Sep 18 20:11:05.376902 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:11:05.376831    2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="fb103bd1-453b-40a1-96b6-93c708f6d7d7" containerName="main"
Sep 18 20:11:05.376902 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:11:05.376840    2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="62e248cd-e0f0-4ea3-a044-2aa5f1261e4c" containerName="main"
Sep 18 20:11:05.380025 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:11:05.380006    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-5547b6/rate-test-model-5547b6-kserve-645446f8f9-dbbz2"
Sep 18 20:11:05.383618 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:11:05.383587    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-5547b6\"/\"kube-root-ca.crt\""
Sep 18 20:11:05.383916 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:11:05.383896    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-5547b6\"/\"openshift-service-ca.crt\""
Sep 18 20:11:05.387933 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:11:05.385144    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-5547b6\"/\"default-dockercfg-fsn7d\""
Sep 18 20:11:05.387933 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:11:05.385477    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-5547b6\"/\"rate-test-model-5547b6-kserve-self-signed-certs\""
Sep 18 20:11:05.390906 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:11:05.390880    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-5547b6/rate-test-model-5547b6-kserve-645446f8f9-dbbz2"]
Sep 18 20:11:05.419310 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:11:05.419271    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cfe5530f-8878-4854-8e41-f7596a3c0af2-dshm\") pod \"rate-test-model-5547b6-kserve-645446f8f9-dbbz2\" (UID: \"cfe5530f-8878-4854-8e41-f7596a3c0af2\") " pod="ai-tenant-e2e-shared-b-w3-5547b6/rate-test-model-5547b6-kserve-645446f8f9-dbbz2"
Sep 18 20:11:05.419310 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:11:05.419315    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cfe5530f-8878-4854-8e41-f7596a3c0af2-tmp-dir\") pod \"rate-test-model-5547b6-kserve-645446f8f9-dbbz2\" (UID: \"cfe5530f-8878-4854-8e41-f7596a3c0af2\") " pod="ai-tenant-e2e-shared-b-w3-5547b6/rate-test-model-5547b6-kserve-645446f8f9-dbbz2"
Sep 18 20:11:05.419543 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:11:05.419423    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cfe5530f-8878-4854-8e41-f7596a3c0af2-model-cache\") pod \"rate-test-model-5547b6-kserve-645446f8f9-dbbz2\" (UID: \"cfe5530f-8878-4854-8e41-f7596a3c0af2\") " pod="ai-tenant-e2e-shared-b-w3-5547b6/rate-test-model-5547b6-kserve-645446f8f9-dbbz2"
Sep 18 20:11:05.419543 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:11:05.419453    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cfe5530f-8878-4854-8e41-f7596a3c0af2-tls-certs\") pod \"rate-test-model-5547b6-kserve-645446f8f9-dbbz2\" (UID: \"cfe5530f-8878-4854-8e41-f7596a3c0af2\") " pod="ai-tenant-e2e-shared-b-w3-5547b6/rate-test-model-5547b6-kserve-645446f8f9-dbbz2"
Sep 18 20:11:05.419543 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:11:05.419487    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cfe5530f-8878-4854-8e41-f7596a3c0af2-home\") pod \"rate-test-model-5547b6-kserve-645446f8f9-dbbz2\" (UID: \"cfe5530f-8878-4854-8e41-f7596a3c0af2\") " pod="ai-tenant-e2e-shared-b-w3-5547b6/rate-test-model-5547b6-kserve-645446f8f9-dbbz2"
Sep 18 20:11:05.419543 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:11:05.419514    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-964zf\" (UniqueName: \"kubernetes.io/projected/cfe5530f-8878-4854-8e41-f7596a3c0af2-kube-api-access-964zf\") pod \"rate-test-model-5547b6-kserve-645446f8f9-dbbz2\" (UID: \"cfe5530f-8878-4854-8e41-f7596a3c0af2\") " pod="ai-tenant-e2e-shared-b-w3-5547b6/rate-test-model-5547b6-kserve-645446f8f9-dbbz2"
Sep 18 20:11:05.520164 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:11:05.520121    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cfe5530f-8878-4854-8e41-f7596a3c0af2-dshm\") pod \"rate-test-model-5547b6-kserve-645446f8f9-dbbz2\" (UID: \"cfe5530f-8878-4854-8e41-f7596a3c0af2\") " pod="ai-tenant-e2e-shared-b-w3-5547b6/rate-test-model-5547b6-kserve-645446f8f9-dbbz2"
Sep 18 20:11:05.520164 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:11:05.520168    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cfe5530f-8878-4854-8e41-f7596a3c0af2-tmp-dir\") pod \"rate-test-model-5547b6-kserve-645446f8f9-dbbz2\" (UID: \"cfe5530f-8878-4854-8e41-f7596a3c0af2\") " pod="ai-tenant-e2e-shared-b-w3-5547b6/rate-test-model-5547b6-kserve-645446f8f9-dbbz2"
Sep 18 20:11:05.520434 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:11:05.520210    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cfe5530f-8878-4854-8e41-f7596a3c0af2-model-cache\") pod \"rate-test-model-5547b6-kserve-645446f8f9-dbbz2\" (UID: \"cfe5530f-8878-4854-8e41-f7596a3c0af2\") " pod="ai-tenant-e2e-shared-b-w3-5547b6/rate-test-model-5547b6-kserve-645446f8f9-dbbz2"
Sep 18 20:11:05.520434 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:11:05.520226    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cfe5530f-8878-4854-8e41-f7596a3c0af2-tls-certs\") pod \"rate-test-model-5547b6-kserve-645446f8f9-dbbz2\" (UID: \"cfe5530f-8878-4854-8e41-f7596a3c0af2\") " pod="ai-tenant-e2e-shared-b-w3-5547b6/rate-test-model-5547b6-kserve-645446f8f9-dbbz2"
Sep 18 20:11:05.520434 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:11:05.520252    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cfe5530f-8878-4854-8e41-f7596a3c0af2-home\") pod \"rate-test-model-5547b6-kserve-645446f8f9-dbbz2\" (UID: \"cfe5530f-8878-4854-8e41-f7596a3c0af2\") " pod="ai-tenant-e2e-shared-b-w3-5547b6/rate-test-model-5547b6-kserve-645446f8f9-dbbz2"
Sep 18 20:11:05.520434 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:11:05.520275    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-964zf\" (UniqueName: \"kubernetes.io/projected/cfe5530f-8878-4854-8e41-f7596a3c0af2-kube-api-access-964zf\") pod \"rate-test-model-5547b6-kserve-645446f8f9-dbbz2\" (UID: \"cfe5530f-8878-4854-8e41-f7596a3c0af2\") " pod="ai-tenant-e2e-shared-b-w3-5547b6/rate-test-model-5547b6-kserve-645446f8f9-dbbz2"
Sep 18 20:11:05.520696 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:11:05.520661    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cfe5530f-8878-4854-8e41-f7596a3c0af2-model-cache\") pod \"rate-test-model-5547b6-kserve-645446f8f9-dbbz2\" (UID: \"cfe5530f-8878-4854-8e41-f7596a3c0af2\") " pod="ai-tenant-e2e-shared-b-w3-5547b6/rate-test-model-5547b6-kserve-645446f8f9-dbbz2"
Sep 18 20:11:05.520847 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:11:05.520678    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cfe5530f-8878-4854-8e41-f7596a3c0af2-tmp-dir\") pod \"rate-test-model-5547b6-kserve-645446f8f9-dbbz2\" (UID: \"cfe5530f-8878-4854-8e41-f7596a3c0af2\") " pod="ai-tenant-e2e-shared-b-w3-5547b6/rate-test-model-5547b6-kserve-645446f8f9-dbbz2"
Sep 18 20:11:05.520847 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:11:05.520729    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cfe5530f-8878-4854-8e41-f7596a3c0af2-home\") pod \"rate-test-model-5547b6-kserve-645446f8f9-dbbz2\" (UID: \"cfe5530f-8878-4854-8e41-f7596a3c0af2\") " pod="ai-tenant-e2e-shared-b-w3-5547b6/rate-test-model-5547b6-kserve-645446f8f9-dbbz2"
Sep 18 20:11:05.522523 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:11:05.522502    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cfe5530f-8878-4854-8e41-f7596a3c0af2-dshm\") pod \"rate-test-model-5547b6-kserve-645446f8f9-dbbz2\" (UID: \"cfe5530f-8878-4854-8e41-f7596a3c0af2\") " pod="ai-tenant-e2e-shared-b-w3-5547b6/rate-test-model-5547b6-kserve-645446f8f9-dbbz2"
Sep 18 20:11:05.522659 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:11:05.522641    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cfe5530f-8878-4854-8e41-f7596a3c0af2-tls-certs\") pod \"rate-test-model-5547b6-kserve-645446f8f9-dbbz2\" (UID: \"cfe5530f-8878-4854-8e41-f7596a3c0af2\") " pod="ai-tenant-e2e-shared-b-w3-5547b6/rate-test-model-5547b6-kserve-645446f8f9-dbbz2"
Sep 18 20:11:05.528392 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:11:05.528364    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-964zf\" (UniqueName: \"kubernetes.io/projected/cfe5530f-8878-4854-8e41-f7596a3c0af2-kube-api-access-964zf\") pod \"rate-test-model-5547b6-kserve-645446f8f9-dbbz2\" (UID: \"cfe5530f-8878-4854-8e41-f7596a3c0af2\") " pod="ai-tenant-e2e-shared-b-w3-5547b6/rate-test-model-5547b6-kserve-645446f8f9-dbbz2"
Sep 18 20:11:05.696265 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:11:05.696222    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-5547b6/rate-test-model-5547b6-kserve-645446f8f9-dbbz2"
Sep 18 20:11:05.970585 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:11:05.970558    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-5547b6/rate-test-model-5547b6-kserve-645446f8f9-dbbz2"]
Sep 18 20:11:05.972902 ip-10-0-136-185 kubenswrapper[2585]: W0918 20:11:05.972872    2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podcfe5530f_8878_4854_8e41_f7596a3c0af2.slice/crio-73180c237fc847ff3b5081d738628441ec71e8f26329fa90c9dceeca26fd4f98 WatchSource:0}: Error finding container 73180c237fc847ff3b5081d738628441ec71e8f26329fa90c9dceeca26fd4f98: Status 404 returned error can't find the container with id 73180c237fc847ff3b5081d738628441ec71e8f26329fa90c9dceeca26fd4f98
Sep 18 20:11:06.708593 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:11:06.708558    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-5547b6/rate-test-model-5547b6-kserve-645446f8f9-dbbz2" event={"ID":"cfe5530f-8878-4854-8e41-f7596a3c0af2","Type":"ContainerStarted","Data":"1eb16c22b85ac752750d50d7aa59818c70be71492097e5833c8feafc43738392"}
Sep 18 20:11:06.708593 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:11:06.708600    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-5547b6/rate-test-model-5547b6-kserve-645446f8f9-dbbz2" event={"ID":"cfe5530f-8878-4854-8e41-f7596a3c0af2","Type":"ContainerStarted","Data":"73180c237fc847ff3b5081d738628441ec71e8f26329fa90c9dceeca26fd4f98"}
Sep 18 20:11:06.732392 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:11:06.732336    2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w3-5547b6/rate-test-model-5547b6-kserve-645446f8f9-dbbz2" podStartSLOduration=1.732318131 podStartE2EDuration="1.732318131s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 20:11:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 20:11:06.730924716 +0000 UTC m=+1908.255530240" watchObservedRunningTime="2026-09-18 20:11:06.732318131 +0000 UTC m=+1908.256923575"
Sep 18 20:11:09.187582 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:11:09.187543    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-xmwh7"]
Sep 18 20:11:15.696466 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:11:15.696413    2585 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w3-5547b6/rate-test-model-5547b6-kserve-645446f8f9-dbbz2"
Sep 18 20:11:15.696466 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:11:15.696468    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w3-5547b6/rate-test-model-5547b6-kserve-645446f8f9-dbbz2"
Sep 18 20:11:15.709118 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:11:15.709092    2585 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w3-5547b6/rate-test-model-5547b6-kserve-645446f8f9-dbbz2"
Sep 18 20:11:15.764658 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:11:15.764631    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w3-5547b6/rate-test-model-5547b6-kserve-645446f8f9-dbbz2"
Sep 18 20:11:23.465851 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:11:23.465809    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-xmwh7"]
Sep 18 20:11:32.783475 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:11:32.783436    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-xmwh7"]
Sep 18 20:11:46.265242 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:11:46.265204    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-xmwh7"]
Sep 18 20:11:51.352034 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:11:51.351993    2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-5547b6/rate-test-model-5547b6-kserve-645446f8f9-dbbz2"]
Sep 18 20:11:56.447603 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:11:56.447552    2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-953438-7bc466dcbf-4zxbk"]
Sep 18 20:11:56.451637 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:11:56.451612    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-953438-7bc466dcbf-4zxbk"
Sep 18 20:11:56.455111 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:11:56.455083    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 18 20:11:56.455932 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:11:56.455720    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-a-953438-serving-cert\""
Sep 18 20:11:56.455932 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:11:56.455732    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-jqjhf\""
Sep 18 20:11:56.455932 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:11:56.455812    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\""
Sep 18 20:11:56.455932 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:11:56.455824    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Sep 18 20:11:56.466973 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:11:56.466943    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-953438-7bc466dcbf-4zxbk"]
Sep 18 20:11:56.494064 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:11:56.494031    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/23a6c2f1-224e-43dc-8e48-ff2725bf6345-maas-api-tls\") pod \"maas-api-e2e-ipp-a-953438-7bc466dcbf-4zxbk\" (UID: \"23a6c2f1-224e-43dc-8e48-ff2725bf6345\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-953438-7bc466dcbf-4zxbk"
Sep 18 20:11:56.494242 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:11:56.494082    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/23a6c2f1-224e-43dc-8e48-ff2725bf6345-metrics-certs\") pod \"maas-api-e2e-ipp-a-953438-7bc466dcbf-4zxbk\" (UID: \"23a6c2f1-224e-43dc-8e48-ff2725bf6345\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-953438-7bc466dcbf-4zxbk"
Sep 18 20:11:56.494242 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:11:56.494198    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-56nbz\" (UniqueName: \"kubernetes.io/projected/23a6c2f1-224e-43dc-8e48-ff2725bf6345-kube-api-access-56nbz\") pod \"maas-api-e2e-ipp-a-953438-7bc466dcbf-4zxbk\" (UID: \"23a6c2f1-224e-43dc-8e48-ff2725bf6345\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-953438-7bc466dcbf-4zxbk"
Sep 18 20:11:56.494336 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:11:56.494273    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/23a6c2f1-224e-43dc-8e48-ff2725bf6345-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-953438-7bc466dcbf-4zxbk\" (UID: \"23a6c2f1-224e-43dc-8e48-ff2725bf6345\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-953438-7bc466dcbf-4zxbk"
Sep 18 20:11:56.595508 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:11:56.595466    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/23a6c2f1-224e-43dc-8e48-ff2725bf6345-maas-api-tls\") pod \"maas-api-e2e-ipp-a-953438-7bc466dcbf-4zxbk\" (UID: \"23a6c2f1-224e-43dc-8e48-ff2725bf6345\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-953438-7bc466dcbf-4zxbk"
Sep 18 20:11:56.595718 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:11:56.595520    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/23a6c2f1-224e-43dc-8e48-ff2725bf6345-metrics-certs\") pod \"maas-api-e2e-ipp-a-953438-7bc466dcbf-4zxbk\" (UID: \"23a6c2f1-224e-43dc-8e48-ff2725bf6345\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-953438-7bc466dcbf-4zxbk"
Sep 18 20:11:56.595718 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:11:56.595573    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-56nbz\" (UniqueName: \"kubernetes.io/projected/23a6c2f1-224e-43dc-8e48-ff2725bf6345-kube-api-access-56nbz\") pod \"maas-api-e2e-ipp-a-953438-7bc466dcbf-4zxbk\" (UID: \"23a6c2f1-224e-43dc-8e48-ff2725bf6345\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-953438-7bc466dcbf-4zxbk"
Sep 18 20:11:56.595718 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:11:56.595620    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/23a6c2f1-224e-43dc-8e48-ff2725bf6345-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-953438-7bc466dcbf-4zxbk\" (UID: \"23a6c2f1-224e-43dc-8e48-ff2725bf6345\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-953438-7bc466dcbf-4zxbk"
Sep 18 20:11:56.595718 ip-10-0-136-185 kubenswrapper[2585]: E0918 20:11:56.595632    2585 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ipp-a-953438-serving-cert: secret "maas-api-e2e-ipp-a-953438-serving-cert" not found
Sep 18 20:11:56.595718 ip-10-0-136-185 kubenswrapper[2585]: E0918 20:11:56.595709    2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/23a6c2f1-224e-43dc-8e48-ff2725bf6345-maas-api-tls podName:23a6c2f1-224e-43dc-8e48-ff2725bf6345 nodeName:}" failed. No retries permitted until 2026-09-18 20:11:57.095689086 +0000 UTC m=+1958.620294486 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/23a6c2f1-224e-43dc-8e48-ff2725bf6345-maas-api-tls") pod "maas-api-e2e-ipp-a-953438-7bc466dcbf-4zxbk" (UID: "23a6c2f1-224e-43dc-8e48-ff2725bf6345") : secret "maas-api-e2e-ipp-a-953438-serving-cert" not found
Sep 18 20:11:56.596403 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:11:56.596379    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/23a6c2f1-224e-43dc-8e48-ff2725bf6345-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-953438-7bc466dcbf-4zxbk\" (UID: \"23a6c2f1-224e-43dc-8e48-ff2725bf6345\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-953438-7bc466dcbf-4zxbk"
Sep 18 20:11:56.598240 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:11:56.598215    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/23a6c2f1-224e-43dc-8e48-ff2725bf6345-metrics-certs\") pod \"maas-api-e2e-ipp-a-953438-7bc466dcbf-4zxbk\" (UID: \"23a6c2f1-224e-43dc-8e48-ff2725bf6345\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-953438-7bc466dcbf-4zxbk"
Sep 18 20:11:56.608081 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:11:56.608056    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-56nbz\" (UniqueName: \"kubernetes.io/projected/23a6c2f1-224e-43dc-8e48-ff2725bf6345-kube-api-access-56nbz\") pod \"maas-api-e2e-ipp-a-953438-7bc466dcbf-4zxbk\" (UID: \"23a6c2f1-224e-43dc-8e48-ff2725bf6345\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-953438-7bc466dcbf-4zxbk"
Sep 18 20:11:57.101167 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:11:57.101126    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/23a6c2f1-224e-43dc-8e48-ff2725bf6345-maas-api-tls\") pod \"maas-api-e2e-ipp-a-953438-7bc466dcbf-4zxbk\" (UID: \"23a6c2f1-224e-43dc-8e48-ff2725bf6345\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-953438-7bc466dcbf-4zxbk"
Sep 18 20:11:57.103559 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:11:57.103526    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/23a6c2f1-224e-43dc-8e48-ff2725bf6345-maas-api-tls\") pod \"maas-api-e2e-ipp-a-953438-7bc466dcbf-4zxbk\" (UID: \"23a6c2f1-224e-43dc-8e48-ff2725bf6345\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-953438-7bc466dcbf-4zxbk"
Sep 18 20:11:57.363924 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:11:57.363820    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-953438-7bc466dcbf-4zxbk"
Sep 18 20:11:57.578183 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:11:57.578143    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-xmwh7"]
Sep 18 20:11:57.634293 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:11:57.634269    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-953438-7bc466dcbf-4zxbk"]
Sep 18 20:11:57.636807 ip-10-0-136-185 kubenswrapper[2585]: W0918 20:11:57.636763    2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod23a6c2f1_224e_43dc_8e48_ff2725bf6345.slice/crio-9f46d0f8e515ed11aabd55eb9439620c4eaac74ad50fc5a49ee338a1e763a01b WatchSource:0}: Error finding container 9f46d0f8e515ed11aabd55eb9439620c4eaac74ad50fc5a49ee338a1e763a01b: Status 404 returned error can't find the container with id 9f46d0f8e515ed11aabd55eb9439620c4eaac74ad50fc5a49ee338a1e763a01b
Sep 18 20:11:57.638251 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:11:57.638231    2585 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 18 20:11:57.961979 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:11:57.961940    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-953438-7bc466dcbf-4zxbk" event={"ID":"23a6c2f1-224e-43dc-8e48-ff2725bf6345","Type":"ContainerStarted","Data":"9f46d0f8e515ed11aabd55eb9439620c4eaac74ad50fc5a49ee338a1e763a01b"}
Sep 18 20:11:58.969207 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:11:58.969171    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-953438-7bc466dcbf-4zxbk" event={"ID":"23a6c2f1-224e-43dc-8e48-ff2725bf6345","Type":"ContainerStarted","Data":"1e6a291615a525b591fe2253f3d6c9ae9f4d05dd3fcde683c1b30cc55589c04c"}
Sep 18 20:11:58.969638 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:11:58.969276    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-953438-7bc466dcbf-4zxbk"
Sep 18 20:11:58.989552 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:11:58.989496    2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-953438-7bc466dcbf-4zxbk" podStartSLOduration=2.528236834 podStartE2EDuration="2.989478033s" totalImagesPullingTime="461.241199ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 20:11:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 20:11:58.987289299 +0000 UTC m=+1960.511894734" watchObservedRunningTime="2026-09-18 20:11:58.989478033 +0000 UTC m=+1960.514083492"
Sep 18 20:12:04.979153 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:12:04.979122    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-953438-7bc466dcbf-4zxbk"
Sep 18 20:12:06.276334 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:12:06.276296    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-xmwh7"]
Sep 18 20:12:06.376244 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:12:06.376204    2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w3-5547b6/rate-test-model-5547b6-kserve-645446f8f9-dbbz2" podUID="cfe5530f-8878-4854-8e41-f7596a3c0af2" containerName="main" containerID="cri-o://1eb16c22b85ac752750d50d7aa59818c70be71492097e5833c8feafc43738392" gracePeriod=45
Sep 18 20:12:06.630456 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:12:06.630427    2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-5547b6/rate-test-model-5547b6-kserve-645446f8f9-dbbz2"
Sep 18 20:12:06.701063 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:12:06.701031    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cfe5530f-8878-4854-8e41-f7596a3c0af2-tmp-dir\") pod \"cfe5530f-8878-4854-8e41-f7596a3c0af2\" (UID: \"cfe5530f-8878-4854-8e41-f7596a3c0af2\") "
Sep 18 20:12:06.701247 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:12:06.701129    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-964zf\" (UniqueName: \"kubernetes.io/projected/cfe5530f-8878-4854-8e41-f7596a3c0af2-kube-api-access-964zf\") pod \"cfe5530f-8878-4854-8e41-f7596a3c0af2\" (UID: \"cfe5530f-8878-4854-8e41-f7596a3c0af2\") "
Sep 18 20:12:06.701247 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:12:06.701174    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cfe5530f-8878-4854-8e41-f7596a3c0af2-model-cache\") pod \"cfe5530f-8878-4854-8e41-f7596a3c0af2\" (UID: \"cfe5530f-8878-4854-8e41-f7596a3c0af2\") "
Sep 18 20:12:06.701247 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:12:06.701193    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cfe5530f-8878-4854-8e41-f7596a3c0af2-dshm\") pod \"cfe5530f-8878-4854-8e41-f7596a3c0af2\" (UID: \"cfe5530f-8878-4854-8e41-f7596a3c0af2\") "
Sep 18 20:12:06.701247 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:12:06.701216    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cfe5530f-8878-4854-8e41-f7596a3c0af2-tls-certs\") pod \"cfe5530f-8878-4854-8e41-f7596a3c0af2\" (UID: \"cfe5530f-8878-4854-8e41-f7596a3c0af2\") "
Sep 18 20:12:06.701466 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:12:06.701268    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cfe5530f-8878-4854-8e41-f7596a3c0af2-home\") pod \"cfe5530f-8878-4854-8e41-f7596a3c0af2\" (UID: \"cfe5530f-8878-4854-8e41-f7596a3c0af2\") "
Sep 18 20:12:06.701466 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:12:06.701357    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cfe5530f-8878-4854-8e41-f7596a3c0af2-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "cfe5530f-8878-4854-8e41-f7596a3c0af2" (UID: "cfe5530f-8878-4854-8e41-f7596a3c0af2"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 20:12:06.701576 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:12:06.701487    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cfe5530f-8878-4854-8e41-f7596a3c0af2-model-cache" (OuterVolumeSpecName: "model-cache") pod "cfe5530f-8878-4854-8e41-f7596a3c0af2" (UID: "cfe5530f-8878-4854-8e41-f7596a3c0af2"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 20:12:06.701576 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:12:06.701550    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cfe5530f-8878-4854-8e41-f7596a3c0af2-home" (OuterVolumeSpecName: "home") pod "cfe5530f-8878-4854-8e41-f7596a3c0af2" (UID: "cfe5530f-8878-4854-8e41-f7596a3c0af2"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 20:12:06.701679 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:12:06.701611    2585 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cfe5530f-8878-4854-8e41-f7596a3c0af2-model-cache\") on node \"ip-10-0-136-185.ec2.internal\" DevicePath \"\""
Sep 18 20:12:06.701679 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:12:06.701627    2585 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cfe5530f-8878-4854-8e41-f7596a3c0af2-home\") on node \"ip-10-0-136-185.ec2.internal\" DevicePath \"\""
Sep 18 20:12:06.701679 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:12:06.701640    2585 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cfe5530f-8878-4854-8e41-f7596a3c0af2-tmp-dir\") on node \"ip-10-0-136-185.ec2.internal\" DevicePath \"\""
Sep 18 20:12:06.703393 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:12:06.703370    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cfe5530f-8878-4854-8e41-f7596a3c0af2-dshm" (OuterVolumeSpecName: "dshm") pod "cfe5530f-8878-4854-8e41-f7596a3c0af2" (UID: "cfe5530f-8878-4854-8e41-f7596a3c0af2"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 20:12:06.703485 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:12:06.703422    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cfe5530f-8878-4854-8e41-f7596a3c0af2-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "cfe5530f-8878-4854-8e41-f7596a3c0af2" (UID: "cfe5530f-8878-4854-8e41-f7596a3c0af2"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 20:12:06.703485 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:12:06.703471    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cfe5530f-8878-4854-8e41-f7596a3c0af2-kube-api-access-964zf" (OuterVolumeSpecName: "kube-api-access-964zf") pod "cfe5530f-8878-4854-8e41-f7596a3c0af2" (UID: "cfe5530f-8878-4854-8e41-f7596a3c0af2"). InnerVolumeSpecName "kube-api-access-964zf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 20:12:06.802603 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:12:06.802514    2585 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-964zf\" (UniqueName: \"kubernetes.io/projected/cfe5530f-8878-4854-8e41-f7596a3c0af2-kube-api-access-964zf\") on node \"ip-10-0-136-185.ec2.internal\" DevicePath \"\""
Sep 18 20:12:06.802603 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:12:06.802548    2585 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cfe5530f-8878-4854-8e41-f7596a3c0af2-dshm\") on node \"ip-10-0-136-185.ec2.internal\" DevicePath \"\""
Sep 18 20:12:06.802603 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:12:06.802557    2585 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cfe5530f-8878-4854-8e41-f7596a3c0af2-tls-certs\") on node \"ip-10-0-136-185.ec2.internal\" DevicePath \"\""
Sep 18 20:12:07.017959 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:12:07.017920    2585 generic.go:356] "Generic (PLEG): container finished" podID="cfe5530f-8878-4854-8e41-f7596a3c0af2" containerID="1eb16c22b85ac752750d50d7aa59818c70be71492097e5833c8feafc43738392" exitCode=0
Sep 18 20:12:07.018172 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:12:07.017967    2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-5547b6/rate-test-model-5547b6-kserve-645446f8f9-dbbz2"
Sep 18 20:12:07.018172 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:12:07.018003    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-5547b6/rate-test-model-5547b6-kserve-645446f8f9-dbbz2" event={"ID":"cfe5530f-8878-4854-8e41-f7596a3c0af2","Type":"ContainerDied","Data":"1eb16c22b85ac752750d50d7aa59818c70be71492097e5833c8feafc43738392"}
Sep 18 20:12:07.018172 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:12:07.018051    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-5547b6/rate-test-model-5547b6-kserve-645446f8f9-dbbz2" event={"ID":"cfe5530f-8878-4854-8e41-f7596a3c0af2","Type":"ContainerDied","Data":"73180c237fc847ff3b5081d738628441ec71e8f26329fa90c9dceeca26fd4f98"}
Sep 18 20:12:07.018172 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:12:07.018068    2585 scope.go:117] "RemoveContainer" containerID="1eb16c22b85ac752750d50d7aa59818c70be71492097e5833c8feafc43738392"
Sep 18 20:12:07.027229 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:12:07.027213    2585 scope.go:117] "RemoveContainer" containerID="1eb16c22b85ac752750d50d7aa59818c70be71492097e5833c8feafc43738392"
Sep 18 20:12:07.027507 ip-10-0-136-185 kubenswrapper[2585]: E0918 20:12:07.027488    2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1eb16c22b85ac752750d50d7aa59818c70be71492097e5833c8feafc43738392\": container with ID starting with 1eb16c22b85ac752750d50d7aa59818c70be71492097e5833c8feafc43738392 not found: ID does not exist" containerID="1eb16c22b85ac752750d50d7aa59818c70be71492097e5833c8feafc43738392"
Sep 18 20:12:07.027558 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:12:07.027517    2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1eb16c22b85ac752750d50d7aa59818c70be71492097e5833c8feafc43738392"} err="failed to get container status \"1eb16c22b85ac752750d50d7aa59818c70be71492097e5833c8feafc43738392\": rpc error: code = NotFound desc = could not find container \"1eb16c22b85ac752750d50d7aa59818c70be71492097e5833c8feafc43738392\": container with ID starting with 1eb16c22b85ac752750d50d7aa59818c70be71492097e5833c8feafc43738392 not found: ID does not exist"
Sep 18 20:12:07.037880 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:12:07.037845    2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-5547b6/rate-test-model-5547b6-kserve-645446f8f9-dbbz2"]
Sep 18 20:12:07.043840 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:12:07.043817    2585 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w3-5547b6/rate-test-model-5547b6-kserve-645446f8f9-dbbz2"]
Sep 18 20:12:08.661598 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:12:08.661564    2585 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="cfe5530f-8878-4854-8e41-f7596a3c0af2" path="/var/lib/kubelet/pods/cfe5530f-8878-4854-8e41-f7596a3c0af2/volumes"
Sep 18 20:12:26.312893 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:12:26.312852    2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-9c0eb2-f78858fcf-z2pxh"]
Sep 18 20:12:26.313356 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:12:26.313319    2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cfe5530f-8878-4854-8e41-f7596a3c0af2" containerName="main"
Sep 18 20:12:26.313356 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:12:26.313337    2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="cfe5530f-8878-4854-8e41-f7596a3c0af2" containerName="main"
Sep 18 20:12:26.313473 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:12:26.313433    2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="cfe5530f-8878-4854-8e41-f7596a3c0af2" containerName="main"
Sep 18 20:12:26.317705 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:12:26.317680    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-9c0eb2-f78858fcf-z2pxh"
Sep 18 20:12:26.320325 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:12:26.320299    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-b-9c0eb2-serving-cert\""
Sep 18 20:12:26.327366 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:12:26.327145    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-9c0eb2-f78858fcf-z2pxh"]
Sep 18 20:12:26.483548 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:12:26.483510    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/29458bcb-e7f0-4cc4-8b9a-0b7d4c51c608-metrics-certs\") pod \"maas-api-e2e-ipp-b-9c0eb2-f78858fcf-z2pxh\" (UID: \"29458bcb-e7f0-4cc4-8b9a-0b7d4c51c608\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-9c0eb2-f78858fcf-z2pxh"
Sep 18 20:12:26.483761 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:12:26.483580    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/29458bcb-e7f0-4cc4-8b9a-0b7d4c51c608-maas-api-tls\") pod \"maas-api-e2e-ipp-b-9c0eb2-f78858fcf-z2pxh\" (UID: \"29458bcb-e7f0-4cc4-8b9a-0b7d4c51c608\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-9c0eb2-f78858fcf-z2pxh"
Sep 18 20:12:26.483761 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:12:26.483618    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zlkpg\" (UniqueName: \"kubernetes.io/projected/29458bcb-e7f0-4cc4-8b9a-0b7d4c51c608-kube-api-access-zlkpg\") pod \"maas-api-e2e-ipp-b-9c0eb2-f78858fcf-z2pxh\" (UID: \"29458bcb-e7f0-4cc4-8b9a-0b7d4c51c608\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-9c0eb2-f78858fcf-z2pxh"
Sep 18 20:12:26.483761 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:12:26.483633    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/29458bcb-e7f0-4cc4-8b9a-0b7d4c51c608-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-9c0eb2-f78858fcf-z2pxh\" (UID: \"29458bcb-e7f0-4cc4-8b9a-0b7d4c51c608\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-9c0eb2-f78858fcf-z2pxh"
Sep 18 20:12:26.584950 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:12:26.584846    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/29458bcb-e7f0-4cc4-8b9a-0b7d4c51c608-maas-api-tls\") pod \"maas-api-e2e-ipp-b-9c0eb2-f78858fcf-z2pxh\" (UID: \"29458bcb-e7f0-4cc4-8b9a-0b7d4c51c608\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-9c0eb2-f78858fcf-z2pxh"
Sep 18 20:12:26.584950 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:12:26.584939    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zlkpg\" (UniqueName: \"kubernetes.io/projected/29458bcb-e7f0-4cc4-8b9a-0b7d4c51c608-kube-api-access-zlkpg\") pod \"maas-api-e2e-ipp-b-9c0eb2-f78858fcf-z2pxh\" (UID: \"29458bcb-e7f0-4cc4-8b9a-0b7d4c51c608\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-9c0eb2-f78858fcf-z2pxh"
Sep 18 20:12:26.585152 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:12:26.584966    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/29458bcb-e7f0-4cc4-8b9a-0b7d4c51c608-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-9c0eb2-f78858fcf-z2pxh\" (UID: \"29458bcb-e7f0-4cc4-8b9a-0b7d4c51c608\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-9c0eb2-f78858fcf-z2pxh"
Sep 18 20:12:26.585152 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:12:26.585024    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/29458bcb-e7f0-4cc4-8b9a-0b7d4c51c608-metrics-certs\") pod \"maas-api-e2e-ipp-b-9c0eb2-f78858fcf-z2pxh\" (UID: \"29458bcb-e7f0-4cc4-8b9a-0b7d4c51c608\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-9c0eb2-f78858fcf-z2pxh"
Sep 18 20:12:26.585735 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:12:26.585709    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/29458bcb-e7f0-4cc4-8b9a-0b7d4c51c608-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-9c0eb2-f78858fcf-z2pxh\" (UID: \"29458bcb-e7f0-4cc4-8b9a-0b7d4c51c608\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-9c0eb2-f78858fcf-z2pxh"
Sep 18 20:12:26.587471 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:12:26.587450    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/29458bcb-e7f0-4cc4-8b9a-0b7d4c51c608-metrics-certs\") pod \"maas-api-e2e-ipp-b-9c0eb2-f78858fcf-z2pxh\" (UID: \"29458bcb-e7f0-4cc4-8b9a-0b7d4c51c608\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-9c0eb2-f78858fcf-z2pxh"
Sep 18 20:12:26.587599 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:12:26.587503    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/29458bcb-e7f0-4cc4-8b9a-0b7d4c51c608-maas-api-tls\") pod \"maas-api-e2e-ipp-b-9c0eb2-f78858fcf-z2pxh\" (UID: \"29458bcb-e7f0-4cc4-8b9a-0b7d4c51c608\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-9c0eb2-f78858fcf-z2pxh"
Sep 18 20:12:26.593107 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:12:26.593070    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zlkpg\" (UniqueName: \"kubernetes.io/projected/29458bcb-e7f0-4cc4-8b9a-0b7d4c51c608-kube-api-access-zlkpg\") pod \"maas-api-e2e-ipp-b-9c0eb2-f78858fcf-z2pxh\" (UID: \"29458bcb-e7f0-4cc4-8b9a-0b7d4c51c608\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-9c0eb2-f78858fcf-z2pxh"
Sep 18 20:12:26.630251 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:12:26.630215    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-9c0eb2-f78858fcf-z2pxh"
Sep 18 20:12:26.905190 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:12:26.905161    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-9c0eb2-f78858fcf-z2pxh"]
Sep 18 20:12:26.907415 ip-10-0-136-185 kubenswrapper[2585]: W0918 20:12:26.907384    2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod29458bcb_e7f0_4cc4_8b9a_0b7d4c51c608.slice/crio-80d8a309f2a5a67e9c3082f9fec0c4213d17888c82d39eff343c844f7e7c7344 WatchSource:0}: Error finding container 80d8a309f2a5a67e9c3082f9fec0c4213d17888c82d39eff343c844f7e7c7344: Status 404 returned error can't find the container with id 80d8a309f2a5a67e9c3082f9fec0c4213d17888c82d39eff343c844f7e7c7344
Sep 18 20:12:26.935551 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:12:26.935517    2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-9c0eb2-765b9f6b74-twlrr"]
Sep 18 20:12:26.940063 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:12:26.940045    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-9c0eb2-765b9f6b74-twlrr"
Sep 18 20:12:26.943126 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:12:26.943105    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-9c0eb2-dockercfg-mlv87\""
Sep 18 20:12:26.943235 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:12:26.943127    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-9c0eb2\""
Sep 18 20:12:26.951600 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:12:26.951577    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-9c0eb2-765b9f6b74-twlrr"]
Sep 18 20:12:26.967055 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:12:26.967025    2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-9c0eb2-674f4f784f-clxft"]
Sep 18 20:12:26.970491 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:12:26.970470    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-9c0eb2-674f4f784f-clxft"
Sep 18 20:12:26.979407 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:12:26.979383    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-9c0eb2-674f4f784f-clxft"]
Sep 18 20:12:27.090315 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:12:27.090276    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/42625f0e-8f4c-4d41-a85a-a0d9bedc7aed-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-9c0eb2-674f4f784f-clxft\" (UID: \"42625f0e-8f4c-4d41-a85a-a0d9bedc7aed\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-9c0eb2-674f4f784f-clxft"
Sep 18 20:12:27.090500 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:12:27.090329    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5768b345-1195-4621-9148-6b63df0b645f-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-9c0eb2-765b9f6b74-twlrr\" (UID: \"5768b345-1195-4621-9148-6b63df0b645f\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-9c0eb2-765b9f6b74-twlrr"
Sep 18 20:12:27.090500 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:12:27.090418    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fcmjw\" (UniqueName: \"kubernetes.io/projected/5768b345-1195-4621-9148-6b63df0b645f-kube-api-access-fcmjw\") pod \"payload-pre-processing-e2e-ipp-b-9c0eb2-765b9f6b74-twlrr\" (UID: \"5768b345-1195-4621-9148-6b63df0b645f\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-9c0eb2-765b9f6b74-twlrr"
Sep 18 20:12:27.090576 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:12:27.090515    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jgdxr\" (UniqueName: \"kubernetes.io/projected/42625f0e-8f4c-4d41-a85a-a0d9bedc7aed-kube-api-access-jgdxr\") pod \"payload-processing-e2e-ipp-b-9c0eb2-674f4f784f-clxft\" (UID: \"42625f0e-8f4c-4d41-a85a-a0d9bedc7aed\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-9c0eb2-674f4f784f-clxft"
Sep 18 20:12:27.119345 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:12:27.119310    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-9c0eb2-f78858fcf-z2pxh" event={"ID":"29458bcb-e7f0-4cc4-8b9a-0b7d4c51c608","Type":"ContainerStarted","Data":"80d8a309f2a5a67e9c3082f9fec0c4213d17888c82d39eff343c844f7e7c7344"}
Sep 18 20:12:27.191856 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:12:27.191729    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jgdxr\" (UniqueName: \"kubernetes.io/projected/42625f0e-8f4c-4d41-a85a-a0d9bedc7aed-kube-api-access-jgdxr\") pod \"payload-processing-e2e-ipp-b-9c0eb2-674f4f784f-clxft\" (UID: \"42625f0e-8f4c-4d41-a85a-a0d9bedc7aed\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-9c0eb2-674f4f784f-clxft"
Sep 18 20:12:27.191856 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:12:27.191781    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/42625f0e-8f4c-4d41-a85a-a0d9bedc7aed-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-9c0eb2-674f4f784f-clxft\" (UID: \"42625f0e-8f4c-4d41-a85a-a0d9bedc7aed\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-9c0eb2-674f4f784f-clxft"
Sep 18 20:12:27.191856 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:12:27.191833    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5768b345-1195-4621-9148-6b63df0b645f-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-9c0eb2-765b9f6b74-twlrr\" (UID: \"5768b345-1195-4621-9148-6b63df0b645f\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-9c0eb2-765b9f6b74-twlrr"
Sep 18 20:12:27.192139 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:12:27.191869    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fcmjw\" (UniqueName: \"kubernetes.io/projected/5768b345-1195-4621-9148-6b63df0b645f-kube-api-access-fcmjw\") pod \"payload-pre-processing-e2e-ipp-b-9c0eb2-765b9f6b74-twlrr\" (UID: \"5768b345-1195-4621-9148-6b63df0b645f\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-9c0eb2-765b9f6b74-twlrr"
Sep 18 20:12:27.192846 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:12:27.192821    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5768b345-1195-4621-9148-6b63df0b645f-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-9c0eb2-765b9f6b74-twlrr\" (UID: \"5768b345-1195-4621-9148-6b63df0b645f\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-9c0eb2-765b9f6b74-twlrr"
Sep 18 20:12:27.192934 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:12:27.192822    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/42625f0e-8f4c-4d41-a85a-a0d9bedc7aed-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-9c0eb2-674f4f784f-clxft\" (UID: \"42625f0e-8f4c-4d41-a85a-a0d9bedc7aed\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-9c0eb2-674f4f784f-clxft"
Sep 18 20:12:27.200620 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:12:27.200589    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fcmjw\" (UniqueName: \"kubernetes.io/projected/5768b345-1195-4621-9148-6b63df0b645f-kube-api-access-fcmjw\") pod \"payload-pre-processing-e2e-ipp-b-9c0eb2-765b9f6b74-twlrr\" (UID: \"5768b345-1195-4621-9148-6b63df0b645f\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-9c0eb2-765b9f6b74-twlrr"
Sep 18 20:12:27.200731 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:12:27.200634    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jgdxr\" (UniqueName: \"kubernetes.io/projected/42625f0e-8f4c-4d41-a85a-a0d9bedc7aed-kube-api-access-jgdxr\") pod \"payload-processing-e2e-ipp-b-9c0eb2-674f4f784f-clxft\" (UID: \"42625f0e-8f4c-4d41-a85a-a0d9bedc7aed\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-9c0eb2-674f4f784f-clxft"
Sep 18 20:12:27.251459 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:12:27.251428    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-9c0eb2-765b9f6b74-twlrr"
Sep 18 20:12:27.280937 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:12:27.280905    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-9c0eb2-674f4f784f-clxft"
Sep 18 20:12:27.523429 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:12:27.523402    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-9c0eb2-765b9f6b74-twlrr"]
Sep 18 20:12:27.525291 ip-10-0-136-185 kubenswrapper[2585]: W0918 20:12:27.525257    2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5768b345_1195_4621_9148_6b63df0b645f.slice/crio-06f0378539ea64cb4296cc4efab4072a5405e4544c547f51a5163518a7a7538e WatchSource:0}: Error finding container 06f0378539ea64cb4296cc4efab4072a5405e4544c547f51a5163518a7a7538e: Status 404 returned error can't find the container with id 06f0378539ea64cb4296cc4efab4072a5405e4544c547f51a5163518a7a7538e
Sep 18 20:12:27.561489 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:12:27.561461    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-9c0eb2-674f4f784f-clxft"]
Sep 18 20:12:27.564053 ip-10-0-136-185 kubenswrapper[2585]: W0918 20:12:27.564022    2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod42625f0e_8f4c_4d41_a85a_a0d9bedc7aed.slice/crio-d20c7521c8dd713f43356727d0197d36b9951295f257b3a87c5871af2faa0a4f WatchSource:0}: Error finding container d20c7521c8dd713f43356727d0197d36b9951295f257b3a87c5871af2faa0a4f: Status 404 returned error can't find the container with id d20c7521c8dd713f43356727d0197d36b9951295f257b3a87c5871af2faa0a4f
Sep 18 20:12:28.129186 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:12:28.129145    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-9c0eb2-765b9f6b74-twlrr" event={"ID":"5768b345-1195-4621-9148-6b63df0b645f","Type":"ContainerStarted","Data":"00394643af4f8e817a5e3980925c435c944b1a9a9241900b7fc24ae155870087"}
Sep 18 20:12:28.129393 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:12:28.129208    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-9c0eb2-765b9f6b74-twlrr" event={"ID":"5768b345-1195-4621-9148-6b63df0b645f","Type":"ContainerStarted","Data":"06f0378539ea64cb4296cc4efab4072a5405e4544c547f51a5163518a7a7538e"}
Sep 18 20:12:28.129393 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:12:28.129368    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-9c0eb2-765b9f6b74-twlrr"
Sep 18 20:12:28.130735 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:12:28.130711    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-9c0eb2-f78858fcf-z2pxh" event={"ID":"29458bcb-e7f0-4cc4-8b9a-0b7d4c51c608","Type":"ContainerStarted","Data":"0b550042d3e56771f65539c912fa224313179d63a1588044afef8a960776bc00"}
Sep 18 20:12:28.130850 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:12:28.130753    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-9c0eb2-f78858fcf-z2pxh"
Sep 18 20:12:28.132321 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:12:28.132295    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-9c0eb2-674f4f784f-clxft" event={"ID":"42625f0e-8f4c-4d41-a85a-a0d9bedc7aed","Type":"ContainerStarted","Data":"22b615dfe60fb730a4f9ce0b59dc814d1f2fc0914def7c703bcdc0ebf5f77e87"}
Sep 18 20:12:28.132425 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:12:28.132345    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-9c0eb2-674f4f784f-clxft"
Sep 18 20:12:28.132425 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:12:28.132362    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-9c0eb2-674f4f784f-clxft" event={"ID":"42625f0e-8f4c-4d41-a85a-a0d9bedc7aed","Type":"ContainerStarted","Data":"d20c7521c8dd713f43356727d0197d36b9951295f257b3a87c5871af2faa0a4f"}
Sep 18 20:12:28.148019 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:12:28.147972    2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-9c0eb2-765b9f6b74-twlrr" podStartSLOduration=2.147955747 podStartE2EDuration="2.147955747s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 20:12:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 20:12:28.146035518 +0000 UTC m=+1989.670640942" watchObservedRunningTime="2026-09-18 20:12:28.147955747 +0000 UTC m=+1989.672561210"
Sep 18 20:12:28.165895 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:12:28.165839    2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-9c0eb2-f78858fcf-z2pxh" podStartSLOduration=1.8375557599999999 podStartE2EDuration="2.165823261s" totalImagesPullingTime="328.267501ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 20:12:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 20:12:28.164482233 +0000 UTC m=+1989.689087656" watchObservedRunningTime="2026-09-18 20:12:28.165823261 +0000 UTC m=+1989.690428686"
Sep 18 20:12:28.182480 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:12:28.182423    2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-9c0eb2-674f4f784f-clxft" podStartSLOduration=2.182405885 podStartE2EDuration="2.182405885s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 20:12:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 20:12:28.180376838 +0000 UTC m=+1989.704982264" watchObservedRunningTime="2026-09-18 20:12:28.182405885 +0000 UTC m=+1989.707011307"
Sep 18 20:12:31.373509 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:12:31.373471    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-xmwh7"]
Sep 18 20:12:34.142853 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:12:34.142814    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-9c0eb2-f78858fcf-z2pxh"
Sep 18 20:12:39.139276 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:12:39.139239    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-9c0eb2-674f4f784f-clxft"
Sep 18 20:12:39.139845 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:12:39.139330    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-9c0eb2-765b9f6b74-twlrr"
Sep 18 20:12:54.308134 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:12:54.308098    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-b-9c0eb2-674f4f784f-clxft_42625f0e-8f4c-4d41-a85a-a0d9bedc7aed/payload-processing/0.log"
Sep 18 20:13:06.474062 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:13:06.474023    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-xmwh7"]
Sep 18 20:13:19.468583 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:13:19.468548    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-xmwh7"]
Sep 18 20:13:20.938407 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:13:20.938373    2585 scope.go:117] "RemoveContainer" containerID="0a6ce61cef2d58e3da0b850db3ca667e0e5f0196deda716dbe67bb9aee152c75"
Sep 18 20:13:20.947341 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:13:20.947320    2585 scope.go:117] "RemoveContainer" containerID="040b95888eae6c3c85c3cf0195147c0df74e0f6f26c560655430274797b5ca3e"
Sep 18 20:13:20.955573 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:13:20.955553    2585 scope.go:117] "RemoveContainer" containerID="5ff5ef87e21db1b4143913570ffcd6b3e9fea319ddb45025e7560cf54c3d9644"
Sep 18 20:13:25.424218 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:13:25.424187    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-b-9c0eb2-674f4f784f-clxft_42625f0e-8f4c-4d41-a85a-a0d9bedc7aed/payload-processing/0.log"
Sep 18 20:13:38.972449 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:13:38.972368    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-xmwh7"]
Sep 18 20:14:15.372160 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:14:15.372119    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-xmwh7"]
Sep 18 20:14:20.718773 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:14:20.718742    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-rctld_cc0e0de2-75d5-43a1-b64b-b071427cd8ce/ovn-acl-logging/0.log"
Sep 18 20:14:20.730021 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:14:20.729997    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-rctld_cc0e0de2-75d5-43a1-b64b-b071427cd8ce/ovn-acl-logging/0.log"
Sep 18 20:14:34.880429 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:14:34.880384    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-xmwh7"]
Sep 18 20:14:37.018126 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:14:37.018089    2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-953438-7bc466dcbf-4zxbk"]
Sep 18 20:14:37.018552 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:14:37.018309    2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-953438-7bc466dcbf-4zxbk" podUID="23a6c2f1-224e-43dc-8e48-ff2725bf6345" containerName="maas-api" containerID="cri-o://1e6a291615a525b591fe2253f3d6c9ae9f4d05dd3fcde683c1b30cc55589c04c" gracePeriod=30
Sep 18 20:14:37.271284 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:14:37.271214    2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-953438-7bc466dcbf-4zxbk"
Sep 18 20:14:37.309849 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:14:37.309820    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-56nbz\" (UniqueName: \"kubernetes.io/projected/23a6c2f1-224e-43dc-8e48-ff2725bf6345-kube-api-access-56nbz\") pod \"23a6c2f1-224e-43dc-8e48-ff2725bf6345\" (UID: \"23a6c2f1-224e-43dc-8e48-ff2725bf6345\") "
Sep 18 20:14:37.310008 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:14:37.309864    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/23a6c2f1-224e-43dc-8e48-ff2725bf6345-openshift-service-ca\") pod \"23a6c2f1-224e-43dc-8e48-ff2725bf6345\" (UID: \"23a6c2f1-224e-43dc-8e48-ff2725bf6345\") "
Sep 18 20:14:37.310008 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:14:37.309900    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/23a6c2f1-224e-43dc-8e48-ff2725bf6345-metrics-certs\") pod \"23a6c2f1-224e-43dc-8e48-ff2725bf6345\" (UID: \"23a6c2f1-224e-43dc-8e48-ff2725bf6345\") "
Sep 18 20:14:37.310123 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:14:37.310016    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/23a6c2f1-224e-43dc-8e48-ff2725bf6345-maas-api-tls\") pod \"23a6c2f1-224e-43dc-8e48-ff2725bf6345\" (UID: \"23a6c2f1-224e-43dc-8e48-ff2725bf6345\") "
Sep 18 20:14:37.310257 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:14:37.310221    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/23a6c2f1-224e-43dc-8e48-ff2725bf6345-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "23a6c2f1-224e-43dc-8e48-ff2725bf6345" (UID: "23a6c2f1-224e-43dc-8e48-ff2725bf6345"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 20:14:37.310392 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:14:37.310377    2585 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/23a6c2f1-224e-43dc-8e48-ff2725bf6345-openshift-service-ca\") on node \"ip-10-0-136-185.ec2.internal\" DevicePath \"\""
Sep 18 20:14:37.312089 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:14:37.312061    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/23a6c2f1-224e-43dc-8e48-ff2725bf6345-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "23a6c2f1-224e-43dc-8e48-ff2725bf6345" (UID: "23a6c2f1-224e-43dc-8e48-ff2725bf6345"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 20:14:37.312089 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:14:37.312082    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/23a6c2f1-224e-43dc-8e48-ff2725bf6345-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "23a6c2f1-224e-43dc-8e48-ff2725bf6345" (UID: "23a6c2f1-224e-43dc-8e48-ff2725bf6345"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 20:14:37.312238 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:14:37.312145    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/23a6c2f1-224e-43dc-8e48-ff2725bf6345-kube-api-access-56nbz" (OuterVolumeSpecName: "kube-api-access-56nbz") pod "23a6c2f1-224e-43dc-8e48-ff2725bf6345" (UID: "23a6c2f1-224e-43dc-8e48-ff2725bf6345"). InnerVolumeSpecName "kube-api-access-56nbz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 20:14:37.411476 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:14:37.411440    2585 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/23a6c2f1-224e-43dc-8e48-ff2725bf6345-metrics-certs\") on node \"ip-10-0-136-185.ec2.internal\" DevicePath \"\""
Sep 18 20:14:37.411476 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:14:37.411471    2585 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/23a6c2f1-224e-43dc-8e48-ff2725bf6345-maas-api-tls\") on node \"ip-10-0-136-185.ec2.internal\" DevicePath \"\""
Sep 18 20:14:37.411476 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:14:37.411481    2585 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-56nbz\" (UniqueName: \"kubernetes.io/projected/23a6c2f1-224e-43dc-8e48-ff2725bf6345-kube-api-access-56nbz\") on node \"ip-10-0-136-185.ec2.internal\" DevicePath \"\""
Sep 18 20:14:37.761726 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:14:37.761693    2585 generic.go:356] "Generic (PLEG): container finished" podID="23a6c2f1-224e-43dc-8e48-ff2725bf6345" containerID="1e6a291615a525b591fe2253f3d6c9ae9f4d05dd3fcde683c1b30cc55589c04c" exitCode=0
Sep 18 20:14:37.761950 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:14:37.761737    2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-953438-7bc466dcbf-4zxbk"
Sep 18 20:14:37.761950 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:14:37.761761    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-953438-7bc466dcbf-4zxbk" event={"ID":"23a6c2f1-224e-43dc-8e48-ff2725bf6345","Type":"ContainerDied","Data":"1e6a291615a525b591fe2253f3d6c9ae9f4d05dd3fcde683c1b30cc55589c04c"}
Sep 18 20:14:37.761950 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:14:37.761818    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-953438-7bc466dcbf-4zxbk" event={"ID":"23a6c2f1-224e-43dc-8e48-ff2725bf6345","Type":"ContainerDied","Data":"9f46d0f8e515ed11aabd55eb9439620c4eaac74ad50fc5a49ee338a1e763a01b"}
Sep 18 20:14:37.761950 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:14:37.761839    2585 scope.go:117] "RemoveContainer" containerID="1e6a291615a525b591fe2253f3d6c9ae9f4d05dd3fcde683c1b30cc55589c04c"
Sep 18 20:14:37.770962 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:14:37.770938    2585 scope.go:117] "RemoveContainer" containerID="1e6a291615a525b591fe2253f3d6c9ae9f4d05dd3fcde683c1b30cc55589c04c"
Sep 18 20:14:37.771230 ip-10-0-136-185 kubenswrapper[2585]: E0918 20:14:37.771211    2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1e6a291615a525b591fe2253f3d6c9ae9f4d05dd3fcde683c1b30cc55589c04c\": container with ID starting with 1e6a291615a525b591fe2253f3d6c9ae9f4d05dd3fcde683c1b30cc55589c04c not found: ID does not exist" containerID="1e6a291615a525b591fe2253f3d6c9ae9f4d05dd3fcde683c1b30cc55589c04c"
Sep 18 20:14:37.771277 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:14:37.771243    2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1e6a291615a525b591fe2253f3d6c9ae9f4d05dd3fcde683c1b30cc55589c04c"} err="failed to get container status \"1e6a291615a525b591fe2253f3d6c9ae9f4d05dd3fcde683c1b30cc55589c04c\": rpc error: code = NotFound desc = could not find container \"1e6a291615a525b591fe2253f3d6c9ae9f4d05dd3fcde683c1b30cc55589c04c\": container with ID starting with 1e6a291615a525b591fe2253f3d6c9ae9f4d05dd3fcde683c1b30cc55589c04c not found: ID does not exist"
Sep 18 20:14:37.812564 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:14:37.812528    2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-953438-7bc466dcbf-4zxbk"]
Sep 18 20:14:37.838026 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:14:37.837984    2585 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-953438-7bc466dcbf-4zxbk"]
Sep 18 20:14:38.662304 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:14:38.662270    2585 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="23a6c2f1-224e-43dc-8e48-ff2725bf6345" path="/var/lib/kubelet/pods/23a6c2f1-224e-43dc-8e48-ff2725bf6345/volumes"
Sep 18 20:14:43.973387 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:14:43.973341    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-xmwh7"]
Sep 18 20:15:00.142911 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:15:00.142877    2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-9c0eb2-29829375-vw9lj"]
Sep 18 20:15:00.143361 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:15:00.143292    2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="23a6c2f1-224e-43dc-8e48-ff2725bf6345" containerName="maas-api"
Sep 18 20:15:00.143361 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:15:00.143305    2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="23a6c2f1-224e-43dc-8e48-ff2725bf6345" containerName="maas-api"
Sep 18 20:15:00.143435 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:15:00.143375    2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="23a6c2f1-224e-43dc-8e48-ff2725bf6345" containerName="maas-api"
Sep 18 20:15:00.146342 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:15:00.146318    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-9c0eb2-29829375-vw9lj"
Sep 18 20:15:00.149563 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:15:00.149537    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-5ttnn\""
Sep 18 20:15:00.157563 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:15:00.157538    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-9c0eb2-29829375-vw9lj"
Sep 18 20:15:00.160170 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:15:00.160145    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-9c0eb2-29829375-vw9lj"]
Sep 18 20:15:00.435042 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:15:00.434882    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-9c0eb2-29829375-vw9lj"]
Sep 18 20:15:00.438103 ip-10-0-136-185 kubenswrapper[2585]: W0918 20:15:00.438075    2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2d79ad1a_0b21_4a05_bcc6_35a18cce0487.slice/crio-dd93d7aa292a042a7702e8de356d64da1027a50edcfe374b0c4151ca0ce71822 WatchSource:0}: Error finding container dd93d7aa292a042a7702e8de356d64da1027a50edcfe374b0c4151ca0ce71822: Status 404 returned error can't find the container with id dd93d7aa292a042a7702e8de356d64da1027a50edcfe374b0c4151ca0ce71822
Sep 18 20:15:00.872517 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:15:00.872483    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-9c0eb2-29829375-vw9lj" event={"ID":"2d79ad1a-0b21-4a05-bcc6-35a18cce0487","Type":"ContainerStarted","Data":"3a3c8818a6eb473fa43689258910a09cda3645a6b90f8e0f4a6ff3ba94e25dc8"}
Sep 18 20:15:00.872688 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:15:00.872534    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-9c0eb2-29829375-vw9lj" event={"ID":"2d79ad1a-0b21-4a05-bcc6-35a18cce0487","Type":"ContainerStarted","Data":"dd93d7aa292a042a7702e8de356d64da1027a50edcfe374b0c4151ca0ce71822"}
Sep 18 20:15:00.887939 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:15:00.887890    2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-9c0eb2-29829375-vw9lj" podStartSLOduration=0.887876165 podStartE2EDuration="887.876165ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 20:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 20:15:00.886644158 +0000 UTC m=+2142.411249582" watchObservedRunningTime="2026-09-18 20:15:00.887876165 +0000 UTC m=+2142.412481588"
Sep 18 20:15:01.879533 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:15:01.879496    2585 generic.go:356] "Generic (PLEG): container finished" podID="2d79ad1a-0b21-4a05-bcc6-35a18cce0487" containerID="3a3c8818a6eb473fa43689258910a09cda3645a6b90f8e0f4a6ff3ba94e25dc8" exitCode=0
Sep 18 20:15:01.879912 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:15:01.879590    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-9c0eb2-29829375-vw9lj" event={"ID":"2d79ad1a-0b21-4a05-bcc6-35a18cce0487","Type":"ContainerDied","Data":"3a3c8818a6eb473fa43689258910a09cda3645a6b90f8e0f4a6ff3ba94e25dc8"}
Sep 18 20:15:03.012452 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:15:03.012428    2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-9c0eb2-29829375-vw9lj"
Sep 18 20:15:03.891321 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:15:03.891291    2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-9c0eb2-29829375-vw9lj"
Sep 18 20:15:03.891321 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:15:03.891308    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-9c0eb2-29829375-vw9lj" event={"ID":"2d79ad1a-0b21-4a05-bcc6-35a18cce0487","Type":"ContainerDied","Data":"dd93d7aa292a042a7702e8de356d64da1027a50edcfe374b0c4151ca0ce71822"}
Sep 18 20:15:03.891537 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:15:03.891352    2585 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="dd93d7aa292a042a7702e8de356d64da1027a50edcfe374b0c4151ca0ce71822"
Sep 18 20:15:37.041482 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:15:37.041447    2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-9c0eb2-f78858fcf-z2pxh"]
Sep 18 20:15:37.041987 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:15:37.041677    2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-9c0eb2-f78858fcf-z2pxh" podUID="29458bcb-e7f0-4cc4-8b9a-0b7d4c51c608" containerName="maas-api" containerID="cri-o://0b550042d3e56771f65539c912fa224313179d63a1588044afef8a960776bc00" gracePeriod=30
Sep 18 20:15:37.071252 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:15:37.071214    2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-9c0eb2-29829375-vw9lj"]
Sep 18 20:15:37.073144 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:15:37.073118    2585 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-9c0eb2-29829375-vw9lj"]
Sep 18 20:15:37.097021 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:15:37.096983    2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-9c0eb2-674f4f784f-clxft"]
Sep 18 20:15:37.097276 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:15:37.097248    2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-9c0eb2-674f4f784f-clxft" podUID="42625f0e-8f4c-4d41-a85a-a0d9bedc7aed" containerName="payload-processing" containerID="cri-o://22b615dfe60fb730a4f9ce0b59dc814d1f2fc0914def7c703bcdc0ebf5f77e87" gracePeriod=30
Sep 18 20:15:37.104410 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:15:37.103962    2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-9c0eb2-765b9f6b74-twlrr"]
Sep 18 20:15:37.104410 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:15:37.104213    2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-9c0eb2-765b9f6b74-twlrr" podUID="5768b345-1195-4621-9148-6b63df0b645f" containerName="payload-pre-processing" containerID="cri-o://00394643af4f8e817a5e3980925c435c944b1a9a9241900b7fc24ae155870087" gracePeriod=30
Sep 18 20:15:37.391982 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:15:37.391953    2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-9c0eb2-f78858fcf-z2pxh"
Sep 18 20:15:37.395675 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:15:37.395643    2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-9c0eb2-674f4f784f-clxft"
Sep 18 20:15:37.402182 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:15:37.402161    2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-9c0eb2-765b9f6b74-twlrr"
Sep 18 20:15:37.467815 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:15:37.467769    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zlkpg\" (UniqueName: \"kubernetes.io/projected/29458bcb-e7f0-4cc4-8b9a-0b7d4c51c608-kube-api-access-zlkpg\") pod \"29458bcb-e7f0-4cc4-8b9a-0b7d4c51c608\" (UID: \"29458bcb-e7f0-4cc4-8b9a-0b7d4c51c608\") "
Sep 18 20:15:37.468004 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:15:37.467855    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/29458bcb-e7f0-4cc4-8b9a-0b7d4c51c608-openshift-service-ca\") pod \"29458bcb-e7f0-4cc4-8b9a-0b7d4c51c608\" (UID: \"29458bcb-e7f0-4cc4-8b9a-0b7d4c51c608\") "
Sep 18 20:15:37.468004 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:15:37.467879    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jgdxr\" (UniqueName: \"kubernetes.io/projected/42625f0e-8f4c-4d41-a85a-a0d9bedc7aed-kube-api-access-jgdxr\") pod \"42625f0e-8f4c-4d41-a85a-a0d9bedc7aed\" (UID: \"42625f0e-8f4c-4d41-a85a-a0d9bedc7aed\") "
Sep 18 20:15:37.468139 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:15:37.468077    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/29458bcb-e7f0-4cc4-8b9a-0b7d4c51c608-metrics-certs\") pod \"29458bcb-e7f0-4cc4-8b9a-0b7d4c51c608\" (UID: \"29458bcb-e7f0-4cc4-8b9a-0b7d4c51c608\") "
Sep 18 20:15:37.468199 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:15:37.468139    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fcmjw\" (UniqueName: \"kubernetes.io/projected/5768b345-1195-4621-9148-6b63df0b645f-kube-api-access-fcmjw\") pod \"5768b345-1195-4621-9148-6b63df0b645f\" (UID: \"5768b345-1195-4621-9148-6b63df0b645f\") "
Sep 18 20:15:37.468258 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:15:37.468207    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/29458bcb-e7f0-4cc4-8b9a-0b7d4c51c608-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "29458bcb-e7f0-4cc4-8b9a-0b7d4c51c608" (UID: "29458bcb-e7f0-4cc4-8b9a-0b7d4c51c608"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 20:15:37.468258 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:15:37.468221    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5768b345-1195-4621-9148-6b63df0b645f-plugins-config-volume\") pod \"5768b345-1195-4621-9148-6b63df0b645f\" (UID: \"5768b345-1195-4621-9148-6b63df0b645f\") "
Sep 18 20:15:37.468378 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:15:37.468280    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/29458bcb-e7f0-4cc4-8b9a-0b7d4c51c608-maas-api-tls\") pod \"29458bcb-e7f0-4cc4-8b9a-0b7d4c51c608\" (UID: \"29458bcb-e7f0-4cc4-8b9a-0b7d4c51c608\") "
Sep 18 20:15:37.468378 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:15:37.468311    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/42625f0e-8f4c-4d41-a85a-a0d9bedc7aed-plugins-config-volume\") pod \"42625f0e-8f4c-4d41-a85a-a0d9bedc7aed\" (UID: \"42625f0e-8f4c-4d41-a85a-a0d9bedc7aed\") "
Sep 18 20:15:37.469407 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:15:37.469370    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/42625f0e-8f4c-4d41-a85a-a0d9bedc7aed-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "42625f0e-8f4c-4d41-a85a-a0d9bedc7aed" (UID: "42625f0e-8f4c-4d41-a85a-a0d9bedc7aed"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 20:15:37.469935 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:15:37.468590    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5768b345-1195-4621-9148-6b63df0b645f-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "5768b345-1195-4621-9148-6b63df0b645f" (UID: "5768b345-1195-4621-9148-6b63df0b645f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 20:15:37.470053 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:15:37.469986    2585 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/42625f0e-8f4c-4d41-a85a-a0d9bedc7aed-plugins-config-volume\") on node \"ip-10-0-136-185.ec2.internal\" DevicePath \"\""
Sep 18 20:15:37.470053 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:15:37.470009    2585 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/29458bcb-e7f0-4cc4-8b9a-0b7d4c51c608-openshift-service-ca\") on node \"ip-10-0-136-185.ec2.internal\" DevicePath \"\""
Sep 18 20:15:37.471305 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:15:37.471120    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/29458bcb-e7f0-4cc4-8b9a-0b7d4c51c608-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "29458bcb-e7f0-4cc4-8b9a-0b7d4c51c608" (UID: "29458bcb-e7f0-4cc4-8b9a-0b7d4c51c608"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 20:15:37.471430 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:15:37.471345    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/29458bcb-e7f0-4cc4-8b9a-0b7d4c51c608-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "29458bcb-e7f0-4cc4-8b9a-0b7d4c51c608" (UID: "29458bcb-e7f0-4cc4-8b9a-0b7d4c51c608"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 20:15:37.471640 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:15:37.471619    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/29458bcb-e7f0-4cc4-8b9a-0b7d4c51c608-kube-api-access-zlkpg" (OuterVolumeSpecName: "kube-api-access-zlkpg") pod "29458bcb-e7f0-4cc4-8b9a-0b7d4c51c608" (UID: "29458bcb-e7f0-4cc4-8b9a-0b7d4c51c608"). InnerVolumeSpecName "kube-api-access-zlkpg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 20:15:37.478469 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:15:37.478437    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/42625f0e-8f4c-4d41-a85a-a0d9bedc7aed-kube-api-access-jgdxr" (OuterVolumeSpecName: "kube-api-access-jgdxr") pod "42625f0e-8f4c-4d41-a85a-a0d9bedc7aed" (UID: "42625f0e-8f4c-4d41-a85a-a0d9bedc7aed"). InnerVolumeSpecName "kube-api-access-jgdxr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 20:15:37.479323 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:15:37.479302    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5768b345-1195-4621-9148-6b63df0b645f-kube-api-access-fcmjw" (OuterVolumeSpecName: "kube-api-access-fcmjw") pod "5768b345-1195-4621-9148-6b63df0b645f" (UID: "5768b345-1195-4621-9148-6b63df0b645f"). InnerVolumeSpecName "kube-api-access-fcmjw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 20:15:37.571115 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:15:37.571078    2585 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fcmjw\" (UniqueName: \"kubernetes.io/projected/5768b345-1195-4621-9148-6b63df0b645f-kube-api-access-fcmjw\") on node \"ip-10-0-136-185.ec2.internal\" DevicePath \"\""
Sep 18 20:15:37.571115 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:15:37.571107    2585 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5768b345-1195-4621-9148-6b63df0b645f-plugins-config-volume\") on node \"ip-10-0-136-185.ec2.internal\" DevicePath \"\""
Sep 18 20:15:37.571115 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:15:37.571117    2585 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/29458bcb-e7f0-4cc4-8b9a-0b7d4c51c608-maas-api-tls\") on node \"ip-10-0-136-185.ec2.internal\" DevicePath \"\""
Sep 18 20:15:37.571115 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:15:37.571126    2585 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zlkpg\" (UniqueName: \"kubernetes.io/projected/29458bcb-e7f0-4cc4-8b9a-0b7d4c51c608-kube-api-access-zlkpg\") on node \"ip-10-0-136-185.ec2.internal\" DevicePath \"\""
Sep 18 20:15:37.571402 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:15:37.571135    2585 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jgdxr\" (UniqueName: \"kubernetes.io/projected/42625f0e-8f4c-4d41-a85a-a0d9bedc7aed-kube-api-access-jgdxr\") on node \"ip-10-0-136-185.ec2.internal\" DevicePath \"\""
Sep 18 20:15:37.571402 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:15:37.571143    2585 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/29458bcb-e7f0-4cc4-8b9a-0b7d4c51c608-metrics-certs\") on node \"ip-10-0-136-185.ec2.internal\" DevicePath \"\""
Sep 18 20:15:38.059447 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:15:38.059418    2585 generic.go:356] "Generic (PLEG): container finished" podID="5768b345-1195-4621-9148-6b63df0b645f" containerID="00394643af4f8e817a5e3980925c435c944b1a9a9241900b7fc24ae155870087" exitCode=0
Sep 18 20:15:38.059850 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:15:38.059508    2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-9c0eb2-765b9f6b74-twlrr"
Sep 18 20:15:38.059850 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:15:38.059525    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-9c0eb2-765b9f6b74-twlrr" event={"ID":"5768b345-1195-4621-9148-6b63df0b645f","Type":"ContainerDied","Data":"00394643af4f8e817a5e3980925c435c944b1a9a9241900b7fc24ae155870087"}
Sep 18 20:15:38.059850 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:15:38.059580    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-9c0eb2-765b9f6b74-twlrr" event={"ID":"5768b345-1195-4621-9148-6b63df0b645f","Type":"ContainerDied","Data":"06f0378539ea64cb4296cc4efab4072a5405e4544c547f51a5163518a7a7538e"}
Sep 18 20:15:38.059850 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:15:38.059615    2585 scope.go:117] "RemoveContainer" containerID="00394643af4f8e817a5e3980925c435c944b1a9a9241900b7fc24ae155870087"
Sep 18 20:15:38.061631 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:15:38.061605    2585 generic.go:356] "Generic (PLEG): container finished" podID="29458bcb-e7f0-4cc4-8b9a-0b7d4c51c608" containerID="0b550042d3e56771f65539c912fa224313179d63a1588044afef8a960776bc00" exitCode=0
Sep 18 20:15:38.061739 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:15:38.061672    2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-9c0eb2-f78858fcf-z2pxh"
Sep 18 20:15:38.062000 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:15:38.061692    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-9c0eb2-f78858fcf-z2pxh" event={"ID":"29458bcb-e7f0-4cc4-8b9a-0b7d4c51c608","Type":"ContainerDied","Data":"0b550042d3e56771f65539c912fa224313179d63a1588044afef8a960776bc00"}
Sep 18 20:15:38.062093 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:15:38.062036    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-9c0eb2-f78858fcf-z2pxh" event={"ID":"29458bcb-e7f0-4cc4-8b9a-0b7d4c51c608","Type":"ContainerDied","Data":"80d8a309f2a5a67e9c3082f9fec0c4213d17888c82d39eff343c844f7e7c7344"}
Sep 18 20:15:38.063630 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:15:38.063604    2585 generic.go:356] "Generic (PLEG): container finished" podID="42625f0e-8f4c-4d41-a85a-a0d9bedc7aed" containerID="22b615dfe60fb730a4f9ce0b59dc814d1f2fc0914def7c703bcdc0ebf5f77e87" exitCode=0
Sep 18 20:15:38.063744 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:15:38.063637    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-9c0eb2-674f4f784f-clxft" event={"ID":"42625f0e-8f4c-4d41-a85a-a0d9bedc7aed","Type":"ContainerDied","Data":"22b615dfe60fb730a4f9ce0b59dc814d1f2fc0914def7c703bcdc0ebf5f77e87"}
Sep 18 20:15:38.063744 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:15:38.063689    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-9c0eb2-674f4f784f-clxft" event={"ID":"42625f0e-8f4c-4d41-a85a-a0d9bedc7aed","Type":"ContainerDied","Data":"d20c7521c8dd713f43356727d0197d36b9951295f257b3a87c5871af2faa0a4f"}
Sep 18 20:15:38.063744 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:15:38.063737    2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-9c0eb2-674f4f784f-clxft"
Sep 18 20:15:38.070136 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:15:38.070061    2585 scope.go:117] "RemoveContainer" containerID="00394643af4f8e817a5e3980925c435c944b1a9a9241900b7fc24ae155870087"
Sep 18 20:15:38.070448 ip-10-0-136-185 kubenswrapper[2585]: E0918 20:15:38.070415    2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"00394643af4f8e817a5e3980925c435c944b1a9a9241900b7fc24ae155870087\": container with ID starting with 00394643af4f8e817a5e3980925c435c944b1a9a9241900b7fc24ae155870087 not found: ID does not exist" containerID="00394643af4f8e817a5e3980925c435c944b1a9a9241900b7fc24ae155870087"
Sep 18 20:15:38.070545 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:15:38.070450    2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"00394643af4f8e817a5e3980925c435c944b1a9a9241900b7fc24ae155870087"} err="failed to get container status \"00394643af4f8e817a5e3980925c435c944b1a9a9241900b7fc24ae155870087\": rpc error: code = NotFound desc = could not find container \"00394643af4f8e817a5e3980925c435c944b1a9a9241900b7fc24ae155870087\": container with ID starting with 00394643af4f8e817a5e3980925c435c944b1a9a9241900b7fc24ae155870087 not found: ID does not exist"
Sep 18 20:15:38.070545 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:15:38.070473    2585 scope.go:117] "RemoveContainer" containerID="0b550042d3e56771f65539c912fa224313179d63a1588044afef8a960776bc00"
Sep 18 20:15:38.079435 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:15:38.079417    2585 scope.go:117] "RemoveContainer" containerID="0b550042d3e56771f65539c912fa224313179d63a1588044afef8a960776bc00"
Sep 18 20:15:38.079704 ip-10-0-136-185 kubenswrapper[2585]: E0918 20:15:38.079688    2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0b550042d3e56771f65539c912fa224313179d63a1588044afef8a960776bc00\": container with ID starting with 0b550042d3e56771f65539c912fa224313179d63a1588044afef8a960776bc00 not found: ID does not exist" containerID="0b550042d3e56771f65539c912fa224313179d63a1588044afef8a960776bc00"
Sep 18 20:15:38.079741 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:15:38.079710    2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0b550042d3e56771f65539c912fa224313179d63a1588044afef8a960776bc00"} err="failed to get container status \"0b550042d3e56771f65539c912fa224313179d63a1588044afef8a960776bc00\": rpc error: code = NotFound desc = could not find container \"0b550042d3e56771f65539c912fa224313179d63a1588044afef8a960776bc00\": container with ID starting with 0b550042d3e56771f65539c912fa224313179d63a1588044afef8a960776bc00 not found: ID does not exist"
Sep 18 20:15:38.079741 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:15:38.079725    2585 scope.go:117] "RemoveContainer" containerID="22b615dfe60fb730a4f9ce0b59dc814d1f2fc0914def7c703bcdc0ebf5f77e87"
Sep 18 20:15:38.082721 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:15:38.082699    2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-9c0eb2-765b9f6b74-twlrr"]
Sep 18 20:15:38.088282 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:15:38.088258    2585 scope.go:117] "RemoveContainer" containerID="22b615dfe60fb730a4f9ce0b59dc814d1f2fc0914def7c703bcdc0ebf5f77e87"
Sep 18 20:15:38.088552 ip-10-0-136-185 kubenswrapper[2585]: E0918 20:15:38.088533    2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"22b615dfe60fb730a4f9ce0b59dc814d1f2fc0914def7c703bcdc0ebf5f77e87\": container with ID starting with 22b615dfe60fb730a4f9ce0b59dc814d1f2fc0914def7c703bcdc0ebf5f77e87 not found: ID does not exist" containerID="22b615dfe60fb730a4f9ce0b59dc814d1f2fc0914def7c703bcdc0ebf5f77e87"
Sep 18 20:15:38.088632 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:15:38.088556    2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"22b615dfe60fb730a4f9ce0b59dc814d1f2fc0914def7c703bcdc0ebf5f77e87"} err="failed to get container status \"22b615dfe60fb730a4f9ce0b59dc814d1f2fc0914def7c703bcdc0ebf5f77e87\": rpc error: code = NotFound desc = could not find container \"22b615dfe60fb730a4f9ce0b59dc814d1f2fc0914def7c703bcdc0ebf5f77e87\": container with ID starting with 22b615dfe60fb730a4f9ce0b59dc814d1f2fc0914def7c703bcdc0ebf5f77e87 not found: ID does not exist"
Sep 18 20:15:38.088821 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:15:38.088800    2585 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-9c0eb2-765b9f6b74-twlrr"]
Sep 18 20:15:38.098067 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:15:38.098044    2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-9c0eb2-f78858fcf-z2pxh"]
Sep 18 20:15:38.100320 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:15:38.100296    2585 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-9c0eb2-f78858fcf-z2pxh"]
Sep 18 20:15:38.110226 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:15:38.110204    2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-9c0eb2-674f4f784f-clxft"]
Sep 18 20:15:38.113941 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:15:38.113920    2585 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-9c0eb2-674f4f784f-clxft"]
Sep 18 20:15:38.661251 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:15:38.661221    2585 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="29458bcb-e7f0-4cc4-8b9a-0b7d4c51c608" path="/var/lib/kubelet/pods/29458bcb-e7f0-4cc4-8b9a-0b7d4c51c608/volumes"
Sep 18 20:15:38.661541 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:15:38.661528    2585 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2d79ad1a-0b21-4a05-bcc6-35a18cce0487" path="/var/lib/kubelet/pods/2d79ad1a-0b21-4a05-bcc6-35a18cce0487/volumes"
Sep 18 20:15:38.661845 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:15:38.661833    2585 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="42625f0e-8f4c-4d41-a85a-a0d9bedc7aed" path="/var/lib/kubelet/pods/42625f0e-8f4c-4d41-a85a-a0d9bedc7aed/volumes"
Sep 18 20:15:38.662176 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:15:38.662165    2585 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5768b345-1195-4621-9148-6b63df0b645f" path="/var/lib/kubelet/pods/5768b345-1195-4621-9148-6b63df0b645f/volumes"
Sep 18 20:15:40.511605 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:15:40.511568    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-xmwh7"]
Sep 18 20:15:54.530179 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:15:54.530122    2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-57b6ce-5cd84f68c5-n6t8k"]
Sep 18 20:15:54.530780 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:15:54.530395    2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-57b6ce-5cd84f68c5-n6t8k" podUID="66260672-3320-4a75-a853-ba1df8d1e872" containerName="payload-processing" containerID="cri-o://9d2b223a9929ac6f6dffbf993bfa61a85e24ca16729b05ca70e225c15406737a" gracePeriod=30
Sep 18 20:15:54.810389 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:15:54.810362    2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-57b6ce-5cd84f68c5-n6t8k"
Sep 18 20:15:54.928567 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:15:54.928529    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xxrjc\" (UniqueName: \"kubernetes.io/projected/66260672-3320-4a75-a853-ba1df8d1e872-kube-api-access-xxrjc\") pod \"66260672-3320-4a75-a853-ba1df8d1e872\" (UID: \"66260672-3320-4a75-a853-ba1df8d1e872\") "
Sep 18 20:15:54.928745 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:15:54.928602    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/66260672-3320-4a75-a853-ba1df8d1e872-plugins-config-volume\") pod \"66260672-3320-4a75-a853-ba1df8d1e872\" (UID: \"66260672-3320-4a75-a853-ba1df8d1e872\") "
Sep 18 20:15:54.929005 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:15:54.928981    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/66260672-3320-4a75-a853-ba1df8d1e872-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "66260672-3320-4a75-a853-ba1df8d1e872" (UID: "66260672-3320-4a75-a853-ba1df8d1e872"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 20:15:54.930772 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:15:54.930751    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/66260672-3320-4a75-a853-ba1df8d1e872-kube-api-access-xxrjc" (OuterVolumeSpecName: "kube-api-access-xxrjc") pod "66260672-3320-4a75-a853-ba1df8d1e872" (UID: "66260672-3320-4a75-a853-ba1df8d1e872"). InnerVolumeSpecName "kube-api-access-xxrjc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 20:15:55.030375 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:15:55.030332    2585 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xxrjc\" (UniqueName: \"kubernetes.io/projected/66260672-3320-4a75-a853-ba1df8d1e872-kube-api-access-xxrjc\") on node \"ip-10-0-136-185.ec2.internal\" DevicePath \"\""
Sep 18 20:15:55.030375 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:15:55.030368    2585 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/66260672-3320-4a75-a853-ba1df8d1e872-plugins-config-volume\") on node \"ip-10-0-136-185.ec2.internal\" DevicePath \"\""
Sep 18 20:15:55.146587 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:15:55.146553    2585 generic.go:356] "Generic (PLEG): container finished" podID="66260672-3320-4a75-a853-ba1df8d1e872" containerID="9d2b223a9929ac6f6dffbf993bfa61a85e24ca16729b05ca70e225c15406737a" exitCode=0
Sep 18 20:15:55.146778 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:15:55.146596    2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-57b6ce-5cd84f68c5-n6t8k"
Sep 18 20:15:55.146778 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:15:55.146608    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-57b6ce-5cd84f68c5-n6t8k" event={"ID":"66260672-3320-4a75-a853-ba1df8d1e872","Type":"ContainerDied","Data":"9d2b223a9929ac6f6dffbf993bfa61a85e24ca16729b05ca70e225c15406737a"}
Sep 18 20:15:55.146778 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:15:55.146648    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-57b6ce-5cd84f68c5-n6t8k" event={"ID":"66260672-3320-4a75-a853-ba1df8d1e872","Type":"ContainerDied","Data":"6b7a38960b32737a7cda49dbd7b19d0ae94fe97c5bfe6a5e7dd3399a2e736936"}
Sep 18 20:15:55.146778 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:15:55.146670    2585 scope.go:117] "RemoveContainer" containerID="9d2b223a9929ac6f6dffbf993bfa61a85e24ca16729b05ca70e225c15406737a"
Sep 18 20:15:55.156123 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:15:55.156104    2585 scope.go:117] "RemoveContainer" containerID="9d2b223a9929ac6f6dffbf993bfa61a85e24ca16729b05ca70e225c15406737a"
Sep 18 20:15:55.156444 ip-10-0-136-185 kubenswrapper[2585]: E0918 20:15:55.156417    2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9d2b223a9929ac6f6dffbf993bfa61a85e24ca16729b05ca70e225c15406737a\": container with ID starting with 9d2b223a9929ac6f6dffbf993bfa61a85e24ca16729b05ca70e225c15406737a not found: ID does not exist" containerID="9d2b223a9929ac6f6dffbf993bfa61a85e24ca16729b05ca70e225c15406737a"
Sep 18 20:15:55.156535 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:15:55.156451    2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9d2b223a9929ac6f6dffbf993bfa61a85e24ca16729b05ca70e225c15406737a"} err="failed to get container status \"9d2b223a9929ac6f6dffbf993bfa61a85e24ca16729b05ca70e225c15406737a\": rpc error: code = NotFound desc = could not find container \"9d2b223a9929ac6f6dffbf993bfa61a85e24ca16729b05ca70e225c15406737a\": container with ID starting with 9d2b223a9929ac6f6dffbf993bfa61a85e24ca16729b05ca70e225c15406737a not found: ID does not exist"
Sep 18 20:15:55.169238 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:15:55.169204    2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-57b6ce-5cd84f68c5-n6t8k"]
Sep 18 20:15:55.175062 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:15:55.175036    2585 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-57b6ce-5cd84f68c5-n6t8k"]
Sep 18 20:15:56.661417 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:15:56.661379    2585 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="66260672-3320-4a75-a853-ba1df8d1e872" path="/var/lib/kubelet/pods/66260672-3320-4a75-a853-ba1df8d1e872/volumes"
Sep 18 20:15:57.774702 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:15:57.774667    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-xmwh7"]
Sep 18 20:16:12.328055 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:16:12.328019    2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-5547b6-576c844cb6-jz58p"]
Sep 18 20:16:12.328518 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:16:12.328271    2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-5547b6-576c844cb6-jz58p" podUID="56b475f5-65d7-4b6d-a8ba-89654ec9adaa" containerName="payload-pre-processing" containerID="cri-o://9be2061167406c237d7862973bfdc207551b0cc8418ff5b307568652fc5341d8" gracePeriod=30
Sep 18 20:16:12.582366 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:16:12.582297    2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-5547b6-576c844cb6-jz58p"
Sep 18 20:16:12.685551 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:16:12.685511    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tnqld\" (UniqueName: \"kubernetes.io/projected/56b475f5-65d7-4b6d-a8ba-89654ec9adaa-kube-api-access-tnqld\") pod \"56b475f5-65d7-4b6d-a8ba-89654ec9adaa\" (UID: \"56b475f5-65d7-4b6d-a8ba-89654ec9adaa\") "
Sep 18 20:16:12.685757 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:16:12.685599    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/56b475f5-65d7-4b6d-a8ba-89654ec9adaa-plugins-config-volume\") pod \"56b475f5-65d7-4b6d-a8ba-89654ec9adaa\" (UID: \"56b475f5-65d7-4b6d-a8ba-89654ec9adaa\") "
Sep 18 20:16:12.686046 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:16:12.686022    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/56b475f5-65d7-4b6d-a8ba-89654ec9adaa-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "56b475f5-65d7-4b6d-a8ba-89654ec9adaa" (UID: "56b475f5-65d7-4b6d-a8ba-89654ec9adaa"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 20:16:12.687850 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:16:12.687829    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/56b475f5-65d7-4b6d-a8ba-89654ec9adaa-kube-api-access-tnqld" (OuterVolumeSpecName: "kube-api-access-tnqld") pod "56b475f5-65d7-4b6d-a8ba-89654ec9adaa" (UID: "56b475f5-65d7-4b6d-a8ba-89654ec9adaa"). InnerVolumeSpecName "kube-api-access-tnqld". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 20:16:12.787156 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:16:12.787114    2585 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tnqld\" (UniqueName: \"kubernetes.io/projected/56b475f5-65d7-4b6d-a8ba-89654ec9adaa-kube-api-access-tnqld\") on node \"ip-10-0-136-185.ec2.internal\" DevicePath \"\""
Sep 18 20:16:12.787156 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:16:12.787147    2585 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/56b475f5-65d7-4b6d-a8ba-89654ec9adaa-plugins-config-volume\") on node \"ip-10-0-136-185.ec2.internal\" DevicePath \"\""
Sep 18 20:16:13.233307 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:16:13.233256    2585 generic.go:356] "Generic (PLEG): container finished" podID="56b475f5-65d7-4b6d-a8ba-89654ec9adaa" containerID="9be2061167406c237d7862973bfdc207551b0cc8418ff5b307568652fc5341d8" exitCode=0
Sep 18 20:16:13.233564 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:16:13.233312    2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-5547b6-576c844cb6-jz58p"
Sep 18 20:16:13.233564 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:16:13.233346    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-5547b6-576c844cb6-jz58p" event={"ID":"56b475f5-65d7-4b6d-a8ba-89654ec9adaa","Type":"ContainerDied","Data":"9be2061167406c237d7862973bfdc207551b0cc8418ff5b307568652fc5341d8"}
Sep 18 20:16:13.233564 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:16:13.233401    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-5547b6-576c844cb6-jz58p" event={"ID":"56b475f5-65d7-4b6d-a8ba-89654ec9adaa","Type":"ContainerDied","Data":"e1043238b5be88bb33bc07ca4558c4c5179ee49d6b1fda17ba9aebc9beb409ea"}
Sep 18 20:16:13.233564 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:16:13.233423    2585 scope.go:117] "RemoveContainer" containerID="9be2061167406c237d7862973bfdc207551b0cc8418ff5b307568652fc5341d8"
Sep 18 20:16:13.242478 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:16:13.242456    2585 scope.go:117] "RemoveContainer" containerID="9be2061167406c237d7862973bfdc207551b0cc8418ff5b307568652fc5341d8"
Sep 18 20:16:13.242897 ip-10-0-136-185 kubenswrapper[2585]: E0918 20:16:13.242782    2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9be2061167406c237d7862973bfdc207551b0cc8418ff5b307568652fc5341d8\": container with ID starting with 9be2061167406c237d7862973bfdc207551b0cc8418ff5b307568652fc5341d8 not found: ID does not exist" containerID="9be2061167406c237d7862973bfdc207551b0cc8418ff5b307568652fc5341d8"
Sep 18 20:16:13.242897 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:16:13.242827    2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9be2061167406c237d7862973bfdc207551b0cc8418ff5b307568652fc5341d8"} err="failed to get container status \"9be2061167406c237d7862973bfdc207551b0cc8418ff5b307568652fc5341d8\": rpc error: code = NotFound desc = could not find container \"9be2061167406c237d7862973bfdc207551b0cc8418ff5b307568652fc5341d8\": container with ID starting with 9be2061167406c237d7862973bfdc207551b0cc8418ff5b307568652fc5341d8 not found: ID does not exist"
Sep 18 20:16:13.253165 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:16:13.253128    2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-5547b6-576c844cb6-jz58p"]
Sep 18 20:16:13.255228 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:16:13.255202    2585 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-5547b6-576c844cb6-jz58p"]
Sep 18 20:16:14.661533 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:16:14.661496    2585 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="56b475f5-65d7-4b6d-a8ba-89654ec9adaa" path="/var/lib/kubelet/pods/56b475f5-65d7-4b6d-a8ba-89654ec9adaa/volumes"
Sep 18 20:16:15.065504 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:16:15.065419    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-xmwh7"]
Sep 18 20:16:23.982949 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:16:23.982910    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-xmwh7"]
Sep 18 20:17:47.178874 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:17:47.178833    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-xmwh7"]
Sep 18 20:18:26.962325 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:18:26.962287    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-xmwh7"]
Sep 18 20:18:35.064752 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:18:35.064714    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-xmwh7"]
Sep 18 20:18:42.969723 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:18:42.969678    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-xmwh7"]
Sep 18 20:19:01.472985 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:19:01.472942    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-xmwh7"]
Sep 18 20:19:20.751692 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:19:20.751659    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-rctld_cc0e0de2-75d5-43a1-b64b-b071427cd8ce/ovn-acl-logging/0.log"
Sep 18 20:19:20.762172 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:19:20.762148    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-rctld_cc0e0de2-75d5-43a1-b64b-b071427cd8ce/ovn-acl-logging/0.log"
Sep 18 20:20:28.568498 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:20:28.568463    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-xmwh7"]
Sep 18 20:20:35.272217 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:20:35.272175    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-xmwh7"]
Sep 18 20:20:42.669400 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:20:42.669353    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-xmwh7"]
Sep 18 20:20:43.466718 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:20:43.466680    2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-same-id-a-ccd585a7-kserve-6f54dc9c75-pprt9"]
Sep 18 20:20:43.467043 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:20:43.467030    2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="56b475f5-65d7-4b6d-a8ba-89654ec9adaa" containerName="payload-pre-processing"
Sep 18 20:20:43.467092 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:20:43.467044    2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="56b475f5-65d7-4b6d-a8ba-89654ec9adaa" containerName="payload-pre-processing"
Sep 18 20:20:43.467092 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:20:43.467065    2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5768b345-1195-4621-9148-6b63df0b645f" containerName="payload-pre-processing"
Sep 18 20:20:43.467092 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:20:43.467072    2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="5768b345-1195-4621-9148-6b63df0b645f" containerName="payload-pre-processing"
Sep 18 20:20:43.467092 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:20:43.467080    2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2d79ad1a-0b21-4a05-bcc6-35a18cce0487" containerName="cleanup"
Sep 18 20:20:43.467092 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:20:43.467086    2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="2d79ad1a-0b21-4a05-bcc6-35a18cce0487" containerName="cleanup"
Sep 18 20:20:43.467252 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:20:43.467093    2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="29458bcb-e7f0-4cc4-8b9a-0b7d4c51c608" containerName="maas-api"
Sep 18 20:20:43.467252 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:20:43.467099    2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="29458bcb-e7f0-4cc4-8b9a-0b7d4c51c608" containerName="maas-api"
Sep 18 20:20:43.467252 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:20:43.467104    2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="66260672-3320-4a75-a853-ba1df8d1e872" containerName="payload-processing"
Sep 18 20:20:43.467252 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:20:43.467111    2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="66260672-3320-4a75-a853-ba1df8d1e872" containerName="payload-processing"
Sep 18 20:20:43.467252 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:20:43.467127    2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="42625f0e-8f4c-4d41-a85a-a0d9bedc7aed" containerName="payload-processing"
Sep 18 20:20:43.467252 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:20:43.467132    2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="42625f0e-8f4c-4d41-a85a-a0d9bedc7aed" containerName="payload-processing"
Sep 18 20:20:43.467252 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:20:43.467182    2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="56b475f5-65d7-4b6d-a8ba-89654ec9adaa" containerName="payload-pre-processing"
Sep 18 20:20:43.467252 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:20:43.467190    2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="29458bcb-e7f0-4cc4-8b9a-0b7d4c51c608" containerName="maas-api"
Sep 18 20:20:43.467252 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:20:43.467195    2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="2d79ad1a-0b21-4a05-bcc6-35a18cce0487" containerName="cleanup"
Sep 18 20:20:43.467252 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:20:43.467203    2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="66260672-3320-4a75-a853-ba1df8d1e872" containerName="payload-processing"
Sep 18 20:20:43.467252 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:20:43.467210    2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="5768b345-1195-4621-9148-6b63df0b645f" containerName="payload-pre-processing"
Sep 18 20:20:43.467252 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:20:43.467215    2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="42625f0e-8f4c-4d41-a85a-a0d9bedc7aed" containerName="payload-processing"
Sep 18 20:20:43.469629 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:20:43.469612    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-ccd585a7-kserve-6f54dc9c75-pprt9"
Sep 18 20:20:43.472124 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:20:43.472100    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-same-id-a-ccd585a7-kserve-self-signed-certs\""
Sep 18 20:20:43.482922 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:20:43.482897    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-a-ccd585a7-kserve-6f54dc9c75-pprt9"]
Sep 18 20:20:43.542884 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:20:43.542851    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f7821a90-17ac-4cd1-8178-eb555e3a9dd3-dshm\") pod \"e2e-same-id-a-ccd585a7-kserve-6f54dc9c75-pprt9\" (UID: \"f7821a90-17ac-4cd1-8178-eb555e3a9dd3\") " pod="llm/e2e-same-id-a-ccd585a7-kserve-6f54dc9c75-pprt9"
Sep 18 20:20:43.542884 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:20:43.542898    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f7821a90-17ac-4cd1-8178-eb555e3a9dd3-tmp-dir\") pod \"e2e-same-id-a-ccd585a7-kserve-6f54dc9c75-pprt9\" (UID: \"f7821a90-17ac-4cd1-8178-eb555e3a9dd3\") " pod="llm/e2e-same-id-a-ccd585a7-kserve-6f54dc9c75-pprt9"
Sep 18 20:20:43.543122 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:20:43.542926    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f7821a90-17ac-4cd1-8178-eb555e3a9dd3-home\") pod \"e2e-same-id-a-ccd585a7-kserve-6f54dc9c75-pprt9\" (UID: \"f7821a90-17ac-4cd1-8178-eb555e3a9dd3\") " pod="llm/e2e-same-id-a-ccd585a7-kserve-6f54dc9c75-pprt9"
Sep 18 20:20:43.543122 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:20:43.543016    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f7821a90-17ac-4cd1-8178-eb555e3a9dd3-tls-certs\") pod \"e2e-same-id-a-ccd585a7-kserve-6f54dc9c75-pprt9\" (UID: \"f7821a90-17ac-4cd1-8178-eb555e3a9dd3\") " pod="llm/e2e-same-id-a-ccd585a7-kserve-6f54dc9c75-pprt9"
Sep 18 20:20:43.543122 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:20:43.543047    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f7821a90-17ac-4cd1-8178-eb555e3a9dd3-model-cache\") pod \"e2e-same-id-a-ccd585a7-kserve-6f54dc9c75-pprt9\" (UID: \"f7821a90-17ac-4cd1-8178-eb555e3a9dd3\") " pod="llm/e2e-same-id-a-ccd585a7-kserve-6f54dc9c75-pprt9"
Sep 18 20:20:43.543122 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:20:43.543067    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z62t2\" (UniqueName: \"kubernetes.io/projected/f7821a90-17ac-4cd1-8178-eb555e3a9dd3-kube-api-access-z62t2\") pod \"e2e-same-id-a-ccd585a7-kserve-6f54dc9c75-pprt9\" (UID: \"f7821a90-17ac-4cd1-8178-eb555e3a9dd3\") " pod="llm/e2e-same-id-a-ccd585a7-kserve-6f54dc9c75-pprt9"
Sep 18 20:20:43.643864 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:20:43.643818    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f7821a90-17ac-4cd1-8178-eb555e3a9dd3-dshm\") pod \"e2e-same-id-a-ccd585a7-kserve-6f54dc9c75-pprt9\" (UID: \"f7821a90-17ac-4cd1-8178-eb555e3a9dd3\") " pod="llm/e2e-same-id-a-ccd585a7-kserve-6f54dc9c75-pprt9"
Sep 18 20:20:43.643864 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:20:43.643868    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f7821a90-17ac-4cd1-8178-eb555e3a9dd3-tmp-dir\") pod \"e2e-same-id-a-ccd585a7-kserve-6f54dc9c75-pprt9\" (UID: \"f7821a90-17ac-4cd1-8178-eb555e3a9dd3\") " pod="llm/e2e-same-id-a-ccd585a7-kserve-6f54dc9c75-pprt9"
Sep 18 20:20:43.644117 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:20:43.643892    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f7821a90-17ac-4cd1-8178-eb555e3a9dd3-home\") pod \"e2e-same-id-a-ccd585a7-kserve-6f54dc9c75-pprt9\" (UID: \"f7821a90-17ac-4cd1-8178-eb555e3a9dd3\") " pod="llm/e2e-same-id-a-ccd585a7-kserve-6f54dc9c75-pprt9"
Sep 18 20:20:43.644117 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:20:43.643926    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f7821a90-17ac-4cd1-8178-eb555e3a9dd3-tls-certs\") pod \"e2e-same-id-a-ccd585a7-kserve-6f54dc9c75-pprt9\" (UID: \"f7821a90-17ac-4cd1-8178-eb555e3a9dd3\") " pod="llm/e2e-same-id-a-ccd585a7-kserve-6f54dc9c75-pprt9"
Sep 18 20:20:43.644117 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:20:43.643942    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f7821a90-17ac-4cd1-8178-eb555e3a9dd3-model-cache\") pod \"e2e-same-id-a-ccd585a7-kserve-6f54dc9c75-pprt9\" (UID: \"f7821a90-17ac-4cd1-8178-eb555e3a9dd3\") " pod="llm/e2e-same-id-a-ccd585a7-kserve-6f54dc9c75-pprt9"
Sep 18 20:20:43.644117 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:20:43.643963    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z62t2\" (UniqueName: \"kubernetes.io/projected/f7821a90-17ac-4cd1-8178-eb555e3a9dd3-kube-api-access-z62t2\") pod \"e2e-same-id-a-ccd585a7-kserve-6f54dc9c75-pprt9\" (UID: \"f7821a90-17ac-4cd1-8178-eb555e3a9dd3\") " pod="llm/e2e-same-id-a-ccd585a7-kserve-6f54dc9c75-pprt9"
Sep 18 20:20:43.644349 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:20:43.644317    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f7821a90-17ac-4cd1-8178-eb555e3a9dd3-tmp-dir\") pod \"e2e-same-id-a-ccd585a7-kserve-6f54dc9c75-pprt9\" (UID: \"f7821a90-17ac-4cd1-8178-eb555e3a9dd3\") " pod="llm/e2e-same-id-a-ccd585a7-kserve-6f54dc9c75-pprt9"
Sep 18 20:20:43.644492 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:20:43.644367    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f7821a90-17ac-4cd1-8178-eb555e3a9dd3-home\") pod \"e2e-same-id-a-ccd585a7-kserve-6f54dc9c75-pprt9\" (UID: \"f7821a90-17ac-4cd1-8178-eb555e3a9dd3\") " pod="llm/e2e-same-id-a-ccd585a7-kserve-6f54dc9c75-pprt9"
Sep 18 20:20:43.644492 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:20:43.644440    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f7821a90-17ac-4cd1-8178-eb555e3a9dd3-model-cache\") pod \"e2e-same-id-a-ccd585a7-kserve-6f54dc9c75-pprt9\" (UID: \"f7821a90-17ac-4cd1-8178-eb555e3a9dd3\") " pod="llm/e2e-same-id-a-ccd585a7-kserve-6f54dc9c75-pprt9"
Sep 18 20:20:43.646314 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:20:43.646293    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f7821a90-17ac-4cd1-8178-eb555e3a9dd3-dshm\") pod \"e2e-same-id-a-ccd585a7-kserve-6f54dc9c75-pprt9\" (UID: \"f7821a90-17ac-4cd1-8178-eb555e3a9dd3\") " pod="llm/e2e-same-id-a-ccd585a7-kserve-6f54dc9c75-pprt9"
Sep 18 20:20:43.646579 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:20:43.646559    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f7821a90-17ac-4cd1-8178-eb555e3a9dd3-tls-certs\") pod \"e2e-same-id-a-ccd585a7-kserve-6f54dc9c75-pprt9\" (UID: \"f7821a90-17ac-4cd1-8178-eb555e3a9dd3\") " pod="llm/e2e-same-id-a-ccd585a7-kserve-6f54dc9c75-pprt9"
Sep 18 20:20:43.651134 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:20:43.651111    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-z62t2\" (UniqueName: \"kubernetes.io/projected/f7821a90-17ac-4cd1-8178-eb555e3a9dd3-kube-api-access-z62t2\") pod \"e2e-same-id-a-ccd585a7-kserve-6f54dc9c75-pprt9\" (UID: \"f7821a90-17ac-4cd1-8178-eb555e3a9dd3\") " pod="llm/e2e-same-id-a-ccd585a7-kserve-6f54dc9c75-pprt9"
Sep 18 20:20:43.779833 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:20:43.779696    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-ccd585a7-kserve-6f54dc9c75-pprt9"
Sep 18 20:20:44.051628 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:20:44.051599    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-a-ccd585a7-kserve-6f54dc9c75-pprt9"]
Sep 18 20:20:44.053642 ip-10-0-136-185 kubenswrapper[2585]: W0918 20:20:44.053597    2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf7821a90_17ac_4cd1_8178_eb555e3a9dd3.slice/crio-e397a4135803bb173ffa6e43ac06a1c71a69927d4bd596fa936d488635213bb5 WatchSource:0}: Error finding container e397a4135803bb173ffa6e43ac06a1c71a69927d4bd596fa936d488635213bb5: Status 404 returned error can't find the container with id e397a4135803bb173ffa6e43ac06a1c71a69927d4bd596fa936d488635213bb5
Sep 18 20:20:44.483976 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:20:44.483935    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-ccd585a7-kserve-6f54dc9c75-pprt9" event={"ID":"f7821a90-17ac-4cd1-8178-eb555e3a9dd3","Type":"ContainerStarted","Data":"ac573cf075fb9effeb5cde77803b295b0ee45497d1133574ebedd5a0d29bc99c"}
Sep 18 20:20:44.483976 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:20:44.483982    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-ccd585a7-kserve-6f54dc9c75-pprt9" event={"ID":"f7821a90-17ac-4cd1-8178-eb555e3a9dd3","Type":"ContainerStarted","Data":"e397a4135803bb173ffa6e43ac06a1c71a69927d4bd596fa936d488635213bb5"}
Sep 18 20:20:44.499254 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:20:44.499196    2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-same-id-a-ccd585a7-kserve-6f54dc9c75-pprt9" podStartSLOduration=1.499178122 podStartE2EDuration="1.499178122s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 20:20:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 20:20:44.498853242 +0000 UTC m=+2486.023458665" watchObservedRunningTime="2026-09-18 20:20:44.499178122 +0000 UTC m=+2486.023783546"
Sep 18 20:20:51.969725 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:20:51.969689    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-xmwh7"]
Sep 18 20:20:53.780634 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:20:53.780594    2585 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-same-id-a-ccd585a7-kserve-6f54dc9c75-pprt9"
Sep 18 20:20:53.780634 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:20:53.780640    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-same-id-a-ccd585a7-kserve-6f54dc9c75-pprt9"
Sep 18 20:20:53.793183 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:20:53.793152    2585 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-same-id-a-ccd585a7-kserve-6f54dc9c75-pprt9"
Sep 18 20:20:54.541322 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:20:54.541285    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-same-id-a-ccd585a7-kserve-6f54dc9c75-pprt9"
Sep 18 20:21:13.937138 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:21:13.937105    2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-a-ccd585a7-kserve-6f54dc9c75-pprt9"]
Sep 18 20:21:17.277725 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:21:17.277689    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-xmwh7"]
Sep 18 20:21:21.053195 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:21:21.053164    2585 scope.go:117] "RemoveContainer" containerID="3a3c8818a6eb473fa43689258910a09cda3645a6b90f8e0f4a6ff3ba94e25dc8"
Sep 18 20:21:28.959249 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:21:28.959211    2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="llm/e2e-same-id-a-ccd585a7-kserve-6f54dc9c75-pprt9" podUID="f7821a90-17ac-4cd1-8178-eb555e3a9dd3" containerName="main" containerID="cri-o://ac573cf075fb9effeb5cde77803b295b0ee45497d1133574ebedd5a0d29bc99c" gracePeriod=45
Sep 18 20:21:29.199478 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:21:29.199452    2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-ccd585a7-kserve-6f54dc9c75-pprt9"
Sep 18 20:21:29.266062 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:21:29.265968    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-z62t2\" (UniqueName: \"kubernetes.io/projected/f7821a90-17ac-4cd1-8178-eb555e3a9dd3-kube-api-access-z62t2\") pod \"f7821a90-17ac-4cd1-8178-eb555e3a9dd3\" (UID: \"f7821a90-17ac-4cd1-8178-eb555e3a9dd3\") "
Sep 18 20:21:29.266062 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:21:29.266054    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f7821a90-17ac-4cd1-8178-eb555e3a9dd3-home\") pod \"f7821a90-17ac-4cd1-8178-eb555e3a9dd3\" (UID: \"f7821a90-17ac-4cd1-8178-eb555e3a9dd3\") "
Sep 18 20:21:29.266268 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:21:29.266079    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f7821a90-17ac-4cd1-8178-eb555e3a9dd3-tmp-dir\") pod \"f7821a90-17ac-4cd1-8178-eb555e3a9dd3\" (UID: \"f7821a90-17ac-4cd1-8178-eb555e3a9dd3\") "
Sep 18 20:21:29.266268 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:21:29.266117    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f7821a90-17ac-4cd1-8178-eb555e3a9dd3-dshm\") pod \"f7821a90-17ac-4cd1-8178-eb555e3a9dd3\" (UID: \"f7821a90-17ac-4cd1-8178-eb555e3a9dd3\") "
Sep 18 20:21:29.266268 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:21:29.266148    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f7821a90-17ac-4cd1-8178-eb555e3a9dd3-tls-certs\") pod \"f7821a90-17ac-4cd1-8178-eb555e3a9dd3\" (UID: \"f7821a90-17ac-4cd1-8178-eb555e3a9dd3\") "
Sep 18 20:21:29.266268 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:21:29.266204    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f7821a90-17ac-4cd1-8178-eb555e3a9dd3-model-cache\") pod \"f7821a90-17ac-4cd1-8178-eb555e3a9dd3\" (UID: \"f7821a90-17ac-4cd1-8178-eb555e3a9dd3\") "
Sep 18 20:21:29.266482 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:21:29.266350    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f7821a90-17ac-4cd1-8178-eb555e3a9dd3-home" (OuterVolumeSpecName: "home") pod "f7821a90-17ac-4cd1-8178-eb555e3a9dd3" (UID: "f7821a90-17ac-4cd1-8178-eb555e3a9dd3"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 20:21:29.266482 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:21:29.266384    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f7821a90-17ac-4cd1-8178-eb555e3a9dd3-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "f7821a90-17ac-4cd1-8178-eb555e3a9dd3" (UID: "f7821a90-17ac-4cd1-8178-eb555e3a9dd3"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 20:21:29.266482 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:21:29.266474    2585 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f7821a90-17ac-4cd1-8178-eb555e3a9dd3-home\") on node \"ip-10-0-136-185.ec2.internal\" DevicePath \"\""
Sep 18 20:21:29.266631 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:21:29.266489    2585 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f7821a90-17ac-4cd1-8178-eb555e3a9dd3-tmp-dir\") on node \"ip-10-0-136-185.ec2.internal\" DevicePath \"\""
Sep 18 20:21:29.266631 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:21:29.266563    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f7821a90-17ac-4cd1-8178-eb555e3a9dd3-model-cache" (OuterVolumeSpecName: "model-cache") pod "f7821a90-17ac-4cd1-8178-eb555e3a9dd3" (UID: "f7821a90-17ac-4cd1-8178-eb555e3a9dd3"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 20:21:29.268343 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:21:29.268316    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f7821a90-17ac-4cd1-8178-eb555e3a9dd3-dshm" (OuterVolumeSpecName: "dshm") pod "f7821a90-17ac-4cd1-8178-eb555e3a9dd3" (UID: "f7821a90-17ac-4cd1-8178-eb555e3a9dd3"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 20:21:29.268743 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:21:29.268710    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f7821a90-17ac-4cd1-8178-eb555e3a9dd3-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "f7821a90-17ac-4cd1-8178-eb555e3a9dd3" (UID: "f7821a90-17ac-4cd1-8178-eb555e3a9dd3"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 20:21:29.268743 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:21:29.268714    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f7821a90-17ac-4cd1-8178-eb555e3a9dd3-kube-api-access-z62t2" (OuterVolumeSpecName: "kube-api-access-z62t2") pod "f7821a90-17ac-4cd1-8178-eb555e3a9dd3" (UID: "f7821a90-17ac-4cd1-8178-eb555e3a9dd3"). InnerVolumeSpecName "kube-api-access-z62t2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 20:21:29.367942 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:21:29.367890    2585 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f7821a90-17ac-4cd1-8178-eb555e3a9dd3-dshm\") on node \"ip-10-0-136-185.ec2.internal\" DevicePath \"\""
Sep 18 20:21:29.367942 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:21:29.367935    2585 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f7821a90-17ac-4cd1-8178-eb555e3a9dd3-tls-certs\") on node \"ip-10-0-136-185.ec2.internal\" DevicePath \"\""
Sep 18 20:21:29.367942 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:21:29.367946    2585 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f7821a90-17ac-4cd1-8178-eb555e3a9dd3-model-cache\") on node \"ip-10-0-136-185.ec2.internal\" DevicePath \"\""
Sep 18 20:21:29.367942 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:21:29.367955    2585 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z62t2\" (UniqueName: \"kubernetes.io/projected/f7821a90-17ac-4cd1-8178-eb555e3a9dd3-kube-api-access-z62t2\") on node \"ip-10-0-136-185.ec2.internal\" DevicePath \"\""
Sep 18 20:21:29.691629 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:21:29.691594    2585 generic.go:356] "Generic (PLEG): container finished" podID="f7821a90-17ac-4cd1-8178-eb555e3a9dd3" containerID="ac573cf075fb9effeb5cde77803b295b0ee45497d1133574ebedd5a0d29bc99c" exitCode=0
Sep 18 20:21:29.691817 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:21:29.691641    2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-ccd585a7-kserve-6f54dc9c75-pprt9"
Sep 18 20:21:29.691817 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:21:29.691677    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-ccd585a7-kserve-6f54dc9c75-pprt9" event={"ID":"f7821a90-17ac-4cd1-8178-eb555e3a9dd3","Type":"ContainerDied","Data":"ac573cf075fb9effeb5cde77803b295b0ee45497d1133574ebedd5a0d29bc99c"}
Sep 18 20:21:29.691817 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:21:29.691727    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-ccd585a7-kserve-6f54dc9c75-pprt9" event={"ID":"f7821a90-17ac-4cd1-8178-eb555e3a9dd3","Type":"ContainerDied","Data":"e397a4135803bb173ffa6e43ac06a1c71a69927d4bd596fa936d488635213bb5"}
Sep 18 20:21:29.691817 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:21:29.691744    2585 scope.go:117] "RemoveContainer" containerID="ac573cf075fb9effeb5cde77803b295b0ee45497d1133574ebedd5a0d29bc99c"
Sep 18 20:21:29.701135 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:21:29.701118    2585 scope.go:117] "RemoveContainer" containerID="ac573cf075fb9effeb5cde77803b295b0ee45497d1133574ebedd5a0d29bc99c"
Sep 18 20:21:29.701420 ip-10-0-136-185 kubenswrapper[2585]: E0918 20:21:29.701401    2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ac573cf075fb9effeb5cde77803b295b0ee45497d1133574ebedd5a0d29bc99c\": container with ID starting with ac573cf075fb9effeb5cde77803b295b0ee45497d1133574ebedd5a0d29bc99c not found: ID does not exist" containerID="ac573cf075fb9effeb5cde77803b295b0ee45497d1133574ebedd5a0d29bc99c"
Sep 18 20:21:29.701470 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:21:29.701428    2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ac573cf075fb9effeb5cde77803b295b0ee45497d1133574ebedd5a0d29bc99c"} err="failed to get container status \"ac573cf075fb9effeb5cde77803b295b0ee45497d1133574ebedd5a0d29bc99c\": rpc error: code = NotFound desc = could not find container \"ac573cf075fb9effeb5cde77803b295b0ee45497d1133574ebedd5a0d29bc99c\": container with ID starting with ac573cf075fb9effeb5cde77803b295b0ee45497d1133574ebedd5a0d29bc99c not found: ID does not exist"
Sep 18 20:21:29.712698 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:21:29.712674    2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-a-ccd585a7-kserve-6f54dc9c75-pprt9"]
Sep 18 20:21:29.716846 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:21:29.716822    2585 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["llm/e2e-same-id-a-ccd585a7-kserve-6f54dc9c75-pprt9"]
Sep 18 20:21:30.066331 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:21:30.066251    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-xmwh7"]
Sep 18 20:21:30.662982 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:21:30.662952    2585 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f7821a90-17ac-4cd1-8178-eb555e3a9dd3" path="/var/lib/kubelet/pods/f7821a90-17ac-4cd1-8178-eb555e3a9dd3/volumes"
Sep 18 20:21:36.875876 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:21:36.875834    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-xmwh7"]
Sep 18 20:21:37.090877 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:21:37.090841    2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-5f6dd68d9c-78tlj"]
Sep 18 20:21:37.091190 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:21:37.091178    2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f7821a90-17ac-4cd1-8178-eb555e3a9dd3" containerName="main"
Sep 18 20:21:37.091249 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:21:37.091193    2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="f7821a90-17ac-4cd1-8178-eb555e3a9dd3" containerName="main"
Sep 18 20:21:37.091286 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:21:37.091250    2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="f7821a90-17ac-4cd1-8178-eb555e3a9dd3" containerName="main"
Sep 18 20:21:37.095162 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:21:37.095146    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-5f6dd68d9c-78tlj"
Sep 18 20:21:37.098310 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:21:37.098284    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 18 20:21:37.098486 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:21:37.098310    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Sep 18 20:21:37.098486 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:21:37.098347    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-serving-cert\""
Sep 18 20:21:37.098486 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:21:37.098379    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-jqjhf\""
Sep 18 20:21:37.098486 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:21:37.098400    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\""
Sep 18 20:21:37.103468 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:21:37.103444    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-5f6dd68d9c-78tlj"]
Sep 18 20:21:37.233730 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:21:37.233690    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b3bd696d-c4d3-4940-a978-944b7df14c6a-openshift-service-ca\") pod \"maas-api-5f6dd68d9c-78tlj\" (UID: \"b3bd696d-c4d3-4940-a978-944b7df14c6a\") " pod="odh-ai-gateway-infra/maas-api-5f6dd68d9c-78tlj"
Sep 18 20:21:37.233938 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:21:37.233766    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b3bd696d-c4d3-4940-a978-944b7df14c6a-metrics-certs\") pod \"maas-api-5f6dd68d9c-78tlj\" (UID: \"b3bd696d-c4d3-4940-a978-944b7df14c6a\") " pod="odh-ai-gateway-infra/maas-api-5f6dd68d9c-78tlj"
Sep 18 20:21:37.233938 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:21:37.233827    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g9lxm\" (UniqueName: \"kubernetes.io/projected/b3bd696d-c4d3-4940-a978-944b7df14c6a-kube-api-access-g9lxm\") pod \"maas-api-5f6dd68d9c-78tlj\" (UID: \"b3bd696d-c4d3-4940-a978-944b7df14c6a\") " pod="odh-ai-gateway-infra/maas-api-5f6dd68d9c-78tlj"
Sep 18 20:21:37.233938 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:21:37.233895    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b3bd696d-c4d3-4940-a978-944b7df14c6a-maas-api-tls\") pod \"maas-api-5f6dd68d9c-78tlj\" (UID: \"b3bd696d-c4d3-4940-a978-944b7df14c6a\") " pod="odh-ai-gateway-infra/maas-api-5f6dd68d9c-78tlj"
Sep 18 20:21:37.334538 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:21:37.334501    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b3bd696d-c4d3-4940-a978-944b7df14c6a-metrics-certs\") pod \"maas-api-5f6dd68d9c-78tlj\" (UID: \"b3bd696d-c4d3-4940-a978-944b7df14c6a\") " pod="odh-ai-gateway-infra/maas-api-5f6dd68d9c-78tlj"
Sep 18 20:21:37.334538 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:21:37.334550    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g9lxm\" (UniqueName: \"kubernetes.io/projected/b3bd696d-c4d3-4940-a978-944b7df14c6a-kube-api-access-g9lxm\") pod \"maas-api-5f6dd68d9c-78tlj\" (UID: \"b3bd696d-c4d3-4940-a978-944b7df14c6a\") " pod="odh-ai-gateway-infra/maas-api-5f6dd68d9c-78tlj"
Sep 18 20:21:37.334835 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:21:37.334589    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b3bd696d-c4d3-4940-a978-944b7df14c6a-maas-api-tls\") pod \"maas-api-5f6dd68d9c-78tlj\" (UID: \"b3bd696d-c4d3-4940-a978-944b7df14c6a\") " pod="odh-ai-gateway-infra/maas-api-5f6dd68d9c-78tlj"
Sep 18 20:21:37.334835 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:21:37.334612    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b3bd696d-c4d3-4940-a978-944b7df14c6a-openshift-service-ca\") pod \"maas-api-5f6dd68d9c-78tlj\" (UID: \"b3bd696d-c4d3-4940-a978-944b7df14c6a\") " pod="odh-ai-gateway-infra/maas-api-5f6dd68d9c-78tlj"
Sep 18 20:21:37.335379 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:21:37.335355    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b3bd696d-c4d3-4940-a978-944b7df14c6a-openshift-service-ca\") pod \"maas-api-5f6dd68d9c-78tlj\" (UID: \"b3bd696d-c4d3-4940-a978-944b7df14c6a\") " pod="odh-ai-gateway-infra/maas-api-5f6dd68d9c-78tlj"
Sep 18 20:21:37.337117 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:21:37.337097    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b3bd696d-c4d3-4940-a978-944b7df14c6a-metrics-certs\") pod \"maas-api-5f6dd68d9c-78tlj\" (UID: \"b3bd696d-c4d3-4940-a978-944b7df14c6a\") " pod="odh-ai-gateway-infra/maas-api-5f6dd68d9c-78tlj"
Sep 18 20:21:37.337202 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:21:37.337126    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b3bd696d-c4d3-4940-a978-944b7df14c6a-maas-api-tls\") pod \"maas-api-5f6dd68d9c-78tlj\" (UID: \"b3bd696d-c4d3-4940-a978-944b7df14c6a\") " pod="odh-ai-gateway-infra/maas-api-5f6dd68d9c-78tlj"
Sep 18 20:21:37.341759 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:21:37.341741    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-g9lxm\" (UniqueName: \"kubernetes.io/projected/b3bd696d-c4d3-4940-a978-944b7df14c6a-kube-api-access-g9lxm\") pod \"maas-api-5f6dd68d9c-78tlj\" (UID: \"b3bd696d-c4d3-4940-a978-944b7df14c6a\") " pod="odh-ai-gateway-infra/maas-api-5f6dd68d9c-78tlj"
Sep 18 20:21:37.407105 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:21:37.407074    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-5f6dd68d9c-78tlj"
Sep 18 20:21:37.676882 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:21:37.676850    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-5f6dd68d9c-78tlj"]
Sep 18 20:21:37.678658 ip-10-0-136-185 kubenswrapper[2585]: W0918 20:21:37.678627    2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb3bd696d_c4d3_4940_a978_944b7df14c6a.slice/crio-2c332605d701bf585998f06f23d91da4fd24289551a46a9ad6ca6e8143f9c7de WatchSource:0}: Error finding container 2c332605d701bf585998f06f23d91da4fd24289551a46a9ad6ca6e8143f9c7de: Status 404 returned error can't find the container with id 2c332605d701bf585998f06f23d91da4fd24289551a46a9ad6ca6e8143f9c7de
Sep 18 20:21:37.680057 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:21:37.680037    2585 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 18 20:21:37.727356 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:21:37.727326    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-5f6dd68d9c-78tlj" event={"ID":"b3bd696d-c4d3-4940-a978-944b7df14c6a","Type":"ContainerStarted","Data":"2c332605d701bf585998f06f23d91da4fd24289551a46a9ad6ca6e8143f9c7de"}
Sep 18 20:21:38.733728 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:21:38.733692    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-5f6dd68d9c-78tlj" event={"ID":"b3bd696d-c4d3-4940-a978-944b7df14c6a","Type":"ContainerStarted","Data":"0fc59136da7d49e7b5d11696d3ada54e1cf75aa39a52e2ec2735dcc56e0944ba"}
Sep 18 20:21:38.734274 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:21:38.733840    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-5f6dd68d9c-78tlj"
Sep 18 20:21:38.772717 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:21:38.772665    2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-5f6dd68d9c-78tlj" podStartSLOduration=1.336357023 podStartE2EDuration="1.772648881s" totalImagesPullingTime="436.291858ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 20:21:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 20:21:38.770489958 +0000 UTC m=+2540.295095382" watchObservedRunningTime="2026-09-18 20:21:38.772648881 +0000 UTC m=+2540.297254304"
Sep 18 20:21:44.701575 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:21:44.701536    2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-5f6dd68d9c-78tlj"]
Sep 18 20:21:44.702071 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:21:44.701722    2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-5f6dd68d9c-78tlj" podUID="b3bd696d-c4d3-4940-a978-944b7df14c6a" containerName="maas-api" containerID="cri-o://0fc59136da7d49e7b5d11696d3ada54e1cf75aa39a52e2ec2735dcc56e0944ba" gracePeriod=30
Sep 18 20:21:44.706812 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:21:44.706775    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-5f6dd68d9c-78tlj"
Sep 18 20:21:44.732779 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:21:44.732749    2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-7f7bf4cfb5-2qw2f"]
Sep 18 20:21:44.735974 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:21:44.735959    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-7f7bf4cfb5-2qw2f"
Sep 18 20:21:44.749530 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:21:44.748879    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-7f7bf4cfb5-2qw2f"]
Sep 18 20:21:44.902585 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:21:44.902547    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/600c067c-9ce8-4988-ac3c-2288090bfe0e-openshift-service-ca\") pod \"maas-api-7f7bf4cfb5-2qw2f\" (UID: \"600c067c-9ce8-4988-ac3c-2288090bfe0e\") " pod="odh-ai-gateway-infra/maas-api-7f7bf4cfb5-2qw2f"
Sep 18 20:21:44.902585 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:21:44.902592    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/600c067c-9ce8-4988-ac3c-2288090bfe0e-maas-api-tls\") pod \"maas-api-7f7bf4cfb5-2qw2f\" (UID: \"600c067c-9ce8-4988-ac3c-2288090bfe0e\") " pod="odh-ai-gateway-infra/maas-api-7f7bf4cfb5-2qw2f"
Sep 18 20:21:44.902865 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:21:44.902645    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/600c067c-9ce8-4988-ac3c-2288090bfe0e-metrics-certs\") pod \"maas-api-7f7bf4cfb5-2qw2f\" (UID: \"600c067c-9ce8-4988-ac3c-2288090bfe0e\") " pod="odh-ai-gateway-infra/maas-api-7f7bf4cfb5-2qw2f"
Sep 18 20:21:44.902865 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:21:44.902665    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hht5w\" (UniqueName: \"kubernetes.io/projected/600c067c-9ce8-4988-ac3c-2288090bfe0e-kube-api-access-hht5w\") pod \"maas-api-7f7bf4cfb5-2qw2f\" (UID: \"600c067c-9ce8-4988-ac3c-2288090bfe0e\") " pod="odh-ai-gateway-infra/maas-api-7f7bf4cfb5-2qw2f"
Sep 18 20:21:44.944442 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:21:44.944414    2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-5f6dd68d9c-78tlj"
Sep 18 20:21:45.003858 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:21:45.003753    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/600c067c-9ce8-4988-ac3c-2288090bfe0e-openshift-service-ca\") pod \"maas-api-7f7bf4cfb5-2qw2f\" (UID: \"600c067c-9ce8-4988-ac3c-2288090bfe0e\") " pod="odh-ai-gateway-infra/maas-api-7f7bf4cfb5-2qw2f"
Sep 18 20:21:45.003858 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:21:45.003818    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/600c067c-9ce8-4988-ac3c-2288090bfe0e-maas-api-tls\") pod \"maas-api-7f7bf4cfb5-2qw2f\" (UID: \"600c067c-9ce8-4988-ac3c-2288090bfe0e\") " pod="odh-ai-gateway-infra/maas-api-7f7bf4cfb5-2qw2f"
Sep 18 20:21:45.004049 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:21:45.003874    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/600c067c-9ce8-4988-ac3c-2288090bfe0e-metrics-certs\") pod \"maas-api-7f7bf4cfb5-2qw2f\" (UID: \"600c067c-9ce8-4988-ac3c-2288090bfe0e\") " pod="odh-ai-gateway-infra/maas-api-7f7bf4cfb5-2qw2f"
Sep 18 20:21:45.004049 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:21:45.003894    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hht5w\" (UniqueName: \"kubernetes.io/projected/600c067c-9ce8-4988-ac3c-2288090bfe0e-kube-api-access-hht5w\") pod \"maas-api-7f7bf4cfb5-2qw2f\" (UID: \"600c067c-9ce8-4988-ac3c-2288090bfe0e\") " pod="odh-ai-gateway-infra/maas-api-7f7bf4cfb5-2qw2f"
Sep 18 20:21:45.004408 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:21:45.004387    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/600c067c-9ce8-4988-ac3c-2288090bfe0e-openshift-service-ca\") pod \"maas-api-7f7bf4cfb5-2qw2f\" (UID: \"600c067c-9ce8-4988-ac3c-2288090bfe0e\") " pod="odh-ai-gateway-infra/maas-api-7f7bf4cfb5-2qw2f"
Sep 18 20:21:45.006383 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:21:45.006352    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/600c067c-9ce8-4988-ac3c-2288090bfe0e-maas-api-tls\") pod \"maas-api-7f7bf4cfb5-2qw2f\" (UID: \"600c067c-9ce8-4988-ac3c-2288090bfe0e\") " pod="odh-ai-gateway-infra/maas-api-7f7bf4cfb5-2qw2f"
Sep 18 20:21:45.006541 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:21:45.006517    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/600c067c-9ce8-4988-ac3c-2288090bfe0e-metrics-certs\") pod \"maas-api-7f7bf4cfb5-2qw2f\" (UID: \"600c067c-9ce8-4988-ac3c-2288090bfe0e\") " pod="odh-ai-gateway-infra/maas-api-7f7bf4cfb5-2qw2f"
Sep 18 20:21:45.012090 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:21:45.012065    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hht5w\" (UniqueName: \"kubernetes.io/projected/600c067c-9ce8-4988-ac3c-2288090bfe0e-kube-api-access-hht5w\") pod \"maas-api-7f7bf4cfb5-2qw2f\" (UID: \"600c067c-9ce8-4988-ac3c-2288090bfe0e\") " pod="odh-ai-gateway-infra/maas-api-7f7bf4cfb5-2qw2f"
Sep 18 20:21:45.073415 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:21:45.073380    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-xmwh7"]
Sep 18 20:21:45.076941 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:21:45.076919    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-7f7bf4cfb5-2qw2f"
Sep 18 20:21:45.104333 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:21:45.104301    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b3bd696d-c4d3-4940-a978-944b7df14c6a-maas-api-tls\") pod \"b3bd696d-c4d3-4940-a978-944b7df14c6a\" (UID: \"b3bd696d-c4d3-4940-a978-944b7df14c6a\") "
Sep 18 20:21:45.104475 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:21:45.104343    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-g9lxm\" (UniqueName: \"kubernetes.io/projected/b3bd696d-c4d3-4940-a978-944b7df14c6a-kube-api-access-g9lxm\") pod \"b3bd696d-c4d3-4940-a978-944b7df14c6a\" (UID: \"b3bd696d-c4d3-4940-a978-944b7df14c6a\") "
Sep 18 20:21:45.104475 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:21:45.104382    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b3bd696d-c4d3-4940-a978-944b7df14c6a-metrics-certs\") pod \"b3bd696d-c4d3-4940-a978-944b7df14c6a\" (UID: \"b3bd696d-c4d3-4940-a978-944b7df14c6a\") "
Sep 18 20:21:45.104475 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:21:45.104414    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b3bd696d-c4d3-4940-a978-944b7df14c6a-openshift-service-ca\") pod \"b3bd696d-c4d3-4940-a978-944b7df14c6a\" (UID: \"b3bd696d-c4d3-4940-a978-944b7df14c6a\") "
Sep 18 20:21:45.104895 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:21:45.104837    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b3bd696d-c4d3-4940-a978-944b7df14c6a-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "b3bd696d-c4d3-4940-a978-944b7df14c6a" (UID: "b3bd696d-c4d3-4940-a978-944b7df14c6a"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 20:21:45.106542 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:21:45.106511    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b3bd696d-c4d3-4940-a978-944b7df14c6a-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "b3bd696d-c4d3-4940-a978-944b7df14c6a" (UID: "b3bd696d-c4d3-4940-a978-944b7df14c6a"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 20:21:45.106648 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:21:45.106544    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b3bd696d-c4d3-4940-a978-944b7df14c6a-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "b3bd696d-c4d3-4940-a978-944b7df14c6a" (UID: "b3bd696d-c4d3-4940-a978-944b7df14c6a"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 20:21:45.106648 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:21:45.106563    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b3bd696d-c4d3-4940-a978-944b7df14c6a-kube-api-access-g9lxm" (OuterVolumeSpecName: "kube-api-access-g9lxm") pod "b3bd696d-c4d3-4940-a978-944b7df14c6a" (UID: "b3bd696d-c4d3-4940-a978-944b7df14c6a"). InnerVolumeSpecName "kube-api-access-g9lxm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 20:21:45.205631 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:21:45.205591    2585 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b3bd696d-c4d3-4940-a978-944b7df14c6a-maas-api-tls\") on node \"ip-10-0-136-185.ec2.internal\" DevicePath \"\""
Sep 18 20:21:45.205631 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:21:45.205632    2585 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g9lxm\" (UniqueName: \"kubernetes.io/projected/b3bd696d-c4d3-4940-a978-944b7df14c6a-kube-api-access-g9lxm\") on node \"ip-10-0-136-185.ec2.internal\" DevicePath \"\""
Sep 18 20:21:45.205891 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:21:45.205646    2585 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b3bd696d-c4d3-4940-a978-944b7df14c6a-metrics-certs\") on node \"ip-10-0-136-185.ec2.internal\" DevicePath \"\""
Sep 18 20:21:45.205891 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:21:45.205659    2585 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b3bd696d-c4d3-4940-a978-944b7df14c6a-openshift-service-ca\") on node \"ip-10-0-136-185.ec2.internal\" DevicePath \"\""
Sep 18 20:21:45.347508 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:21:45.347453    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-7f7bf4cfb5-2qw2f"]
Sep 18 20:21:45.349857 ip-10-0-136-185 kubenswrapper[2585]: W0918 20:21:45.349831    2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod600c067c_9ce8_4988_ac3c_2288090bfe0e.slice/crio-a1a42e09af2669d8ea13169920317075fb1be3ef55d8053e909de83724eed0a8 WatchSource:0}: Error finding container a1a42e09af2669d8ea13169920317075fb1be3ef55d8053e909de83724eed0a8: Status 404 returned error can't find the container with id a1a42e09af2669d8ea13169920317075fb1be3ef55d8053e909de83724eed0a8
Sep 18 20:21:45.767065 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:21:45.767032    2585 generic.go:356] "Generic (PLEG): container finished" podID="b3bd696d-c4d3-4940-a978-944b7df14c6a" containerID="0fc59136da7d49e7b5d11696d3ada54e1cf75aa39a52e2ec2735dcc56e0944ba" exitCode=0
Sep 18 20:21:45.767495 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:21:45.767081    2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-5f6dd68d9c-78tlj"
Sep 18 20:21:45.767495 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:21:45.767115    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-5f6dd68d9c-78tlj" event={"ID":"b3bd696d-c4d3-4940-a978-944b7df14c6a","Type":"ContainerDied","Data":"0fc59136da7d49e7b5d11696d3ada54e1cf75aa39a52e2ec2735dcc56e0944ba"}
Sep 18 20:21:45.767495 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:21:45.767179    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-5f6dd68d9c-78tlj" event={"ID":"b3bd696d-c4d3-4940-a978-944b7df14c6a","Type":"ContainerDied","Data":"2c332605d701bf585998f06f23d91da4fd24289551a46a9ad6ca6e8143f9c7de"}
Sep 18 20:21:45.767495 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:21:45.767212    2585 scope.go:117] "RemoveContainer" containerID="0fc59136da7d49e7b5d11696d3ada54e1cf75aa39a52e2ec2735dcc56e0944ba"
Sep 18 20:21:45.768600 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:21:45.768400    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-7f7bf4cfb5-2qw2f" event={"ID":"600c067c-9ce8-4988-ac3c-2288090bfe0e","Type":"ContainerStarted","Data":"a1a42e09af2669d8ea13169920317075fb1be3ef55d8053e909de83724eed0a8"}
Sep 18 20:21:45.776152 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:21:45.776135    2585 scope.go:117] "RemoveContainer" containerID="0fc59136da7d49e7b5d11696d3ada54e1cf75aa39a52e2ec2735dcc56e0944ba"
Sep 18 20:21:45.776410 ip-10-0-136-185 kubenswrapper[2585]: E0918 20:21:45.776395    2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0fc59136da7d49e7b5d11696d3ada54e1cf75aa39a52e2ec2735dcc56e0944ba\": container with ID starting with 0fc59136da7d49e7b5d11696d3ada54e1cf75aa39a52e2ec2735dcc56e0944ba not found: ID does not exist" containerID="0fc59136da7d49e7b5d11696d3ada54e1cf75aa39a52e2ec2735dcc56e0944ba"
Sep 18 20:21:45.776487 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:21:45.776414    2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0fc59136da7d49e7b5d11696d3ada54e1cf75aa39a52e2ec2735dcc56e0944ba"} err="failed to get container status \"0fc59136da7d49e7b5d11696d3ada54e1cf75aa39a52e2ec2735dcc56e0944ba\": rpc error: code = NotFound desc = could not find container \"0fc59136da7d49e7b5d11696d3ada54e1cf75aa39a52e2ec2735dcc56e0944ba\": container with ID starting with 0fc59136da7d49e7b5d11696d3ada54e1cf75aa39a52e2ec2735dcc56e0944ba not found: ID does not exist"
Sep 18 20:21:45.787733 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:21:45.787713    2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-5f6dd68d9c-78tlj"]
Sep 18 20:21:45.792694 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:21:45.792670    2585 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-5f6dd68d9c-78tlj"]
Sep 18 20:21:46.662002 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:21:46.661968    2585 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b3bd696d-c4d3-4940-a978-944b7df14c6a" path="/var/lib/kubelet/pods/b3bd696d-c4d3-4940-a978-944b7df14c6a/volumes"
Sep 18 20:21:46.775285 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:21:46.775249    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-7f7bf4cfb5-2qw2f" event={"ID":"600c067c-9ce8-4988-ac3c-2288090bfe0e","Type":"ContainerStarted","Data":"abc216059a11d6470a6f57dc77603be7d7b1de221f47431a038da1266cdbacc8"}
Sep 18 20:21:46.775689 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:21:46.775366    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-7f7bf4cfb5-2qw2f"
Sep 18 20:21:46.792045 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:21:46.791993    2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-7f7bf4cfb5-2qw2f" podStartSLOduration=2.367469131 podStartE2EDuration="2.791977994s" totalImagesPullingTime="424.508863ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 20:21:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 20:21:46.791572198 +0000 UTC m=+2548.316177623" watchObservedRunningTime="2026-09-18 20:21:46.791977994 +0000 UTC m=+2548.316583416"
Sep 18 20:21:51.979812 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:21:51.979761    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-xmwh7"]
Sep 18 20:21:52.785049 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:21:52.785020    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-7f7bf4cfb5-2qw2f"
Sep 18 20:21:57.777916 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:21:57.777876    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-xmwh7"]
Sep 18 20:22:04.769445 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:22:04.769407    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-xmwh7"]
Sep 18 20:22:13.901013 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:22:13.900983    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-bd4fcf9c7-qlrdn_66b774a3-01a1-4190-979b-1a956d8b5049/copy-manifests/0.log"
Sep 18 20:22:13.910637 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:22:13.910608    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-bd4fcf9c7-qlrdn_66b774a3-01a1-4190-979b-1a956d8b5049/manager/0.log"
Sep 18 20:22:14.428164 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:22:14.428131    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-7f7bf4cfb5-2qw2f_600c067c-9ce8-4988-ac3c-2288090bfe0e/maas-api/0.log"
Sep 18 20:22:16.355067 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:22:16.355036    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-84994f7ffc-xmwh7_d1b251b4-2bbc-43d9-9ea1-5fc90e16f6e7/limitador/0.log"
Sep 18 20:22:16.474057 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:22:16.474022    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-74495c69f8-nj8ds_3b8173f6-26db-4ca9-b53b-471b9692b149/manager/0.log"
Sep 18 20:22:17.174092 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:22:17.174059    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-6bc78fcbdb-tr229_5d1ac900-f718-4110-b637-069335895e44/kube-auth-proxy/0.log"
Sep 18 20:22:18.354487 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:22:18.354447    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-trlp-test-simulated-kserve-598466548d-96nlc_407f9e49-a524-4758-ac91-66120cb274ea/main/0.log"
Sep 18 20:22:18.591424 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:22:18.591394    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_facebook-opt-125m-simulated-kserve-74b68447df-wvw2g_1220608c-ad1c-46cd-8fa7-cb983458aa4e/main/0.log"
Sep 18 20:22:18.711250 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:22:18.711199    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_premium-simulated-simulated-premium-kserve-57fc66f697-kmnjt_ed909c73-5b8a-4187-9369-4d321ce59a9e/main/0.log"
Sep 18 20:22:27.366976 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:22:27.366921    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-npsrh_7e47b713-e1a2-48ff-b961-ad41e706a225/global-pull-secret-syncer/0.log"
Sep 18 20:22:27.564116 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:22:27.564059    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-mbhzp_d2fa3583-1483-4960-a44b-694496d13f45/konnectivity-agent/0.log"
Sep 18 20:22:27.611414 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:22:27.611382    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-136-185.ec2.internal_5989c800ad96c7a3440a3290994469f4/haproxy/0.log"
Sep 18 20:22:32.448278 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:22:32.448243    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-84994f7ffc-xmwh7_d1b251b4-2bbc-43d9-9ea1-5fc90e16f6e7/limitador/0.log"
Sep 18 20:22:32.537934 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:22:32.537893    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-74495c69f8-nj8ds_3b8173f6-26db-4ca9-b53b-471b9692b149/manager/0.log"
Sep 18 20:22:34.274401 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:22:34.274369    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_monitoring-plugin-5984459ff8-dmghk_4cf62754-c2f2-4fb8-81bb-5302f58e0cf5/monitoring-plugin/0.log"
Sep 18 20:22:34.318204 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:22:34.318174    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-m9c72_ff4afcd9-ff15-4e03-a63a-7927c4b458f4/node-exporter/0.log"
Sep 18 20:22:34.341333 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:22:34.341266    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-m9c72_ff4afcd9-ff15-4e03-a63a-7927c4b458f4/kube-rbac-proxy/0.log"
Sep 18 20:22:34.367186 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:22:34.367158    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-m9c72_ff4afcd9-ff15-4e03-a63a-7927c4b458f4/init-textfile/0.log"
Sep 18 20:22:34.592000 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:22:34.591911    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-86c5bc79d6-q45br_d6506887-aec3-4244-911e-b0a4fd696601/kube-rbac-proxy-main/0.log"
Sep 18 20:22:34.614664 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:22:34.614642    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-86c5bc79d6-q45br_d6506887-aec3-4244-911e-b0a4fd696601/kube-rbac-proxy-self/0.log"
Sep 18 20:22:34.635162 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:22:34.635138    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-86c5bc79d6-q45br_d6506887-aec3-4244-911e-b0a4fd696601/openshift-state-metrics/0.log"
Sep 18 20:22:34.683242 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:22:34.683213    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_a1205f9f-d121-45f4-baf3-b56d70ccb24b/prometheus/0.log"
Sep 18 20:22:34.706449 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:22:34.706422    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_a1205f9f-d121-45f4-baf3-b56d70ccb24b/config-reloader/0.log"
Sep 18 20:22:34.725779 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:22:34.725755    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_a1205f9f-d121-45f4-baf3-b56d70ccb24b/thanos-sidecar/0.log"
Sep 18 20:22:34.745216 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:22:34.745193    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_a1205f9f-d121-45f4-baf3-b56d70ccb24b/kube-rbac-proxy-web/0.log"
Sep 18 20:22:34.770899 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:22:34.770863    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_a1205f9f-d121-45f4-baf3-b56d70ccb24b/kube-rbac-proxy/0.log"
Sep 18 20:22:34.793644 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:22:34.793619    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_a1205f9f-d121-45f4-baf3-b56d70ccb24b/kube-rbac-proxy-thanos/0.log"
Sep 18 20:22:34.813868 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:22:34.813842    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_a1205f9f-d121-45f4-baf3-b56d70ccb24b/init-config-reloader/0.log"
Sep 18 20:22:34.896329 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:22:34.896299    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-admission-webhook-5c6747dc7b-59t4f_bddff53b-d5cc-4cf2-8e76-9ab8ef1dd7ef/prometheus-operator-admission-webhook/0.log"
Sep 18 20:22:37.260721 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:22:37.260687    2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-ts5cd/perf-node-gather-daemonset-kzbsj"]
Sep 18 20:22:37.261117 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:22:37.261050    2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b3bd696d-c4d3-4940-a978-944b7df14c6a" containerName="maas-api"
Sep 18 20:22:37.261117 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:22:37.261062    2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="b3bd696d-c4d3-4940-a978-944b7df14c6a" containerName="maas-api"
Sep 18 20:22:37.261190 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:22:37.261129    2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="b3bd696d-c4d3-4940-a978-944b7df14c6a" containerName="maas-api"
Sep 18 20:22:37.265474 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:22:37.265450    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-ts5cd/perf-node-gather-daemonset-kzbsj"
Sep 18 20:22:37.267721 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:22:37.267700    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-ts5cd\"/\"openshift-service-ca.crt\""
Sep 18 20:22:37.268548 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:22:37.268527    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-ts5cd\"/\"default-dockercfg-4nfxp\""
Sep 18 20:22:37.268548 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:22:37.268542    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-ts5cd\"/\"kube-root-ca.crt\""
Sep 18 20:22:37.272022 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:22:37.271997    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-ts5cd/perf-node-gather-daemonset-kzbsj"]
Sep 18 20:22:37.372089 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:22:37.372053    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/7eb241ca-869b-4097-a5b8-69757e24dca2-podres\") pod \"perf-node-gather-daemonset-kzbsj\" (UID: \"7eb241ca-869b-4097-a5b8-69757e24dca2\") " pod="openshift-must-gather-ts5cd/perf-node-gather-daemonset-kzbsj"
Sep 18 20:22:37.372276 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:22:37.372115    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/7eb241ca-869b-4097-a5b8-69757e24dca2-sys\") pod \"perf-node-gather-daemonset-kzbsj\" (UID: \"7eb241ca-869b-4097-a5b8-69757e24dca2\") " pod="openshift-must-gather-ts5cd/perf-node-gather-daemonset-kzbsj"
Sep 18 20:22:37.372276 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:22:37.372141    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/7eb241ca-869b-4097-a5b8-69757e24dca2-lib-modules\") pod \"perf-node-gather-daemonset-kzbsj\" (UID: \"7eb241ca-869b-4097-a5b8-69757e24dca2\") " pod="openshift-must-gather-ts5cd/perf-node-gather-daemonset-kzbsj"
Sep 18 20:22:37.372276 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:22:37.372169    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/7eb241ca-869b-4097-a5b8-69757e24dca2-proc\") pod \"perf-node-gather-daemonset-kzbsj\" (UID: \"7eb241ca-869b-4097-a5b8-69757e24dca2\") " pod="openshift-must-gather-ts5cd/perf-node-gather-daemonset-kzbsj"
Sep 18 20:22:37.372276 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:22:37.372185    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6rp97\" (UniqueName: \"kubernetes.io/projected/7eb241ca-869b-4097-a5b8-69757e24dca2-kube-api-access-6rp97\") pod \"perf-node-gather-daemonset-kzbsj\" (UID: \"7eb241ca-869b-4097-a5b8-69757e24dca2\") " pod="openshift-must-gather-ts5cd/perf-node-gather-daemonset-kzbsj"
Sep 18 20:22:37.472911 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:22:37.472875    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/7eb241ca-869b-4097-a5b8-69757e24dca2-podres\") pod \"perf-node-gather-daemonset-kzbsj\" (UID: \"7eb241ca-869b-4097-a5b8-69757e24dca2\") " pod="openshift-must-gather-ts5cd/perf-node-gather-daemonset-kzbsj"
Sep 18 20:22:37.473088 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:22:37.472929    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/7eb241ca-869b-4097-a5b8-69757e24dca2-sys\") pod \"perf-node-gather-daemonset-kzbsj\" (UID: \"7eb241ca-869b-4097-a5b8-69757e24dca2\") " pod="openshift-must-gather-ts5cd/perf-node-gather-daemonset-kzbsj"
Sep 18 20:22:37.473088 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:22:37.472949    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/7eb241ca-869b-4097-a5b8-69757e24dca2-lib-modules\") pod \"perf-node-gather-daemonset-kzbsj\" (UID: \"7eb241ca-869b-4097-a5b8-69757e24dca2\") " pod="openshift-must-gather-ts5cd/perf-node-gather-daemonset-kzbsj"
Sep 18 20:22:37.473088 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:22:37.472974    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/7eb241ca-869b-4097-a5b8-69757e24dca2-proc\") pod \"perf-node-gather-daemonset-kzbsj\" (UID: \"7eb241ca-869b-4097-a5b8-69757e24dca2\") " pod="openshift-must-gather-ts5cd/perf-node-gather-daemonset-kzbsj"
Sep 18 20:22:37.473088 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:22:37.472992    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6rp97\" (UniqueName: \"kubernetes.io/projected/7eb241ca-869b-4097-a5b8-69757e24dca2-kube-api-access-6rp97\") pod \"perf-node-gather-daemonset-kzbsj\" (UID: \"7eb241ca-869b-4097-a5b8-69757e24dca2\") " pod="openshift-must-gather-ts5cd/perf-node-gather-daemonset-kzbsj"
Sep 18 20:22:37.473088 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:22:37.473064    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/7eb241ca-869b-4097-a5b8-69757e24dca2-podres\") pod \"perf-node-gather-daemonset-kzbsj\" (UID: \"7eb241ca-869b-4097-a5b8-69757e24dca2\") " pod="openshift-must-gather-ts5cd/perf-node-gather-daemonset-kzbsj"
Sep 18 20:22:37.473088 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:22:37.473082    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/7eb241ca-869b-4097-a5b8-69757e24dca2-sys\") pod \"perf-node-gather-daemonset-kzbsj\" (UID: \"7eb241ca-869b-4097-a5b8-69757e24dca2\") " pod="openshift-must-gather-ts5cd/perf-node-gather-daemonset-kzbsj"
Sep 18 20:22:37.473298 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:22:37.473080    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/7eb241ca-869b-4097-a5b8-69757e24dca2-proc\") pod \"perf-node-gather-daemonset-kzbsj\" (UID: \"7eb241ca-869b-4097-a5b8-69757e24dca2\") " pod="openshift-must-gather-ts5cd/perf-node-gather-daemonset-kzbsj"
Sep 18 20:22:37.473298 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:22:37.473119    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/7eb241ca-869b-4097-a5b8-69757e24dca2-lib-modules\") pod \"perf-node-gather-daemonset-kzbsj\" (UID: \"7eb241ca-869b-4097-a5b8-69757e24dca2\") " pod="openshift-must-gather-ts5cd/perf-node-gather-daemonset-kzbsj"
Sep 18 20:22:37.480824 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:22:37.480804    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6rp97\" (UniqueName: \"kubernetes.io/projected/7eb241ca-869b-4097-a5b8-69757e24dca2-kube-api-access-6rp97\") pod \"perf-node-gather-daemonset-kzbsj\" (UID: \"7eb241ca-869b-4097-a5b8-69757e24dca2\") " pod="openshift-must-gather-ts5cd/perf-node-gather-daemonset-kzbsj"
Sep 18 20:22:37.577649 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:22:37.577563    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-ts5cd/perf-node-gather-daemonset-kzbsj"
Sep 18 20:22:37.741914 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:22:37.741885    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_downloads-6594f4c75-p7gk4_1f4af1db-93c0-48d0-81df-868ea52c8a1e/download-server/0.log"
Sep 18 20:22:37.845981 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:22:37.845957    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-ts5cd/perf-node-gather-daemonset-kzbsj"]
Sep 18 20:22:37.848067 ip-10-0-136-185 kubenswrapper[2585]: W0918 20:22:37.848041    2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod7eb241ca_869b_4097_a5b8_69757e24dca2.slice/crio-498360e4b491d49f7820c0cbbba9cd8c21722ef09670dbcc212bb09a102ca3ea WatchSource:0}: Error finding container 498360e4b491d49f7820c0cbbba9cd8c21722ef09670dbcc212bb09a102ca3ea: Status 404 returned error can't find the container with id 498360e4b491d49f7820c0cbbba9cd8c21722ef09670dbcc212bb09a102ca3ea
Sep 18 20:22:38.011161 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:22:38.011123    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-ts5cd/perf-node-gather-daemonset-kzbsj" event={"ID":"7eb241ca-869b-4097-a5b8-69757e24dca2","Type":"ContainerStarted","Data":"42b907317db557c1f4956694d07e6ab70d8ef2ad38a6ed2701778f88bb2cf731"}
Sep 18 20:22:38.011353 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:22:38.011176    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-ts5cd/perf-node-gather-daemonset-kzbsj" event={"ID":"7eb241ca-869b-4097-a5b8-69757e24dca2","Type":"ContainerStarted","Data":"498360e4b491d49f7820c0cbbba9cd8c21722ef09670dbcc212bb09a102ca3ea"}
Sep 18 20:22:38.011353 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:22:38.011198    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-ts5cd/perf-node-gather-daemonset-kzbsj"
Sep 18 20:22:38.026568 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:22:38.026518    2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-ts5cd/perf-node-gather-daemonset-kzbsj" podStartSLOduration=1.026501572 podStartE2EDuration="1.026501572s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-18 20:22:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 20:22:38.026438289 +0000 UTC m=+2599.551043713" watchObservedRunningTime="2026-09-18 20:22:38.026501572 +0000 UTC m=+2599.551106994"
Sep 18 20:22:38.384717 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:22:38.384684    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-storage-operator_volume-data-source-validator-5c4db8bdf9-gpcjl_16c53ed7-e3cd-4eb5-8afa-3ee430dbe58f/volume-data-source-validator/0.log"
Sep 18 20:22:39.566389 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:22:39.566360    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-grfwt_13a57b4c-73f9-4960-89d4-e9c25d392279/dns/0.log"
Sep 18 20:22:39.585690 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:22:39.585659    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-grfwt_13a57b4c-73f9-4960-89d4-e9c25d392279/kube-rbac-proxy/0.log"
Sep 18 20:22:39.702196 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:22:39.702162    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-ltq6c_bdda9ef7-2f71-4b51-84a9-56da43dd64d9/dns-node-resolver/0.log"
Sep 18 20:22:40.258654 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:22:40.258618    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-llt9z_3a884214-2df1-4958-a6b9-e3f2c903ae22/node-ca/0.log"
Sep 18 20:22:41.809452 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:22:41.809427    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-6bc78fcbdb-tr229_5d1ac900-f718-4110-b637-069335895e44/kube-auth-proxy/0.log"
Sep 18 20:22:42.825044 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:22:42.825005    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-kxzjt_a797d295-00fd-4085-b1cc-bdfffc6af82f/serve-healthcheck-canary/0.log"
Sep 18 20:22:43.336163 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:22:43.336126    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-6f9c644dd9-6bc55_d6560c03-8e80-4d69-b52c-73e63032f2ec/insights-operator/1.log"
Sep 18 20:22:43.336370 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:22:43.336340    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-6f9c644dd9-6bc55_d6560c03-8e80-4d69-b52c-73e63032f2ec/insights-operator/0.log"
Sep 18 20:22:43.357126 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:22:43.357095    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-7czbj_f3c00964-2160-44d6-8699-354dae4ac4a5/kube-rbac-proxy/0.log"
Sep 18 20:22:43.376595 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:22:43.376570    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-7czbj_f3c00964-2160-44d6-8699-354dae4ac4a5/exporter/0.log"
Sep 18 20:22:43.396230 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:22:43.396205    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-7czbj_f3c00964-2160-44d6-8699-354dae4ac4a5/extractor/0.log"
Sep 18 20:22:44.025736 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:22:44.025700    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-ts5cd/perf-node-gather-daemonset-kzbsj"
Sep 18 20:22:46.154011 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:22:46.153984    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-bd4fcf9c7-qlrdn_66b774a3-01a1-4190-979b-1a956d8b5049/manager/0.log"
Sep 18 20:22:46.175365 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:22:46.175334    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-bd4fcf9c7-qlrdn_66b774a3-01a1-4190-979b-1a956d8b5049/copy-manifests/0.log"
Sep 18 20:22:47.735760 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:22:47.735721    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_lws-controller-manager-5947d9d788-w2hwk_799a1d4a-ff39-4bba-84c2-893ef3b7fbf7/manager/0.log"
Sep 18 20:22:48.868493 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:22:48.868405    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-6bc78fcbdb-tr229_5d1ac900-f718-4110-b637-069335895e44/kube-auth-proxy/0.log"
Sep 18 20:22:51.062103 ip-10-0-136-185 kubenswrapper[2585]: I0918 20:22:51.062067    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-6bc78fcbdb-tr229_5d1ac900-f718-4110-b637-069335895e44/kube-auth-proxy/0.log"