Sep 11 17:18:42.313872 ip-10-0-141-139 systemd[1]: Starting Kubernetes Kubelet...
Sep 11 17:18:42.689053 ip-10-0-141-139 kubenswrapper[2577]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 11 17:18:42.689053 ip-10-0-141-139 kubenswrapper[2577]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Sep 11 17:18:42.689053 ip-10-0-141-139 kubenswrapper[2577]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 11 17:18:42.689053 ip-10-0-141-139 kubenswrapper[2577]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI.
Sep 11 17:18:42.689053 ip-10-0-141-139 kubenswrapper[2577]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 11 17:18:42.690712 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:18:42.690626    2577 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime"
Sep 11 17:18:42.695465 ip-10-0-141-139 kubenswrapper[2577]: W0911 17:18:42.695079    2577 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 11 17:18:42.695465 ip-10-0-141-139 kubenswrapper[2577]: W0911 17:18:42.695106    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 11 17:18:42.695465 ip-10-0-141-139 kubenswrapper[2577]: W0911 17:18:42.695112    2577 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 11 17:18:42.695465 ip-10-0-141-139 kubenswrapper[2577]: W0911 17:18:42.695117    2577 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 11 17:18:42.695465 ip-10-0-141-139 kubenswrapper[2577]: W0911 17:18:42.695122    2577 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 11 17:18:42.695465 ip-10-0-141-139 kubenswrapper[2577]: W0911 17:18:42.695126    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 11 17:18:42.695465 ip-10-0-141-139 kubenswrapper[2577]: W0911 17:18:42.695132    2577 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 11 17:18:42.695465 ip-10-0-141-139 kubenswrapper[2577]: W0911 17:18:42.695136    2577 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 11 17:18:42.695465 ip-10-0-141-139 kubenswrapper[2577]: W0911 17:18:42.695141    2577 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 11 17:18:42.695465 ip-10-0-141-139 kubenswrapper[2577]: W0911 17:18:42.695151    2577 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 11 17:18:42.695465 ip-10-0-141-139 kubenswrapper[2577]: W0911 17:18:42.695155    2577 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 11 17:18:42.695465 ip-10-0-141-139 kubenswrapper[2577]: W0911 17:18:42.695159    2577 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 11 17:18:42.695465 ip-10-0-141-139 kubenswrapper[2577]: W0911 17:18:42.695164    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 11 17:18:42.695465 ip-10-0-141-139 kubenswrapper[2577]: W0911 17:18:42.695168    2577 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 11 17:18:42.695465 ip-10-0-141-139 kubenswrapper[2577]: W0911 17:18:42.695172    2577 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 11 17:18:42.695465 ip-10-0-141-139 kubenswrapper[2577]: W0911 17:18:42.695176    2577 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 11 17:18:42.695465 ip-10-0-141-139 kubenswrapper[2577]: W0911 17:18:42.695180    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 11 17:18:42.695465 ip-10-0-141-139 kubenswrapper[2577]: W0911 17:18:42.695184    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 11 17:18:42.695465 ip-10-0-141-139 kubenswrapper[2577]: W0911 17:18:42.695189    2577 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 11 17:18:42.696242 ip-10-0-141-139 kubenswrapper[2577]: W0911 17:18:42.695193    2577 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 11 17:18:42.696242 ip-10-0-141-139 kubenswrapper[2577]: W0911 17:18:42.695198    2577 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 11 17:18:42.696242 ip-10-0-141-139 kubenswrapper[2577]: W0911 17:18:42.695202    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 11 17:18:42.696242 ip-10-0-141-139 kubenswrapper[2577]: W0911 17:18:42.695211    2577 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 11 17:18:42.696242 ip-10-0-141-139 kubenswrapper[2577]: W0911 17:18:42.695216    2577 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 11 17:18:42.696242 ip-10-0-141-139 kubenswrapper[2577]: W0911 17:18:42.695220    2577 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 11 17:18:42.696242 ip-10-0-141-139 kubenswrapper[2577]: W0911 17:18:42.695224    2577 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 11 17:18:42.696242 ip-10-0-141-139 kubenswrapper[2577]: W0911 17:18:42.695229    2577 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 11 17:18:42.696242 ip-10-0-141-139 kubenswrapper[2577]: W0911 17:18:42.695233    2577 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 11 17:18:42.696242 ip-10-0-141-139 kubenswrapper[2577]: W0911 17:18:42.695237    2577 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 11 17:18:42.696242 ip-10-0-141-139 kubenswrapper[2577]: W0911 17:18:42.695241    2577 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 11 17:18:42.696242 ip-10-0-141-139 kubenswrapper[2577]: W0911 17:18:42.695246    2577 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 11 17:18:42.696242 ip-10-0-141-139 kubenswrapper[2577]: W0911 17:18:42.695252    2577 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 11 17:18:42.696242 ip-10-0-141-139 kubenswrapper[2577]: W0911 17:18:42.695257    2577 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 11 17:18:42.696242 ip-10-0-141-139 kubenswrapper[2577]: W0911 17:18:42.695261    2577 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 11 17:18:42.696242 ip-10-0-141-139 kubenswrapper[2577]: W0911 17:18:42.695272    2577 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 11 17:18:42.696242 ip-10-0-141-139 kubenswrapper[2577]: W0911 17:18:42.695276    2577 feature_gate.go:328] unrecognized feature gate: Example
Sep 11 17:18:42.696242 ip-10-0-141-139 kubenswrapper[2577]: W0911 17:18:42.695280    2577 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 11 17:18:42.696242 ip-10-0-141-139 kubenswrapper[2577]: W0911 17:18:42.695285    2577 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 11 17:18:42.696242 ip-10-0-141-139 kubenswrapper[2577]: W0911 17:18:42.695290    2577 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 11 17:18:42.696739 ip-10-0-141-139 kubenswrapper[2577]: W0911 17:18:42.695297    2577 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 11 17:18:42.696739 ip-10-0-141-139 kubenswrapper[2577]: W0911 17:18:42.695304    2577 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 11 17:18:42.696739 ip-10-0-141-139 kubenswrapper[2577]: W0911 17:18:42.695308    2577 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 11 17:18:42.696739 ip-10-0-141-139 kubenswrapper[2577]: W0911 17:18:42.695313    2577 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 11 17:18:42.696739 ip-10-0-141-139 kubenswrapper[2577]: W0911 17:18:42.695317    2577 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 11 17:18:42.696739 ip-10-0-141-139 kubenswrapper[2577]: W0911 17:18:42.695321    2577 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 11 17:18:42.696739 ip-10-0-141-139 kubenswrapper[2577]: W0911 17:18:42.695331    2577 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 11 17:18:42.696739 ip-10-0-141-139 kubenswrapper[2577]: W0911 17:18:42.695336    2577 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 11 17:18:42.696739 ip-10-0-141-139 kubenswrapper[2577]: W0911 17:18:42.695340    2577 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 11 17:18:42.696739 ip-10-0-141-139 kubenswrapper[2577]: W0911 17:18:42.695344    2577 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 11 17:18:42.696739 ip-10-0-141-139 kubenswrapper[2577]: W0911 17:18:42.695348    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 11 17:18:42.696739 ip-10-0-141-139 kubenswrapper[2577]: W0911 17:18:42.695353    2577 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 11 17:18:42.696739 ip-10-0-141-139 kubenswrapper[2577]: W0911 17:18:42.695358    2577 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 11 17:18:42.696739 ip-10-0-141-139 kubenswrapper[2577]: W0911 17:18:42.695362    2577 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 11 17:18:42.696739 ip-10-0-141-139 kubenswrapper[2577]: W0911 17:18:42.695366    2577 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 11 17:18:42.696739 ip-10-0-141-139 kubenswrapper[2577]: W0911 17:18:42.695370    2577 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 11 17:18:42.696739 ip-10-0-141-139 kubenswrapper[2577]: W0911 17:18:42.695374    2577 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 11 17:18:42.696739 ip-10-0-141-139 kubenswrapper[2577]: W0911 17:18:42.695379    2577 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 11 17:18:42.696739 ip-10-0-141-139 kubenswrapper[2577]: W0911 17:18:42.695383    2577 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 11 17:18:42.697256 ip-10-0-141-139 kubenswrapper[2577]: W0911 17:18:42.695393    2577 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 11 17:18:42.697256 ip-10-0-141-139 kubenswrapper[2577]: W0911 17:18:42.695397    2577 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 11 17:18:42.697256 ip-10-0-141-139 kubenswrapper[2577]: W0911 17:18:42.695401    2577 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 11 17:18:42.697256 ip-10-0-141-139 kubenswrapper[2577]: W0911 17:18:42.695405    2577 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 11 17:18:42.697256 ip-10-0-141-139 kubenswrapper[2577]: W0911 17:18:42.695409    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 11 17:18:42.697256 ip-10-0-141-139 kubenswrapper[2577]: W0911 17:18:42.695414    2577 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 11 17:18:42.697256 ip-10-0-141-139 kubenswrapper[2577]: W0911 17:18:42.695418    2577 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 11 17:18:42.697256 ip-10-0-141-139 kubenswrapper[2577]: W0911 17:18:42.695423    2577 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 11 17:18:42.697256 ip-10-0-141-139 kubenswrapper[2577]: W0911 17:18:42.695427    2577 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 11 17:18:42.697256 ip-10-0-141-139 kubenswrapper[2577]: W0911 17:18:42.695431    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 11 17:18:42.697256 ip-10-0-141-139 kubenswrapper[2577]: W0911 17:18:42.695438    2577 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 11 17:18:42.697256 ip-10-0-141-139 kubenswrapper[2577]: W0911 17:18:42.695442    2577 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 11 17:18:42.697256 ip-10-0-141-139 kubenswrapper[2577]: W0911 17:18:42.695453    2577 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 11 17:18:42.697256 ip-10-0-141-139 kubenswrapper[2577]: W0911 17:18:42.695497    2577 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 11 17:18:42.697256 ip-10-0-141-139 kubenswrapper[2577]: W0911 17:18:42.695529    2577 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 11 17:18:42.697256 ip-10-0-141-139 kubenswrapper[2577]: W0911 17:18:42.695534    2577 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 11 17:18:42.697256 ip-10-0-141-139 kubenswrapper[2577]: W0911 17:18:42.695540    2577 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 11 17:18:42.697256 ip-10-0-141-139 kubenswrapper[2577]: W0911 17:18:42.695546    2577 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 11 17:18:42.697256 ip-10-0-141-139 kubenswrapper[2577]: W0911 17:18:42.695550    2577 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 11 17:18:42.697256 ip-10-0-141-139 kubenswrapper[2577]: W0911 17:18:42.695557    2577 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 11 17:18:42.697754 ip-10-0-141-139 kubenswrapper[2577]: W0911 17:18:42.695561    2577 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 11 17:18:42.697754 ip-10-0-141-139 kubenswrapper[2577]: W0911 17:18:42.695566    2577 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 11 17:18:42.697754 ip-10-0-141-139 kubenswrapper[2577]: W0911 17:18:42.695570    2577 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 11 17:18:42.697754 ip-10-0-141-139 kubenswrapper[2577]: W0911 17:18:42.695575    2577 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 11 17:18:42.697754 ip-10-0-141-139 kubenswrapper[2577]: W0911 17:18:42.695580    2577 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 11 17:18:42.697754 ip-10-0-141-139 kubenswrapper[2577]: W0911 17:18:42.695584    2577 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 11 17:18:42.697754 ip-10-0-141-139 kubenswrapper[2577]: W0911 17:18:42.695594    2577 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 11 17:18:42.697754 ip-10-0-141-139 kubenswrapper[2577]: W0911 17:18:42.695598    2577 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 11 17:18:42.697754 ip-10-0-141-139 kubenswrapper[2577]: W0911 17:18:42.695603    2577 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 11 17:18:42.697754 ip-10-0-141-139 kubenswrapper[2577]: W0911 17:18:42.695607    2577 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 11 17:18:42.697754 ip-10-0-141-139 kubenswrapper[2577]: W0911 17:18:42.695613    2577 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 11 17:18:42.697754 ip-10-0-141-139 kubenswrapper[2577]: W0911 17:18:42.695621    2577 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 11 17:18:42.697754 ip-10-0-141-139 kubenswrapper[2577]: W0911 17:18:42.695626    2577 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 11 17:18:42.697754 ip-10-0-141-139 kubenswrapper[2577]: W0911 17:18:42.695630    2577 feature_gate.go:328] unrecognized feature gate: Example2
Sep 11 17:18:42.697754 ip-10-0-141-139 kubenswrapper[2577]: W0911 17:18:42.695635    2577 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 11 17:18:42.697754 ip-10-0-141-139 kubenswrapper[2577]: W0911 17:18:42.695641    2577 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 11 17:18:42.697754 ip-10-0-141-139 kubenswrapper[2577]: W0911 17:18:42.695645    2577 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 11 17:18:42.697754 ip-10-0-141-139 kubenswrapper[2577]: W0911 17:18:42.695649    2577 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 11 17:18:42.697754 ip-10-0-141-139 kubenswrapper[2577]: W0911 17:18:42.695659    2577 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 11 17:18:42.698258 ip-10-0-141-139 kubenswrapper[2577]: W0911 17:18:42.695663    2577 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 11 17:18:42.698258 ip-10-0-141-139 kubenswrapper[2577]: W0911 17:18:42.695667    2577 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 11 17:18:42.698258 ip-10-0-141-139 kubenswrapper[2577]: W0911 17:18:42.697654    2577 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 11 17:18:42.698258 ip-10-0-141-139 kubenswrapper[2577]: W0911 17:18:42.697666    2577 feature_gate.go:328] unrecognized feature gate: Example
Sep 11 17:18:42.698258 ip-10-0-141-139 kubenswrapper[2577]: W0911 17:18:42.697670    2577 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 11 17:18:42.698258 ip-10-0-141-139 kubenswrapper[2577]: W0911 17:18:42.697674    2577 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 11 17:18:42.698258 ip-10-0-141-139 kubenswrapper[2577]: W0911 17:18:42.697677    2577 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 11 17:18:42.698258 ip-10-0-141-139 kubenswrapper[2577]: W0911 17:18:42.697680    2577 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 11 17:18:42.698258 ip-10-0-141-139 kubenswrapper[2577]: W0911 17:18:42.697684    2577 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 11 17:18:42.698258 ip-10-0-141-139 kubenswrapper[2577]: W0911 17:18:42.697688    2577 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 11 17:18:42.698258 ip-10-0-141-139 kubenswrapper[2577]: W0911 17:18:42.697690    2577 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 11 17:18:42.698258 ip-10-0-141-139 kubenswrapper[2577]: W0911 17:18:42.697693    2577 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 11 17:18:42.698258 ip-10-0-141-139 kubenswrapper[2577]: W0911 17:18:42.697696    2577 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 11 17:18:42.698258 ip-10-0-141-139 kubenswrapper[2577]: W0911 17:18:42.697699    2577 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 11 17:18:42.698258 ip-10-0-141-139 kubenswrapper[2577]: W0911 17:18:42.697702    2577 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 11 17:18:42.698258 ip-10-0-141-139 kubenswrapper[2577]: W0911 17:18:42.697704    2577 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 11 17:18:42.698258 ip-10-0-141-139 kubenswrapper[2577]: W0911 17:18:42.697712    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 11 17:18:42.698258 ip-10-0-141-139 kubenswrapper[2577]: W0911 17:18:42.697715    2577 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 11 17:18:42.698258 ip-10-0-141-139 kubenswrapper[2577]: W0911 17:18:42.697720    2577 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 11 17:18:42.698802 ip-10-0-141-139 kubenswrapper[2577]: W0911 17:18:42.697724    2577 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 11 17:18:42.698802 ip-10-0-141-139 kubenswrapper[2577]: W0911 17:18:42.697727    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 11 17:18:42.698802 ip-10-0-141-139 kubenswrapper[2577]: W0911 17:18:42.697730    2577 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 11 17:18:42.698802 ip-10-0-141-139 kubenswrapper[2577]: W0911 17:18:42.697734    2577 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 11 17:18:42.698802 ip-10-0-141-139 kubenswrapper[2577]: W0911 17:18:42.697737    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 11 17:18:42.698802 ip-10-0-141-139 kubenswrapper[2577]: W0911 17:18:42.697740    2577 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 11 17:18:42.698802 ip-10-0-141-139 kubenswrapper[2577]: W0911 17:18:42.697743    2577 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 11 17:18:42.698802 ip-10-0-141-139 kubenswrapper[2577]: W0911 17:18:42.697746    2577 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 11 17:18:42.698802 ip-10-0-141-139 kubenswrapper[2577]: W0911 17:18:42.697748    2577 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 11 17:18:42.698802 ip-10-0-141-139 kubenswrapper[2577]: W0911 17:18:42.697751    2577 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 11 17:18:42.698802 ip-10-0-141-139 kubenswrapper[2577]: W0911 17:18:42.697754    2577 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 11 17:18:42.698802 ip-10-0-141-139 kubenswrapper[2577]: W0911 17:18:42.697757    2577 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 11 17:18:42.698802 ip-10-0-141-139 kubenswrapper[2577]: W0911 17:18:42.697759    2577 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 11 17:18:42.698802 ip-10-0-141-139 kubenswrapper[2577]: W0911 17:18:42.697762    2577 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 11 17:18:42.698802 ip-10-0-141-139 kubenswrapper[2577]: W0911 17:18:42.697765    2577 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 11 17:18:42.698802 ip-10-0-141-139 kubenswrapper[2577]: W0911 17:18:42.697768    2577 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 11 17:18:42.698802 ip-10-0-141-139 kubenswrapper[2577]: W0911 17:18:42.697772    2577 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 11 17:18:42.698802 ip-10-0-141-139 kubenswrapper[2577]: W0911 17:18:42.697775    2577 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 11 17:18:42.698802 ip-10-0-141-139 kubenswrapper[2577]: W0911 17:18:42.697777    2577 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 11 17:18:42.698802 ip-10-0-141-139 kubenswrapper[2577]: W0911 17:18:42.697780    2577 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 11 17:18:42.699305 ip-10-0-141-139 kubenswrapper[2577]: W0911 17:18:42.697783    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 11 17:18:42.699305 ip-10-0-141-139 kubenswrapper[2577]: W0911 17:18:42.697785    2577 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 11 17:18:42.699305 ip-10-0-141-139 kubenswrapper[2577]: W0911 17:18:42.697788    2577 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 11 17:18:42.699305 ip-10-0-141-139 kubenswrapper[2577]: W0911 17:18:42.697791    2577 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 11 17:18:42.699305 ip-10-0-141-139 kubenswrapper[2577]: W0911 17:18:42.697793    2577 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 11 17:18:42.699305 ip-10-0-141-139 kubenswrapper[2577]: W0911 17:18:42.697796    2577 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 11 17:18:42.699305 ip-10-0-141-139 kubenswrapper[2577]: W0911 17:18:42.697799    2577 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 11 17:18:42.699305 ip-10-0-141-139 kubenswrapper[2577]: W0911 17:18:42.697802    2577 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 11 17:18:42.699305 ip-10-0-141-139 kubenswrapper[2577]: W0911 17:18:42.697804    2577 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 11 17:18:42.699305 ip-10-0-141-139 kubenswrapper[2577]: W0911 17:18:42.697807    2577 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 11 17:18:42.699305 ip-10-0-141-139 kubenswrapper[2577]: W0911 17:18:42.697811    2577 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 11 17:18:42.699305 ip-10-0-141-139 kubenswrapper[2577]: W0911 17:18:42.697815    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 11 17:18:42.699305 ip-10-0-141-139 kubenswrapper[2577]: W0911 17:18:42.697818    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 11 17:18:42.699305 ip-10-0-141-139 kubenswrapper[2577]: W0911 17:18:42.697821    2577 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 11 17:18:42.699305 ip-10-0-141-139 kubenswrapper[2577]: W0911 17:18:42.697825    2577 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 11 17:18:42.699305 ip-10-0-141-139 kubenswrapper[2577]: W0911 17:18:42.697827    2577 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 11 17:18:42.699305 ip-10-0-141-139 kubenswrapper[2577]: W0911 17:18:42.697830    2577 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 11 17:18:42.699305 ip-10-0-141-139 kubenswrapper[2577]: W0911 17:18:42.697834    2577 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 11 17:18:42.699305 ip-10-0-141-139 kubenswrapper[2577]: W0911 17:18:42.697836    2577 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 11 17:18:42.699305 ip-10-0-141-139 kubenswrapper[2577]: W0911 17:18:42.697839    2577 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 11 17:18:42.699824 ip-10-0-141-139 kubenswrapper[2577]: W0911 17:18:42.697841    2577 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 11 17:18:42.699824 ip-10-0-141-139 kubenswrapper[2577]: W0911 17:18:42.697844    2577 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 11 17:18:42.699824 ip-10-0-141-139 kubenswrapper[2577]: W0911 17:18:42.697846    2577 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 11 17:18:42.699824 ip-10-0-141-139 kubenswrapper[2577]: W0911 17:18:42.697849    2577 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 11 17:18:42.699824 ip-10-0-141-139 kubenswrapper[2577]: W0911 17:18:42.697851    2577 feature_gate.go:328] unrecognized feature gate: Example2
Sep 11 17:18:42.699824 ip-10-0-141-139 kubenswrapper[2577]: W0911 17:18:42.697854    2577 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 11 17:18:42.699824 ip-10-0-141-139 kubenswrapper[2577]: W0911 17:18:42.697857    2577 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 11 17:18:42.699824 ip-10-0-141-139 kubenswrapper[2577]: W0911 17:18:42.697859    2577 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 11 17:18:42.699824 ip-10-0-141-139 kubenswrapper[2577]: W0911 17:18:42.697862    2577 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 11 17:18:42.699824 ip-10-0-141-139 kubenswrapper[2577]: W0911 17:18:42.697865    2577 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 11 17:18:42.699824 ip-10-0-141-139 kubenswrapper[2577]: W0911 17:18:42.697867    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 11 17:18:42.699824 ip-10-0-141-139 kubenswrapper[2577]: W0911 17:18:42.697870    2577 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 11 17:18:42.699824 ip-10-0-141-139 kubenswrapper[2577]: W0911 17:18:42.697873    2577 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 11 17:18:42.699824 ip-10-0-141-139 kubenswrapper[2577]: W0911 17:18:42.697876    2577 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 11 17:18:42.699824 ip-10-0-141-139 kubenswrapper[2577]: W0911 17:18:42.697879    2577 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 11 17:18:42.699824 ip-10-0-141-139 kubenswrapper[2577]: W0911 17:18:42.697882    2577 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 11 17:18:42.699824 ip-10-0-141-139 kubenswrapper[2577]: W0911 17:18:42.697885    2577 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 11 17:18:42.699824 ip-10-0-141-139 kubenswrapper[2577]: W0911 17:18:42.697888    2577 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 11 17:18:42.699824 ip-10-0-141-139 kubenswrapper[2577]: W0911 17:18:42.697890    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 11 17:18:42.699824 ip-10-0-141-139 kubenswrapper[2577]: W0911 17:18:42.697893    2577 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 11 17:18:42.700326 ip-10-0-141-139 kubenswrapper[2577]: W0911 17:18:42.697895    2577 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 11 17:18:42.700326 ip-10-0-141-139 kubenswrapper[2577]: W0911 17:18:42.697898    2577 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 11 17:18:42.700326 ip-10-0-141-139 kubenswrapper[2577]: W0911 17:18:42.697900    2577 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 11 17:18:42.700326 ip-10-0-141-139 kubenswrapper[2577]: W0911 17:18:42.697903    2577 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 11 17:18:42.700326 ip-10-0-141-139 kubenswrapper[2577]: W0911 17:18:42.697905    2577 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 11 17:18:42.700326 ip-10-0-141-139 kubenswrapper[2577]: W0911 17:18:42.697909    2577 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 11 17:18:42.700326 ip-10-0-141-139 kubenswrapper[2577]: W0911 17:18:42.697914    2577 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 11 17:18:42.700326 ip-10-0-141-139 kubenswrapper[2577]: W0911 17:18:42.697917    2577 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 11 17:18:42.700326 ip-10-0-141-139 kubenswrapper[2577]: W0911 17:18:42.697920    2577 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 11 17:18:42.700326 ip-10-0-141-139 kubenswrapper[2577]: W0911 17:18:42.697923    2577 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 11 17:18:42.700326 ip-10-0-141-139 kubenswrapper[2577]: W0911 17:18:42.697925    2577 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 11 17:18:42.700326 ip-10-0-141-139 kubenswrapper[2577]: W0911 17:18:42.697928    2577 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 11 17:18:42.700326 ip-10-0-141-139 kubenswrapper[2577]: W0911 17:18:42.697930    2577 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 11 17:18:42.700326 ip-10-0-141-139 kubenswrapper[2577]: W0911 17:18:42.697933    2577 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 11 17:18:42.700326 ip-10-0-141-139 kubenswrapper[2577]: W0911 17:18:42.697935    2577 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 11 17:18:42.700326 ip-10-0-141-139 kubenswrapper[2577]: W0911 17:18:42.697938    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 11 17:18:42.700326 ip-10-0-141-139 kubenswrapper[2577]: W0911 17:18:42.697940    2577 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 11 17:18:42.700326 ip-10-0-141-139 kubenswrapper[2577]: W0911 17:18:42.697943    2577 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 11 17:18:42.700326 ip-10-0-141-139 kubenswrapper[2577]: W0911 17:18:42.697945    2577 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 11 17:18:42.700326 ip-10-0-141-139 kubenswrapper[2577]: W0911 17:18:42.697948    2577 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 11 17:18:42.700836 ip-10-0-141-139 kubenswrapper[2577]: W0911 17:18:42.697951    2577 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 11 17:18:42.700836 ip-10-0-141-139 kubenswrapper[2577]: W0911 17:18:42.697954    2577 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 11 17:18:42.700836 ip-10-0-141-139 kubenswrapper[2577]: W0911 17:18:42.698395    2577 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 11 17:18:42.700836 ip-10-0-141-139 kubenswrapper[2577]: W0911 17:18:42.698400    2577 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 11 17:18:42.700836 ip-10-0-141-139 kubenswrapper[2577]: W0911 17:18:42.698404    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 11 17:18:42.700836 ip-10-0-141-139 kubenswrapper[2577]: W0911 17:18:42.698407    2577 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 11 17:18:42.700836 ip-10-0-141-139 kubenswrapper[2577]: W0911 17:18:42.698409    2577 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 11 17:18:42.700836 ip-10-0-141-139 kubenswrapper[2577]: W0911 17:18:42.698412    2577 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 11 17:18:42.700836 ip-10-0-141-139 kubenswrapper[2577]: W0911 17:18:42.698415    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 11 17:18:42.700836 ip-10-0-141-139 kubenswrapper[2577]: W0911 17:18:42.698418    2577 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 11 17:18:42.700836 ip-10-0-141-139 kubenswrapper[2577]: W0911 17:18:42.698421    2577 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 11 17:18:42.700836 ip-10-0-141-139 kubenswrapper[2577]: W0911 17:18:42.698423    2577 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 11 17:18:42.700836 ip-10-0-141-139 kubenswrapper[2577]: W0911 17:18:42.698426    2577 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 11 17:18:42.700836 ip-10-0-141-139 kubenswrapper[2577]: W0911 17:18:42.698428    2577 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 11 17:18:42.700836 ip-10-0-141-139 kubenswrapper[2577]: W0911 17:18:42.698431    2577 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 11 17:18:42.700836 ip-10-0-141-139 kubenswrapper[2577]: W0911 17:18:42.698434    2577 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 11 17:18:42.700836 ip-10-0-141-139 kubenswrapper[2577]: W0911 17:18:42.698436    2577 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 11 17:18:42.700836 ip-10-0-141-139 kubenswrapper[2577]: W0911 17:18:42.698439    2577 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 11 17:18:42.700836 ip-10-0-141-139 kubenswrapper[2577]: W0911 17:18:42.698443    2577 feature_gate.go:328] unrecognized feature gate: Example2
Sep 11 17:18:42.700836 ip-10-0-141-139 kubenswrapper[2577]: W0911 17:18:42.698446    2577 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 11 17:18:42.701348 ip-10-0-141-139 kubenswrapper[2577]: W0911 17:18:42.698450    2577 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 11 17:18:42.701348 ip-10-0-141-139 kubenswrapper[2577]: W0911 17:18:42.698452    2577 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 11 17:18:42.701348 ip-10-0-141-139 kubenswrapper[2577]: W0911 17:18:42.698455    2577 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 11 17:18:42.701348 ip-10-0-141-139 kubenswrapper[2577]: W0911 17:18:42.698458    2577 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 11 17:18:42.701348 ip-10-0-141-139 kubenswrapper[2577]: W0911 17:18:42.698461    2577 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 11 17:18:42.701348 ip-10-0-141-139 kubenswrapper[2577]: W0911 17:18:42.698463    2577 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 11 17:18:42.701348 ip-10-0-141-139 kubenswrapper[2577]: W0911 17:18:42.698466    2577 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 11 17:18:42.701348 ip-10-0-141-139 kubenswrapper[2577]: W0911 17:18:42.698469    2577 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 11 17:18:42.701348 ip-10-0-141-139 kubenswrapper[2577]: W0911 17:18:42.698471    2577 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 11 17:18:42.701348 ip-10-0-141-139 kubenswrapper[2577]: W0911 17:18:42.698474    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 11 17:18:42.701348 ip-10-0-141-139 kubenswrapper[2577]: W0911 17:18:42.698476    2577 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 11 17:18:42.701348 ip-10-0-141-139 kubenswrapper[2577]: W0911 17:18:42.698479    2577 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 11 17:18:42.701348 ip-10-0-141-139 kubenswrapper[2577]: W0911 17:18:42.698482    2577 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 11 17:18:42.701348 ip-10-0-141-139 kubenswrapper[2577]: W0911 17:18:42.698484    2577 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 11 17:18:42.701348 ip-10-0-141-139 kubenswrapper[2577]: W0911 17:18:42.698487    2577 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 11 17:18:42.701348 ip-10-0-141-139 kubenswrapper[2577]: W0911 17:18:42.698490    2577 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 11 17:18:42.701348 ip-10-0-141-139 kubenswrapper[2577]: W0911 17:18:42.698492    2577 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 11 17:18:42.701348 ip-10-0-141-139 kubenswrapper[2577]: W0911 17:18:42.698495    2577 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 11 17:18:42.701348 ip-10-0-141-139 kubenswrapper[2577]: W0911 17:18:42.698497    2577 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 11 17:18:42.701348 ip-10-0-141-139 kubenswrapper[2577]: W0911 17:18:42.698515    2577 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 11 17:18:42.701877 ip-10-0-141-139 kubenswrapper[2577]: W0911 17:18:42.698518    2577 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 11 17:18:42.701877 ip-10-0-141-139 kubenswrapper[2577]: W0911 17:18:42.698520    2577 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 11 17:18:42.701877 ip-10-0-141-139 kubenswrapper[2577]: W0911 17:18:42.698523    2577 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 11 17:18:42.701877 ip-10-0-141-139 kubenswrapper[2577]: W0911 17:18:42.698526    2577 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 11 17:18:42.701877 ip-10-0-141-139 kubenswrapper[2577]: W0911 17:18:42.698528    2577 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 11 17:18:42.701877 ip-10-0-141-139 kubenswrapper[2577]: W0911 17:18:42.698531    2577 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 11 17:18:42.701877 ip-10-0-141-139 kubenswrapper[2577]: W0911 17:18:42.698535    2577 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 11 17:18:42.701877 ip-10-0-141-139 kubenswrapper[2577]: W0911 17:18:42.698539    2577 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 11 17:18:42.701877 ip-10-0-141-139 kubenswrapper[2577]: W0911 17:18:42.698542    2577 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 11 17:18:42.701877 ip-10-0-141-139 kubenswrapper[2577]: W0911 17:18:42.698544    2577 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 11 17:18:42.701877 ip-10-0-141-139 kubenswrapper[2577]: W0911 17:18:42.698547    2577 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 11 17:18:42.701877 ip-10-0-141-139 kubenswrapper[2577]: W0911 17:18:42.698551    2577 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 11 17:18:42.701877 ip-10-0-141-139 kubenswrapper[2577]: W0911 17:18:42.698554    2577 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 11 17:18:42.701877 ip-10-0-141-139 kubenswrapper[2577]: W0911 17:18:42.698556    2577 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 11 17:18:42.701877 ip-10-0-141-139 kubenswrapper[2577]: W0911 17:18:42.698559    2577 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 11 17:18:42.701877 ip-10-0-141-139 kubenswrapper[2577]: W0911 17:18:42.698562    2577 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 11 17:18:42.701877 ip-10-0-141-139 kubenswrapper[2577]: W0911 17:18:42.698565    2577 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 11 17:18:42.701877 ip-10-0-141-139 kubenswrapper[2577]: W0911 17:18:42.698567    2577 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 11 17:18:42.701877 ip-10-0-141-139 kubenswrapper[2577]: W0911 17:18:42.698571    2577 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 11 17:18:42.702338 ip-10-0-141-139 kubenswrapper[2577]: W0911 17:18:42.698575    2577 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 11 17:18:42.702338 ip-10-0-141-139 kubenswrapper[2577]: W0911 17:18:42.698579    2577 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 11 17:18:42.702338 ip-10-0-141-139 kubenswrapper[2577]: W0911 17:18:42.698583    2577 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 11 17:18:42.702338 ip-10-0-141-139 kubenswrapper[2577]: W0911 17:18:42.698586    2577 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 11 17:18:42.702338 ip-10-0-141-139 kubenswrapper[2577]: W0911 17:18:42.698589    2577 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 11 17:18:42.702338 ip-10-0-141-139 kubenswrapper[2577]: W0911 17:18:42.698592    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 11 17:18:42.702338 ip-10-0-141-139 kubenswrapper[2577]: W0911 17:18:42.698595    2577 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 11 17:18:42.702338 ip-10-0-141-139 kubenswrapper[2577]: W0911 17:18:42.698598    2577 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 11 17:18:42.702338 ip-10-0-141-139 kubenswrapper[2577]: W0911 17:18:42.698601    2577 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 11 17:18:42.702338 ip-10-0-141-139 kubenswrapper[2577]: W0911 17:18:42.698603    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 11 17:18:42.702338 ip-10-0-141-139 kubenswrapper[2577]: W0911 17:18:42.698606    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 11 17:18:42.702338 ip-10-0-141-139 kubenswrapper[2577]: W0911 17:18:42.698609    2577 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 11 17:18:42.702338 ip-10-0-141-139 kubenswrapper[2577]: W0911 17:18:42.698611    2577 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 11 17:18:42.702338 ip-10-0-141-139 kubenswrapper[2577]: W0911 17:18:42.698614    2577 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 11 17:18:42.702338 ip-10-0-141-139 kubenswrapper[2577]: W0911 17:18:42.698616    2577 feature_gate.go:328] unrecognized feature gate: Example
Sep 11 17:18:42.702338 ip-10-0-141-139 kubenswrapper[2577]: W0911 17:18:42.698619    2577 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 11 17:18:42.702338 ip-10-0-141-139 kubenswrapper[2577]: W0911 17:18:42.698621    2577 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 11 17:18:42.702338 ip-10-0-141-139 kubenswrapper[2577]: W0911 17:18:42.698624    2577 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 11 17:18:42.702338 ip-10-0-141-139 kubenswrapper[2577]: W0911 17:18:42.698626    2577 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 11 17:18:42.702338 ip-10-0-141-139 kubenswrapper[2577]: W0911 17:18:42.698629    2577 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 11 17:18:42.702846 ip-10-0-141-139 kubenswrapper[2577]: W0911 17:18:42.698632    2577 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 11 17:18:42.702846 ip-10-0-141-139 kubenswrapper[2577]: W0911 17:18:42.698634    2577 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 11 17:18:42.702846 ip-10-0-141-139 kubenswrapper[2577]: W0911 17:18:42.698637    2577 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 11 17:18:42.702846 ip-10-0-141-139 kubenswrapper[2577]: W0911 17:18:42.698640    2577 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 11 17:18:42.702846 ip-10-0-141-139 kubenswrapper[2577]: W0911 17:18:42.698643    2577 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 11 17:18:42.702846 ip-10-0-141-139 kubenswrapper[2577]: W0911 17:18:42.698646    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 11 17:18:42.702846 ip-10-0-141-139 kubenswrapper[2577]: W0911 17:18:42.698649    2577 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 11 17:18:42.702846 ip-10-0-141-139 kubenswrapper[2577]: W0911 17:18:42.698651    2577 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 11 17:18:42.702846 ip-10-0-141-139 kubenswrapper[2577]: W0911 17:18:42.698654    2577 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 11 17:18:42.702846 ip-10-0-141-139 kubenswrapper[2577]: W0911 17:18:42.698658    2577 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 11 17:18:42.702846 ip-10-0-141-139 kubenswrapper[2577]: W0911 17:18:42.698661    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 11 17:18:42.702846 ip-10-0-141-139 kubenswrapper[2577]: W0911 17:18:42.698663    2577 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 11 17:18:42.702846 ip-10-0-141-139 kubenswrapper[2577]: W0911 17:18:42.698666    2577 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 11 17:18:42.702846 ip-10-0-141-139 kubenswrapper[2577]: W0911 17:18:42.698668    2577 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 11 17:18:42.702846 ip-10-0-141-139 kubenswrapper[2577]: W0911 17:18:42.698671    2577 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 11 17:18:42.702846 ip-10-0-141-139 kubenswrapper[2577]: W0911 17:18:42.698674    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 11 17:18:42.702846 ip-10-0-141-139 kubenswrapper[2577]: W0911 17:18:42.698676    2577 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 11 17:18:42.702846 ip-10-0-141-139 kubenswrapper[2577]: W0911 17:18:42.698679    2577 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 11 17:18:42.702846 ip-10-0-141-139 kubenswrapper[2577]: W0911 17:18:42.698682    2577 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 11 17:18:42.702846 ip-10-0-141-139 kubenswrapper[2577]: W0911 17:18:42.698684    2577 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 11 17:18:42.703330 ip-10-0-141-139 kubenswrapper[2577]: W0911 17:18:42.698687    2577 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 11 17:18:42.703330 ip-10-0-141-139 kubenswrapper[2577]: W0911 17:18:42.698689    2577 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 11 17:18:42.703330 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:18:42.698774    2577 flags.go:64] FLAG: --address="0.0.0.0"
Sep 11 17:18:42.703330 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:18:42.698823    2577 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Sep 11 17:18:42.703330 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:18:42.698830    2577 flags.go:64] FLAG: --anonymous-auth="true"
Sep 11 17:18:42.703330 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:18:42.698835    2577 flags.go:64] FLAG: --application-metrics-count-limit="100"
Sep 11 17:18:42.703330 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:18:42.698839    2577 flags.go:64] FLAG: --authentication-token-webhook="false"
Sep 11 17:18:42.703330 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:18:42.698843    2577 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Sep 11 17:18:42.703330 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:18:42.698848    2577 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Sep 11 17:18:42.703330 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:18:42.698853    2577 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Sep 11 17:18:42.703330 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:18:42.698856    2577 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Sep 11 17:18:42.703330 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:18:42.698859    2577 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Sep 11 17:18:42.703330 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:18:42.698863    2577 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Sep 11 17:18:42.703330 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:18:42.698866    2577 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Sep 11 17:18:42.703330 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:18:42.698869    2577 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Sep 11 17:18:42.703330 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:18:42.698872    2577 flags.go:64] FLAG: --cgroup-root=""
Sep 11 17:18:42.703330 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:18:42.698876    2577 flags.go:64] FLAG: --cgroups-per-qos="true"
Sep 11 17:18:42.703330 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:18:42.698879    2577 flags.go:64] FLAG: --client-ca-file=""
Sep 11 17:18:42.703330 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:18:42.698882    2577 flags.go:64] FLAG: --cloud-provider="external"
Sep 11 17:18:42.703330 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:18:42.698886    2577 flags.go:64] FLAG: --cluster-dns="[]"
Sep 11 17:18:42.703330 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:18:42.698890    2577 flags.go:64] FLAG: --cluster-domain=""
Sep 11 17:18:42.703330 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:18:42.698893    2577 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Sep 11 17:18:42.703330 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:18:42.698896    2577 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Sep 11 17:18:42.703951 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:18:42.698899    2577 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Sep 11 17:18:42.703951 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:18:42.698903    2577 flags.go:64] FLAG: --container-log-max-files="5"
Sep 11 17:18:42.703951 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:18:42.698909    2577 flags.go:64] FLAG: --container-log-max-size="10Mi"
Sep 11 17:18:42.703951 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:18:42.698913    2577 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Sep 11 17:18:42.703951 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:18:42.698916    2577 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Sep 11 17:18:42.703951 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:18:42.698919    2577 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Sep 11 17:18:42.703951 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:18:42.698923    2577 flags.go:64] FLAG: --contention-profiling="false"
Sep 11 17:18:42.703951 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:18:42.698926    2577 flags.go:64] FLAG: --cpu-cfs-quota="true"
Sep 11 17:18:42.703951 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:18:42.698929    2577 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Sep 11 17:18:42.703951 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:18:42.698932    2577 flags.go:64] FLAG: --cpu-manager-policy="none"
Sep 11 17:18:42.703951 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:18:42.698935    2577 flags.go:64] FLAG: --cpu-manager-policy-options=""
Sep 11 17:18:42.703951 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:18:42.698939    2577 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Sep 11 17:18:42.703951 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:18:42.698943    2577 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Sep 11 17:18:42.703951 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:18:42.698946    2577 flags.go:64] FLAG: --enable-debugging-handlers="true"
Sep 11 17:18:42.703951 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:18:42.698948    2577 flags.go:64] FLAG: --enable-load-reader="false"
Sep 11 17:18:42.703951 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:18:42.698951    2577 flags.go:64] FLAG: --enable-server="true"
Sep 11 17:18:42.703951 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:18:42.698954    2577 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Sep 11 17:18:42.703951 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:18:42.698960    2577 flags.go:64] FLAG: --event-burst="100"
Sep 11 17:18:42.703951 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:18:42.698963    2577 flags.go:64] FLAG: --event-qps="50"
Sep 11 17:18:42.703951 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:18:42.698966    2577 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Sep 11 17:18:42.703951 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:18:42.698969    2577 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Sep 11 17:18:42.703951 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:18:42.698972    2577 flags.go:64] FLAG: --eviction-hard=""
Sep 11 17:18:42.703951 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:18:42.698976    2577 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Sep 11 17:18:42.703951 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:18:42.698979    2577 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Sep 11 17:18:42.704580 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:18:42.698983    2577 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Sep 11 17:18:42.704580 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:18:42.698986    2577 flags.go:64] FLAG: --eviction-soft=""
Sep 11 17:18:42.704580 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:18:42.698989    2577 flags.go:64] FLAG: --eviction-soft-grace-period=""
Sep 11 17:18:42.704580 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:18:42.698991    2577 flags.go:64] FLAG: --exit-on-lock-contention="false"
Sep 11 17:18:42.704580 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:18:42.698995    2577 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Sep 11 17:18:42.704580 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:18:42.698998    2577 flags.go:64] FLAG: --experimental-mounter-path=""
Sep 11 17:18:42.704580 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:18:42.699001    2577 flags.go:64] FLAG: --fail-cgroupv1="false"
Sep 11 17:18:42.704580 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:18:42.699004    2577 flags.go:64] FLAG: --fail-swap-on="true"
Sep 11 17:18:42.704580 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:18:42.699007    2577 flags.go:64] FLAG: --feature-gates=""
Sep 11 17:18:42.704580 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:18:42.699011    2577 flags.go:64] FLAG: --file-check-frequency="20s"
Sep 11 17:18:42.704580 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:18:42.699014    2577 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Sep 11 17:18:42.704580 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:18:42.699017    2577 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Sep 11 17:18:42.704580 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:18:42.699036    2577 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Sep 11 17:18:42.704580 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:18:42.699040    2577 flags.go:64] FLAG: --healthz-port="10248"
Sep 11 17:18:42.704580 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:18:42.699043    2577 flags.go:64] FLAG: --help="false"
Sep 11 17:18:42.704580 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:18:42.699047    2577 flags.go:64] FLAG: --hostname-override="ip-10-0-141-139.ec2.internal"
Sep 11 17:18:42.704580 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:18:42.699050    2577 flags.go:64] FLAG: --housekeeping-interval="10s"
Sep 11 17:18:42.704580 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:18:42.699054    2577 flags.go:64] FLAG: --http-check-frequency="20s"
Sep 11 17:18:42.704580 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:18:42.699057    2577 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Sep 11 17:18:42.704580 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:18:42.699060    2577 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Sep 11 17:18:42.704580 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:18:42.699064    2577 flags.go:64] FLAG: --image-gc-high-threshold="85"
Sep 11 17:18:42.704580 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:18:42.699067    2577 flags.go:64] FLAG: --image-gc-low-threshold="80"
Sep 11 17:18:42.704580 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:18:42.699070    2577 flags.go:64] FLAG: --image-service-endpoint=""
Sep 11 17:18:42.705168 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:18:42.699073    2577 flags.go:64] FLAG: --kernel-memcg-notification="false"
Sep 11 17:18:42.705168 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:18:42.699076    2577 flags.go:64] FLAG: --kube-api-burst="100"
Sep 11 17:18:42.705168 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:18:42.699079    2577 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Sep 11 17:18:42.705168 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:18:42.699082    2577 flags.go:64] FLAG: --kube-api-qps="50"
Sep 11 17:18:42.705168 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:18:42.699085    2577 flags.go:64] FLAG: --kube-reserved=""
Sep 11 17:18:42.705168 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:18:42.699088    2577 flags.go:64] FLAG: --kube-reserved-cgroup=""
Sep 11 17:18:42.705168 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:18:42.699091    2577 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Sep 11 17:18:42.705168 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:18:42.699094    2577 flags.go:64] FLAG: --kubelet-cgroups=""
Sep 11 17:18:42.705168 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:18:42.699097    2577 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Sep 11 17:18:42.705168 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:18:42.699100    2577 flags.go:64] FLAG: --lock-file=""
Sep 11 17:18:42.705168 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:18:42.699102    2577 flags.go:64] FLAG: --log-cadvisor-usage="false"
Sep 11 17:18:42.705168 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:18:42.699105    2577 flags.go:64] FLAG: --log-flush-frequency="5s"
Sep 11 17:18:42.705168 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:18:42.699108    2577 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Sep 11 17:18:42.705168 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:18:42.699114    2577 flags.go:64] FLAG: --log-json-split-stream="false"
Sep 11 17:18:42.705168 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:18:42.699118    2577 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Sep 11 17:18:42.705168 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:18:42.699120    2577 flags.go:64] FLAG: --log-text-split-stream="false"
Sep 11 17:18:42.705168 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:18:42.699123    2577 flags.go:64] FLAG: --logging-format="text"
Sep 11 17:18:42.705168 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:18:42.699126    2577 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Sep 11 17:18:42.705168 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:18:42.699130    2577 flags.go:64] FLAG: --make-iptables-util-chains="true"
Sep 11 17:18:42.705168 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:18:42.699133    2577 flags.go:64] FLAG: --manifest-url=""
Sep 11 17:18:42.705168 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:18:42.699136    2577 flags.go:64] FLAG: --manifest-url-header=""
Sep 11 17:18:42.705168 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:18:42.699140    2577 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Sep 11 17:18:42.705168 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:18:42.699143    2577 flags.go:64] FLAG: --max-open-files="1000000"
Sep 11 17:18:42.705168 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:18:42.699147    2577 flags.go:64] FLAG: --max-pods="110"
Sep 11 17:18:42.705168 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:18:42.699151    2577 flags.go:64] FLAG: --maximum-dead-containers="-1"
Sep 11 17:18:42.705811 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:18:42.699153    2577 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Sep 11 17:18:42.705811 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:18:42.699156    2577 flags.go:64] FLAG: --memory-manager-policy="None"
Sep 11 17:18:42.705811 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:18:42.699160    2577 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Sep 11 17:18:42.705811 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:18:42.699163    2577 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Sep 11 17:18:42.705811 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:18:42.699166    2577 flags.go:64] FLAG: --node-ip="0.0.0.0"
Sep 11 17:18:42.705811 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:18:42.699169    2577 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Sep 11 17:18:42.705811 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:18:42.699177    2577 flags.go:64] FLAG: --node-status-max-images="50"
Sep 11 17:18:42.705811 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:18:42.699180    2577 flags.go:64] FLAG: --node-status-update-frequency="10s"
Sep 11 17:18:42.705811 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:18:42.699183    2577 flags.go:64] FLAG: --oom-score-adj="-999"
Sep 11 17:18:42.705811 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:18:42.699186    2577 flags.go:64] FLAG: --pod-cidr=""
Sep 11 17:18:42.705811 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:18:42.699189    2577 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:eee392dbe3ac2ed4a38bca29ae3bd486f5b7780406c62c600154cd1906871b10"
Sep 11 17:18:42.705811 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:18:42.699195    2577 flags.go:64] FLAG: --pod-manifest-path=""
Sep 11 17:18:42.705811 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:18:42.699198    2577 flags.go:64] FLAG: --pod-max-pids="-1"
Sep 11 17:18:42.705811 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:18:42.699201    2577 flags.go:64] FLAG: --pods-per-core="0"
Sep 11 17:18:42.705811 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:18:42.699204    2577 flags.go:64] FLAG: --port="10250"
Sep 11 17:18:42.705811 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:18:42.699207    2577 flags.go:64] FLAG: --protect-kernel-defaults="false"
Sep 11 17:18:42.705811 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:18:42.699210    2577 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0d8e048f3c3e9e189"
Sep 11 17:18:42.705811 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:18:42.699213    2577 flags.go:64] FLAG: --qos-reserved=""
Sep 11 17:18:42.705811 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:18:42.699216    2577 flags.go:64] FLAG: --read-only-port="10255"
Sep 11 17:18:42.705811 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:18:42.699219    2577 flags.go:64] FLAG: --register-node="true"
Sep 11 17:18:42.705811 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:18:42.699222    2577 flags.go:64] FLAG: --register-with-taints=""
Sep 11 17:18:42.705811 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:18:42.699226    2577 flags.go:64] FLAG: --registry-burst="10"
Sep 11 17:18:42.705811 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:18:42.699229    2577 flags.go:64] FLAG: --registry-qps="5"
Sep 11 17:18:42.705811 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:18:42.699232    2577 flags.go:64] FLAG: --reserved-cpus=""
Sep 11 17:18:42.706429 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:18:42.699235    2577 flags.go:64] FLAG: --reserved-memory=""
Sep 11 17:18:42.706429 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:18:42.699241    2577 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Sep 11 17:18:42.706429 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:18:42.699244    2577 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Sep 11 17:18:42.706429 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:18:42.699247    2577 flags.go:64] FLAG: --rotate-certificates="false"
Sep 11 17:18:42.706429 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:18:42.699250    2577 flags.go:64] FLAG: --rotate-server-certificates="false"
Sep 11 17:18:42.706429 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:18:42.699254    2577 flags.go:64] FLAG: --runonce="false"
Sep 11 17:18:42.706429 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:18:42.699257    2577 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Sep 11 17:18:42.706429 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:18:42.699260    2577 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Sep 11 17:18:42.706429 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:18:42.699263    2577 flags.go:64] FLAG: --seccomp-default="false"
Sep 11 17:18:42.706429 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:18:42.699267    2577 flags.go:64] FLAG: --serialize-image-pulls="true"
Sep 11 17:18:42.706429 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:18:42.699270    2577 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Sep 11 17:18:42.706429 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:18:42.699273    2577 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Sep 11 17:18:42.706429 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:18:42.699276    2577 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Sep 11 17:18:42.706429 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:18:42.699279    2577 flags.go:64] FLAG: --storage-driver-password="root"
Sep 11 17:18:42.706429 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:18:42.699282    2577 flags.go:64] FLAG: --storage-driver-secure="false"
Sep 11 17:18:42.706429 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:18:42.699285    2577 flags.go:64] FLAG: --storage-driver-table="stats"
Sep 11 17:18:42.706429 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:18:42.699287    2577 flags.go:64] FLAG: --storage-driver-user="root"
Sep 11 17:18:42.706429 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:18:42.699290    2577 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Sep 11 17:18:42.706429 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:18:42.699294    2577 flags.go:64] FLAG: --sync-frequency="1m0s"
Sep 11 17:18:42.706429 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:18:42.699297    2577 flags.go:64] FLAG: --system-cgroups=""
Sep 11 17:18:42.706429 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:18:42.699300    2577 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Sep 11 17:18:42.706429 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:18:42.699305    2577 flags.go:64] FLAG: --system-reserved-cgroup=""
Sep 11 17:18:42.706429 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:18:42.699308    2577 flags.go:64] FLAG: --tls-cert-file=""
Sep 11 17:18:42.706429 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:18:42.699311    2577 flags.go:64] FLAG: --tls-cipher-suites="[]"
Sep 11 17:18:42.706429 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:18:42.699315    2577 flags.go:64] FLAG: --tls-min-version=""
Sep 11 17:18:42.707137 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:18:42.699318    2577 flags.go:64] FLAG: --tls-private-key-file=""
Sep 11 17:18:42.707137 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:18:42.699321    2577 flags.go:64] FLAG: --topology-manager-policy="none"
Sep 11 17:18:42.707137 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:18:42.699324    2577 flags.go:64] FLAG: --topology-manager-policy-options=""
Sep 11 17:18:42.707137 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:18:42.699327    2577 flags.go:64] FLAG: --topology-manager-scope="container"
Sep 11 17:18:42.707137 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:18:42.699330    2577 flags.go:64] FLAG: --v="2"
Sep 11 17:18:42.707137 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:18:42.699334    2577 flags.go:64] FLAG: --version="false"
Sep 11 17:18:42.707137 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:18:42.699338    2577 flags.go:64] FLAG: --vmodule=""
Sep 11 17:18:42.707137 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:18:42.699342    2577 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Sep 11 17:18:42.707137 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:18:42.699346    2577 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Sep 11 17:18:42.707137 ip-10-0-141-139 kubenswrapper[2577]: W0911 17:18:42.699491    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 11 17:18:42.707137 ip-10-0-141-139 kubenswrapper[2577]: W0911 17:18:42.699495    2577 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 11 17:18:42.707137 ip-10-0-141-139 kubenswrapper[2577]: W0911 17:18:42.699514    2577 feature_gate.go:328] unrecognized feature gate: Example2
Sep 11 17:18:42.707137 ip-10-0-141-139 kubenswrapper[2577]: W0911 17:18:42.699521    2577 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 11 17:18:42.707137 ip-10-0-141-139 kubenswrapper[2577]: W0911 17:18:42.699524    2577 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 11 17:18:42.707137 ip-10-0-141-139 kubenswrapper[2577]: W0911 17:18:42.699527    2577 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 11 17:18:42.707137 ip-10-0-141-139 kubenswrapper[2577]: W0911 17:18:42.699529    2577 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 11 17:18:42.707137 ip-10-0-141-139 kubenswrapper[2577]: W0911 17:18:42.699533    2577 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 11 17:18:42.707137 ip-10-0-141-139 kubenswrapper[2577]: W0911 17:18:42.699535    2577 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 11 17:18:42.707137 ip-10-0-141-139 kubenswrapper[2577]: W0911 17:18:42.699538    2577 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 11 17:18:42.707137 ip-10-0-141-139 kubenswrapper[2577]: W0911 17:18:42.699541    2577 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 11 17:18:42.707137 ip-10-0-141-139 kubenswrapper[2577]: W0911 17:18:42.699543    2577 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 11 17:18:42.707137 ip-10-0-141-139 kubenswrapper[2577]: W0911 17:18:42.699546    2577 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 11 17:18:42.707792 ip-10-0-141-139 kubenswrapper[2577]: W0911 17:18:42.699549    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 11 17:18:42.707792 ip-10-0-141-139 kubenswrapper[2577]: W0911 17:18:42.699552    2577 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 11 17:18:42.707792 ip-10-0-141-139 kubenswrapper[2577]: W0911 17:18:42.699554    2577 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 11 17:18:42.707792 ip-10-0-141-139 kubenswrapper[2577]: W0911 17:18:42.699557    2577 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 11 17:18:42.707792 ip-10-0-141-139 kubenswrapper[2577]: W0911 17:18:42.699560    2577 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 11 17:18:42.707792 ip-10-0-141-139 kubenswrapper[2577]: W0911 17:18:42.699562    2577 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 11 17:18:42.707792 ip-10-0-141-139 kubenswrapper[2577]: W0911 17:18:42.699565    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 11 17:18:42.707792 ip-10-0-141-139 kubenswrapper[2577]: W0911 17:18:42.699569    2577 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 11 17:18:42.707792 ip-10-0-141-139 kubenswrapper[2577]: W0911 17:18:42.699572    2577 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 11 17:18:42.707792 ip-10-0-141-139 kubenswrapper[2577]: W0911 17:18:42.699575    2577 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 11 17:18:42.707792 ip-10-0-141-139 kubenswrapper[2577]: W0911 17:18:42.699579    2577 feature_gate.go:328] unrecognized feature gate: Example
Sep 11 17:18:42.707792 ip-10-0-141-139 kubenswrapper[2577]: W0911 17:18:42.699582    2577 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 11 17:18:42.707792 ip-10-0-141-139 kubenswrapper[2577]: W0911 17:18:42.699585    2577 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 11 17:18:42.707792 ip-10-0-141-139 kubenswrapper[2577]: W0911 17:18:42.699588    2577 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 11 17:18:42.707792 ip-10-0-141-139 kubenswrapper[2577]: W0911 17:18:42.699592    2577 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 11 17:18:42.707792 ip-10-0-141-139 kubenswrapper[2577]: W0911 17:18:42.699595    2577 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 11 17:18:42.707792 ip-10-0-141-139 kubenswrapper[2577]: W0911 17:18:42.699597    2577 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 11 17:18:42.707792 ip-10-0-141-139 kubenswrapper[2577]: W0911 17:18:42.699601    2577 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 11 17:18:42.707792 ip-10-0-141-139 kubenswrapper[2577]: W0911 17:18:42.699605    2577 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 11 17:18:42.708292 ip-10-0-141-139 kubenswrapper[2577]: W0911 17:18:42.699607    2577 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 11 17:18:42.708292 ip-10-0-141-139 kubenswrapper[2577]: W0911 17:18:42.699610    2577 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 11 17:18:42.708292 ip-10-0-141-139 kubenswrapper[2577]: W0911 17:18:42.699613    2577 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 11 17:18:42.708292 ip-10-0-141-139 kubenswrapper[2577]: W0911 17:18:42.699617    2577 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 11 17:18:42.708292 ip-10-0-141-139 kubenswrapper[2577]: W0911 17:18:42.699620    2577 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 11 17:18:42.708292 ip-10-0-141-139 kubenswrapper[2577]: W0911 17:18:42.699622    2577 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 11 17:18:42.708292 ip-10-0-141-139 kubenswrapper[2577]: W0911 17:18:42.699625    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 11 17:18:42.708292 ip-10-0-141-139 kubenswrapper[2577]: W0911 17:18:42.699628    2577 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 11 17:18:42.708292 ip-10-0-141-139 kubenswrapper[2577]: W0911 17:18:42.699630    2577 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 11 17:18:42.708292 ip-10-0-141-139 kubenswrapper[2577]: W0911 17:18:42.699633    2577 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 11 17:18:42.708292 ip-10-0-141-139 kubenswrapper[2577]: W0911 17:18:42.699635    2577 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 11 17:18:42.708292 ip-10-0-141-139 kubenswrapper[2577]: W0911 17:18:42.699638    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 11 17:18:42.708292 ip-10-0-141-139 kubenswrapper[2577]: W0911 17:18:42.699640    2577 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 11 17:18:42.708292 ip-10-0-141-139 kubenswrapper[2577]: W0911 17:18:42.699643    2577 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 11 17:18:42.708292 ip-10-0-141-139 kubenswrapper[2577]: W0911 17:18:42.699646    2577 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 11 17:18:42.708292 ip-10-0-141-139 kubenswrapper[2577]: W0911 17:18:42.699649    2577 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 11 17:18:42.708292 ip-10-0-141-139 kubenswrapper[2577]: W0911 17:18:42.699651    2577 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 11 17:18:42.708292 ip-10-0-141-139 kubenswrapper[2577]: W0911 17:18:42.699654    2577 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 11 17:18:42.708292 ip-10-0-141-139 kubenswrapper[2577]: W0911 17:18:42.699657    2577 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 11 17:18:42.708292 ip-10-0-141-139 kubenswrapper[2577]: W0911 17:18:42.699659    2577 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 11 17:18:42.708812 ip-10-0-141-139 kubenswrapper[2577]: W0911 17:18:42.699663    2577 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 11 17:18:42.708812 ip-10-0-141-139 kubenswrapper[2577]: W0911 17:18:42.699665    2577 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 11 17:18:42.708812 ip-10-0-141-139 kubenswrapper[2577]: W0911 17:18:42.699668    2577 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 11 17:18:42.708812 ip-10-0-141-139 kubenswrapper[2577]: W0911 17:18:42.699671    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 11 17:18:42.708812 ip-10-0-141-139 kubenswrapper[2577]: W0911 17:18:42.699673    2577 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 11 17:18:42.708812 ip-10-0-141-139 kubenswrapper[2577]: W0911 17:18:42.699676    2577 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 11 17:18:42.708812 ip-10-0-141-139 kubenswrapper[2577]: W0911 17:18:42.699680    2577 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 11 17:18:42.708812 ip-10-0-141-139 kubenswrapper[2577]: W0911 17:18:42.699683    2577 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 11 17:18:42.708812 ip-10-0-141-139 kubenswrapper[2577]: W0911 17:18:42.699686    2577 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 11 17:18:42.708812 ip-10-0-141-139 kubenswrapper[2577]: W0911 17:18:42.699689    2577 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 11 17:18:42.708812 ip-10-0-141-139 kubenswrapper[2577]: W0911 17:18:42.699692    2577 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 11 17:18:42.708812 ip-10-0-141-139 kubenswrapper[2577]: W0911 17:18:42.699704    2577 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 11 17:18:42.708812 ip-10-0-141-139 kubenswrapper[2577]: W0911 17:18:42.699706    2577 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 11 17:18:42.708812 ip-10-0-141-139 kubenswrapper[2577]: W0911 17:18:42.699709    2577 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 11 17:18:42.708812 ip-10-0-141-139 kubenswrapper[2577]: W0911 17:18:42.699711    2577 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 11 17:18:42.708812 ip-10-0-141-139 kubenswrapper[2577]: W0911 17:18:42.699715    2577 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 11 17:18:42.708812 ip-10-0-141-139 kubenswrapper[2577]: W0911 17:18:42.699718    2577 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 11 17:18:42.708812 ip-10-0-141-139 kubenswrapper[2577]: W0911 17:18:42.699720    2577 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 11 17:18:42.708812 ip-10-0-141-139 kubenswrapper[2577]: W0911 17:18:42.699723    2577 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 11 17:18:42.708812 ip-10-0-141-139 kubenswrapper[2577]: W0911 17:18:42.699725    2577 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 11 17:18:42.709300 ip-10-0-141-139 kubenswrapper[2577]: W0911 17:18:42.699728    2577 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 11 17:18:42.709300 ip-10-0-141-139 kubenswrapper[2577]: W0911 17:18:42.699731    2577 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 11 17:18:42.709300 ip-10-0-141-139 kubenswrapper[2577]: W0911 17:18:42.699734    2577 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 11 17:18:42.709300 ip-10-0-141-139 kubenswrapper[2577]: W0911 17:18:42.699736    2577 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 11 17:18:42.709300 ip-10-0-141-139 kubenswrapper[2577]: W0911 17:18:42.699739    2577 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 11 17:18:42.709300 ip-10-0-141-139 kubenswrapper[2577]: W0911 17:18:42.699742    2577 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 11 17:18:42.709300 ip-10-0-141-139 kubenswrapper[2577]: W0911 17:18:42.699744    2577 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 11 17:18:42.709300 ip-10-0-141-139 kubenswrapper[2577]: W0911 17:18:42.699747    2577 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 11 17:18:42.709300 ip-10-0-141-139 kubenswrapper[2577]: W0911 17:18:42.699750    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 11 17:18:42.709300 ip-10-0-141-139 kubenswrapper[2577]: W0911 17:18:42.699753    2577 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 11 17:18:42.709300 ip-10-0-141-139 kubenswrapper[2577]: W0911 17:18:42.699755    2577 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 11 17:18:42.709300 ip-10-0-141-139 kubenswrapper[2577]: W0911 17:18:42.699758    2577 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 11 17:18:42.709300 ip-10-0-141-139 kubenswrapper[2577]: W0911 17:18:42.699761    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 11 17:18:42.709300 ip-10-0-141-139 kubenswrapper[2577]: W0911 17:18:42.699763    2577 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 11 17:18:42.709300 ip-10-0-141-139 kubenswrapper[2577]: W0911 17:18:42.699766    2577 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 11 17:18:42.709300 ip-10-0-141-139 kubenswrapper[2577]: W0911 17:18:42.699768    2577 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 11 17:18:42.709300 ip-10-0-141-139 kubenswrapper[2577]: W0911 17:18:42.699771    2577 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 11 17:18:42.709300 ip-10-0-141-139 kubenswrapper[2577]: W0911 17:18:42.699773    2577 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 11 17:18:42.709300 ip-10-0-141-139 kubenswrapper[2577]: W0911 17:18:42.699776    2577 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 11 17:18:42.709785 ip-10-0-141-139 kubenswrapper[2577]: W0911 17:18:42.699779    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 11 17:18:42.709785 ip-10-0-141-139 kubenswrapper[2577]: W0911 17:18:42.699781    2577 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 11 17:18:42.709785 ip-10-0-141-139 kubenswrapper[2577]: W0911 17:18:42.699784    2577 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 11 17:18:42.709785 ip-10-0-141-139 kubenswrapper[2577]: W0911 17:18:42.699787    2577 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 11 17:18:42.709785 ip-10-0-141-139 kubenswrapper[2577]: W0911 17:18:42.699791    2577 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 11 17:18:42.709785 ip-10-0-141-139 kubenswrapper[2577]: W0911 17:18:42.699794    2577 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 11 17:18:42.709785 ip-10-0-141-139 kubenswrapper[2577]: W0911 17:18:42.699797    2577 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 11 17:18:42.709785 ip-10-0-141-139 kubenswrapper[2577]: W0911 17:18:42.699799    2577 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 11 17:18:42.709785 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:18:42.700334    2577 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 11 17:18:42.709785 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:18:42.708031    2577 server.go:529] "Kubelet version" kubeletVersion="v1.34.9"
Sep 11 17:18:42.709785 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:18:42.708050    2577 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Sep 11 17:18:42.709785 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:18:42.708813    2577 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Sep 11 17:18:42.709785 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:18:42.708820    2577 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Sep 11 17:18:42.709785 ip-10-0-141-139 kubenswrapper[2577]: W0911 17:18:42.708895    2577 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 11 17:18:42.710161 ip-10-0-141-139 kubenswrapper[2577]: W0911 17:18:42.708901    2577 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 11 17:18:42.710161 ip-10-0-141-139 kubenswrapper[2577]: W0911 17:18:42.708904    2577 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 11 17:18:42.710161 ip-10-0-141-139 kubenswrapper[2577]: W0911 17:18:42.708907    2577 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 11 17:18:42.710161 ip-10-0-141-139 kubenswrapper[2577]: W0911 17:18:42.708910    2577 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 11 17:18:42.710161 ip-10-0-141-139 kubenswrapper[2577]: W0911 17:18:42.708913    2577 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 11 17:18:42.710161 ip-10-0-141-139 kubenswrapper[2577]: W0911 17:18:42.708916    2577 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 11 17:18:42.710161 ip-10-0-141-139 kubenswrapper[2577]: W0911 17:18:42.708918    2577 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 11 17:18:42.710161 ip-10-0-141-139 kubenswrapper[2577]: W0911 17:18:42.708921    2577 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 11 17:18:42.710161 ip-10-0-141-139 kubenswrapper[2577]: W0911 17:18:42.708924    2577 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 11 17:18:42.710161 ip-10-0-141-139 kubenswrapper[2577]: W0911 17:18:42.708927    2577 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 11 17:18:42.710161 ip-10-0-141-139 kubenswrapper[2577]: W0911 17:18:42.708929    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 11 17:18:42.710161 ip-10-0-141-139 kubenswrapper[2577]: W0911 17:18:42.708932    2577 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 11 17:18:42.710161 ip-10-0-141-139 kubenswrapper[2577]: W0911 17:18:42.708934    2577 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 11 17:18:42.710161 ip-10-0-141-139 kubenswrapper[2577]: W0911 17:18:42.708937    2577 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 11 17:18:42.710161 ip-10-0-141-139 kubenswrapper[2577]: W0911 17:18:42.708939    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 11 17:18:42.710161 ip-10-0-141-139 kubenswrapper[2577]: W0911 17:18:42.708942    2577 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 11 17:18:42.710161 ip-10-0-141-139 kubenswrapper[2577]: W0911 17:18:42.708944    2577 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 11 17:18:42.710161 ip-10-0-141-139 kubenswrapper[2577]: W0911 17:18:42.708946    2577 feature_gate.go:328] unrecognized feature gate: Example2
Sep 11 17:18:42.710161 ip-10-0-141-139 kubenswrapper[2577]: W0911 17:18:42.708949    2577 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 11 17:18:42.710161 ip-10-0-141-139 kubenswrapper[2577]: W0911 17:18:42.708952    2577 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 11 17:18:42.710666 ip-10-0-141-139 kubenswrapper[2577]: W0911 17:18:42.708954    2577 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 11 17:18:42.710666 ip-10-0-141-139 kubenswrapper[2577]: W0911 17:18:42.708957    2577 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 11 17:18:42.710666 ip-10-0-141-139 kubenswrapper[2577]: W0911 17:18:42.708959    2577 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 11 17:18:42.710666 ip-10-0-141-139 kubenswrapper[2577]: W0911 17:18:42.708964    2577 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 11 17:18:42.710666 ip-10-0-141-139 kubenswrapper[2577]: W0911 17:18:42.708966    2577 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 11 17:18:42.710666 ip-10-0-141-139 kubenswrapper[2577]: W0911 17:18:42.708969    2577 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 11 17:18:42.710666 ip-10-0-141-139 kubenswrapper[2577]: W0911 17:18:42.708972    2577 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 11 17:18:42.710666 ip-10-0-141-139 kubenswrapper[2577]: W0911 17:18:42.708975    2577 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 11 17:18:42.710666 ip-10-0-141-139 kubenswrapper[2577]: W0911 17:18:42.708978    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 11 17:18:42.710666 ip-10-0-141-139 kubenswrapper[2577]: W0911 17:18:42.708981    2577 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 11 17:18:42.710666 ip-10-0-141-139 kubenswrapper[2577]: W0911 17:18:42.708983    2577 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 11 17:18:42.710666 ip-10-0-141-139 kubenswrapper[2577]: W0911 17:18:42.708986    2577 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 11 17:18:42.710666 ip-10-0-141-139 kubenswrapper[2577]: W0911 17:18:42.708988    2577 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 11 17:18:42.710666 ip-10-0-141-139 kubenswrapper[2577]: W0911 17:18:42.708991    2577 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 11 17:18:42.710666 ip-10-0-141-139 kubenswrapper[2577]: W0911 17:18:42.708994    2577 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 11 17:18:42.710666 ip-10-0-141-139 kubenswrapper[2577]: W0911 17:18:42.708997    2577 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 11 17:18:42.710666 ip-10-0-141-139 kubenswrapper[2577]: W0911 17:18:42.708999    2577 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 11 17:18:42.710666 ip-10-0-141-139 kubenswrapper[2577]: W0911 17:18:42.709002    2577 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 11 17:18:42.710666 ip-10-0-141-139 kubenswrapper[2577]: W0911 17:18:42.709004    2577 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 11 17:18:42.710666 ip-10-0-141-139 kubenswrapper[2577]: W0911 17:18:42.709007    2577 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 11 17:18:42.711155 ip-10-0-141-139 kubenswrapper[2577]: W0911 17:18:42.709009    2577 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 11 17:18:42.711155 ip-10-0-141-139 kubenswrapper[2577]: W0911 17:18:42.709013    2577 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 11 17:18:42.711155 ip-10-0-141-139 kubenswrapper[2577]: W0911 17:18:42.709018    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 11 17:18:42.711155 ip-10-0-141-139 kubenswrapper[2577]: W0911 17:18:42.709021    2577 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 11 17:18:42.711155 ip-10-0-141-139 kubenswrapper[2577]: W0911 17:18:42.709024    2577 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 11 17:18:42.711155 ip-10-0-141-139 kubenswrapper[2577]: W0911 17:18:42.709027    2577 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 11 17:18:42.711155 ip-10-0-141-139 kubenswrapper[2577]: W0911 17:18:42.709030    2577 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 11 17:18:42.711155 ip-10-0-141-139 kubenswrapper[2577]: W0911 17:18:42.709033    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 11 17:18:42.711155 ip-10-0-141-139 kubenswrapper[2577]: W0911 17:18:42.709036    2577 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 11 17:18:42.711155 ip-10-0-141-139 kubenswrapper[2577]: W0911 17:18:42.709038    2577 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 11 17:18:42.711155 ip-10-0-141-139 kubenswrapper[2577]: W0911 17:18:42.709041    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 11 17:18:42.711155 ip-10-0-141-139 kubenswrapper[2577]: W0911 17:18:42.709044    2577 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 11 17:18:42.711155 ip-10-0-141-139 kubenswrapper[2577]: W0911 17:18:42.709046    2577 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 11 17:18:42.711155 ip-10-0-141-139 kubenswrapper[2577]: W0911 17:18:42.709048    2577 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 11 17:18:42.711155 ip-10-0-141-139 kubenswrapper[2577]: W0911 17:18:42.709051    2577 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 11 17:18:42.711155 ip-10-0-141-139 kubenswrapper[2577]: W0911 17:18:42.709054    2577 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 11 17:18:42.711155 ip-10-0-141-139 kubenswrapper[2577]: W0911 17:18:42.709057    2577 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 11 17:18:42.711155 ip-10-0-141-139 kubenswrapper[2577]: W0911 17:18:42.709061    2577 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 11 17:18:42.711155 ip-10-0-141-139 kubenswrapper[2577]: W0911 17:18:42.709065    2577 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 11 17:18:42.711628 ip-10-0-141-139 kubenswrapper[2577]: W0911 17:18:42.709068    2577 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 11 17:18:42.711628 ip-10-0-141-139 kubenswrapper[2577]: W0911 17:18:42.709072    2577 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 11 17:18:42.711628 ip-10-0-141-139 kubenswrapper[2577]: W0911 17:18:42.709077    2577 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 11 17:18:42.711628 ip-10-0-141-139 kubenswrapper[2577]: W0911 17:18:42.709080    2577 feature_gate.go:328] unrecognized feature gate: Example
Sep 11 17:18:42.711628 ip-10-0-141-139 kubenswrapper[2577]: W0911 17:18:42.709082    2577 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 11 17:18:42.711628 ip-10-0-141-139 kubenswrapper[2577]: W0911 17:18:42.709085    2577 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 11 17:18:42.711628 ip-10-0-141-139 kubenswrapper[2577]: W0911 17:18:42.709088    2577 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 11 17:18:42.711628 ip-10-0-141-139 kubenswrapper[2577]: W0911 17:18:42.709090    2577 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 11 17:18:42.711628 ip-10-0-141-139 kubenswrapper[2577]: W0911 17:18:42.709093    2577 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 11 17:18:42.711628 ip-10-0-141-139 kubenswrapper[2577]: W0911 17:18:42.709095    2577 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 11 17:18:42.711628 ip-10-0-141-139 kubenswrapper[2577]: W0911 17:18:42.709098    2577 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 11 17:18:42.711628 ip-10-0-141-139 kubenswrapper[2577]: W0911 17:18:42.709100    2577 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 11 17:18:42.711628 ip-10-0-141-139 kubenswrapper[2577]: W0911 17:18:42.709103    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 11 17:18:42.711628 ip-10-0-141-139 kubenswrapper[2577]: W0911 17:18:42.709106    2577 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 11 17:18:42.711628 ip-10-0-141-139 kubenswrapper[2577]: W0911 17:18:42.709108    2577 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 11 17:18:42.711628 ip-10-0-141-139 kubenswrapper[2577]: W0911 17:18:42.709111    2577 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 11 17:18:42.711628 ip-10-0-141-139 kubenswrapper[2577]: W0911 17:18:42.709114    2577 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 11 17:18:42.711628 ip-10-0-141-139 kubenswrapper[2577]: W0911 17:18:42.709116    2577 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 11 17:18:42.711628 ip-10-0-141-139 kubenswrapper[2577]: W0911 17:18:42.709119    2577 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 11 17:18:42.712114 ip-10-0-141-139 kubenswrapper[2577]: W0911 17:18:42.709122    2577 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 11 17:18:42.712114 ip-10-0-141-139 kubenswrapper[2577]: W0911 17:18:42.709125    2577 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 11 17:18:42.712114 ip-10-0-141-139 kubenswrapper[2577]: W0911 17:18:42.709128    2577 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 11 17:18:42.712114 ip-10-0-141-139 kubenswrapper[2577]: W0911 17:18:42.709131    2577 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 11 17:18:42.712114 ip-10-0-141-139 kubenswrapper[2577]: W0911 17:18:42.709133    2577 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 11 17:18:42.712114 ip-10-0-141-139 kubenswrapper[2577]: W0911 17:18:42.709136    2577 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 11 17:18:42.712114 ip-10-0-141-139 kubenswrapper[2577]: W0911 17:18:42.709138    2577 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 11 17:18:42.712114 ip-10-0-141-139 kubenswrapper[2577]: W0911 17:18:42.709141    2577 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 11 17:18:42.712114 ip-10-0-141-139 kubenswrapper[2577]: W0911 17:18:42.709144    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 11 17:18:42.712114 ip-10-0-141-139 kubenswrapper[2577]: W0911 17:18:42.709146    2577 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 11 17:18:42.712114 ip-10-0-141-139 kubenswrapper[2577]: W0911 17:18:42.709150    2577 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 11 17:18:42.712114 ip-10-0-141-139 kubenswrapper[2577]: W0911 17:18:42.709153    2577 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 11 17:18:42.712114 ip-10-0-141-139 kubenswrapper[2577]: W0911 17:18:42.709156    2577 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 11 17:18:42.712114 ip-10-0-141-139 kubenswrapper[2577]: W0911 17:18:42.709160    2577 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 11 17:18:42.712114 ip-10-0-141-139 kubenswrapper[2577]: W0911 17:18:42.709163    2577 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 11 17:18:42.712114 ip-10-0-141-139 kubenswrapper[2577]: W0911 17:18:42.709165    2577 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 11 17:18:42.712114 ip-10-0-141-139 kubenswrapper[2577]: W0911 17:18:42.709168    2577 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 11 17:18:42.712114 ip-10-0-141-139 kubenswrapper[2577]: W0911 17:18:42.709170    2577 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 11 17:18:42.712114 ip-10-0-141-139 kubenswrapper[2577]: W0911 17:18:42.709173    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 11 17:18:42.712114 ip-10-0-141-139 kubenswrapper[2577]: W0911 17:18:42.709176    2577 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 11 17:18:42.712619 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:18:42.709181    2577 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 11 17:18:42.712619 ip-10-0-141-139 kubenswrapper[2577]: W0911 17:18:42.709322    2577 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 11 17:18:42.712619 ip-10-0-141-139 kubenswrapper[2577]: W0911 17:18:42.709326    2577 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 11 17:18:42.712619 ip-10-0-141-139 kubenswrapper[2577]: W0911 17:18:42.709329    2577 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 11 17:18:42.712619 ip-10-0-141-139 kubenswrapper[2577]: W0911 17:18:42.709332    2577 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 11 17:18:42.712619 ip-10-0-141-139 kubenswrapper[2577]: W0911 17:18:42.709335    2577 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 11 17:18:42.712619 ip-10-0-141-139 kubenswrapper[2577]: W0911 17:18:42.709338    2577 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 11 17:18:42.712619 ip-10-0-141-139 kubenswrapper[2577]: W0911 17:18:42.709341    2577 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 11 17:18:42.712619 ip-10-0-141-139 kubenswrapper[2577]: W0911 17:18:42.709343    2577 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 11 17:18:42.712619 ip-10-0-141-139 kubenswrapper[2577]: W0911 17:18:42.709346    2577 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 11 17:18:42.712619 ip-10-0-141-139 kubenswrapper[2577]: W0911 17:18:42.709348    2577 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 11 17:18:42.712619 ip-10-0-141-139 kubenswrapper[2577]: W0911 17:18:42.709351    2577 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 11 17:18:42.712619 ip-10-0-141-139 kubenswrapper[2577]: W0911 17:18:42.709353    2577 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 11 17:18:42.712619 ip-10-0-141-139 kubenswrapper[2577]: W0911 17:18:42.709356    2577 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 11 17:18:42.712619 ip-10-0-141-139 kubenswrapper[2577]: W0911 17:18:42.709359    2577 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 11 17:18:42.712987 ip-10-0-141-139 kubenswrapper[2577]: W0911 17:18:42.709361    2577 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 11 17:18:42.712987 ip-10-0-141-139 kubenswrapper[2577]: W0911 17:18:42.709364    2577 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 11 17:18:42.712987 ip-10-0-141-139 kubenswrapper[2577]: W0911 17:18:42.709366    2577 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 11 17:18:42.712987 ip-10-0-141-139 kubenswrapper[2577]: W0911 17:18:42.709369    2577 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 11 17:18:42.712987 ip-10-0-141-139 kubenswrapper[2577]: W0911 17:18:42.709372    2577 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 11 17:18:42.712987 ip-10-0-141-139 kubenswrapper[2577]: W0911 17:18:42.709374    2577 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 11 17:18:42.712987 ip-10-0-141-139 kubenswrapper[2577]: W0911 17:18:42.709377    2577 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 11 17:18:42.712987 ip-10-0-141-139 kubenswrapper[2577]: W0911 17:18:42.709379    2577 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 11 17:18:42.712987 ip-10-0-141-139 kubenswrapper[2577]: W0911 17:18:42.709382    2577 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 11 17:18:42.712987 ip-10-0-141-139 kubenswrapper[2577]: W0911 17:18:42.709385    2577 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 11 17:18:42.712987 ip-10-0-141-139 kubenswrapper[2577]: W0911 17:18:42.709388    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 11 17:18:42.712987 ip-10-0-141-139 kubenswrapper[2577]: W0911 17:18:42.709391    2577 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 11 17:18:42.712987 ip-10-0-141-139 kubenswrapper[2577]: W0911 17:18:42.709393    2577 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 11 17:18:42.712987 ip-10-0-141-139 kubenswrapper[2577]: W0911 17:18:42.709396    2577 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 11 17:18:42.712987 ip-10-0-141-139 kubenswrapper[2577]: W0911 17:18:42.709398    2577 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 11 17:18:42.712987 ip-10-0-141-139 kubenswrapper[2577]: W0911 17:18:42.709401    2577 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 11 17:18:42.712987 ip-10-0-141-139 kubenswrapper[2577]: W0911 17:18:42.709404    2577 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 11 17:18:42.712987 ip-10-0-141-139 kubenswrapper[2577]: W0911 17:18:42.709406    2577 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 11 17:18:42.712987 ip-10-0-141-139 kubenswrapper[2577]: W0911 17:18:42.709409    2577 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 11 17:18:42.712987 ip-10-0-141-139 kubenswrapper[2577]: W0911 17:18:42.709411    2577 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 11 17:18:42.713471 ip-10-0-141-139 kubenswrapper[2577]: W0911 17:18:42.709414    2577 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 11 17:18:42.713471 ip-10-0-141-139 kubenswrapper[2577]: W0911 17:18:42.709416    2577 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 11 17:18:42.713471 ip-10-0-141-139 kubenswrapper[2577]: W0911 17:18:42.709419    2577 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 11 17:18:42.713471 ip-10-0-141-139 kubenswrapper[2577]: W0911 17:18:42.709421    2577 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 11 17:18:42.713471 ip-10-0-141-139 kubenswrapper[2577]: W0911 17:18:42.709423    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 11 17:18:42.713471 ip-10-0-141-139 kubenswrapper[2577]: W0911 17:18:42.709426    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 11 17:18:42.713471 ip-10-0-141-139 kubenswrapper[2577]: W0911 17:18:42.709428    2577 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 11 17:18:42.713471 ip-10-0-141-139 kubenswrapper[2577]: W0911 17:18:42.709431    2577 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 11 17:18:42.713471 ip-10-0-141-139 kubenswrapper[2577]: W0911 17:18:42.709433    2577 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 11 17:18:42.713471 ip-10-0-141-139 kubenswrapper[2577]: W0911 17:18:42.709436    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 11 17:18:42.713471 ip-10-0-141-139 kubenswrapper[2577]: W0911 17:18:42.709439    2577 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 11 17:18:42.713471 ip-10-0-141-139 kubenswrapper[2577]: W0911 17:18:42.709441    2577 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 11 17:18:42.713471 ip-10-0-141-139 kubenswrapper[2577]: W0911 17:18:42.709444    2577 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 11 17:18:42.713471 ip-10-0-141-139 kubenswrapper[2577]: W0911 17:18:42.709446    2577 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 11 17:18:42.713471 ip-10-0-141-139 kubenswrapper[2577]: W0911 17:18:42.709449    2577 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 11 17:18:42.713471 ip-10-0-141-139 kubenswrapper[2577]: W0911 17:18:42.709451    2577 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 11 17:18:42.713471 ip-10-0-141-139 kubenswrapper[2577]: W0911 17:18:42.709454    2577 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 11 17:18:42.713471 ip-10-0-141-139 kubenswrapper[2577]: W0911 17:18:42.709456    2577 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 11 17:18:42.713471 ip-10-0-141-139 kubenswrapper[2577]: W0911 17:18:42.709459    2577 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 11 17:18:42.713960 ip-10-0-141-139 kubenswrapper[2577]: W0911 17:18:42.709463    2577 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 11 17:18:42.713960 ip-10-0-141-139 kubenswrapper[2577]: W0911 17:18:42.709466    2577 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 11 17:18:42.713960 ip-10-0-141-139 kubenswrapper[2577]: W0911 17:18:42.709470    2577 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 11 17:18:42.713960 ip-10-0-141-139 kubenswrapper[2577]: W0911 17:18:42.709474    2577 feature_gate.go:328] unrecognized feature gate: Example
Sep 11 17:18:42.713960 ip-10-0-141-139 kubenswrapper[2577]: W0911 17:18:42.709477    2577 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 11 17:18:42.713960 ip-10-0-141-139 kubenswrapper[2577]: W0911 17:18:42.709480    2577 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 11 17:18:42.713960 ip-10-0-141-139 kubenswrapper[2577]: W0911 17:18:42.709482    2577 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 11 17:18:42.713960 ip-10-0-141-139 kubenswrapper[2577]: W0911 17:18:42.709485    2577 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 11 17:18:42.713960 ip-10-0-141-139 kubenswrapper[2577]: W0911 17:18:42.709488    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 11 17:18:42.713960 ip-10-0-141-139 kubenswrapper[2577]: W0911 17:18:42.709491    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 11 17:18:42.713960 ip-10-0-141-139 kubenswrapper[2577]: W0911 17:18:42.709494    2577 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 11 17:18:42.713960 ip-10-0-141-139 kubenswrapper[2577]: W0911 17:18:42.709497    2577 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 11 17:18:42.713960 ip-10-0-141-139 kubenswrapper[2577]: W0911 17:18:42.709517    2577 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 11 17:18:42.713960 ip-10-0-141-139 kubenswrapper[2577]: W0911 17:18:42.709520    2577 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 11 17:18:42.713960 ip-10-0-141-139 kubenswrapper[2577]: W0911 17:18:42.709523    2577 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 11 17:18:42.713960 ip-10-0-141-139 kubenswrapper[2577]: W0911 17:18:42.709525    2577 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 11 17:18:42.713960 ip-10-0-141-139 kubenswrapper[2577]: W0911 17:18:42.709528    2577 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 11 17:18:42.713960 ip-10-0-141-139 kubenswrapper[2577]: W0911 17:18:42.709531    2577 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 11 17:18:42.713960 ip-10-0-141-139 kubenswrapper[2577]: W0911 17:18:42.709534    2577 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 11 17:18:42.714445 ip-10-0-141-139 kubenswrapper[2577]: W0911 17:18:42.709537    2577 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 11 17:18:42.714445 ip-10-0-141-139 kubenswrapper[2577]: W0911 17:18:42.709540    2577 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 11 17:18:42.714445 ip-10-0-141-139 kubenswrapper[2577]: W0911 17:18:42.709543    2577 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 11 17:18:42.714445 ip-10-0-141-139 kubenswrapper[2577]: W0911 17:18:42.709546    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 11 17:18:42.714445 ip-10-0-141-139 kubenswrapper[2577]: W0911 17:18:42.709549    2577 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 11 17:18:42.714445 ip-10-0-141-139 kubenswrapper[2577]: W0911 17:18:42.709552    2577 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 11 17:18:42.714445 ip-10-0-141-139 kubenswrapper[2577]: W0911 17:18:42.709555    2577 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 11 17:18:42.714445 ip-10-0-141-139 kubenswrapper[2577]: W0911 17:18:42.709557    2577 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 11 17:18:42.714445 ip-10-0-141-139 kubenswrapper[2577]: W0911 17:18:42.709560    2577 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 11 17:18:42.714445 ip-10-0-141-139 kubenswrapper[2577]: W0911 17:18:42.709563    2577 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 11 17:18:42.714445 ip-10-0-141-139 kubenswrapper[2577]: W0911 17:18:42.709565    2577 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 11 17:18:42.714445 ip-10-0-141-139 kubenswrapper[2577]: W0911 17:18:42.709568    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 11 17:18:42.714445 ip-10-0-141-139 kubenswrapper[2577]: W0911 17:18:42.709570    2577 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 11 17:18:42.714445 ip-10-0-141-139 kubenswrapper[2577]: W0911 17:18:42.709572    2577 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 11 17:18:42.714445 ip-10-0-141-139 kubenswrapper[2577]: W0911 17:18:42.709575    2577 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 11 17:18:42.714445 ip-10-0-141-139 kubenswrapper[2577]: W0911 17:18:42.709578    2577 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 11 17:18:42.714445 ip-10-0-141-139 kubenswrapper[2577]: W0911 17:18:42.709580    2577 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 11 17:18:42.714445 ip-10-0-141-139 kubenswrapper[2577]: W0911 17:18:42.709584    2577 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 11 17:18:42.714445 ip-10-0-141-139 kubenswrapper[2577]: W0911 17:18:42.709587    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 11 17:18:42.714445 ip-10-0-141-139 kubenswrapper[2577]: W0911 17:18:42.709589    2577 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 11 17:18:42.715020 ip-10-0-141-139 kubenswrapper[2577]: W0911 17:18:42.709592    2577 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 11 17:18:42.715020 ip-10-0-141-139 kubenswrapper[2577]: W0911 17:18:42.709594    2577 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 11 17:18:42.715020 ip-10-0-141-139 kubenswrapper[2577]: W0911 17:18:42.709596    2577 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 11 17:18:42.715020 ip-10-0-141-139 kubenswrapper[2577]: W0911 17:18:42.709599    2577 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 11 17:18:42.715020 ip-10-0-141-139 kubenswrapper[2577]: W0911 17:18:42.709601    2577 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 11 17:18:42.715020 ip-10-0-141-139 kubenswrapper[2577]: W0911 17:18:42.709604    2577 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 11 17:18:42.715020 ip-10-0-141-139 kubenswrapper[2577]: W0911 17:18:42.709606    2577 feature_gate.go:328] unrecognized feature gate: Example2
Sep 11 17:18:42.715020 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:18:42.709611    2577 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 11 17:18:42.715020 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:18:42.709739    2577 server.go:962] "Client rotation is on, will bootstrap in background"
Sep 11 17:18:42.715020 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:18:42.711880    2577 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Sep 11 17:18:42.715020 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:18:42.712826    2577 server.go:1019] "Starting client certificate rotation"
Sep 11 17:18:42.715020 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:18:42.712944    2577 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 11 17:18:42.715020 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:18:42.712991    2577 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 11 17:18:42.715350 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:18:42.715335    2577 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Sep 11 17:18:42.718913 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:18:42.718892    2577 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Sep 11 17:18:42.737631 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:18:42.737607    2577 log.go:25] "Validated CRI v1 runtime API"
Sep 11 17:18:42.741568 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:18:42.741553    2577 log.go:25] "Validated CRI v1 image API"
Sep 11 17:18:42.742456 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:18:42.742432    2577 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Sep 11 17:18:42.745856 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:18:42.745840    2577 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Sep 11 17:18:42.749593 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:18:42.749573    2577 fs.go:135] Filesystem UUIDs: map[0d8bc2e7-04ba-4a8a-acb8-e2f5a82c3e9d:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2 b113fdb2-1f7a-47ff-a62a-76616ce487b2:/dev/nvme0n1p4]
Sep 11 17:18:42.749638 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:18:42.749593    2577 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}]
Sep 11 17:18:42.754767 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:18:42.754658    2577 manager.go:216] Machine: {Timestamp:2026-09-11 17:18:42.754156422 +0000 UTC m=+0.327376525 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3095321 MemoryCapacity:32812146688 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec21d67f4b69ac08aeee0f325a3f0d8e SystemUUID:ec21d67f-4b69-ac08-aeee-0f325a3f0d8e BootID:0b4b4dc4-327d-4f66-a3f6-a1a04f0a8eb6 Filesystems:[{Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16406073344 Type:vfs Inodes:4005389 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6562430976 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16406073344 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:54:b3:ba:98:a7 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:54:b3:ba:98:a7 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:06:f5:72:29:dc:0e Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:32812146688 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:34603008 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Sep 11 17:18:42.754767 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:18:42.754762    2577 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Sep 11 17:18:42.754870 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:18:42.754844    2577 manager.go:232] Version: {KernelVersion:5.14.0-570.140.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260908-1 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Sep 11 17:18:42.756375 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:18:42.756342    2577 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Sep 11 17:18:42.756542 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:18:42.756378    2577 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-141-139.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Sep 11 17:18:42.757034 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:18:42.757022    2577 topology_manager.go:138] "Creating topology manager with none policy"
Sep 11 17:18:42.757075 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:18:42.757035    2577 container_manager_linux.go:309] "Creating device plugin manager"
Sep 11 17:18:42.757075 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:18:42.757042    2577 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 11 17:18:42.757635 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:18:42.757624    2577 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 11 17:18:42.758484 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:18:42.758472    2577 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager"
Sep 11 17:18:42.762962 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:18:42.762945    2577 state_mem.go:36] "Initialized new in-memory state store"
Sep 11 17:18:42.763097 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:18:42.763087    2577 server.go:1240] "Using root directory" path="/var/lib/kubelet"
Sep 11 17:18:42.763136 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:18:42.763126    2577 kubelet.go:486] "Attempting to sync node with API server"
Sep 11 17:18:42.763171 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:18:42.763142    2577 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests"
Sep 11 17:18:42.763204 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:18:42.763198    2577 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Sep 11 17:18:42.763234 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:18:42.763214    2577 kubelet.go:398] "Adding apiserver pod source"
Sep 11 17:18:42.763265 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:18:42.763236    2577 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Sep 11 17:18:42.763784 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:18:42.763768    2577 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Sep 11 17:18:42.763837 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:18:42.763793    2577 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Sep 11 17:18:42.766156 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:18:42.766140    2577 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.12-5.rhaos4.21.gitd1ec63c.el9" apiVersion="v1"
Sep 11 17:18:42.768760 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:18:42.768746    2577 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Sep 11 17:18:42.768814 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:18:42.768777    2577 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Sep 11 17:18:42.770680 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:18:42.770667    2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Sep 11 17:18:42.770712 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:18:42.770688    2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Sep 11 17:18:42.770712 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:18:42.770694    2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Sep 11 17:18:42.770712 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:18:42.770700    2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Sep 11 17:18:42.770712 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:18:42.770710    2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Sep 11 17:18:42.770821 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:18:42.770716    2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Sep 11 17:18:42.770821 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:18:42.770730    2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Sep 11 17:18:42.770821 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:18:42.770735    2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Sep 11 17:18:42.770821 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:18:42.770741    2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Sep 11 17:18:42.770821 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:18:42.770746    2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Sep 11 17:18:42.770821 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:18:42.770759    2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Sep 11 17:18:42.770821 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:18:42.770765    2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Sep 11 17:18:42.771321 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:18:42.771312    2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Sep 11 17:18:42.771354 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:18:42.771322    2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Sep 11 17:18:42.775381 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:18:42.775362    2577 server.go:1268] "Started kubelet"
Sep 11 17:18:42.775483 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:18:42.775426    2577 server.go:180] "Starting to listen" address="0.0.0.0" port=10250
Sep 11 17:18:42.775602 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:18:42.775495    2577 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Sep 11 17:18:42.775670 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:18:42.775655    2577 server_v1.go:49] "podresources" method="list" useActivePods=true
Sep 11 17:18:42.776327 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:18:42.776071    2577 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-141-139.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 11 17:18:42.776327 ip-10-0-141-139 kubenswrapper[2577]: E0911 17:18:42.776077    2577 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
Sep 11 17:18:42.776327 ip-10-0-141-139 kubenswrapper[2577]: E0911 17:18:42.776175    2577 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-141-139.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
Sep 11 17:18:42.776537 ip-10-0-141-139 systemd[1]: Started Kubernetes Kubelet.
Sep 11 17:18:42.776883 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:18:42.776850    2577 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Sep 11 17:18:42.782854 ip-10-0-141-139 kubenswrapper[2577]: E0911 17:18:42.782827    2577 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Sep 11 17:18:42.783394 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:18:42.783376    2577 server.go:310] "Adding debug handlers to kubelet server"
Sep 11 17:18:42.785324 ip-10-0-141-139 kubenswrapper[2577]: E0911 17:18:42.784436    2577 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-141-139.ec2.internal.18d453d42fef8a77  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-141-139.ec2.internal,UID:ip-10-0-141-139.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-141-139.ec2.internal,},FirstTimestamp:2026-09-11 17:18:42.775329399 +0000 UTC m=+0.348549505,LastTimestamp:2026-09-11 17:18:42.775329399 +0000 UTC m=+0.348549505,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-141-139.ec2.internal,}"
Sep 11 17:18:42.787155 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:18:42.787128    2577 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Sep 11 17:18:42.788371 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:18:42.788351    2577 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
Sep 11 17:18:42.788532 ip-10-0-141-139 kubenswrapper[2577]: E0911 17:18:42.788467    2577 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-139.ec2.internal\" not found"
Sep 11 17:18:42.788903 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:18:42.788888    2577 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Sep 11 17:18:42.788967 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:18:42.788890    2577 volume_manager.go:311] "The desired_state_of_world populator starts"
Sep 11 17:18:42.788967 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:18:42.788920    2577 volume_manager.go:313] "Starting Kubelet Volume Manager"
Sep 11 17:18:42.789067 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:18:42.789004    2577 reconstruct.go:97] "Volume reconstruction finished"
Sep 11 17:18:42.789067 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:18:42.789014    2577 reconciler.go:29] "Reconciler: start to sync state"
Sep 11 17:18:42.789273 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:18:42.789252    2577 factory.go:55] Registering systemd factory
Sep 11 17:18:42.789352 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:18:42.789276    2577 factory.go:223] Registration of the systemd container factory successfully
Sep 11 17:18:42.789567 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:18:42.789551    2577 factory.go:153] Registering CRI-O factory
Sep 11 17:18:42.789662 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:18:42.789571    2577 factory.go:223] Registration of the crio container factory successfully
Sep 11 17:18:42.789662 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:18:42.789624    2577 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Sep 11 17:18:42.789662 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:18:42.789640    2577 factory.go:103] Registering Raw factory
Sep 11 17:18:42.789662 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:18:42.789650    2577 manager.go:1196] Started watching for new ooms in manager
Sep 11 17:18:42.789990 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:18:42.789980    2577 manager.go:349] Starting recovery of all containers
Sep 11 17:18:42.791854 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:18:42.791819    2577 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-987b5"
Sep 11 17:18:42.798486 ip-10-0-141-139 kubenswrapper[2577]: E0911 17:18:42.798455    2577 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-141-139.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
Sep 11 17:18:42.798752 ip-10-0-141-139 kubenswrapper[2577]: E0911 17:18:42.798710    2577 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver"
Sep 11 17:18:42.799924 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:18:42.799908    2577 manager.go:354] Recovery completed
Sep 11 17:18:42.803581 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:18:42.803564    2577 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-987b5"
Sep 11 17:18:42.804030 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:18:42.804018    2577 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 11 17:18:42.806116 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:18:42.806099    2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-139.ec2.internal" event="NodeHasSufficientMemory"
Sep 11 17:18:42.806184 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:18:42.806131    2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-139.ec2.internal" event="NodeHasNoDiskPressure"
Sep 11 17:18:42.806184 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:18:42.806143    2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-139.ec2.internal" event="NodeHasSufficientPID"
Sep 11 17:18:42.806728 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:18:42.806715    2577 cpu_manager.go:222] "Starting CPU manager" policy="none"
Sep 11 17:18:42.806728 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:18:42.806726    2577 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s"
Sep 11 17:18:42.806873 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:18:42.806741    2577 state_mem.go:36] "Initialized new in-memory state store"
Sep 11 17:18:42.806873 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:18:42.806770    2577 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 11 17:18:42.807636 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:18:42.807620    2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-139.ec2.internal" event="NodeHasSufficientMemory"
Sep 11 17:18:42.807709 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:18:42.807653    2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-139.ec2.internal" event="NodeHasNoDiskPressure"
Sep 11 17:18:42.807709 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:18:42.807667    2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-139.ec2.internal" event="NodeHasSufficientPID"
Sep 11 17:18:42.808191 ip-10-0-141-139 kubenswrapper[2577]: E0911 17:18:42.808095    2577 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-141-139.ec2.internal.18d453d431c54d5c  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-141-139.ec2.internal,UID:ip-10-0-141-139.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-141-139.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-141-139.ec2.internal,},FirstTimestamp:2026-09-11 17:18:42.806115676 +0000 UTC m=+0.379335780,LastTimestamp:2026-09-11 17:18:42.806115676 +0000 UTC m=+0.379335780,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-141-139.ec2.internal,}"
Sep 11 17:18:42.808859 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:18:42.808841    2577 policy_none.go:49] "None policy: Start"
Sep 11 17:18:42.808859 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:18:42.808858    2577 memory_manager.go:187] "Starting memorymanager" policy="None"
Sep 11 17:18:42.808945 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:18:42.808867    2577 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Sep 11 17:18:42.810824 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:18:42.810813    2577 policy_none.go:47] "Start"
Sep 11 17:18:42.863651 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:18:42.846102    2577 manager.go:337] "Starting Device Plugin manager"
Sep 11 17:18:42.863651 ip-10-0-141-139 kubenswrapper[2577]: E0911 17:18:42.846146    2577 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Sep 11 17:18:42.863651 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:18:42.846156    2577 server.go:87] "Starting device plugin registration server"
Sep 11 17:18:42.863651 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:18:42.846465    2577 eviction_manager.go:189] "Eviction manager: starting control loop"
Sep 11 17:18:42.863651 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:18:42.846477    2577 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Sep 11 17:18:42.863651 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:18:42.846590    2577 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Sep 11 17:18:42.863651 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:18:42.846673    2577 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts"
Sep 11 17:18:42.863651 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:18:42.846683    2577 plugin_manager.go:118] "Starting Kubelet Plugin Manager"
Sep 11 17:18:42.863651 ip-10-0-141-139 kubenswrapper[2577]: E0911 17:18:42.847182    2577 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Sep 11 17:18:42.863651 ip-10-0-141-139 kubenswrapper[2577]: E0911 17:18:42.847232    2577 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-141-139.ec2.internal\" not found"
Sep 11 17:18:42.918142 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:18:42.918090    2577 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Sep 11 17:18:42.919492 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:18:42.919470    2577 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Sep 11 17:18:42.919640 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:18:42.919546    2577 status_manager.go:244] "Starting to sync pod status with apiserver"
Sep 11 17:18:42.919640 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:18:42.919580    2577 kubelet.go:2448] "Starting kubelet main sync loop"
Sep 11 17:18:42.919640 ip-10-0-141-139 kubenswrapper[2577]: E0911 17:18:42.919622    2577 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Sep 11 17:18:42.923077 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:18:42.923013    2577 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 11 17:18:42.947541 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:18:42.947489    2577 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 11 17:18:42.948593 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:18:42.948575    2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-139.ec2.internal" event="NodeHasSufficientMemory"
Sep 11 17:18:42.948696 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:18:42.948606    2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-139.ec2.internal" event="NodeHasNoDiskPressure"
Sep 11 17:18:42.948696 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:18:42.948617    2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-139.ec2.internal" event="NodeHasSufficientPID"
Sep 11 17:18:42.948696 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:18:42.948639    2577 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-141-139.ec2.internal"
Sep 11 17:18:42.962059 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:18:42.962034    2577 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-141-139.ec2.internal"
Sep 11 17:18:42.962201 ip-10-0-141-139 kubenswrapper[2577]: E0911 17:18:42.962066    2577 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-141-139.ec2.internal\": node \"ip-10-0-141-139.ec2.internal\" not found"
Sep 11 17:18:42.998117 ip-10-0-141-139 kubenswrapper[2577]: E0911 17:18:42.998084    2577 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-139.ec2.internal\" not found"
Sep 11 17:18:43.020483 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:18:43.020453    2577 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["kube-system/kube-apiserver-proxy-ip-10-0-141-139.ec2.internal","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-139.ec2.internal"]
Sep 11 17:18:43.020631 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:18:43.020555    2577 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 11 17:18:43.021585 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:18:43.021569    2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-139.ec2.internal" event="NodeHasSufficientMemory"
Sep 11 17:18:43.021673 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:18:43.021598    2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-139.ec2.internal" event="NodeHasNoDiskPressure"
Sep 11 17:18:43.021673 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:18:43.021607    2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-139.ec2.internal" event="NodeHasSufficientPID"
Sep 11 17:18:43.023535 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:18:43.023524    2577 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 11 17:18:43.023679 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:18:43.023662    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-139.ec2.internal"
Sep 11 17:18:43.023716 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:18:43.023695    2577 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 11 17:18:43.024310 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:18:43.024293    2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-139.ec2.internal" event="NodeHasSufficientMemory"
Sep 11 17:18:43.024377 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:18:43.024320    2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-139.ec2.internal" event="NodeHasNoDiskPressure"
Sep 11 17:18:43.024377 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:18:43.024331    2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-139.ec2.internal" event="NodeHasSufficientPID"
Sep 11 17:18:43.024377 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:18:43.024348    2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-139.ec2.internal" event="NodeHasSufficientMemory"
Sep 11 17:18:43.024377 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:18:43.024367    2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-139.ec2.internal" event="NodeHasNoDiskPressure"
Sep 11 17:18:43.024522 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:18:43.024380    2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-139.ec2.internal" event="NodeHasSufficientPID"
Sep 11 17:18:43.026235 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:18:43.026219    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-139.ec2.internal"
Sep 11 17:18:43.026290 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:18:43.026250    2577 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 11 17:18:43.026957 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:18:43.026943    2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-139.ec2.internal" event="NodeHasSufficientMemory"
Sep 11 17:18:43.027023 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:18:43.026969    2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-139.ec2.internal" event="NodeHasNoDiskPressure"
Sep 11 17:18:43.027023 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:18:43.026979    2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-139.ec2.internal" event="NodeHasSufficientPID"
Sep 11 17:18:43.050345 ip-10-0-141-139 kubenswrapper[2577]: E0911 17:18:43.050322    2577 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-141-139.ec2.internal\" not found" node="ip-10-0-141-139.ec2.internal"
Sep 11 17:18:43.054757 ip-10-0-141-139 kubenswrapper[2577]: E0911 17:18:43.054740    2577 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-141-139.ec2.internal\" not found" node="ip-10-0-141-139.ec2.internal"
Sep 11 17:18:43.098619 ip-10-0-141-139 kubenswrapper[2577]: E0911 17:18:43.098591    2577 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-139.ec2.internal\" not found"
Sep 11 17:18:43.190316 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:18:43.190259    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/8b76f1b2a16574ce38927f7f5da97679-config\") pod \"kube-apiserver-proxy-ip-10-0-141-139.ec2.internal\" (UID: \"8b76f1b2a16574ce38927f7f5da97679\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-141-139.ec2.internal"
Sep 11 17:18:43.190316 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:18:43.190293    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/628f3dc115ebc1acc6ed7da71915e638-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-141-139.ec2.internal\" (UID: \"628f3dc115ebc1acc6ed7da71915e638\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-139.ec2.internal"
Sep 11 17:18:43.190316 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:18:43.190310    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/628f3dc115ebc1acc6ed7da71915e638-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-141-139.ec2.internal\" (UID: \"628f3dc115ebc1acc6ed7da71915e638\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-139.ec2.internal"
Sep 11 17:18:43.199333 ip-10-0-141-139 kubenswrapper[2577]: E0911 17:18:43.199309    2577 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-139.ec2.internal\" not found"
Sep 11 17:18:43.291026 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:18:43.290990    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/8b76f1b2a16574ce38927f7f5da97679-config\") pod \"kube-apiserver-proxy-ip-10-0-141-139.ec2.internal\" (UID: \"8b76f1b2a16574ce38927f7f5da97679\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-141-139.ec2.internal"
Sep 11 17:18:43.291088 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:18:43.291055    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/628f3dc115ebc1acc6ed7da71915e638-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-141-139.ec2.internal\" (UID: \"628f3dc115ebc1acc6ed7da71915e638\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-139.ec2.internal"
Sep 11 17:18:43.291121 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:18:43.291094    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/628f3dc115ebc1acc6ed7da71915e638-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-141-139.ec2.internal\" (UID: \"628f3dc115ebc1acc6ed7da71915e638\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-139.ec2.internal"
Sep 11 17:18:43.291163 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:18:43.291155    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/628f3dc115ebc1acc6ed7da71915e638-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-141-139.ec2.internal\" (UID: \"628f3dc115ebc1acc6ed7da71915e638\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-139.ec2.internal"
Sep 11 17:18:43.291197 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:18:43.291158    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/8b76f1b2a16574ce38927f7f5da97679-config\") pod \"kube-apiserver-proxy-ip-10-0-141-139.ec2.internal\" (UID: \"8b76f1b2a16574ce38927f7f5da97679\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-141-139.ec2.internal"
Sep 11 17:18:43.291197 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:18:43.291163    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/628f3dc115ebc1acc6ed7da71915e638-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-141-139.ec2.internal\" (UID: \"628f3dc115ebc1acc6ed7da71915e638\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-139.ec2.internal"
Sep 11 17:18:43.299817 ip-10-0-141-139 kubenswrapper[2577]: E0911 17:18:43.299793    2577 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-139.ec2.internal\" not found"
Sep 11 17:18:43.352971 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:18:43.352943    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-139.ec2.internal"
Sep 11 17:18:43.357809 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:18:43.357354    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-139.ec2.internal"
Sep 11 17:18:43.401003 ip-10-0-141-139 kubenswrapper[2577]: E0911 17:18:43.400827    2577 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-139.ec2.internal\" not found"
Sep 11 17:18:43.501424 ip-10-0-141-139 kubenswrapper[2577]: E0911 17:18:43.501337    2577 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-139.ec2.internal\" not found"
Sep 11 17:18:43.601888 ip-10-0-141-139 kubenswrapper[2577]: E0911 17:18:43.601857    2577 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-139.ec2.internal\" not found"
Sep 11 17:18:43.621074 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:18:43.621052    2577 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 11 17:18:43.672491 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:18:43.672463    2577 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 11 17:18:43.702779 ip-10-0-141-139 kubenswrapper[2577]: E0911 17:18:43.702748    2577 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-139.ec2.internal\" not found"
Sep 11 17:18:43.712849 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:18:43.712821    2577 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Sep 11 17:18:43.712990 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:18:43.712974    2577 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Sep 11 17:18:43.713033 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:18:43.713002    2577 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Sep 11 17:18:43.713033 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:18:43.713016    2577 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Sep 11 17:18:43.745427 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:18:43.745399    2577 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 11 17:18:43.763423 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:18:43.763362    2577 apiserver.go:52] "Watching apiserver"
Sep 11 17:18:43.771908 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:18:43.771887    2577 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Sep 11 17:18:43.773327 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:18:43.773302    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-multus/multus-nltl2","openshift-multus/network-metrics-daemon-bl4vr","openshift-network-diagnostics/network-check-target-xfn47","openshift-ovn-kubernetes/ovnkube-node-sn5kw","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hxl5l","openshift-dns/node-resolver-9v9dt","openshift-multus/multus-additional-cni-plugins-6shpl","openshift-network-operator/iptables-alerter-vmxwz","kube-system/global-pull-secret-syncer-fsh8r","kube-system/konnectivity-agent-g6gcb","openshift-cluster-node-tuning-operator/tuned-d4gm9","openshift-image-registry/node-ca-k5j4d"]
Sep 11 17:18:43.775871 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:18:43.775841    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-nltl2"
Sep 11 17:18:43.778163 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:18:43.778145    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-sn5kw"
Sep 11 17:18:43.778559 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:18:43.778538    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Sep 11 17:18:43.778731 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:18:43.778714    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Sep 11 17:18:43.779060 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:18:43.779045    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Sep 11 17:18:43.779109 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:18:43.779042    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Sep 11 17:18:43.779714 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:18:43.779651    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-xm9gh\""
Sep 11 17:18:43.780778 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:18:43.780759    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Sep 11 17:18:43.780939 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:18:43.780917    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hxl5l"
Sep 11 17:18:43.782226 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:18:43.782202    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Sep 11 17:18:43.782678 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:18:43.782648    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-f42nh\""
Sep 11 17:18:43.782789 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:18:43.782773    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Sep 11 17:18:43.782935 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:18:43.782919    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Sep 11 17:18:43.783046 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:18:43.783032    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Sep 11 17:18:43.783152 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:18:43.783138    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Sep 11 17:18:43.783334 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:18:43.783316    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Sep 11 17:18:43.784045 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:18:43.784024    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-9v9dt"
Sep 11 17:18:43.784140 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:18:43.784043    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Sep 11 17:18:43.784140 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:18:43.784086    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Sep 11 17:18:43.784140 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:18:43.784093    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-jcpk2\""
Sep 11 17:18:43.786439 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:18:43.786423    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-6shpl"
Sep 11 17:18:43.786757 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:18:43.786691    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Sep 11 17:18:43.786956 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:18:43.786942    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-bjlc9\""
Sep 11 17:18:43.787006 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:18:43.786959    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Sep 11 17:18:43.787927 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:18:43.787909    2577 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Sep 11 17:18:43.788744 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:18:43.788729    2577 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-139.ec2.internal"
Sep 11 17:18:43.788880 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:18:43.788864    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-vmxwz"
Sep 11 17:18:43.790937 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:18:43.790919    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-bp2mg\""
Sep 11 17:18:43.791263 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:18:43.791242    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-g6gcb"
Sep 11 17:18:43.791356 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:18:43.791342    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Sep 11 17:18:43.791673 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:18:43.791657    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Sep 11 17:18:43.791881 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:18:43.791864    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Sep 11 17:18:43.791935 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:18:43.791892    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Sep 11 17:18:43.791988 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:18:43.791973    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Sep 11 17:18:43.792537 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:18:43.792523    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-km2wv\""
Sep 11 17:18:43.793460 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:18:43.793425    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-qtt2x\""
Sep 11 17:18:43.793598 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:18:43.793524    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Sep 11 17:18:43.793790 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:18:43.793768    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f8tfp\" (UniqueName: \"kubernetes.io/projected/9208373a-bbca-49fc-91ed-d29c0a833261-kube-api-access-f8tfp\") pod \"multus-nltl2\" (UID: \"9208373a-bbca-49fc-91ed-d29c0a833261\") " pod="openshift-multus/multus-nltl2"
Sep 11 17:18:43.793855 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:18:43.793808    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/bb06c328-82b1-4690-aa4a-6584fb149952-host-slash\") pod \"ovnkube-node-sn5kw\" (UID: \"bb06c328-82b1-4690-aa4a-6584fb149952\") " pod="openshift-ovn-kubernetes/ovnkube-node-sn5kw"
Sep 11 17:18:43.793855 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:18:43.793833    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/bb06c328-82b1-4690-aa4a-6584fb149952-run-openvswitch\") pod \"ovnkube-node-sn5kw\" (UID: \"bb06c328-82b1-4690-aa4a-6584fb149952\") " pod="openshift-ovn-kubernetes/ovnkube-node-sn5kw"
Sep 11 17:18:43.793932 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:18:43.793857    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/bb06c328-82b1-4690-aa4a-6584fb149952-host-run-ovn-kubernetes\") pod \"ovnkube-node-sn5kw\" (UID: \"bb06c328-82b1-4690-aa4a-6584fb149952\") " pod="openshift-ovn-kubernetes/ovnkube-node-sn5kw"
Sep 11 17:18:43.793932 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:18:43.793885    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/daeb9a26-f9e6-4d68-bc7d-f31b5f10d621-kubelet-dir\") pod \"aws-ebs-csi-driver-node-hxl5l\" (UID: \"daeb9a26-f9e6-4d68-bc7d-f31b5f10d621\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hxl5l"
Sep 11 17:18:43.793932 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:18:43.793909    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/daeb9a26-f9e6-4d68-bc7d-f31b5f10d621-sys-fs\") pod \"aws-ebs-csi-driver-node-hxl5l\" (UID: \"daeb9a26-f9e6-4d68-bc7d-f31b5f10d621\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hxl5l"
Sep 11 17:18:43.794140 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:18:43.793944    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/bb06c328-82b1-4690-aa4a-6584fb149952-var-lib-openvswitch\") pod \"ovnkube-node-sn5kw\" (UID: \"bb06c328-82b1-4690-aa4a-6584fb149952\") " pod="openshift-ovn-kubernetes/ovnkube-node-sn5kw"
Sep 11 17:18:43.794140 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:18:43.793966    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/bb06c328-82b1-4690-aa4a-6584fb149952-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-sn5kw\" (UID: \"bb06c328-82b1-4690-aa4a-6584fb149952\") " pod="openshift-ovn-kubernetes/ovnkube-node-sn5kw"
Sep 11 17:18:43.794140 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:18:43.793989    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/bb06c328-82b1-4690-aa4a-6584fb149952-host-run-netns\") pod \"ovnkube-node-sn5kw\" (UID: \"bb06c328-82b1-4690-aa4a-6584fb149952\") " pod="openshift-ovn-kubernetes/ovnkube-node-sn5kw"
Sep 11 17:18:43.794140 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:18:43.794008    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/daeb9a26-f9e6-4d68-bc7d-f31b5f10d621-registration-dir\") pod \"aws-ebs-csi-driver-node-hxl5l\" (UID: \"daeb9a26-f9e6-4d68-bc7d-f31b5f10d621\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hxl5l"
Sep 11 17:18:43.794140 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:18:43.794040    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/bb06c328-82b1-4690-aa4a-6584fb149952-ovnkube-config\") pod \"ovnkube-node-sn5kw\" (UID: \"bb06c328-82b1-4690-aa4a-6584fb149952\") " pod="openshift-ovn-kubernetes/ovnkube-node-sn5kw"
Sep 11 17:18:43.794140 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:18:43.794061    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/daeb9a26-f9e6-4d68-bc7d-f31b5f10d621-socket-dir\") pod \"aws-ebs-csi-driver-node-hxl5l\" (UID: \"daeb9a26-f9e6-4d68-bc7d-f31b5f10d621\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hxl5l"
Sep 11 17:18:43.794140 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:18:43.794102    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nbw97\" (UniqueName: \"kubernetes.io/projected/66d2b8f8-a4f7-4811-8034-2523351499ee-kube-api-access-nbw97\") pod \"node-resolver-9v9dt\" (UID: \"66d2b8f8-a4f7-4811-8034-2523351499ee\") " pod="openshift-dns/node-resolver-9v9dt"
Sep 11 17:18:43.794140 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:18:43.794126    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/9208373a-bbca-49fc-91ed-d29c0a833261-multus-socket-dir-parent\") pod \"multus-nltl2\" (UID: \"9208373a-bbca-49fc-91ed-d29c0a833261\") " pod="openshift-multus/multus-nltl2"
Sep 11 17:18:43.794563 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:18:43.794149    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/9208373a-bbca-49fc-91ed-d29c0a833261-host-run-multus-certs\") pod \"multus-nltl2\" (UID: \"9208373a-bbca-49fc-91ed-d29c0a833261\") " pod="openshift-multus/multus-nltl2"
Sep 11 17:18:43.794563 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:18:43.794157    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Sep 11 17:18:43.794563 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:18:43.794172    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/337b4304-78e3-4abd-afbf-b866f7104056-agent-certs\") pod \"konnectivity-agent-g6gcb\" (UID: \"337b4304-78e3-4abd-afbf-b866f7104056\") " pod="kube-system/konnectivity-agent-g6gcb"
Sep 11 17:18:43.794563 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:18:43.794196    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/bb06c328-82b1-4690-aa4a-6584fb149952-etc-openvswitch\") pod \"ovnkube-node-sn5kw\" (UID: \"bb06c328-82b1-4690-aa4a-6584fb149952\") " pod="openshift-ovn-kubernetes/ovnkube-node-sn5kw"
Sep 11 17:18:43.794563 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:18:43.794219    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/bb06c328-82b1-4690-aa4a-6584fb149952-log-socket\") pod \"ovnkube-node-sn5kw\" (UID: \"bb06c328-82b1-4690-aa4a-6584fb149952\") " pod="openshift-ovn-kubernetes/ovnkube-node-sn5kw"
Sep 11 17:18:43.794563 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:18:43.794255    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/337b4304-78e3-4abd-afbf-b866f7104056-konnectivity-ca\") pod \"konnectivity-agent-g6gcb\" (UID: \"337b4304-78e3-4abd-afbf-b866f7104056\") " pod="kube-system/konnectivity-agent-g6gcb"
Sep 11 17:18:43.794563 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:18:43.794278    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-smgzf\" (UniqueName: \"kubernetes.io/projected/daeb9a26-f9e6-4d68-bc7d-f31b5f10d621-kube-api-access-smgzf\") pod \"aws-ebs-csi-driver-node-hxl5l\" (UID: \"daeb9a26-f9e6-4d68-bc7d-f31b5f10d621\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hxl5l"
Sep 11 17:18:43.794563 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:18:43.794304    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/9208373a-bbca-49fc-91ed-d29c0a833261-multus-cni-dir\") pod \"multus-nltl2\" (UID: \"9208373a-bbca-49fc-91ed-d29c0a833261\") " pod="openshift-multus/multus-nltl2"
Sep 11 17:18:43.794563 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:18:43.794327    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/9208373a-bbca-49fc-91ed-d29c0a833261-host-var-lib-cni-bin\") pod \"multus-nltl2\" (UID: \"9208373a-bbca-49fc-91ed-d29c0a833261\") " pod="openshift-multus/multus-nltl2"
Sep 11 17:18:43.794563 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:18:43.794354    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/bb06c328-82b1-4690-aa4a-6584fb149952-host-cni-bin\") pod \"ovnkube-node-sn5kw\" (UID: \"bb06c328-82b1-4690-aa4a-6584fb149952\") " pod="openshift-ovn-kubernetes/ovnkube-node-sn5kw"
Sep 11 17:18:43.794563 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:18:43.794416    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/9208373a-bbca-49fc-91ed-d29c0a833261-host-var-lib-kubelet\") pod \"multus-nltl2\" (UID: \"9208373a-bbca-49fc-91ed-d29c0a833261\") " pod="openshift-multus/multus-nltl2"
Sep 11 17:18:43.794563 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:18:43.794424    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-d4gm9"
Sep 11 17:18:43.794563 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:18:43.794441    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/a1ce372a-94c9-4089-8d8b-e4dee9b7e782-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-6shpl\" (UID: \"a1ce372a-94c9-4089-8d8b-e4dee9b7e782\") " pod="openshift-multus/multus-additional-cni-plugins-6shpl"
Sep 11 17:18:43.794563 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:18:43.794458    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/9208373a-bbca-49fc-91ed-d29c0a833261-multus-conf-dir\") pod \"multus-nltl2\" (UID: \"9208373a-bbca-49fc-91ed-d29c0a833261\") " pod="openshift-multus/multus-nltl2"
Sep 11 17:18:43.794563 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:18:43.794476    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/66d2b8f8-a4f7-4811-8034-2523351499ee-hosts-file\") pod \"node-resolver-9v9dt\" (UID: \"66d2b8f8-a4f7-4811-8034-2523351499ee\") " pod="openshift-dns/node-resolver-9v9dt"
Sep 11 17:18:43.794563 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:18:43.794517    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-47h2q\" (UniqueName: \"kubernetes.io/projected/a1ce372a-94c9-4089-8d8b-e4dee9b7e782-kube-api-access-47h2q\") pod \"multus-additional-cni-plugins-6shpl\" (UID: \"a1ce372a-94c9-4089-8d8b-e4dee9b7e782\") " pod="openshift-multus/multus-additional-cni-plugins-6shpl"
Sep 11 17:18:43.794563 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:18:43.794556    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/9208373a-bbca-49fc-91ed-d29c0a833261-cni-binary-copy\") pod \"multus-nltl2\" (UID: \"9208373a-bbca-49fc-91ed-d29c0a833261\") " pod="openshift-multus/multus-nltl2"
Sep 11 17:18:43.795282 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:18:43.794590    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/daeb9a26-f9e6-4d68-bc7d-f31b5f10d621-device-dir\") pod \"aws-ebs-csi-driver-node-hxl5l\" (UID: \"daeb9a26-f9e6-4d68-bc7d-f31b5f10d621\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hxl5l"
Sep 11 17:18:43.795282 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:18:43.794613    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/daeb9a26-f9e6-4d68-bc7d-f31b5f10d621-etc-selinux\") pod \"aws-ebs-csi-driver-node-hxl5l\" (UID: \"daeb9a26-f9e6-4d68-bc7d-f31b5f10d621\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hxl5l"
Sep 11 17:18:43.795282 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:18:43.794634    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a1ce372a-94c9-4089-8d8b-e4dee9b7e782-system-cni-dir\") pod \"multus-additional-cni-plugins-6shpl\" (UID: \"a1ce372a-94c9-4089-8d8b-e4dee9b7e782\") " pod="openshift-multus/multus-additional-cni-plugins-6shpl"
Sep 11 17:18:43.795282 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:18:43.794661    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2bn2z\" (UniqueName: \"kubernetes.io/projected/d64f20b5-ec23-4fdd-b8cc-ec63c443faa3-kube-api-access-2bn2z\") pod \"iptables-alerter-vmxwz\" (UID: \"d64f20b5-ec23-4fdd-b8cc-ec63c443faa3\") " pod="openshift-network-operator/iptables-alerter-vmxwz"
Sep 11 17:18:43.795282 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:18:43.794689    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/9208373a-bbca-49fc-91ed-d29c0a833261-host-run-k8s-cni-cncf-io\") pod \"multus-nltl2\" (UID: \"9208373a-bbca-49fc-91ed-d29c0a833261\") " pod="openshift-multus/multus-nltl2"
Sep 11 17:18:43.795282 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:18:43.794712    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/bb06c328-82b1-4690-aa4a-6584fb149952-systemd-units\") pod \"ovnkube-node-sn5kw\" (UID: \"bb06c328-82b1-4690-aa4a-6584fb149952\") " pod="openshift-ovn-kubernetes/ovnkube-node-sn5kw"
Sep 11 17:18:43.795282 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:18:43.794734    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8x4f2\" (UniqueName: \"kubernetes.io/projected/bb06c328-82b1-4690-aa4a-6584fb149952-kube-api-access-8x4f2\") pod \"ovnkube-node-sn5kw\" (UID: \"bb06c328-82b1-4690-aa4a-6584fb149952\") " pod="openshift-ovn-kubernetes/ovnkube-node-sn5kw"
Sep 11 17:18:43.795282 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:18:43.794763    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/bb06c328-82b1-4690-aa4a-6584fb149952-host-cni-netd\") pod \"ovnkube-node-sn5kw\" (UID: \"bb06c328-82b1-4690-aa4a-6584fb149952\") " pod="openshift-ovn-kubernetes/ovnkube-node-sn5kw"
Sep 11 17:18:43.795282 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:18:43.794786    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/9208373a-bbca-49fc-91ed-d29c0a833261-system-cni-dir\") pod \"multus-nltl2\" (UID: \"9208373a-bbca-49fc-91ed-d29c0a833261\") " pod="openshift-multus/multus-nltl2"
Sep 11 17:18:43.795282 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:18:43.794830    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/9208373a-bbca-49fc-91ed-d29c0a833261-cnibin\") pod \"multus-nltl2\" (UID: \"9208373a-bbca-49fc-91ed-d29c0a833261\") " pod="openshift-multus/multus-nltl2"
Sep 11 17:18:43.795282 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:18:43.794854    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/9208373a-bbca-49fc-91ed-d29c0a833261-host-var-lib-cni-multus\") pod \"multus-nltl2\" (UID: \"9208373a-bbca-49fc-91ed-d29c0a833261\") " pod="openshift-multus/multus-nltl2"
Sep 11 17:18:43.795282 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:18:43.794879    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/a1ce372a-94c9-4089-8d8b-e4dee9b7e782-cni-binary-copy\") pod \"multus-additional-cni-plugins-6shpl\" (UID: \"a1ce372a-94c9-4089-8d8b-e4dee9b7e782\") " pod="openshift-multus/multus-additional-cni-plugins-6shpl"
Sep 11 17:18:43.795282 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:18:43.794906    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/a1ce372a-94c9-4089-8d8b-e4dee9b7e782-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-6shpl\" (UID: \"a1ce372a-94c9-4089-8d8b-e4dee9b7e782\") " pod="openshift-multus/multus-additional-cni-plugins-6shpl"
Sep 11 17:18:43.795282 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:18:43.794931    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/d64f20b5-ec23-4fdd-b8cc-ec63c443faa3-iptables-alerter-script\") pod \"iptables-alerter-vmxwz\" (UID: \"d64f20b5-ec23-4fdd-b8cc-ec63c443faa3\") " pod="openshift-network-operator/iptables-alerter-vmxwz"
Sep 11 17:18:43.795282 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:18:43.794958    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/9208373a-bbca-49fc-91ed-d29c0a833261-host-run-netns\") pod \"multus-nltl2\" (UID: \"9208373a-bbca-49fc-91ed-d29c0a833261\") " pod="openshift-multus/multus-nltl2"
Sep 11 17:18:43.795282 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:18:43.794976    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/9208373a-bbca-49fc-91ed-d29c0a833261-hostroot\") pod \"multus-nltl2\" (UID: \"9208373a-bbca-49fc-91ed-d29c0a833261\") " pod="openshift-multus/multus-nltl2"
Sep 11 17:18:43.795822 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:18:43.794989    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/bb06c328-82b1-4690-aa4a-6584fb149952-run-systemd\") pod \"ovnkube-node-sn5kw\" (UID: \"bb06c328-82b1-4690-aa4a-6584fb149952\") " pod="openshift-ovn-kubernetes/ovnkube-node-sn5kw"
Sep 11 17:18:43.795822 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:18:43.795002    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/bb06c328-82b1-4690-aa4a-6584fb149952-run-ovn\") pod \"ovnkube-node-sn5kw\" (UID: \"bb06c328-82b1-4690-aa4a-6584fb149952\") " pod="openshift-ovn-kubernetes/ovnkube-node-sn5kw"
Sep 11 17:18:43.795822 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:18:43.795022    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/bb06c328-82b1-4690-aa4a-6584fb149952-node-log\") pod \"ovnkube-node-sn5kw\" (UID: \"bb06c328-82b1-4690-aa4a-6584fb149952\") " pod="openshift-ovn-kubernetes/ovnkube-node-sn5kw"
Sep 11 17:18:43.795822 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:18:43.795045    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/66d2b8f8-a4f7-4811-8034-2523351499ee-tmp-dir\") pod \"node-resolver-9v9dt\" (UID: \"66d2b8f8-a4f7-4811-8034-2523351499ee\") " pod="openshift-dns/node-resolver-9v9dt"
Sep 11 17:18:43.795822 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:18:43.795063    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/a1ce372a-94c9-4089-8d8b-e4dee9b7e782-tuning-conf-dir\") pod \"multus-additional-cni-plugins-6shpl\" (UID: \"a1ce372a-94c9-4089-8d8b-e4dee9b7e782\") " pod="openshift-multus/multus-additional-cni-plugins-6shpl"
Sep 11 17:18:43.795822 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:18:43.795081    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/d64f20b5-ec23-4fdd-b8cc-ec63c443faa3-host-slash\") pod \"iptables-alerter-vmxwz\" (UID: \"d64f20b5-ec23-4fdd-b8cc-ec63c443faa3\") " pod="openshift-network-operator/iptables-alerter-vmxwz"
Sep 11 17:18:43.795822 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:18:43.795098    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/9208373a-bbca-49fc-91ed-d29c0a833261-multus-daemon-config\") pod \"multus-nltl2\" (UID: \"9208373a-bbca-49fc-91ed-d29c0a833261\") " pod="openshift-multus/multus-nltl2"
Sep 11 17:18:43.795822 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:18:43.795130    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9208373a-bbca-49fc-91ed-d29c0a833261-etc-kubernetes\") pod \"multus-nltl2\" (UID: \"9208373a-bbca-49fc-91ed-d29c0a833261\") " pod="openshift-multus/multus-nltl2"
Sep 11 17:18:43.795822 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:18:43.795144    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/bb06c328-82b1-4690-aa4a-6584fb149952-host-kubelet\") pod \"ovnkube-node-sn5kw\" (UID: \"bb06c328-82b1-4690-aa4a-6584fb149952\") " pod="openshift-ovn-kubernetes/ovnkube-node-sn5kw"
Sep 11 17:18:43.795822 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:18:43.795163    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/bb06c328-82b1-4690-aa4a-6584fb149952-env-overrides\") pod \"ovnkube-node-sn5kw\" (UID: \"bb06c328-82b1-4690-aa4a-6584fb149952\") " pod="openshift-ovn-kubernetes/ovnkube-node-sn5kw"
Sep 11 17:18:43.795822 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:18:43.795193    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/bb06c328-82b1-4690-aa4a-6584fb149952-ovn-node-metrics-cert\") pod \"ovnkube-node-sn5kw\" (UID: \"bb06c328-82b1-4690-aa4a-6584fb149952\") " pod="openshift-ovn-kubernetes/ovnkube-node-sn5kw"
Sep 11 17:18:43.795822 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:18:43.795210    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/a1ce372a-94c9-4089-8d8b-e4dee9b7e782-cnibin\") pod \"multus-additional-cni-plugins-6shpl\" (UID: \"a1ce372a-94c9-4089-8d8b-e4dee9b7e782\") " pod="openshift-multus/multus-additional-cni-plugins-6shpl"
Sep 11 17:18:43.795822 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:18:43.795226    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/bb06c328-82b1-4690-aa4a-6584fb149952-ovnkube-script-lib\") pod \"ovnkube-node-sn5kw\" (UID: \"bb06c328-82b1-4690-aa4a-6584fb149952\") " pod="openshift-ovn-kubernetes/ovnkube-node-sn5kw"
Sep 11 17:18:43.796456 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:18:43.796440    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Sep 11 17:18:43.796528 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:18:43.796440    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Sep 11 17:18:43.796573 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:18:43.796526    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-kh2s9\""
Sep 11 17:18:43.797074 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:18:43.797060    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-k5j4d"
Sep 11 17:18:43.799054 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:18:43.799037    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Sep 11 17:18:43.799415 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:18:43.799392    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bl4vr"
Sep 11 17:18:43.799531 ip-10-0-141-139 kubenswrapper[2577]: E0911 17:18:43.799494    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bl4vr" podUID="5c1ac293-cc19-4446-8244-a8ca7fc8a8f4"
Sep 11 17:18:43.799922 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:18:43.799893    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Sep 11 17:18:43.799996 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:18:43.799946    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Sep 11 17:18:43.800081 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:18:43.800038    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-sjwwm\""
Sep 11 17:18:43.802138 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:18:43.802118    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xfn47"
Sep 11 17:18:43.802246 ip-10-0-141-139 kubenswrapper[2577]: E0911 17:18:43.802204    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xfn47" podUID="1e5d9e0c-7bc0-4fcf-8845-8873df5b1bee"
Sep 11 17:18:43.804286 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:18:43.804269    2577 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Sep 11 17:18:43.804417 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:18:43.804371    2577 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-139.ec2.internal"
Sep 11 17:18:43.804735 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:18:43.804717    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kube-system/kube-apiserver-proxy-ip-10-0-141-139.ec2.internal"]
Sep 11 17:18:43.804834 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:18:43.804795    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fsh8r"
Sep 11 17:18:43.805021 ip-10-0-141-139 kubenswrapper[2577]: E0911 17:18:43.804861    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-fsh8r" podUID="73dd9a34-fac4-44c9-96d3-b2c8968ed271"
Sep 11 17:18:43.806448 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:18:43.806429    2577 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Sep 11 17:18:43.808064 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:18:43.808012    2577 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-09-10 17:13:43 +0000 UTC" deadline="2028-03-14 07:12:02.553332734 +0000 UTC"
Sep 11 17:18:43.808064 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:18:43.808063    2577 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="13189h53m18.745272138s"
Sep 11 17:18:43.815166 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:18:43.815152    2577 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Sep 11 17:18:43.815346 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:18:43.815330    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-139.ec2.internal"]
Sep 11 17:18:43.825239 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:18:43.825218    2577 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-h7g7r"
Sep 11 17:18:43.833064 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:18:43.833043    2577 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-h7g7r"
Sep 11 17:18:43.890034 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:18:43.889997    2577 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Sep 11 17:18:43.895478 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:18:43.895446    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/daeb9a26-f9e6-4d68-bc7d-f31b5f10d621-device-dir\") pod \"aws-ebs-csi-driver-node-hxl5l\" (UID: \"daeb9a26-f9e6-4d68-bc7d-f31b5f10d621\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hxl5l"
Sep 11 17:18:43.895649 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:18:43.895488    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/daeb9a26-f9e6-4d68-bc7d-f31b5f10d621-etc-selinux\") pod \"aws-ebs-csi-driver-node-hxl5l\" (UID: \"daeb9a26-f9e6-4d68-bc7d-f31b5f10d621\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hxl5l"
Sep 11 17:18:43.895649 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:18:43.895534    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a1ce372a-94c9-4089-8d8b-e4dee9b7e782-system-cni-dir\") pod \"multus-additional-cni-plugins-6shpl\" (UID: \"a1ce372a-94c9-4089-8d8b-e4dee9b7e782\") " pod="openshift-multus/multus-additional-cni-plugins-6shpl"
Sep 11 17:18:43.895649 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:18:43.895559    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2bn2z\" (UniqueName: \"kubernetes.io/projected/d64f20b5-ec23-4fdd-b8cc-ec63c443faa3-kube-api-access-2bn2z\") pod \"iptables-alerter-vmxwz\" (UID: \"d64f20b5-ec23-4fdd-b8cc-ec63c443faa3\") " pod="openshift-network-operator/iptables-alerter-vmxwz"
Sep 11 17:18:43.895649 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:18:43.895576    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/daeb9a26-f9e6-4d68-bc7d-f31b5f10d621-device-dir\") pod \"aws-ebs-csi-driver-node-hxl5l\" (UID: \"daeb9a26-f9e6-4d68-bc7d-f31b5f10d621\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hxl5l"
Sep 11 17:18:43.895649 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:18:43.895583    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/9208373a-bbca-49fc-91ed-d29c0a833261-host-run-k8s-cni-cncf-io\") pod \"multus-nltl2\" (UID: \"9208373a-bbca-49fc-91ed-d29c0a833261\") " pod="openshift-multus/multus-nltl2"
Sep 11 17:18:43.895649 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:18:43.895628    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/9208373a-bbca-49fc-91ed-d29c0a833261-host-run-k8s-cni-cncf-io\") pod \"multus-nltl2\" (UID: \"9208373a-bbca-49fc-91ed-d29c0a833261\") " pod="openshift-multus/multus-nltl2"
Sep 11 17:18:43.895649 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:18:43.895637    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/bb06c328-82b1-4690-aa4a-6584fb149952-systemd-units\") pod \"ovnkube-node-sn5kw\" (UID: \"bb06c328-82b1-4690-aa4a-6584fb149952\") " pod="openshift-ovn-kubernetes/ovnkube-node-sn5kw"
Sep 11 17:18:43.896182 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:18:43.895665    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8x4f2\" (UniqueName: \"kubernetes.io/projected/bb06c328-82b1-4690-aa4a-6584fb149952-kube-api-access-8x4f2\") pod \"ovnkube-node-sn5kw\" (UID: \"bb06c328-82b1-4690-aa4a-6584fb149952\") " pod="openshift-ovn-kubernetes/ovnkube-node-sn5kw"
Sep 11 17:18:43.896182 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:18:43.895696    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/2fe5e836-a8a9-402a-bce2-da7ddc7418fa-etc-sysconfig\") pod \"tuned-d4gm9\" (UID: \"2fe5e836-a8a9-402a-bce2-da7ddc7418fa\") " pod="openshift-cluster-node-tuning-operator/tuned-d4gm9"
Sep 11 17:18:43.896182 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:18:43.895719    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/2fe5e836-a8a9-402a-bce2-da7ddc7418fa-var-lib-kubelet\") pod \"tuned-d4gm9\" (UID: \"2fe5e836-a8a9-402a-bce2-da7ddc7418fa\") " pod="openshift-cluster-node-tuning-operator/tuned-d4gm9"
Sep 11 17:18:43.896182 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:18:43.895740    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/bb06c328-82b1-4690-aa4a-6584fb149952-host-cni-netd\") pod \"ovnkube-node-sn5kw\" (UID: \"bb06c328-82b1-4690-aa4a-6584fb149952\") " pod="openshift-ovn-kubernetes/ovnkube-node-sn5kw"
Sep 11 17:18:43.896182 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:18:43.895767    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/daeb9a26-f9e6-4d68-bc7d-f31b5f10d621-etc-selinux\") pod \"aws-ebs-csi-driver-node-hxl5l\" (UID: \"daeb9a26-f9e6-4d68-bc7d-f31b5f10d621\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hxl5l"
Sep 11 17:18:43.896182 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:18:43.895806    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/bb06c328-82b1-4690-aa4a-6584fb149952-systemd-units\") pod \"ovnkube-node-sn5kw\" (UID: \"bb06c328-82b1-4690-aa4a-6584fb149952\") " pod="openshift-ovn-kubernetes/ovnkube-node-sn5kw"
Sep 11 17:18:43.896182 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:18:43.895818    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/9208373a-bbca-49fc-91ed-d29c0a833261-system-cni-dir\") pod \"multus-nltl2\" (UID: \"9208373a-bbca-49fc-91ed-d29c0a833261\") " pod="openshift-multus/multus-nltl2"
Sep 11 17:18:43.896182 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:18:43.895768    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/9208373a-bbca-49fc-91ed-d29c0a833261-system-cni-dir\") pod \"multus-nltl2\" (UID: \"9208373a-bbca-49fc-91ed-d29c0a833261\") " pod="openshift-multus/multus-nltl2"
Sep 11 17:18:43.896182 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:18:43.895854    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/9208373a-bbca-49fc-91ed-d29c0a833261-cnibin\") pod \"multus-nltl2\" (UID: \"9208373a-bbca-49fc-91ed-d29c0a833261\") " pod="openshift-multus/multus-nltl2"
Sep 11 17:18:43.896182 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:18:43.895878    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/9208373a-bbca-49fc-91ed-d29c0a833261-host-var-lib-cni-multus\") pod \"multus-nltl2\" (UID: \"9208373a-bbca-49fc-91ed-d29c0a833261\") " pod="openshift-multus/multus-nltl2"
Sep 11 17:18:43.896182 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:18:43.895899    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/c5878f27-a359-47d1-b223-08f4bd829380-host\") pod \"node-ca-k5j4d\" (UID: \"c5878f27-a359-47d1-b223-08f4bd829380\") " pod="openshift-image-registry/node-ca-k5j4d"
Sep 11 17:18:43.896182 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:18:43.895937    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/a1ce372a-94c9-4089-8d8b-e4dee9b7e782-cni-binary-copy\") pod \"multus-additional-cni-plugins-6shpl\" (UID: \"a1ce372a-94c9-4089-8d8b-e4dee9b7e782\") " pod="openshift-multus/multus-additional-cni-plugins-6shpl"
Sep 11 17:18:43.896182 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:18:43.895950    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/9208373a-bbca-49fc-91ed-d29c0a833261-cnibin\") pod \"multus-nltl2\" (UID: \"9208373a-bbca-49fc-91ed-d29c0a833261\") " pod="openshift-multus/multus-nltl2"
Sep 11 17:18:43.896182 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:18:43.895664    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a1ce372a-94c9-4089-8d8b-e4dee9b7e782-system-cni-dir\") pod \"multus-additional-cni-plugins-6shpl\" (UID: \"a1ce372a-94c9-4089-8d8b-e4dee9b7e782\") " pod="openshift-multus/multus-additional-cni-plugins-6shpl"
Sep 11 17:18:43.896182 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:18:43.895987    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/9208373a-bbca-49fc-91ed-d29c0a833261-host-var-lib-cni-multus\") pod \"multus-nltl2\" (UID: \"9208373a-bbca-49fc-91ed-d29c0a833261\") " pod="openshift-multus/multus-nltl2"
Sep 11 17:18:43.896182 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:18:43.896016    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/a1ce372a-94c9-4089-8d8b-e4dee9b7e782-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-6shpl\" (UID: \"a1ce372a-94c9-4089-8d8b-e4dee9b7e782\") " pod="openshift-multus/multus-additional-cni-plugins-6shpl"
Sep 11 17:18:43.896182 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:18:43.896102    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/d64f20b5-ec23-4fdd-b8cc-ec63c443faa3-iptables-alerter-script\") pod \"iptables-alerter-vmxwz\" (UID: \"d64f20b5-ec23-4fdd-b8cc-ec63c443faa3\") " pod="openshift-network-operator/iptables-alerter-vmxwz"
Sep 11 17:18:43.896840 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:18:43.896159    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/9208373a-bbca-49fc-91ed-d29c0a833261-host-run-netns\") pod \"multus-nltl2\" (UID: \"9208373a-bbca-49fc-91ed-d29c0a833261\") " pod="openshift-multus/multus-nltl2"
Sep 11 17:18:43.896840 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:18:43.896203    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/bb06c328-82b1-4690-aa4a-6584fb149952-host-cni-netd\") pod \"ovnkube-node-sn5kw\" (UID: \"bb06c328-82b1-4690-aa4a-6584fb149952\") " pod="openshift-ovn-kubernetes/ovnkube-node-sn5kw"
Sep 11 17:18:43.896840 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:18:43.896239    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/9208373a-bbca-49fc-91ed-d29c0a833261-hostroot\") pod \"multus-nltl2\" (UID: \"9208373a-bbca-49fc-91ed-d29c0a833261\") " pod="openshift-multus/multus-nltl2"
Sep 11 17:18:43.896840 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:18:43.896248    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/9208373a-bbca-49fc-91ed-d29c0a833261-host-run-netns\") pod \"multus-nltl2\" (UID: \"9208373a-bbca-49fc-91ed-d29c0a833261\") " pod="openshift-multus/multus-nltl2"
Sep 11 17:18:43.896840 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:18:43.896287    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/bb06c328-82b1-4690-aa4a-6584fb149952-run-systemd\") pod \"ovnkube-node-sn5kw\" (UID: \"bb06c328-82b1-4690-aa4a-6584fb149952\") " pod="openshift-ovn-kubernetes/ovnkube-node-sn5kw"
Sep 11 17:18:43.896840 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:18:43.896313    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/bb06c328-82b1-4690-aa4a-6584fb149952-run-ovn\") pod \"ovnkube-node-sn5kw\" (UID: \"bb06c328-82b1-4690-aa4a-6584fb149952\") " pod="openshift-ovn-kubernetes/ovnkube-node-sn5kw"
Sep 11 17:18:43.896840 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:18:43.896355    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/bb06c328-82b1-4690-aa4a-6584fb149952-run-ovn\") pod \"ovnkube-node-sn5kw\" (UID: \"bb06c328-82b1-4690-aa4a-6584fb149952\") " pod="openshift-ovn-kubernetes/ovnkube-node-sn5kw"
Sep 11 17:18:43.896840 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:18:43.896362    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/bb06c328-82b1-4690-aa4a-6584fb149952-node-log\") pod \"ovnkube-node-sn5kw\" (UID: \"bb06c328-82b1-4690-aa4a-6584fb149952\") " pod="openshift-ovn-kubernetes/ovnkube-node-sn5kw"
Sep 11 17:18:43.896840 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:18:43.896385    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/9208373a-bbca-49fc-91ed-d29c0a833261-hostroot\") pod \"multus-nltl2\" (UID: \"9208373a-bbca-49fc-91ed-d29c0a833261\") " pod="openshift-multus/multus-nltl2"
Sep 11 17:18:43.896840 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:18:43.896409    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/bb06c328-82b1-4690-aa4a-6584fb149952-run-systemd\") pod \"ovnkube-node-sn5kw\" (UID: \"bb06c328-82b1-4690-aa4a-6584fb149952\") " pod="openshift-ovn-kubernetes/ovnkube-node-sn5kw"
Sep 11 17:18:43.896840 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:18:43.896444    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/66d2b8f8-a4f7-4811-8034-2523351499ee-tmp-dir\") pod \"node-resolver-9v9dt\" (UID: \"66d2b8f8-a4f7-4811-8034-2523351499ee\") " pod="openshift-dns/node-resolver-9v9dt"
Sep 11 17:18:43.896840 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:18:43.896444    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/bb06c328-82b1-4690-aa4a-6584fb149952-node-log\") pod \"ovnkube-node-sn5kw\" (UID: \"bb06c328-82b1-4690-aa4a-6584fb149952\") " pod="openshift-ovn-kubernetes/ovnkube-node-sn5kw"
Sep 11 17:18:43.896840 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:18:43.896475    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/a1ce372a-94c9-4089-8d8b-e4dee9b7e782-tuning-conf-dir\") pod \"multus-additional-cni-plugins-6shpl\" (UID: \"a1ce372a-94c9-4089-8d8b-e4dee9b7e782\") " pod="openshift-multus/multus-additional-cni-plugins-6shpl"
Sep 11 17:18:43.896840 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:18:43.896547    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/d64f20b5-ec23-4fdd-b8cc-ec63c443faa3-host-slash\") pod \"iptables-alerter-vmxwz\" (UID: \"d64f20b5-ec23-4fdd-b8cc-ec63c443faa3\") " pod="openshift-network-operator/iptables-alerter-vmxwz"
Sep 11 17:18:43.896840 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:18:43.896548    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/a1ce372a-94c9-4089-8d8b-e4dee9b7e782-cni-binary-copy\") pod \"multus-additional-cni-plugins-6shpl\" (UID: \"a1ce372a-94c9-4089-8d8b-e4dee9b7e782\") " pod="openshift-multus/multus-additional-cni-plugins-6shpl"
Sep 11 17:18:43.896840 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:18:43.896627    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/d64f20b5-ec23-4fdd-b8cc-ec63c443faa3-host-slash\") pod \"iptables-alerter-vmxwz\" (UID: \"d64f20b5-ec23-4fdd-b8cc-ec63c443faa3\") " pod="openshift-network-operator/iptables-alerter-vmxwz"
Sep 11 17:18:43.896840 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:18:43.896631    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/9208373a-bbca-49fc-91ed-d29c0a833261-multus-daemon-config\") pod \"multus-nltl2\" (UID: \"9208373a-bbca-49fc-91ed-d29c0a833261\") " pod="openshift-multus/multus-nltl2"
Sep 11 17:18:43.896840 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:18:43.896634    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/a1ce372a-94c9-4089-8d8b-e4dee9b7e782-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-6shpl\" (UID: \"a1ce372a-94c9-4089-8d8b-e4dee9b7e782\") " pod="openshift-multus/multus-additional-cni-plugins-6shpl"
Sep 11 17:18:43.897678 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:18:43.896653    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/a1ce372a-94c9-4089-8d8b-e4dee9b7e782-tuning-conf-dir\") pod \"multus-additional-cni-plugins-6shpl\" (UID: \"a1ce372a-94c9-4089-8d8b-e4dee9b7e782\") " pod="openshift-multus/multus-additional-cni-plugins-6shpl"
Sep 11 17:18:43.897678 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:18:43.896671    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9208373a-bbca-49fc-91ed-d29c0a833261-etc-kubernetes\") pod \"multus-nltl2\" (UID: \"9208373a-bbca-49fc-91ed-d29c0a833261\") " pod="openshift-multus/multus-nltl2"
Sep 11 17:18:43.897678 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:18:43.896691    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/d64f20b5-ec23-4fdd-b8cc-ec63c443faa3-iptables-alerter-script\") pod \"iptables-alerter-vmxwz\" (UID: \"d64f20b5-ec23-4fdd-b8cc-ec63c443faa3\") " pod="openshift-network-operator/iptables-alerter-vmxwz"
Sep 11 17:18:43.897678 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:18:43.896703    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/bb06c328-82b1-4690-aa4a-6584fb149952-host-kubelet\") pod \"ovnkube-node-sn5kw\" (UID: \"bb06c328-82b1-4690-aa4a-6584fb149952\") " pod="openshift-ovn-kubernetes/ovnkube-node-sn5kw"
Sep 11 17:18:43.897678 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:18:43.896730    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/bb06c328-82b1-4690-aa4a-6584fb149952-host-kubelet\") pod \"ovnkube-node-sn5kw\" (UID: \"bb06c328-82b1-4690-aa4a-6584fb149952\") " pod="openshift-ovn-kubernetes/ovnkube-node-sn5kw"
Sep 11 17:18:43.897678 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:18:43.896728    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/bb06c328-82b1-4690-aa4a-6584fb149952-env-overrides\") pod \"ovnkube-node-sn5kw\" (UID: \"bb06c328-82b1-4690-aa4a-6584fb149952\") " pod="openshift-ovn-kubernetes/ovnkube-node-sn5kw"
Sep 11 17:18:43.897678 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:18:43.896706    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9208373a-bbca-49fc-91ed-d29c0a833261-etc-kubernetes\") pod \"multus-nltl2\" (UID: \"9208373a-bbca-49fc-91ed-d29c0a833261\") " pod="openshift-multus/multus-nltl2"
Sep 11 17:18:43.897678 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:18:43.896768    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/bb06c328-82b1-4690-aa4a-6584fb149952-ovn-node-metrics-cert\") pod \"ovnkube-node-sn5kw\" (UID: \"bb06c328-82b1-4690-aa4a-6584fb149952\") " pod="openshift-ovn-kubernetes/ovnkube-node-sn5kw"
Sep 11 17:18:43.897678 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:18:43.896788    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/a1ce372a-94c9-4089-8d8b-e4dee9b7e782-cnibin\") pod \"multus-additional-cni-plugins-6shpl\" (UID: \"a1ce372a-94c9-4089-8d8b-e4dee9b7e782\") " pod="openshift-multus/multus-additional-cni-plugins-6shpl"
Sep 11 17:18:43.897678 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:18:43.896784    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/66d2b8f8-a4f7-4811-8034-2523351499ee-tmp-dir\") pod \"node-resolver-9v9dt\" (UID: \"66d2b8f8-a4f7-4811-8034-2523351499ee\") " pod="openshift-dns/node-resolver-9v9dt"
Sep 11 17:18:43.897678 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:18:43.896802    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/bb06c328-82b1-4690-aa4a-6584fb149952-ovnkube-script-lib\") pod \"ovnkube-node-sn5kw\" (UID: \"bb06c328-82b1-4690-aa4a-6584fb149952\") " pod="openshift-ovn-kubernetes/ovnkube-node-sn5kw"
Sep 11 17:18:43.897678 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:18:43.896826    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/2fe5e836-a8a9-402a-bce2-da7ddc7418fa-etc-sysctl-d\") pod \"tuned-d4gm9\" (UID: \"2fe5e836-a8a9-402a-bce2-da7ddc7418fa\") " pod="openshift-cluster-node-tuning-operator/tuned-d4gm9"
Sep 11 17:18:43.897678 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:18:43.896845    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/2fe5e836-a8a9-402a-bce2-da7ddc7418fa-run\") pod \"tuned-d4gm9\" (UID: \"2fe5e836-a8a9-402a-bce2-da7ddc7418fa\") " pod="openshift-cluster-node-tuning-operator/tuned-d4gm9"
Sep 11 17:18:43.897678 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:18:43.896848    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/a1ce372a-94c9-4089-8d8b-e4dee9b7e782-cnibin\") pod \"multus-additional-cni-plugins-6shpl\" (UID: \"a1ce372a-94c9-4089-8d8b-e4dee9b7e782\") " pod="openshift-multus/multus-additional-cni-plugins-6shpl"
Sep 11 17:18:43.897678 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:18:43.896860    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/c5878f27-a359-47d1-b223-08f4bd829380-serviceca\") pod \"node-ca-k5j4d\" (UID: \"c5878f27-a359-47d1-b223-08f4bd829380\") " pod="openshift-image-registry/node-ca-k5j4d"
Sep 11 17:18:43.897678 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:18:43.896882    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hhmg9\" (UniqueName: \"kubernetes.io/projected/c5878f27-a359-47d1-b223-08f4bd829380-kube-api-access-hhmg9\") pod \"node-ca-k5j4d\" (UID: \"c5878f27-a359-47d1-b223-08f4bd829380\") " pod="openshift-image-registry/node-ca-k5j4d"
Sep 11 17:18:43.897678 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:18:43.896926    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h2trs\" (UniqueName: \"kubernetes.io/projected/1e5d9e0c-7bc0-4fcf-8845-8873df5b1bee-kube-api-access-h2trs\") pod \"network-check-target-xfn47\" (UID: \"1e5d9e0c-7bc0-4fcf-8845-8873df5b1bee\") " pod="openshift-network-diagnostics/network-check-target-xfn47"
Sep 11 17:18:43.898477 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:18:43.896951    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f8tfp\" (UniqueName: \"kubernetes.io/projected/9208373a-bbca-49fc-91ed-d29c0a833261-kube-api-access-f8tfp\") pod \"multus-nltl2\" (UID: \"9208373a-bbca-49fc-91ed-d29c0a833261\") " pod="openshift-multus/multus-nltl2"
Sep 11 17:18:43.898477 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:18:43.896993    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/bb06c328-82b1-4690-aa4a-6584fb149952-host-slash\") pod \"ovnkube-node-sn5kw\" (UID: \"bb06c328-82b1-4690-aa4a-6584fb149952\") " pod="openshift-ovn-kubernetes/ovnkube-node-sn5kw"
Sep 11 17:18:43.898477 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:18:43.897017    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/bb06c328-82b1-4690-aa4a-6584fb149952-run-openvswitch\") pod \"ovnkube-node-sn5kw\" (UID: \"bb06c328-82b1-4690-aa4a-6584fb149952\") " pod="openshift-ovn-kubernetes/ovnkube-node-sn5kw"
Sep 11 17:18:43.898477 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:18:43.897043    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/bb06c328-82b1-4690-aa4a-6584fb149952-host-run-ovn-kubernetes\") pod \"ovnkube-node-sn5kw\" (UID: \"bb06c328-82b1-4690-aa4a-6584fb149952\") " pod="openshift-ovn-kubernetes/ovnkube-node-sn5kw"
Sep 11 17:18:43.898477 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:18:43.897067    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/daeb9a26-f9e6-4d68-bc7d-f31b5f10d621-kubelet-dir\") pod \"aws-ebs-csi-driver-node-hxl5l\" (UID: \"daeb9a26-f9e6-4d68-bc7d-f31b5f10d621\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hxl5l"
Sep 11 17:18:43.898477 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:18:43.897091    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/daeb9a26-f9e6-4d68-bc7d-f31b5f10d621-sys-fs\") pod \"aws-ebs-csi-driver-node-hxl5l\" (UID: \"daeb9a26-f9e6-4d68-bc7d-f31b5f10d621\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hxl5l"
Sep 11 17:18:43.898477 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:18:43.897116    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/2fe5e836-a8a9-402a-bce2-da7ddc7418fa-etc-tuned\") pod \"tuned-d4gm9\" (UID: \"2fe5e836-a8a9-402a-bce2-da7ddc7418fa\") " pod="openshift-cluster-node-tuning-operator/tuned-d4gm9"
Sep 11 17:18:43.898477 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:18:43.897129    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/bb06c328-82b1-4690-aa4a-6584fb149952-env-overrides\") pod \"ovnkube-node-sn5kw\" (UID: \"bb06c328-82b1-4690-aa4a-6584fb149952\") " pod="openshift-ovn-kubernetes/ovnkube-node-sn5kw"
Sep 11 17:18:43.898477 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:18:43.897140    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/73dd9a34-fac4-44c9-96d3-b2c8968ed271-dbus\") pod \"global-pull-secret-syncer-fsh8r\" (UID: \"73dd9a34-fac4-44c9-96d3-b2c8968ed271\") " pod="kube-system/global-pull-secret-syncer-fsh8r"
Sep 11 17:18:43.898477 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:18:43.897130    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/9208373a-bbca-49fc-91ed-d29c0a833261-multus-daemon-config\") pod \"multus-nltl2\" (UID: \"9208373a-bbca-49fc-91ed-d29c0a833261\") " pod="openshift-multus/multus-nltl2"
Sep 11 17:18:43.898477 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:18:43.897166    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/bb06c328-82b1-4690-aa4a-6584fb149952-var-lib-openvswitch\") pod \"ovnkube-node-sn5kw\" (UID: \"bb06c328-82b1-4690-aa4a-6584fb149952\") " pod="openshift-ovn-kubernetes/ovnkube-node-sn5kw"
Sep 11 17:18:43.898477 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:18:43.897190    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/bb06c328-82b1-4690-aa4a-6584fb149952-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-sn5kw\" (UID: \"bb06c328-82b1-4690-aa4a-6584fb149952\") " pod="openshift-ovn-kubernetes/ovnkube-node-sn5kw"
Sep 11 17:18:43.898477 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:18:43.897176    2577 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Sep 11 17:18:43.898477 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:18:43.897215    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/bb06c328-82b1-4690-aa4a-6584fb149952-host-slash\") pod \"ovnkube-node-sn5kw\" (UID: \"bb06c328-82b1-4690-aa4a-6584fb149952\") " pod="openshift-ovn-kubernetes/ovnkube-node-sn5kw"
Sep 11 17:18:43.898477 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:18:43.897217    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t89tx\" (UniqueName: \"kubernetes.io/projected/2fe5e836-a8a9-402a-bce2-da7ddc7418fa-kube-api-access-t89tx\") pod \"tuned-d4gm9\" (UID: \"2fe5e836-a8a9-402a-bce2-da7ddc7418fa\") " pod="openshift-cluster-node-tuning-operator/tuned-d4gm9"
Sep 11 17:18:43.898477 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:18:43.897237    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/daeb9a26-f9e6-4d68-bc7d-f31b5f10d621-kubelet-dir\") pod \"aws-ebs-csi-driver-node-hxl5l\" (UID: \"daeb9a26-f9e6-4d68-bc7d-f31b5f10d621\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hxl5l"
Sep 11 17:18:43.898477 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:18:43.897249    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/bb06c328-82b1-4690-aa4a-6584fb149952-host-run-netns\") pod \"ovnkube-node-sn5kw\" (UID: \"bb06c328-82b1-4690-aa4a-6584fb149952\") " pod="openshift-ovn-kubernetes/ovnkube-node-sn5kw"
Sep 11 17:18:43.899141 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:18:43.897220    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/bb06c328-82b1-4690-aa4a-6584fb149952-run-openvswitch\") pod \"ovnkube-node-sn5kw\" (UID: \"bb06c328-82b1-4690-aa4a-6584fb149952\") " pod="openshift-ovn-kubernetes/ovnkube-node-sn5kw"
Sep 11 17:18:43.899141 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:18:43.897255    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/bb06c328-82b1-4690-aa4a-6584fb149952-host-run-ovn-kubernetes\") pod \"ovnkube-node-sn5kw\" (UID: \"bb06c328-82b1-4690-aa4a-6584fb149952\") " pod="openshift-ovn-kubernetes/ovnkube-node-sn5kw"
Sep 11 17:18:43.899141 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:18:43.897290    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/bb06c328-82b1-4690-aa4a-6584fb149952-var-lib-openvswitch\") pod \"ovnkube-node-sn5kw\" (UID: \"bb06c328-82b1-4690-aa4a-6584fb149952\") " pod="openshift-ovn-kubernetes/ovnkube-node-sn5kw"
Sep 11 17:18:43.899141 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:18:43.897306    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/daeb9a26-f9e6-4d68-bc7d-f31b5f10d621-sys-fs\") pod \"aws-ebs-csi-driver-node-hxl5l\" (UID: \"daeb9a26-f9e6-4d68-bc7d-f31b5f10d621\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hxl5l"
Sep 11 17:18:43.899141 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:18:43.897308    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/bb06c328-82b1-4690-aa4a-6584fb149952-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-sn5kw\" (UID: \"bb06c328-82b1-4690-aa4a-6584fb149952\") " pod="openshift-ovn-kubernetes/ovnkube-node-sn5kw"
Sep 11 17:18:43.899141 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:18:43.897318    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/2fe5e836-a8a9-402a-bce2-da7ddc7418fa-tmp\") pod \"tuned-d4gm9\" (UID: \"2fe5e836-a8a9-402a-bce2-da7ddc7418fa\") " pod="openshift-cluster-node-tuning-operator/tuned-d4gm9"
Sep 11 17:18:43.899141 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:18:43.897343    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/daeb9a26-f9e6-4d68-bc7d-f31b5f10d621-registration-dir\") pod \"aws-ebs-csi-driver-node-hxl5l\" (UID: \"daeb9a26-f9e6-4d68-bc7d-f31b5f10d621\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hxl5l"
Sep 11 17:18:43.899141 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:18:43.897382    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/bb06c328-82b1-4690-aa4a-6584fb149952-ovnkube-config\") pod \"ovnkube-node-sn5kw\" (UID: \"bb06c328-82b1-4690-aa4a-6584fb149952\") " pod="openshift-ovn-kubernetes/ovnkube-node-sn5kw"
Sep 11 17:18:43.899141 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:18:43.897380    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/bb06c328-82b1-4690-aa4a-6584fb149952-ovnkube-script-lib\") pod \"ovnkube-node-sn5kw\" (UID: \"bb06c328-82b1-4690-aa4a-6584fb149952\") " pod="openshift-ovn-kubernetes/ovnkube-node-sn5kw"
Sep 11 17:18:43.899141 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:18:43.897432    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/daeb9a26-f9e6-4d68-bc7d-f31b5f10d621-registration-dir\") pod \"aws-ebs-csi-driver-node-hxl5l\" (UID: \"daeb9a26-f9e6-4d68-bc7d-f31b5f10d621\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hxl5l"
Sep 11 17:18:43.899141 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:18:43.897433    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/daeb9a26-f9e6-4d68-bc7d-f31b5f10d621-socket-dir\") pod \"aws-ebs-csi-driver-node-hxl5l\" (UID: \"daeb9a26-f9e6-4d68-bc7d-f31b5f10d621\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hxl5l"
Sep 11 17:18:43.899141 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:18:43.897454    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/bb06c328-82b1-4690-aa4a-6584fb149952-host-run-netns\") pod \"ovnkube-node-sn5kw\" (UID: \"bb06c328-82b1-4690-aa4a-6584fb149952\") " pod="openshift-ovn-kubernetes/ovnkube-node-sn5kw"
Sep 11 17:18:43.899141 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:18:43.897496    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nbw97\" (UniqueName: \"kubernetes.io/projected/66d2b8f8-a4f7-4811-8034-2523351499ee-kube-api-access-nbw97\") pod \"node-resolver-9v9dt\" (UID: \"66d2b8f8-a4f7-4811-8034-2523351499ee\") " pod="openshift-dns/node-resolver-9v9dt"
Sep 11 17:18:43.899141 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:18:43.897556    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/9208373a-bbca-49fc-91ed-d29c0a833261-multus-socket-dir-parent\") pod \"multus-nltl2\" (UID: \"9208373a-bbca-49fc-91ed-d29c0a833261\") " pod="openshift-multus/multus-nltl2"
Sep 11 17:18:43.899141 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:18:43.897605    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/daeb9a26-f9e6-4d68-bc7d-f31b5f10d621-socket-dir\") pod \"aws-ebs-csi-driver-node-hxl5l\" (UID: \"daeb9a26-f9e6-4d68-bc7d-f31b5f10d621\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hxl5l"
Sep 11 17:18:43.899141 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:18:43.897659    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/9208373a-bbca-49fc-91ed-d29c0a833261-host-run-multus-certs\") pod \"multus-nltl2\" (UID: \"9208373a-bbca-49fc-91ed-d29c0a833261\") " pod="openshift-multus/multus-nltl2"
Sep 11 17:18:43.899141 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:18:43.897701    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/2fe5e836-a8a9-402a-bce2-da7ddc7418fa-etc-modprobe-d\") pod \"tuned-d4gm9\" (UID: \"2fe5e836-a8a9-402a-bce2-da7ddc7418fa\") " pod="openshift-cluster-node-tuning-operator/tuned-d4gm9"
Sep 11 17:18:43.899687 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:18:43.897729    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/2fe5e836-a8a9-402a-bce2-da7ddc7418fa-host\") pod \"tuned-d4gm9\" (UID: \"2fe5e836-a8a9-402a-bce2-da7ddc7418fa\") " pod="openshift-cluster-node-tuning-operator/tuned-d4gm9"
Sep 11 17:18:43.899687 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:18:43.897745    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/9208373a-bbca-49fc-91ed-d29c0a833261-host-run-multus-certs\") pod \"multus-nltl2\" (UID: \"9208373a-bbca-49fc-91ed-d29c0a833261\") " pod="openshift-multus/multus-nltl2"
Sep 11 17:18:43.899687 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:18:43.897758    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/337b4304-78e3-4abd-afbf-b866f7104056-agent-certs\") pod \"konnectivity-agent-g6gcb\" (UID: \"337b4304-78e3-4abd-afbf-b866f7104056\") " pod="kube-system/konnectivity-agent-g6gcb"
Sep 11 17:18:43.899687 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:18:43.897766    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/9208373a-bbca-49fc-91ed-d29c0a833261-multus-socket-dir-parent\") pod \"multus-nltl2\" (UID: \"9208373a-bbca-49fc-91ed-d29c0a833261\") " pod="openshift-multus/multus-nltl2"
Sep 11 17:18:43.899687 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:18:43.897792    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/bb06c328-82b1-4690-aa4a-6584fb149952-etc-openvswitch\") pod \"ovnkube-node-sn5kw\" (UID: \"bb06c328-82b1-4690-aa4a-6584fb149952\") " pod="openshift-ovn-kubernetes/ovnkube-node-sn5kw"
Sep 11 17:18:43.899687 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:18:43.897815    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/bb06c328-82b1-4690-aa4a-6584fb149952-log-socket\") pod \"ovnkube-node-sn5kw\" (UID: \"bb06c328-82b1-4690-aa4a-6584fb149952\") " pod="openshift-ovn-kubernetes/ovnkube-node-sn5kw"
Sep 11 17:18:43.899687 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:18:43.897838    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/73dd9a34-fac4-44c9-96d3-b2c8968ed271-original-pull-secret\") pod \"global-pull-secret-syncer-fsh8r\" (UID: \"73dd9a34-fac4-44c9-96d3-b2c8968ed271\") " pod="kube-system/global-pull-secret-syncer-fsh8r"
Sep 11 17:18:43.899687 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:18:43.897841    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/bb06c328-82b1-4690-aa4a-6584fb149952-ovnkube-config\") pod \"ovnkube-node-sn5kw\" (UID: \"bb06c328-82b1-4690-aa4a-6584fb149952\") " pod="openshift-ovn-kubernetes/ovnkube-node-sn5kw"
Sep 11 17:18:43.899687 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:18:43.897861    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/337b4304-78e3-4abd-afbf-b866f7104056-konnectivity-ca\") pod \"konnectivity-agent-g6gcb\" (UID: \"337b4304-78e3-4abd-afbf-b866f7104056\") " pod="kube-system/konnectivity-agent-g6gcb"
Sep 11 17:18:43.899687 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:18:43.897895    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/bb06c328-82b1-4690-aa4a-6584fb149952-etc-openvswitch\") pod \"ovnkube-node-sn5kw\" (UID: \"bb06c328-82b1-4690-aa4a-6584fb149952\") " pod="openshift-ovn-kubernetes/ovnkube-node-sn5kw"
Sep 11 17:18:43.899687 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:18:43.897895    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/bb06c328-82b1-4690-aa4a-6584fb149952-log-socket\") pod \"ovnkube-node-sn5kw\" (UID: \"bb06c328-82b1-4690-aa4a-6584fb149952\") " pod="openshift-ovn-kubernetes/ovnkube-node-sn5kw"
Sep 11 17:18:43.899687 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:18:43.897912    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-smgzf\" (UniqueName: \"kubernetes.io/projected/daeb9a26-f9e6-4d68-bc7d-f31b5f10d621-kube-api-access-smgzf\") pod \"aws-ebs-csi-driver-node-hxl5l\" (UID: \"daeb9a26-f9e6-4d68-bc7d-f31b5f10d621\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hxl5l"
Sep 11 17:18:43.899687 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:18:43.897940    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-45kgz\" (UniqueName: \"kubernetes.io/projected/5c1ac293-cc19-4446-8244-a8ca7fc8a8f4-kube-api-access-45kgz\") pod \"network-metrics-daemon-bl4vr\" (UID: \"5c1ac293-cc19-4446-8244-a8ca7fc8a8f4\") " pod="openshift-multus/network-metrics-daemon-bl4vr"
Sep 11 17:18:43.899687 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:18:43.897968    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/9208373a-bbca-49fc-91ed-d29c0a833261-multus-cni-dir\") pod \"multus-nltl2\" (UID: \"9208373a-bbca-49fc-91ed-d29c0a833261\") " pod="openshift-multus/multus-nltl2"
Sep 11 17:18:43.899687 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:18:43.897990    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/9208373a-bbca-49fc-91ed-d29c0a833261-host-var-lib-cni-bin\") pod \"multus-nltl2\" (UID: \"9208373a-bbca-49fc-91ed-d29c0a833261\") " pod="openshift-multus/multus-nltl2"
Sep 11 17:18:43.899687 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:18:43.898010    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/bb06c328-82b1-4690-aa4a-6584fb149952-host-cni-bin\") pod \"ovnkube-node-sn5kw\" (UID: \"bb06c328-82b1-4690-aa4a-6584fb149952\") " pod="openshift-ovn-kubernetes/ovnkube-node-sn5kw"
Sep 11 17:18:43.899687 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:18:43.898041    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/2fe5e836-a8a9-402a-bce2-da7ddc7418fa-etc-systemd\") pod \"tuned-d4gm9\" (UID: \"2fe5e836-a8a9-402a-bce2-da7ddc7418fa\") " pod="openshift-cluster-node-tuning-operator/tuned-d4gm9"
Sep 11 17:18:43.900269 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:18:43.898062    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/73dd9a34-fac4-44c9-96d3-b2c8968ed271-kubelet-config\") pod \"global-pull-secret-syncer-fsh8r\" (UID: \"73dd9a34-fac4-44c9-96d3-b2c8968ed271\") " pod="kube-system/global-pull-secret-syncer-fsh8r"
Sep 11 17:18:43.900269 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:18:43.898087    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/9208373a-bbca-49fc-91ed-d29c0a833261-host-var-lib-kubelet\") pod \"multus-nltl2\" (UID: \"9208373a-bbca-49fc-91ed-d29c0a833261\") " pod="openshift-multus/multus-nltl2"
Sep 11 17:18:43.900269 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:18:43.898109    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/a1ce372a-94c9-4089-8d8b-e4dee9b7e782-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-6shpl\" (UID: \"a1ce372a-94c9-4089-8d8b-e4dee9b7e782\") " pod="openshift-multus/multus-additional-cni-plugins-6shpl"
Sep 11 17:18:43.900269 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:18:43.898131    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/9208373a-bbca-49fc-91ed-d29c0a833261-multus-conf-dir\") pod \"multus-nltl2\" (UID: \"9208373a-bbca-49fc-91ed-d29c0a833261\") " pod="openshift-multus/multus-nltl2"
Sep 11 17:18:43.900269 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:18:43.898140    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/bb06c328-82b1-4690-aa4a-6584fb149952-host-cni-bin\") pod \"ovnkube-node-sn5kw\" (UID: \"bb06c328-82b1-4690-aa4a-6584fb149952\") " pod="openshift-ovn-kubernetes/ovnkube-node-sn5kw"
Sep 11 17:18:43.900269 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:18:43.898152    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/2fe5e836-a8a9-402a-bce2-da7ddc7418fa-etc-sysctl-conf\") pod \"tuned-d4gm9\" (UID: \"2fe5e836-a8a9-402a-bce2-da7ddc7418fa\") " pod="openshift-cluster-node-tuning-operator/tuned-d4gm9"
Sep 11 17:18:43.900269 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:18:43.898191    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/9208373a-bbca-49fc-91ed-d29c0a833261-host-var-lib-kubelet\") pod \"multus-nltl2\" (UID: \"9208373a-bbca-49fc-91ed-d29c0a833261\") " pod="openshift-multus/multus-nltl2"
Sep 11 17:18:43.900269 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:18:43.898283    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/9208373a-bbca-49fc-91ed-d29c0a833261-multus-cni-dir\") pod \"multus-nltl2\" (UID: \"9208373a-bbca-49fc-91ed-d29c0a833261\") " pod="openshift-multus/multus-nltl2"
Sep 11 17:18:43.900269 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:18:43.898330    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/9208373a-bbca-49fc-91ed-d29c0a833261-host-var-lib-cni-bin\") pod \"multus-nltl2\" (UID: \"9208373a-bbca-49fc-91ed-d29c0a833261\") " pod="openshift-multus/multus-nltl2"
Sep 11 17:18:43.900269 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:18:43.898356    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/2fe5e836-a8a9-402a-bce2-da7ddc7418fa-sys\") pod \"tuned-d4gm9\" (UID: \"2fe5e836-a8a9-402a-bce2-da7ddc7418fa\") " pod="openshift-cluster-node-tuning-operator/tuned-d4gm9"
Sep 11 17:18:43.900269 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:18:43.898378    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/2fe5e836-a8a9-402a-bce2-da7ddc7418fa-lib-modules\") pod \"tuned-d4gm9\" (UID: \"2fe5e836-a8a9-402a-bce2-da7ddc7418fa\") " pod="openshift-cluster-node-tuning-operator/tuned-d4gm9"
Sep 11 17:18:43.900269 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:18:43.898403    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/66d2b8f8-a4f7-4811-8034-2523351499ee-hosts-file\") pod \"node-resolver-9v9dt\" (UID: \"66d2b8f8-a4f7-4811-8034-2523351499ee\") " pod="openshift-dns/node-resolver-9v9dt"
Sep 11 17:18:43.900269 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:18:43.898426    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-47h2q\" (UniqueName: \"kubernetes.io/projected/a1ce372a-94c9-4089-8d8b-e4dee9b7e782-kube-api-access-47h2q\") pod \"multus-additional-cni-plugins-6shpl\" (UID: \"a1ce372a-94c9-4089-8d8b-e4dee9b7e782\") " pod="openshift-multus/multus-additional-cni-plugins-6shpl"
Sep 11 17:18:43.900269 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:18:43.898432    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/9208373a-bbca-49fc-91ed-d29c0a833261-multus-conf-dir\") pod \"multus-nltl2\" (UID: \"9208373a-bbca-49fc-91ed-d29c0a833261\") " pod="openshift-multus/multus-nltl2"
Sep 11 17:18:43.900269 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:18:43.898450    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/9208373a-bbca-49fc-91ed-d29c0a833261-cni-binary-copy\") pod \"multus-nltl2\" (UID: \"9208373a-bbca-49fc-91ed-d29c0a833261\") " pod="openshift-multus/multus-nltl2"
Sep 11 17:18:43.900269 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:18:43.898489    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/66d2b8f8-a4f7-4811-8034-2523351499ee-hosts-file\") pod \"node-resolver-9v9dt\" (UID: \"66d2b8f8-a4f7-4811-8034-2523351499ee\") " pod="openshift-dns/node-resolver-9v9dt"
Sep 11 17:18:43.900269 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:18:43.898492    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2fe5e836-a8a9-402a-bce2-da7ddc7418fa-etc-kubernetes\") pod \"tuned-d4gm9\" (UID: \"2fe5e836-a8a9-402a-bce2-da7ddc7418fa\") " pod="openshift-cluster-node-tuning-operator/tuned-d4gm9"
Sep 11 17:18:43.900855 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:18:43.898537    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5c1ac293-cc19-4446-8244-a8ca7fc8a8f4-metrics-certs\") pod \"network-metrics-daemon-bl4vr\" (UID: \"5c1ac293-cc19-4446-8244-a8ca7fc8a8f4\") " pod="openshift-multus/network-metrics-daemon-bl4vr"
Sep 11 17:18:43.900855 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:18:43.898607    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/337b4304-78e3-4abd-afbf-b866f7104056-konnectivity-ca\") pod \"konnectivity-agent-g6gcb\" (UID: \"337b4304-78e3-4abd-afbf-b866f7104056\") " pod="kube-system/konnectivity-agent-g6gcb"
Sep 11 17:18:43.900855 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:18:43.899135    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/a1ce372a-94c9-4089-8d8b-e4dee9b7e782-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-6shpl\" (UID: \"a1ce372a-94c9-4089-8d8b-e4dee9b7e782\") " pod="openshift-multus/multus-additional-cni-plugins-6shpl"
Sep 11 17:18:43.900855 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:18:43.899151    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/9208373a-bbca-49fc-91ed-d29c0a833261-cni-binary-copy\") pod \"multus-nltl2\" (UID: \"9208373a-bbca-49fc-91ed-d29c0a833261\") " pod="openshift-multus/multus-nltl2"
Sep 11 17:18:43.900855 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:18:43.900801    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/bb06c328-82b1-4690-aa4a-6584fb149952-ovn-node-metrics-cert\") pod \"ovnkube-node-sn5kw\" (UID: \"bb06c328-82b1-4690-aa4a-6584fb149952\") " pod="openshift-ovn-kubernetes/ovnkube-node-sn5kw"
Sep 11 17:18:43.901063 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:18:43.901043    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/337b4304-78e3-4abd-afbf-b866f7104056-agent-certs\") pod \"konnectivity-agent-g6gcb\" (UID: \"337b4304-78e3-4abd-afbf-b866f7104056\") " pod="kube-system/konnectivity-agent-g6gcb"
Sep 11 17:18:43.910126 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:18:43.910093    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-smgzf\" (UniqueName: \"kubernetes.io/projected/daeb9a26-f9e6-4d68-bc7d-f31b5f10d621-kube-api-access-smgzf\") pod \"aws-ebs-csi-driver-node-hxl5l\" (UID: \"daeb9a26-f9e6-4d68-bc7d-f31b5f10d621\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hxl5l"
Sep 11 17:18:43.910259 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:18:43.910210    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-f8tfp\" (UniqueName: \"kubernetes.io/projected/9208373a-bbca-49fc-91ed-d29c0a833261-kube-api-access-f8tfp\") pod \"multus-nltl2\" (UID: \"9208373a-bbca-49fc-91ed-d29c0a833261\") " pod="openshift-multus/multus-nltl2"
Sep 11 17:18:43.910564 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:18:43.910541    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2bn2z\" (UniqueName: \"kubernetes.io/projected/d64f20b5-ec23-4fdd-b8cc-ec63c443faa3-kube-api-access-2bn2z\") pod \"iptables-alerter-vmxwz\" (UID: \"d64f20b5-ec23-4fdd-b8cc-ec63c443faa3\") " pod="openshift-network-operator/iptables-alerter-vmxwz"
Sep 11 17:18:43.910674 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:18:43.910566    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nbw97\" (UniqueName: \"kubernetes.io/projected/66d2b8f8-a4f7-4811-8034-2523351499ee-kube-api-access-nbw97\") pod \"node-resolver-9v9dt\" (UID: \"66d2b8f8-a4f7-4811-8034-2523351499ee\") " pod="openshift-dns/node-resolver-9v9dt"
Sep 11 17:18:43.910770 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:18:43.910754    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-47h2q\" (UniqueName: \"kubernetes.io/projected/a1ce372a-94c9-4089-8d8b-e4dee9b7e782-kube-api-access-47h2q\") pod \"multus-additional-cni-plugins-6shpl\" (UID: \"a1ce372a-94c9-4089-8d8b-e4dee9b7e782\") " pod="openshift-multus/multus-additional-cni-plugins-6shpl"
Sep 11 17:18:43.911282 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:18:43.911265    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8x4f2\" (UniqueName: \"kubernetes.io/projected/bb06c328-82b1-4690-aa4a-6584fb149952-kube-api-access-8x4f2\") pod \"ovnkube-node-sn5kw\" (UID: \"bb06c328-82b1-4690-aa4a-6584fb149952\") " pod="openshift-ovn-kubernetes/ovnkube-node-sn5kw"
Sep 11 17:18:43.918161 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:18:43.918141    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-g6gcb"
Sep 11 17:18:43.981925 ip-10-0-141-139 kubenswrapper[2577]: W0911 17:18:43.981891    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod628f3dc115ebc1acc6ed7da71915e638.slice/crio-9e2a33d81f13e4e0b29153d4a0baeefba1f530f9434f5754635f5afde9aa7853 WatchSource:0}: Error finding container 9e2a33d81f13e4e0b29153d4a0baeefba1f530f9434f5754635f5afde9aa7853: Status 404 returned error can't find the container with id 9e2a33d81f13e4e0b29153d4a0baeefba1f530f9434f5754635f5afde9aa7853
Sep 11 17:18:43.985264 ip-10-0-141-139 kubenswrapper[2577]: W0911 17:18:43.984480    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8b76f1b2a16574ce38927f7f5da97679.slice/crio-34c0bbbcf87bc00c0ae06fa4572a57e31830daf2393222626c4fc7cd694fb661 WatchSource:0}: Error finding container 34c0bbbcf87bc00c0ae06fa4572a57e31830daf2393222626c4fc7cd694fb661: Status 404 returned error can't find the container with id 34c0bbbcf87bc00c0ae06fa4572a57e31830daf2393222626c4fc7cd694fb661
Sep 11 17:18:43.988186 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:18:43.988164    2577 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 11 17:18:43.999146 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:18:43.999121    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/c5878f27-a359-47d1-b223-08f4bd829380-host\") pod \"node-ca-k5j4d\" (UID: \"c5878f27-a359-47d1-b223-08f4bd829380\") " pod="openshift-image-registry/node-ca-k5j4d"
Sep 11 17:18:43.999290 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:18:43.999167    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/2fe5e836-a8a9-402a-bce2-da7ddc7418fa-etc-sysctl-d\") pod \"tuned-d4gm9\" (UID: \"2fe5e836-a8a9-402a-bce2-da7ddc7418fa\") " pod="openshift-cluster-node-tuning-operator/tuned-d4gm9"
Sep 11 17:18:43.999290 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:18:43.999186    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/2fe5e836-a8a9-402a-bce2-da7ddc7418fa-run\") pod \"tuned-d4gm9\" (UID: \"2fe5e836-a8a9-402a-bce2-da7ddc7418fa\") " pod="openshift-cluster-node-tuning-operator/tuned-d4gm9"
Sep 11 17:18:43.999290 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:18:43.999243    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/2fe5e836-a8a9-402a-bce2-da7ddc7418fa-run\") pod \"tuned-d4gm9\" (UID: \"2fe5e836-a8a9-402a-bce2-da7ddc7418fa\") " pod="openshift-cluster-node-tuning-operator/tuned-d4gm9"
Sep 11 17:18:43.999290 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:18:43.999243    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/c5878f27-a359-47d1-b223-08f4bd829380-host\") pod \"node-ca-k5j4d\" (UID: \"c5878f27-a359-47d1-b223-08f4bd829380\") " pod="openshift-image-registry/node-ca-k5j4d"
Sep 11 17:18:43.999449 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:18:43.999289    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/c5878f27-a359-47d1-b223-08f4bd829380-serviceca\") pod \"node-ca-k5j4d\" (UID: \"c5878f27-a359-47d1-b223-08f4bd829380\") " pod="openshift-image-registry/node-ca-k5j4d"
Sep 11 17:18:43.999449 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:18:43.999318    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/2fe5e836-a8a9-402a-bce2-da7ddc7418fa-etc-sysctl-d\") pod \"tuned-d4gm9\" (UID: \"2fe5e836-a8a9-402a-bce2-da7ddc7418fa\") " pod="openshift-cluster-node-tuning-operator/tuned-d4gm9"
Sep 11 17:18:43.999449 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:18:43.999320    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hhmg9\" (UniqueName: \"kubernetes.io/projected/c5878f27-a359-47d1-b223-08f4bd829380-kube-api-access-hhmg9\") pod \"node-ca-k5j4d\" (UID: \"c5878f27-a359-47d1-b223-08f4bd829380\") " pod="openshift-image-registry/node-ca-k5j4d"
Sep 11 17:18:43.999449 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:18:43.999388    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h2trs\" (UniqueName: \"kubernetes.io/projected/1e5d9e0c-7bc0-4fcf-8845-8873df5b1bee-kube-api-access-h2trs\") pod \"network-check-target-xfn47\" (UID: \"1e5d9e0c-7bc0-4fcf-8845-8873df5b1bee\") " pod="openshift-network-diagnostics/network-check-target-xfn47"
Sep 11 17:18:43.999449 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:18:43.999425    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/2fe5e836-a8a9-402a-bce2-da7ddc7418fa-etc-tuned\") pod \"tuned-d4gm9\" (UID: \"2fe5e836-a8a9-402a-bce2-da7ddc7418fa\") " pod="openshift-cluster-node-tuning-operator/tuned-d4gm9"
Sep 11 17:18:43.999449 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:18:43.999447    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/73dd9a34-fac4-44c9-96d3-b2c8968ed271-dbus\") pod \"global-pull-secret-syncer-fsh8r\" (UID: \"73dd9a34-fac4-44c9-96d3-b2c8968ed271\") " pod="kube-system/global-pull-secret-syncer-fsh8r"
Sep 11 17:18:43.999746 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:18:43.999473    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t89tx\" (UniqueName: \"kubernetes.io/projected/2fe5e836-a8a9-402a-bce2-da7ddc7418fa-kube-api-access-t89tx\") pod \"tuned-d4gm9\" (UID: \"2fe5e836-a8a9-402a-bce2-da7ddc7418fa\") " pod="openshift-cluster-node-tuning-operator/tuned-d4gm9"
Sep 11 17:18:43.999746 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:18:43.999494    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/2fe5e836-a8a9-402a-bce2-da7ddc7418fa-tmp\") pod \"tuned-d4gm9\" (UID: \"2fe5e836-a8a9-402a-bce2-da7ddc7418fa\") " pod="openshift-cluster-node-tuning-operator/tuned-d4gm9"
Sep 11 17:18:43.999746 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:18:43.999540    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/2fe5e836-a8a9-402a-bce2-da7ddc7418fa-etc-modprobe-d\") pod \"tuned-d4gm9\" (UID: \"2fe5e836-a8a9-402a-bce2-da7ddc7418fa\") " pod="openshift-cluster-node-tuning-operator/tuned-d4gm9"
Sep 11 17:18:43.999746 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:18:43.999561    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/2fe5e836-a8a9-402a-bce2-da7ddc7418fa-host\") pod \"tuned-d4gm9\" (UID: \"2fe5e836-a8a9-402a-bce2-da7ddc7418fa\") " pod="openshift-cluster-node-tuning-operator/tuned-d4gm9"
Sep 11 17:18:43.999746 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:18:43.999564    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/73dd9a34-fac4-44c9-96d3-b2c8968ed271-dbus\") pod \"global-pull-secret-syncer-fsh8r\" (UID: \"73dd9a34-fac4-44c9-96d3-b2c8968ed271\") " pod="kube-system/global-pull-secret-syncer-fsh8r"
Sep 11 17:18:43.999746 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:18:43.999615    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/73dd9a34-fac4-44c9-96d3-b2c8968ed271-original-pull-secret\") pod \"global-pull-secret-syncer-fsh8r\" (UID: \"73dd9a34-fac4-44c9-96d3-b2c8968ed271\") " pod="kube-system/global-pull-secret-syncer-fsh8r"
Sep 11 17:18:43.999746 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:18:43.999644    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/2fe5e836-a8a9-402a-bce2-da7ddc7418fa-host\") pod \"tuned-d4gm9\" (UID: \"2fe5e836-a8a9-402a-bce2-da7ddc7418fa\") " pod="openshift-cluster-node-tuning-operator/tuned-d4gm9"
Sep 11 17:18:43.999746 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:18:43.999675    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-45kgz\" (UniqueName: \"kubernetes.io/projected/5c1ac293-cc19-4446-8244-a8ca7fc8a8f4-kube-api-access-45kgz\") pod \"network-metrics-daemon-bl4vr\" (UID: \"5c1ac293-cc19-4446-8244-a8ca7fc8a8f4\") " pod="openshift-multus/network-metrics-daemon-bl4vr"
Sep 11 17:18:43.999746 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:18:43.999690    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/2fe5e836-a8a9-402a-bce2-da7ddc7418fa-etc-modprobe-d\") pod \"tuned-d4gm9\" (UID: \"2fe5e836-a8a9-402a-bce2-da7ddc7418fa\") " pod="openshift-cluster-node-tuning-operator/tuned-d4gm9"
Sep 11 17:18:43.999746 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:18:43.999704    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/2fe5e836-a8a9-402a-bce2-da7ddc7418fa-etc-systemd\") pod \"tuned-d4gm9\" (UID: \"2fe5e836-a8a9-402a-bce2-da7ddc7418fa\") " pod="openshift-cluster-node-tuning-operator/tuned-d4gm9"
Sep 11 17:18:43.999746 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:18:43.999731    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/73dd9a34-fac4-44c9-96d3-b2c8968ed271-kubelet-config\") pod \"global-pull-secret-syncer-fsh8r\" (UID: \"73dd9a34-fac4-44c9-96d3-b2c8968ed271\") " pod="kube-system/global-pull-secret-syncer-fsh8r"
Sep 11 17:18:43.999746 ip-10-0-141-139 kubenswrapper[2577]: E0911 17:18:43.999739    2577 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 11 17:18:44.000399 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:18:43.999758    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/2fe5e836-a8a9-402a-bce2-da7ddc7418fa-etc-sysctl-conf\") pod \"tuned-d4gm9\" (UID: \"2fe5e836-a8a9-402a-bce2-da7ddc7418fa\") " pod="openshift-cluster-node-tuning-operator/tuned-d4gm9"
Sep 11 17:18:44.000399 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:18:43.999783    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/2fe5e836-a8a9-402a-bce2-da7ddc7418fa-sys\") pod \"tuned-d4gm9\" (UID: \"2fe5e836-a8a9-402a-bce2-da7ddc7418fa\") " pod="openshift-cluster-node-tuning-operator/tuned-d4gm9"
Sep 11 17:18:44.000399 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:18:43.999795    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/c5878f27-a359-47d1-b223-08f4bd829380-serviceca\") pod \"node-ca-k5j4d\" (UID: \"c5878f27-a359-47d1-b223-08f4bd829380\") " pod="openshift-image-registry/node-ca-k5j4d"
Sep 11 17:18:44.000399 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:18:43.999833    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/2fe5e836-a8a9-402a-bce2-da7ddc7418fa-sys\") pod \"tuned-d4gm9\" (UID: \"2fe5e836-a8a9-402a-bce2-da7ddc7418fa\") " pod="openshift-cluster-node-tuning-operator/tuned-d4gm9"
Sep 11 17:18:44.000399 ip-10-0-141-139 kubenswrapper[2577]: E0911 17:18:43.999840    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/73dd9a34-fac4-44c9-96d3-b2c8968ed271-original-pull-secret podName:73dd9a34-fac4-44c9-96d3-b2c8968ed271 nodeName:}" failed. No retries permitted until 2026-09-11 17:18:44.499791046 +0000 UTC m=+2.073011136 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/73dd9a34-fac4-44c9-96d3-b2c8968ed271-original-pull-secret") pod "global-pull-secret-syncer-fsh8r" (UID: "73dd9a34-fac4-44c9-96d3-b2c8968ed271") : object "kube-system"/"original-pull-secret" not registered
Sep 11 17:18:44.000399 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:18:43.999867    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/2fe5e836-a8a9-402a-bce2-da7ddc7418fa-etc-systemd\") pod \"tuned-d4gm9\" (UID: \"2fe5e836-a8a9-402a-bce2-da7ddc7418fa\") " pod="openshift-cluster-node-tuning-operator/tuned-d4gm9"
Sep 11 17:18:44.000399 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:18:43.999882    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/73dd9a34-fac4-44c9-96d3-b2c8968ed271-kubelet-config\") pod \"global-pull-secret-syncer-fsh8r\" (UID: \"73dd9a34-fac4-44c9-96d3-b2c8968ed271\") " pod="kube-system/global-pull-secret-syncer-fsh8r"
Sep 11 17:18:44.000399 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:18:43.999883    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/2fe5e836-a8a9-402a-bce2-da7ddc7418fa-lib-modules\") pod \"tuned-d4gm9\" (UID: \"2fe5e836-a8a9-402a-bce2-da7ddc7418fa\") " pod="openshift-cluster-node-tuning-operator/tuned-d4gm9"
Sep 11 17:18:44.000399 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:18:43.999920    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2fe5e836-a8a9-402a-bce2-da7ddc7418fa-etc-kubernetes\") pod \"tuned-d4gm9\" (UID: \"2fe5e836-a8a9-402a-bce2-da7ddc7418fa\") " pod="openshift-cluster-node-tuning-operator/tuned-d4gm9"
Sep 11 17:18:44.000399 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:18:43.999940    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5c1ac293-cc19-4446-8244-a8ca7fc8a8f4-metrics-certs\") pod \"network-metrics-daemon-bl4vr\" (UID: \"5c1ac293-cc19-4446-8244-a8ca7fc8a8f4\") " pod="openshift-multus/network-metrics-daemon-bl4vr"
Sep 11 17:18:44.000399 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:18:43.999977    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/2fe5e836-a8a9-402a-bce2-da7ddc7418fa-etc-sysconfig\") pod \"tuned-d4gm9\" (UID: \"2fe5e836-a8a9-402a-bce2-da7ddc7418fa\") " pod="openshift-cluster-node-tuning-operator/tuned-d4gm9"
Sep 11 17:18:44.000399 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:18:44.000002    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/2fe5e836-a8a9-402a-bce2-da7ddc7418fa-var-lib-kubelet\") pod \"tuned-d4gm9\" (UID: \"2fe5e836-a8a9-402a-bce2-da7ddc7418fa\") " pod="openshift-cluster-node-tuning-operator/tuned-d4gm9"
Sep 11 17:18:44.000399 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:18:44.000002    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/2fe5e836-a8a9-402a-bce2-da7ddc7418fa-etc-sysctl-conf\") pod \"tuned-d4gm9\" (UID: \"2fe5e836-a8a9-402a-bce2-da7ddc7418fa\") " pod="openshift-cluster-node-tuning-operator/tuned-d4gm9"
Sep 11 17:18:44.000399 ip-10-0-141-139 kubenswrapper[2577]: E0911 17:18:44.000045    2577 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 11 17:18:44.000399 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:18:44.000101    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/2fe5e836-a8a9-402a-bce2-da7ddc7418fa-lib-modules\") pod \"tuned-d4gm9\" (UID: \"2fe5e836-a8a9-402a-bce2-da7ddc7418fa\") " pod="openshift-cluster-node-tuning-operator/tuned-d4gm9"
Sep 11 17:18:44.000399 ip-10-0-141-139 kubenswrapper[2577]: E0911 17:18:44.000114    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5c1ac293-cc19-4446-8244-a8ca7fc8a8f4-metrics-certs podName:5c1ac293-cc19-4446-8244-a8ca7fc8a8f4 nodeName:}" failed. No retries permitted until 2026-09-11 17:18:44.500097265 +0000 UTC m=+2.073317398 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/5c1ac293-cc19-4446-8244-a8ca7fc8a8f4-metrics-certs") pod "network-metrics-daemon-bl4vr" (UID: "5c1ac293-cc19-4446-8244-a8ca7fc8a8f4") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 11 17:18:44.001289 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:18:44.000121    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2fe5e836-a8a9-402a-bce2-da7ddc7418fa-etc-kubernetes\") pod \"tuned-d4gm9\" (UID: \"2fe5e836-a8a9-402a-bce2-da7ddc7418fa\") " pod="openshift-cluster-node-tuning-operator/tuned-d4gm9"
Sep 11 17:18:44.001289 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:18:44.000169    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/2fe5e836-a8a9-402a-bce2-da7ddc7418fa-var-lib-kubelet\") pod \"tuned-d4gm9\" (UID: \"2fe5e836-a8a9-402a-bce2-da7ddc7418fa\") " pod="openshift-cluster-node-tuning-operator/tuned-d4gm9"
Sep 11 17:18:44.001289 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:18:44.000227    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/2fe5e836-a8a9-402a-bce2-da7ddc7418fa-etc-sysconfig\") pod \"tuned-d4gm9\" (UID: \"2fe5e836-a8a9-402a-bce2-da7ddc7418fa\") " pod="openshift-cluster-node-tuning-operator/tuned-d4gm9"
Sep 11 17:18:44.002165 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:18:44.002144    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/2fe5e836-a8a9-402a-bce2-da7ddc7418fa-etc-tuned\") pod \"tuned-d4gm9\" (UID: \"2fe5e836-a8a9-402a-bce2-da7ddc7418fa\") " pod="openshift-cluster-node-tuning-operator/tuned-d4gm9"
Sep 11 17:18:44.002306 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:18:44.002173    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/2fe5e836-a8a9-402a-bce2-da7ddc7418fa-tmp\") pod \"tuned-d4gm9\" (UID: \"2fe5e836-a8a9-402a-bce2-da7ddc7418fa\") " pod="openshift-cluster-node-tuning-operator/tuned-d4gm9"
Sep 11 17:18:44.010571 ip-10-0-141-139 kubenswrapper[2577]: E0911 17:18:44.010551    2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 11 17:18:44.010571 ip-10-0-141-139 kubenswrapper[2577]: E0911 17:18:44.010573    2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 11 17:18:44.010713 ip-10-0-141-139 kubenswrapper[2577]: E0911 17:18:44.010584    2577 projected.go:196] Error preparing data for projected volume kube-api-access-h2trs for pod openshift-network-diagnostics/network-check-target-xfn47: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 11 17:18:44.010713 ip-10-0-141-139 kubenswrapper[2577]: E0911 17:18:44.010633    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/1e5d9e0c-7bc0-4fcf-8845-8873df5b1bee-kube-api-access-h2trs podName:1e5d9e0c-7bc0-4fcf-8845-8873df5b1bee nodeName:}" failed. No retries permitted until 2026-09-11 17:18:44.510619887 +0000 UTC m=+2.083839976 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-h2trs" (UniqueName: "kubernetes.io/projected/1e5d9e0c-7bc0-4fcf-8845-8873df5b1bee-kube-api-access-h2trs") pod "network-check-target-xfn47" (UID: "1e5d9e0c-7bc0-4fcf-8845-8873df5b1bee") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 11 17:18:44.014039 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:18:44.013974    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-45kgz\" (UniqueName: \"kubernetes.io/projected/5c1ac293-cc19-4446-8244-a8ca7fc8a8f4-kube-api-access-45kgz\") pod \"network-metrics-daemon-bl4vr\" (UID: \"5c1ac293-cc19-4446-8244-a8ca7fc8a8f4\") " pod="openshift-multus/network-metrics-daemon-bl4vr"
Sep 11 17:18:44.014542 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:18:44.014526    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-t89tx\" (UniqueName: \"kubernetes.io/projected/2fe5e836-a8a9-402a-bce2-da7ddc7418fa-kube-api-access-t89tx\") pod \"tuned-d4gm9\" (UID: \"2fe5e836-a8a9-402a-bce2-da7ddc7418fa\") " pod="openshift-cluster-node-tuning-operator/tuned-d4gm9"
Sep 11 17:18:44.015050 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:18:44.015037    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hhmg9\" (UniqueName: \"kubernetes.io/projected/c5878f27-a359-47d1-b223-08f4bd829380-kube-api-access-hhmg9\") pod \"node-ca-k5j4d\" (UID: \"c5878f27-a359-47d1-b223-08f4bd829380\") " pod="openshift-image-registry/node-ca-k5j4d"
Sep 11 17:18:44.104834 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:18:44.104795    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-nltl2"
Sep 11 17:18:44.115963 ip-10-0-141-139 kubenswrapper[2577]: W0911 17:18:44.115936    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9208373a_bbca_49fc_91ed_d29c0a833261.slice/crio-cec7b094a11dd4aa5a25c58b1fc508de4cc24aa982b48a233f5570aaab9555b3 WatchSource:0}: Error finding container cec7b094a11dd4aa5a25c58b1fc508de4cc24aa982b48a233f5570aaab9555b3: Status 404 returned error can't find the container with id cec7b094a11dd4aa5a25c58b1fc508de4cc24aa982b48a233f5570aaab9555b3
Sep 11 17:18:44.118191 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:18:44.118157    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-sn5kw"
Sep 11 17:18:44.127434 ip-10-0-141-139 kubenswrapper[2577]: W0911 17:18:44.127407    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbb06c328_82b1_4690_aa4a_6584fb149952.slice/crio-b62474850a5514ef233b59119c327419eaa7318b09714fe72c0e528906b4977c WatchSource:0}: Error finding container b62474850a5514ef233b59119c327419eaa7318b09714fe72c0e528906b4977c: Status 404 returned error can't find the container with id b62474850a5514ef233b59119c327419eaa7318b09714fe72c0e528906b4977c
Sep 11 17:18:44.138859 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:18:44.138830    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hxl5l"
Sep 11 17:18:44.144265 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:18:44.144242    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-9v9dt"
Sep 11 17:18:44.149096 ip-10-0-141-139 kubenswrapper[2577]: W0911 17:18:44.149069    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddaeb9a26_f9e6_4d68_bc7d_f31b5f10d621.slice/crio-380bc78bfbf5bd0ca9e51b40e44262fd0f60d217c599baf749c4bd6d8bb65ce4 WatchSource:0}: Error finding container 380bc78bfbf5bd0ca9e51b40e44262fd0f60d217c599baf749c4bd6d8bb65ce4: Status 404 returned error can't find the container with id 380bc78bfbf5bd0ca9e51b40e44262fd0f60d217c599baf749c4bd6d8bb65ce4
Sep 11 17:18:44.154517 ip-10-0-141-139 kubenswrapper[2577]: W0911 17:18:44.154480    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod66d2b8f8_a4f7_4811_8034_2523351499ee.slice/crio-83ef3b6787c3e27674c57717c10faa9849cc3fd9f170a3e5fcbb4cb127870a6d WatchSource:0}: Error finding container 83ef3b6787c3e27674c57717c10faa9849cc3fd9f170a3e5fcbb4cb127870a6d: Status 404 returned error can't find the container with id 83ef3b6787c3e27674c57717c10faa9849cc3fd9f170a3e5fcbb4cb127870a6d
Sep 11 17:18:44.168849 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:18:44.168817    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-6shpl"
Sep 11 17:18:44.179693 ip-10-0-141-139 kubenswrapper[2577]: W0911 17:18:44.179666    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda1ce372a_94c9_4089_8d8b_e4dee9b7e782.slice/crio-1c0256c5ff62bf8116ea523ec3f784d583778dee9fa02e75e65d4f454e037b79 WatchSource:0}: Error finding container 1c0256c5ff62bf8116ea523ec3f784d583778dee9fa02e75e65d4f454e037b79: Status 404 returned error can't find the container with id 1c0256c5ff62bf8116ea523ec3f784d583778dee9fa02e75e65d4f454e037b79
Sep 11 17:18:44.192446 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:18:44.192419    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-vmxwz"
Sep 11 17:18:44.201675 ip-10-0-141-139 kubenswrapper[2577]: W0911 17:18:44.201647    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd64f20b5_ec23_4fdd_b8cc_ec63c443faa3.slice/crio-aca76955c05ffc856eeddfe8f7bcb268e496cfef0f311c5cd47617a4b0f7279a WatchSource:0}: Error finding container aca76955c05ffc856eeddfe8f7bcb268e496cfef0f311c5cd47617a4b0f7279a: Status 404 returned error can't find the container with id aca76955c05ffc856eeddfe8f7bcb268e496cfef0f311c5cd47617a4b0f7279a
Sep 11 17:18:44.230436 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:18:44.230405    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-d4gm9"
Sep 11 17:18:44.235453 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:18:44.235432    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-k5j4d"
Sep 11 17:18:44.256840 ip-10-0-141-139 kubenswrapper[2577]: W0911 17:18:44.256811    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc5878f27_a359_47d1_b223_08f4bd829380.slice/crio-63374893d4e99d33cad273c01caf10bb605d52e77c756b824f671880bb7af1eb WatchSource:0}: Error finding container 63374893d4e99d33cad273c01caf10bb605d52e77c756b824f671880bb7af1eb: Status 404 returned error can't find the container with id 63374893d4e99d33cad273c01caf10bb605d52e77c756b824f671880bb7af1eb
Sep 11 17:18:44.257550 ip-10-0-141-139 kubenswrapper[2577]: W0911 17:18:44.257502    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod337b4304_78e3_4abd_afbf_b866f7104056.slice/crio-a03397f9c63c99f41bccf3c405adbe137299ba76d105ab57a792a5d611452a3c WatchSource:0}: Error finding container a03397f9c63c99f41bccf3c405adbe137299ba76d105ab57a792a5d611452a3c: Status 404 returned error can't find the container with id a03397f9c63c99f41bccf3c405adbe137299ba76d105ab57a792a5d611452a3c
Sep 11 17:18:44.259355 ip-10-0-141-139 kubenswrapper[2577]: W0911 17:18:44.259334    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2fe5e836_a8a9_402a_bce2_da7ddc7418fa.slice/crio-fdda03b2d6020b176978192463763a1d511ab917e17e2eb8091582c33ee79749 WatchSource:0}: Error finding container fdda03b2d6020b176978192463763a1d511ab917e17e2eb8091582c33ee79749: Status 404 returned error can't find the container with id fdda03b2d6020b176978192463763a1d511ab917e17e2eb8091582c33ee79749
Sep 11 17:18:44.503246 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:18:44.503150    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/73dd9a34-fac4-44c9-96d3-b2c8968ed271-original-pull-secret\") pod \"global-pull-secret-syncer-fsh8r\" (UID: \"73dd9a34-fac4-44c9-96d3-b2c8968ed271\") " pod="kube-system/global-pull-secret-syncer-fsh8r"
Sep 11 17:18:44.503246 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:18:44.503194    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5c1ac293-cc19-4446-8244-a8ca7fc8a8f4-metrics-certs\") pod \"network-metrics-daemon-bl4vr\" (UID: \"5c1ac293-cc19-4446-8244-a8ca7fc8a8f4\") " pod="openshift-multus/network-metrics-daemon-bl4vr"
Sep 11 17:18:44.503460 ip-10-0-141-139 kubenswrapper[2577]: E0911 17:18:44.503294    2577 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 11 17:18:44.503460 ip-10-0-141-139 kubenswrapper[2577]: E0911 17:18:44.503308    2577 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 11 17:18:44.503460 ip-10-0-141-139 kubenswrapper[2577]: E0911 17:18:44.503349    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5c1ac293-cc19-4446-8244-a8ca7fc8a8f4-metrics-certs podName:5c1ac293-cc19-4446-8244-a8ca7fc8a8f4 nodeName:}" failed. No retries permitted until 2026-09-11 17:18:45.503334251 +0000 UTC m=+3.076554342 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/5c1ac293-cc19-4446-8244-a8ca7fc8a8f4-metrics-certs") pod "network-metrics-daemon-bl4vr" (UID: "5c1ac293-cc19-4446-8244-a8ca7fc8a8f4") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 11 17:18:44.503460 ip-10-0-141-139 kubenswrapper[2577]: E0911 17:18:44.503372    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/73dd9a34-fac4-44c9-96d3-b2c8968ed271-original-pull-secret podName:73dd9a34-fac4-44c9-96d3-b2c8968ed271 nodeName:}" failed. No retries permitted until 2026-09-11 17:18:45.503354572 +0000 UTC m=+3.076574663 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/73dd9a34-fac4-44c9-96d3-b2c8968ed271-original-pull-secret") pod "global-pull-secret-syncer-fsh8r" (UID: "73dd9a34-fac4-44c9-96d3-b2c8968ed271") : object "kube-system"/"original-pull-secret" not registered
Sep 11 17:18:44.603890 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:18:44.603855    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h2trs\" (UniqueName: \"kubernetes.io/projected/1e5d9e0c-7bc0-4fcf-8845-8873df5b1bee-kube-api-access-h2trs\") pod \"network-check-target-xfn47\" (UID: \"1e5d9e0c-7bc0-4fcf-8845-8873df5b1bee\") " pod="openshift-network-diagnostics/network-check-target-xfn47"
Sep 11 17:18:44.604075 ip-10-0-141-139 kubenswrapper[2577]: E0911 17:18:44.604007    2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 11 17:18:44.604075 ip-10-0-141-139 kubenswrapper[2577]: E0911 17:18:44.604030    2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 11 17:18:44.604075 ip-10-0-141-139 kubenswrapper[2577]: E0911 17:18:44.604044    2577 projected.go:196] Error preparing data for projected volume kube-api-access-h2trs for pod openshift-network-diagnostics/network-check-target-xfn47: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 11 17:18:44.604224 ip-10-0-141-139 kubenswrapper[2577]: E0911 17:18:44.604109    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/1e5d9e0c-7bc0-4fcf-8845-8873df5b1bee-kube-api-access-h2trs podName:1e5d9e0c-7bc0-4fcf-8845-8873df5b1bee nodeName:}" failed. No retries permitted until 2026-09-11 17:18:45.604092396 +0000 UTC m=+3.177312493 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-h2trs" (UniqueName: "kubernetes.io/projected/1e5d9e0c-7bc0-4fcf-8845-8873df5b1bee-kube-api-access-h2trs") pod "network-check-target-xfn47" (UID: "1e5d9e0c-7bc0-4fcf-8845-8873df5b1bee") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 11 17:18:44.835645 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:18:44.835538    2577 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-10 17:13:44 +0000 UTC" deadline="2028-03-07 03:15:34.990483776 +0000 UTC"
Sep 11 17:18:44.835645 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:18:44.835586    2577 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13017h56m50.154903664s"
Sep 11 17:18:44.943048 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:18:44.942915    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-6shpl" event={"ID":"a1ce372a-94c9-4089-8d8b-e4dee9b7e782","Type":"ContainerStarted","Data":"1c0256c5ff62bf8116ea523ec3f784d583778dee9fa02e75e65d4f454e037b79"}
Sep 11 17:18:44.948370 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:18:44.948330    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-9v9dt" event={"ID":"66d2b8f8-a4f7-4811-8034-2523351499ee","Type":"ContainerStarted","Data":"83ef3b6787c3e27674c57717c10faa9849cc3fd9f170a3e5fcbb4cb127870a6d"}
Sep 11 17:18:44.952831 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:18:44.952794    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-sn5kw" event={"ID":"bb06c328-82b1-4690-aa4a-6584fb149952","Type":"ContainerStarted","Data":"b62474850a5514ef233b59119c327419eaa7318b09714fe72c0e528906b4977c"}
Sep 11 17:18:44.958079 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:18:44.958042    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-139.ec2.internal" event={"ID":"8b76f1b2a16574ce38927f7f5da97679","Type":"ContainerStarted","Data":"34c0bbbcf87bc00c0ae06fa4572a57e31830daf2393222626c4fc7cd694fb661"}
Sep 11 17:18:44.963053 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:18:44.963018    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hxl5l" event={"ID":"daeb9a26-f9e6-4d68-bc7d-f31b5f10d621","Type":"ContainerStarted","Data":"380bc78bfbf5bd0ca9e51b40e44262fd0f60d217c599baf749c4bd6d8bb65ce4"}
Sep 11 17:18:44.973875 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:18:44.973836    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-nltl2" event={"ID":"9208373a-bbca-49fc-91ed-d29c0a833261","Type":"ContainerStarted","Data":"cec7b094a11dd4aa5a25c58b1fc508de4cc24aa982b48a233f5570aaab9555b3"}
Sep 11 17:18:44.981159 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:18:44.981120    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-139.ec2.internal" event={"ID":"628f3dc115ebc1acc6ed7da71915e638","Type":"ContainerStarted","Data":"9e2a33d81f13e4e0b29153d4a0baeefba1f530f9434f5754635f5afde9aa7853"}
Sep 11 17:18:44.985118 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:18:44.985091    2577 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 11 17:18:44.997880 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:18:44.997648    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-g6gcb" event={"ID":"337b4304-78e3-4abd-afbf-b866f7104056","Type":"ContainerStarted","Data":"a03397f9c63c99f41bccf3c405adbe137299ba76d105ab57a792a5d611452a3c"}
Sep 11 17:18:45.007238 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:18:45.007160    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-d4gm9" event={"ID":"2fe5e836-a8a9-402a-bce2-da7ddc7418fa","Type":"ContainerStarted","Data":"fdda03b2d6020b176978192463763a1d511ab917e17e2eb8091582c33ee79749"}
Sep 11 17:18:45.021465 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:18:45.021422    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-k5j4d" event={"ID":"c5878f27-a359-47d1-b223-08f4bd829380","Type":"ContainerStarted","Data":"63374893d4e99d33cad273c01caf10bb605d52e77c756b824f671880bb7af1eb"}
Sep 11 17:18:45.036122 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:18:45.036083    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-vmxwz" event={"ID":"d64f20b5-ec23-4fdd-b8cc-ec63c443faa3","Type":"ContainerStarted","Data":"aca76955c05ffc856eeddfe8f7bcb268e496cfef0f311c5cd47617a4b0f7279a"}
Sep 11 17:18:45.512597 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:18:45.512539    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/73dd9a34-fac4-44c9-96d3-b2c8968ed271-original-pull-secret\") pod \"global-pull-secret-syncer-fsh8r\" (UID: \"73dd9a34-fac4-44c9-96d3-b2c8968ed271\") " pod="kube-system/global-pull-secret-syncer-fsh8r"
Sep 11 17:18:45.512815 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:18:45.512608    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5c1ac293-cc19-4446-8244-a8ca7fc8a8f4-metrics-certs\") pod \"network-metrics-daemon-bl4vr\" (UID: \"5c1ac293-cc19-4446-8244-a8ca7fc8a8f4\") " pod="openshift-multus/network-metrics-daemon-bl4vr"
Sep 11 17:18:45.512815 ip-10-0-141-139 kubenswrapper[2577]: E0911 17:18:45.512765    2577 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 11 17:18:45.512938 ip-10-0-141-139 kubenswrapper[2577]: E0911 17:18:45.512835    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5c1ac293-cc19-4446-8244-a8ca7fc8a8f4-metrics-certs podName:5c1ac293-cc19-4446-8244-a8ca7fc8a8f4 nodeName:}" failed. No retries permitted until 2026-09-11 17:18:47.512815616 +0000 UTC m=+5.086035723 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/5c1ac293-cc19-4446-8244-a8ca7fc8a8f4-metrics-certs") pod "network-metrics-daemon-bl4vr" (UID: "5c1ac293-cc19-4446-8244-a8ca7fc8a8f4") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 11 17:18:45.513428 ip-10-0-141-139 kubenswrapper[2577]: E0911 17:18:45.513400    2577 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 11 17:18:45.513518 ip-10-0-141-139 kubenswrapper[2577]: E0911 17:18:45.513477    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/73dd9a34-fac4-44c9-96d3-b2c8968ed271-original-pull-secret podName:73dd9a34-fac4-44c9-96d3-b2c8968ed271 nodeName:}" failed. No retries permitted until 2026-09-11 17:18:47.513455017 +0000 UTC m=+5.086675124 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/73dd9a34-fac4-44c9-96d3-b2c8968ed271-original-pull-secret") pod "global-pull-secret-syncer-fsh8r" (UID: "73dd9a34-fac4-44c9-96d3-b2c8968ed271") : object "kube-system"/"original-pull-secret" not registered
Sep 11 17:18:45.613919 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:18:45.613877    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h2trs\" (UniqueName: \"kubernetes.io/projected/1e5d9e0c-7bc0-4fcf-8845-8873df5b1bee-kube-api-access-h2trs\") pod \"network-check-target-xfn47\" (UID: \"1e5d9e0c-7bc0-4fcf-8845-8873df5b1bee\") " pod="openshift-network-diagnostics/network-check-target-xfn47"
Sep 11 17:18:45.614129 ip-10-0-141-139 kubenswrapper[2577]: E0911 17:18:45.614080    2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 11 17:18:45.614129 ip-10-0-141-139 kubenswrapper[2577]: E0911 17:18:45.614101    2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 11 17:18:45.614129 ip-10-0-141-139 kubenswrapper[2577]: E0911 17:18:45.614114    2577 projected.go:196] Error preparing data for projected volume kube-api-access-h2trs for pod openshift-network-diagnostics/network-check-target-xfn47: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 11 17:18:45.614312 ip-10-0-141-139 kubenswrapper[2577]: E0911 17:18:45.614180    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/1e5d9e0c-7bc0-4fcf-8845-8873df5b1bee-kube-api-access-h2trs podName:1e5d9e0c-7bc0-4fcf-8845-8873df5b1bee nodeName:}" failed. No retries permitted until 2026-09-11 17:18:47.61415938 +0000 UTC m=+5.187379477 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-h2trs" (UniqueName: "kubernetes.io/projected/1e5d9e0c-7bc0-4fcf-8845-8873df5b1bee-kube-api-access-h2trs") pod "network-check-target-xfn47" (UID: "1e5d9e0c-7bc0-4fcf-8845-8873df5b1bee") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 11 17:18:45.836897 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:18:45.836762    2577 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-10 17:13:44 +0000 UTC" deadline="2028-03-07 21:24:22.596352366 +0000 UTC"
Sep 11 17:18:45.836897 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:18:45.836833    2577 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13036h5m36.759524221s"
Sep 11 17:18:45.920076 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:18:45.920037    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fsh8r"
Sep 11 17:18:45.920280 ip-10-0-141-139 kubenswrapper[2577]: E0911 17:18:45.920200    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-fsh8r" podUID="73dd9a34-fac4-44c9-96d3-b2c8968ed271"
Sep 11 17:18:45.920837 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:18:45.920692    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bl4vr"
Sep 11 17:18:45.920837 ip-10-0-141-139 kubenswrapper[2577]: E0911 17:18:45.920818    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bl4vr" podUID="5c1ac293-cc19-4446-8244-a8ca7fc8a8f4"
Sep 11 17:18:45.921005 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:18:45.920884    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xfn47"
Sep 11 17:18:45.921005 ip-10-0-141-139 kubenswrapper[2577]: E0911 17:18:45.920956    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xfn47" podUID="1e5d9e0c-7bc0-4fcf-8845-8873df5b1bee"
Sep 11 17:18:46.397905 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:18:46.397869    2577 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 11 17:18:46.544723 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:18:46.544035    2577 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 11 17:18:47.533591 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:18:47.533512    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5c1ac293-cc19-4446-8244-a8ca7fc8a8f4-metrics-certs\") pod \"network-metrics-daemon-bl4vr\" (UID: \"5c1ac293-cc19-4446-8244-a8ca7fc8a8f4\") " pod="openshift-multus/network-metrics-daemon-bl4vr"
Sep 11 17:18:47.534297 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:18:47.533656    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/73dd9a34-fac4-44c9-96d3-b2c8968ed271-original-pull-secret\") pod \"global-pull-secret-syncer-fsh8r\" (UID: \"73dd9a34-fac4-44c9-96d3-b2c8968ed271\") " pod="kube-system/global-pull-secret-syncer-fsh8r"
Sep 11 17:18:47.534297 ip-10-0-141-139 kubenswrapper[2577]: E0911 17:18:47.533817    2577 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 11 17:18:47.534297 ip-10-0-141-139 kubenswrapper[2577]: E0911 17:18:47.533887    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/73dd9a34-fac4-44c9-96d3-b2c8968ed271-original-pull-secret podName:73dd9a34-fac4-44c9-96d3-b2c8968ed271 nodeName:}" failed. No retries permitted until 2026-09-11 17:18:51.533867424 +0000 UTC m=+9.107087533 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/73dd9a34-fac4-44c9-96d3-b2c8968ed271-original-pull-secret") pod "global-pull-secret-syncer-fsh8r" (UID: "73dd9a34-fac4-44c9-96d3-b2c8968ed271") : object "kube-system"/"original-pull-secret" not registered
Sep 11 17:18:47.534673 ip-10-0-141-139 kubenswrapper[2577]: E0911 17:18:47.534652    2577 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 11 17:18:47.534765 ip-10-0-141-139 kubenswrapper[2577]: E0911 17:18:47.534725    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5c1ac293-cc19-4446-8244-a8ca7fc8a8f4-metrics-certs podName:5c1ac293-cc19-4446-8244-a8ca7fc8a8f4 nodeName:}" failed. No retries permitted until 2026-09-11 17:18:51.534707483 +0000 UTC m=+9.107927591 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/5c1ac293-cc19-4446-8244-a8ca7fc8a8f4-metrics-certs") pod "network-metrics-daemon-bl4vr" (UID: "5c1ac293-cc19-4446-8244-a8ca7fc8a8f4") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 11 17:18:47.634452 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:18:47.634358    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h2trs\" (UniqueName: \"kubernetes.io/projected/1e5d9e0c-7bc0-4fcf-8845-8873df5b1bee-kube-api-access-h2trs\") pod \"network-check-target-xfn47\" (UID: \"1e5d9e0c-7bc0-4fcf-8845-8873df5b1bee\") " pod="openshift-network-diagnostics/network-check-target-xfn47"
Sep 11 17:18:47.634702 ip-10-0-141-139 kubenswrapper[2577]: E0911 17:18:47.634569    2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 11 17:18:47.634702 ip-10-0-141-139 kubenswrapper[2577]: E0911 17:18:47.634603    2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 11 17:18:47.634702 ip-10-0-141-139 kubenswrapper[2577]: E0911 17:18:47.634619    2577 projected.go:196] Error preparing data for projected volume kube-api-access-h2trs for pod openshift-network-diagnostics/network-check-target-xfn47: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 11 17:18:47.634702 ip-10-0-141-139 kubenswrapper[2577]: E0911 17:18:47.634682    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/1e5d9e0c-7bc0-4fcf-8845-8873df5b1bee-kube-api-access-h2trs podName:1e5d9e0c-7bc0-4fcf-8845-8873df5b1bee nodeName:}" failed. No retries permitted until 2026-09-11 17:18:51.634661418 +0000 UTC m=+9.207881526 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-h2trs" (UniqueName: "kubernetes.io/projected/1e5d9e0c-7bc0-4fcf-8845-8873df5b1bee-kube-api-access-h2trs") pod "network-check-target-xfn47" (UID: "1e5d9e0c-7bc0-4fcf-8845-8873df5b1bee") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 11 17:18:47.921527 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:18:47.920561    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fsh8r"
Sep 11 17:18:47.921527 ip-10-0-141-139 kubenswrapper[2577]: E0911 17:18:47.920707    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-fsh8r" podUID="73dd9a34-fac4-44c9-96d3-b2c8968ed271"
Sep 11 17:18:47.921527 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:18:47.921113    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bl4vr"
Sep 11 17:18:47.921527 ip-10-0-141-139 kubenswrapper[2577]: E0911 17:18:47.921230    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bl4vr" podUID="5c1ac293-cc19-4446-8244-a8ca7fc8a8f4"
Sep 11 17:18:47.921527 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:18:47.921285    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xfn47"
Sep 11 17:18:47.921527 ip-10-0-141-139 kubenswrapper[2577]: E0911 17:18:47.921378    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xfn47" podUID="1e5d9e0c-7bc0-4fcf-8845-8873df5b1bee"
Sep 11 17:18:49.921299 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:18:49.920467    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fsh8r"
Sep 11 17:18:49.921299 ip-10-0-141-139 kubenswrapper[2577]: E0911 17:18:49.920641    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-fsh8r" podUID="73dd9a34-fac4-44c9-96d3-b2c8968ed271"
Sep 11 17:18:49.921299 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:18:49.920734    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bl4vr"
Sep 11 17:18:49.921299 ip-10-0-141-139 kubenswrapper[2577]: E0911 17:18:49.920819    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bl4vr" podUID="5c1ac293-cc19-4446-8244-a8ca7fc8a8f4"
Sep 11 17:18:49.921299 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:18:49.920875    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xfn47"
Sep 11 17:18:49.921299 ip-10-0-141-139 kubenswrapper[2577]: E0911 17:18:49.920941    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xfn47" podUID="1e5d9e0c-7bc0-4fcf-8845-8873df5b1bee"
Sep 11 17:18:51.573971 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:18:51.573218    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/73dd9a34-fac4-44c9-96d3-b2c8968ed271-original-pull-secret\") pod \"global-pull-secret-syncer-fsh8r\" (UID: \"73dd9a34-fac4-44c9-96d3-b2c8968ed271\") " pod="kube-system/global-pull-secret-syncer-fsh8r"
Sep 11 17:18:51.573971 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:18:51.573280    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5c1ac293-cc19-4446-8244-a8ca7fc8a8f4-metrics-certs\") pod \"network-metrics-daemon-bl4vr\" (UID: \"5c1ac293-cc19-4446-8244-a8ca7fc8a8f4\") " pod="openshift-multus/network-metrics-daemon-bl4vr"
Sep 11 17:18:51.573971 ip-10-0-141-139 kubenswrapper[2577]: E0911 17:18:51.573399    2577 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 11 17:18:51.573971 ip-10-0-141-139 kubenswrapper[2577]: E0911 17:18:51.573455    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5c1ac293-cc19-4446-8244-a8ca7fc8a8f4-metrics-certs podName:5c1ac293-cc19-4446-8244-a8ca7fc8a8f4 nodeName:}" failed. No retries permitted until 2026-09-11 17:18:59.573437046 +0000 UTC m=+17.146657152 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/5c1ac293-cc19-4446-8244-a8ca7fc8a8f4-metrics-certs") pod "network-metrics-daemon-bl4vr" (UID: "5c1ac293-cc19-4446-8244-a8ca7fc8a8f4") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 11 17:18:51.573971 ip-10-0-141-139 kubenswrapper[2577]: E0911 17:18:51.573882    2577 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 11 17:18:51.573971 ip-10-0-141-139 kubenswrapper[2577]: E0911 17:18:51.573941    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/73dd9a34-fac4-44c9-96d3-b2c8968ed271-original-pull-secret podName:73dd9a34-fac4-44c9-96d3-b2c8968ed271 nodeName:}" failed. No retries permitted until 2026-09-11 17:18:59.573912173 +0000 UTC m=+17.147132279 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/73dd9a34-fac4-44c9-96d3-b2c8968ed271-original-pull-secret") pod "global-pull-secret-syncer-fsh8r" (UID: "73dd9a34-fac4-44c9-96d3-b2c8968ed271") : object "kube-system"/"original-pull-secret" not registered
Sep 11 17:18:51.674443 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:18:51.674339    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h2trs\" (UniqueName: \"kubernetes.io/projected/1e5d9e0c-7bc0-4fcf-8845-8873df5b1bee-kube-api-access-h2trs\") pod \"network-check-target-xfn47\" (UID: \"1e5d9e0c-7bc0-4fcf-8845-8873df5b1bee\") " pod="openshift-network-diagnostics/network-check-target-xfn47"
Sep 11 17:18:51.674650 ip-10-0-141-139 kubenswrapper[2577]: E0911 17:18:51.674580    2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 11 17:18:51.674650 ip-10-0-141-139 kubenswrapper[2577]: E0911 17:18:51.674609    2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 11 17:18:51.674650 ip-10-0-141-139 kubenswrapper[2577]: E0911 17:18:51.674623    2577 projected.go:196] Error preparing data for projected volume kube-api-access-h2trs for pod openshift-network-diagnostics/network-check-target-xfn47: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 11 17:18:51.674802 ip-10-0-141-139 kubenswrapper[2577]: E0911 17:18:51.674688    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/1e5d9e0c-7bc0-4fcf-8845-8873df5b1bee-kube-api-access-h2trs podName:1e5d9e0c-7bc0-4fcf-8845-8873df5b1bee nodeName:}" failed. No retries permitted until 2026-09-11 17:18:59.674668222 +0000 UTC m=+17.247888327 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-h2trs" (UniqueName: "kubernetes.io/projected/1e5d9e0c-7bc0-4fcf-8845-8873df5b1bee-kube-api-access-h2trs") pod "network-check-target-xfn47" (UID: "1e5d9e0c-7bc0-4fcf-8845-8873df5b1bee") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 11 17:18:51.920524 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:18:51.920473    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fsh8r"
Sep 11 17:18:51.920524 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:18:51.920500    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bl4vr"
Sep 11 17:18:51.920792 ip-10-0-141-139 kubenswrapper[2577]: E0911 17:18:51.920630    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-fsh8r" podUID="73dd9a34-fac4-44c9-96d3-b2c8968ed271"
Sep 11 17:18:51.920792 ip-10-0-141-139 kubenswrapper[2577]: E0911 17:18:51.920675    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bl4vr" podUID="5c1ac293-cc19-4446-8244-a8ca7fc8a8f4"
Sep 11 17:18:51.920792 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:18:51.920720    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xfn47"
Sep 11 17:18:51.920932 ip-10-0-141-139 kubenswrapper[2577]: E0911 17:18:51.920799    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xfn47" podUID="1e5d9e0c-7bc0-4fcf-8845-8873df5b1bee"
Sep 11 17:18:53.920067 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:18:53.920025    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fsh8r"
Sep 11 17:18:53.920555 ip-10-0-141-139 kubenswrapper[2577]: E0911 17:18:53.920157    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-fsh8r" podUID="73dd9a34-fac4-44c9-96d3-b2c8968ed271"
Sep 11 17:18:53.920555 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:18:53.920542    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bl4vr"
Sep 11 17:18:53.920698 ip-10-0-141-139 kubenswrapper[2577]: E0911 17:18:53.920650    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bl4vr" podUID="5c1ac293-cc19-4446-8244-a8ca7fc8a8f4"
Sep 11 17:18:53.920755 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:18:53.920710    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xfn47"
Sep 11 17:18:53.920806 ip-10-0-141-139 kubenswrapper[2577]: E0911 17:18:53.920769    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xfn47" podUID="1e5d9e0c-7bc0-4fcf-8845-8873df5b1bee"
Sep 11 17:18:55.920155 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:18:55.920117    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fsh8r"
Sep 11 17:18:55.920658 ip-10-0-141-139 kubenswrapper[2577]: E0911 17:18:55.920255    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-fsh8r" podUID="73dd9a34-fac4-44c9-96d3-b2c8968ed271"
Sep 11 17:18:55.920727 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:18:55.920686    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bl4vr"
Sep 11 17:18:55.920818 ip-10-0-141-139 kubenswrapper[2577]: E0911 17:18:55.920797    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bl4vr" podUID="5c1ac293-cc19-4446-8244-a8ca7fc8a8f4"
Sep 11 17:18:55.920882 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:18:55.920852    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xfn47"
Sep 11 17:18:55.920944 ip-10-0-141-139 kubenswrapper[2577]: E0911 17:18:55.920922    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xfn47" podUID="1e5d9e0c-7bc0-4fcf-8845-8873df5b1bee"
Sep 11 17:18:57.920176 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:18:57.920136    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fsh8r"
Sep 11 17:18:57.920649 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:18:57.920152    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xfn47"
Sep 11 17:18:57.920649 ip-10-0-141-139 kubenswrapper[2577]: E0911 17:18:57.920277    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-fsh8r" podUID="73dd9a34-fac4-44c9-96d3-b2c8968ed271"
Sep 11 17:18:57.920649 ip-10-0-141-139 kubenswrapper[2577]: E0911 17:18:57.920373    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xfn47" podUID="1e5d9e0c-7bc0-4fcf-8845-8873df5b1bee"
Sep 11 17:18:57.920649 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:18:57.920155    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bl4vr"
Sep 11 17:18:57.920649 ip-10-0-141-139 kubenswrapper[2577]: E0911 17:18:57.920516    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bl4vr" podUID="5c1ac293-cc19-4446-8244-a8ca7fc8a8f4"
Sep 11 17:18:59.638153 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:18:59.638110    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/73dd9a34-fac4-44c9-96d3-b2c8968ed271-original-pull-secret\") pod \"global-pull-secret-syncer-fsh8r\" (UID: \"73dd9a34-fac4-44c9-96d3-b2c8968ed271\") " pod="kube-system/global-pull-secret-syncer-fsh8r"
Sep 11 17:18:59.638678 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:18:59.638169    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5c1ac293-cc19-4446-8244-a8ca7fc8a8f4-metrics-certs\") pod \"network-metrics-daemon-bl4vr\" (UID: \"5c1ac293-cc19-4446-8244-a8ca7fc8a8f4\") " pod="openshift-multus/network-metrics-daemon-bl4vr"
Sep 11 17:18:59.638678 ip-10-0-141-139 kubenswrapper[2577]: E0911 17:18:59.638264    2577 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 11 17:18:59.638678 ip-10-0-141-139 kubenswrapper[2577]: E0911 17:18:59.638324    2577 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 11 17:18:59.638678 ip-10-0-141-139 kubenswrapper[2577]: E0911 17:18:59.638349    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/73dd9a34-fac4-44c9-96d3-b2c8968ed271-original-pull-secret podName:73dd9a34-fac4-44c9-96d3-b2c8968ed271 nodeName:}" failed. No retries permitted until 2026-09-11 17:19:15.638327365 +0000 UTC m=+33.211547456 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/73dd9a34-fac4-44c9-96d3-b2c8968ed271-original-pull-secret") pod "global-pull-secret-syncer-fsh8r" (UID: "73dd9a34-fac4-44c9-96d3-b2c8968ed271") : object "kube-system"/"original-pull-secret" not registered
Sep 11 17:18:59.638678 ip-10-0-141-139 kubenswrapper[2577]: E0911 17:18:59.638389    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5c1ac293-cc19-4446-8244-a8ca7fc8a8f4-metrics-certs podName:5c1ac293-cc19-4446-8244-a8ca7fc8a8f4 nodeName:}" failed. No retries permitted until 2026-09-11 17:19:15.638370292 +0000 UTC m=+33.211590403 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/5c1ac293-cc19-4446-8244-a8ca7fc8a8f4-metrics-certs") pod "network-metrics-daemon-bl4vr" (UID: "5c1ac293-cc19-4446-8244-a8ca7fc8a8f4") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 11 17:18:59.738898 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:18:59.738854    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h2trs\" (UniqueName: \"kubernetes.io/projected/1e5d9e0c-7bc0-4fcf-8845-8873df5b1bee-kube-api-access-h2trs\") pod \"network-check-target-xfn47\" (UID: \"1e5d9e0c-7bc0-4fcf-8845-8873df5b1bee\") " pod="openshift-network-diagnostics/network-check-target-xfn47"
Sep 11 17:18:59.739090 ip-10-0-141-139 kubenswrapper[2577]: E0911 17:18:59.739031    2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 11 17:18:59.739090 ip-10-0-141-139 kubenswrapper[2577]: E0911 17:18:59.739049    2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 11 17:18:59.739090 ip-10-0-141-139 kubenswrapper[2577]: E0911 17:18:59.739059    2577 projected.go:196] Error preparing data for projected volume kube-api-access-h2trs for pod openshift-network-diagnostics/network-check-target-xfn47: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 11 17:18:59.739214 ip-10-0-141-139 kubenswrapper[2577]: E0911 17:18:59.739122    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/1e5d9e0c-7bc0-4fcf-8845-8873df5b1bee-kube-api-access-h2trs podName:1e5d9e0c-7bc0-4fcf-8845-8873df5b1bee nodeName:}" failed. No retries permitted until 2026-09-11 17:19:15.739105935 +0000 UTC m=+33.312326030 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-h2trs" (UniqueName: "kubernetes.io/projected/1e5d9e0c-7bc0-4fcf-8845-8873df5b1bee-kube-api-access-h2trs") pod "network-check-target-xfn47" (UID: "1e5d9e0c-7bc0-4fcf-8845-8873df5b1bee") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 11 17:18:59.920419 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:18:59.920340    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fsh8r"
Sep 11 17:18:59.920419 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:18:59.920372    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xfn47"
Sep 11 17:18:59.920740 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:18:59.920339    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bl4vr"
Sep 11 17:18:59.920740 ip-10-0-141-139 kubenswrapper[2577]: E0911 17:18:59.920460    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-fsh8r" podUID="73dd9a34-fac4-44c9-96d3-b2c8968ed271"
Sep 11 17:18:59.920740 ip-10-0-141-139 kubenswrapper[2577]: E0911 17:18:59.920580    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xfn47" podUID="1e5d9e0c-7bc0-4fcf-8845-8873df5b1bee"
Sep 11 17:18:59.920740 ip-10-0-141-139 kubenswrapper[2577]: E0911 17:18:59.920672    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bl4vr" podUID="5c1ac293-cc19-4446-8244-a8ca7fc8a8f4"
Sep 11 17:19:01.920011 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:19:01.919970    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xfn47"
Sep 11 17:19:01.920011 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:19:01.920004    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fsh8r"
Sep 11 17:19:01.920605 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:19:01.919970    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bl4vr"
Sep 11 17:19:01.920605 ip-10-0-141-139 kubenswrapper[2577]: E0911 17:19:01.920091    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xfn47" podUID="1e5d9e0c-7bc0-4fcf-8845-8873df5b1bee"
Sep 11 17:19:01.920605 ip-10-0-141-139 kubenswrapper[2577]: E0911 17:19:01.920198    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bl4vr" podUID="5c1ac293-cc19-4446-8244-a8ca7fc8a8f4"
Sep 11 17:19:01.920605 ip-10-0-141-139 kubenswrapper[2577]: E0911 17:19:01.920280    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-fsh8r" podUID="73dd9a34-fac4-44c9-96d3-b2c8968ed271"
Sep 11 17:19:03.920046 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:19:03.920007    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bl4vr"
Sep 11 17:19:03.920518 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:19:03.920006    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fsh8r"
Sep 11 17:19:03.920518 ip-10-0-141-139 kubenswrapper[2577]: E0911 17:19:03.920149    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bl4vr" podUID="5c1ac293-cc19-4446-8244-a8ca7fc8a8f4"
Sep 11 17:19:03.920518 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:19:03.920007    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xfn47"
Sep 11 17:19:03.920518 ip-10-0-141-139 kubenswrapper[2577]: E0911 17:19:03.920237    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-fsh8r" podUID="73dd9a34-fac4-44c9-96d3-b2c8968ed271"
Sep 11 17:19:03.920518 ip-10-0-141-139 kubenswrapper[2577]: E0911 17:19:03.920281    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xfn47" podUID="1e5d9e0c-7bc0-4fcf-8845-8873df5b1bee"
Sep 11 17:19:05.088246 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:19:05.087778    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-g6gcb" event={"ID":"337b4304-78e3-4abd-afbf-b866f7104056","Type":"ContainerStarted","Data":"83baa7253dff76a3ffa2cdc5be2e5833c193443157f2b74d254eba7696a6e9d8"}
Sep 11 17:19:05.089696 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:19:05.089670    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-d4gm9" event={"ID":"2fe5e836-a8a9-402a-bce2-da7ddc7418fa","Type":"ContainerStarted","Data":"0eb34edf809a8a19f09bcc6ae4726724055f5955d45a536e6b9b7ae28d86fdd9"}
Sep 11 17:19:05.091384 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:19:05.091338    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-k5j4d" event={"ID":"c5878f27-a359-47d1-b223-08f4bd829380","Type":"ContainerStarted","Data":"69aa44053967f544d627e9bf24e0cf12141ef72e35190f0874aec8da67aa3d54"}
Sep 11 17:19:05.093088 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:19:05.093067    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-6shpl" event={"ID":"a1ce372a-94c9-4089-8d8b-e4dee9b7e782","Type":"ContainerStarted","Data":"0be1c794661e39179ec6ad6f9174617ec67bafba82ecadf40f26ff359eb2bca2"}
Sep 11 17:19:05.094425 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:19:05.094401    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-9v9dt" event={"ID":"66d2b8f8-a4f7-4811-8034-2523351499ee","Type":"ContainerStarted","Data":"93bcc4ff8a4bc047388d4bc548280f89077909cf3570f3604216c2e61a5549f8"}
Sep 11 17:19:05.097644 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:19:05.097622    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-sn5kw" event={"ID":"bb06c328-82b1-4690-aa4a-6584fb149952","Type":"ContainerStarted","Data":"1d6d4c4732e54ab1a109d7e4b0ad6077afc62289dd99df0f85c76ae111b501fa"}
Sep 11 17:19:05.097768 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:19:05.097690    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-sn5kw" event={"ID":"bb06c328-82b1-4690-aa4a-6584fb149952","Type":"ContainerStarted","Data":"5385b59249a17d36c7254e2b6213bf8a23c9a1a2491b8751f4eb0c984f20e30e"}
Sep 11 17:19:05.097768 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:19:05.097737    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-sn5kw" event={"ID":"bb06c328-82b1-4690-aa4a-6584fb149952","Type":"ContainerStarted","Data":"e2bce9b788e951c2bb5d263dc6f9d19af53b12fe87ce35893df26fd0f9ef50cc"}
Sep 11 17:19:05.099372 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:19:05.099339    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-139.ec2.internal" event={"ID":"8b76f1b2a16574ce38927f7f5da97679","Type":"ContainerStarted","Data":"8b78f14dd2ee58fd6fb4c4f64d7382dac15125c26c72f8eb42b8bc0b0d3d5574"}
Sep 11 17:19:05.100833 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:19:05.100814    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hxl5l" event={"ID":"daeb9a26-f9e6-4d68-bc7d-f31b5f10d621","Type":"ContainerStarted","Data":"019e9d8ec54c181eecf0b19d010e416db159b5293ef0dd3848f50b9b800c6d27"}
Sep 11 17:19:05.104381 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:19:05.104325    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-nltl2" event={"ID":"9208373a-bbca-49fc-91ed-d29c0a833261","Type":"ContainerStarted","Data":"ca4d7155be22a98c48f9d9f7c7c5a93ccf58380c7ce8c5b9f8a295ccdf22d143"}
Sep 11 17:19:05.106461 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:19:05.106436    2577 generic.go:356] "Generic (PLEG): container finished" podID="628f3dc115ebc1acc6ed7da71915e638" containerID="5167c3835d8bbaff80a52f61fc79d0c94810d4aa1211b304cc49b72800b4217c" exitCode=0
Sep 11 17:19:05.106569 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:19:05.106475    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-139.ec2.internal" event={"ID":"628f3dc115ebc1acc6ed7da71915e638","Type":"ContainerDied","Data":"5167c3835d8bbaff80a52f61fc79d0c94810d4aa1211b304cc49b72800b4217c"}
Sep 11 17:19:05.109202 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:19:05.109165    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-g6gcb" podStartSLOduration=1.764354926 podStartE2EDuration="22.109152774s" totalImagesPullingTime="20.344797848s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 17:18:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 17:19:05.10896725 +0000 UTC m=+22.682187362" watchObservedRunningTime="2026-09-11 17:19:05.109152774 +0000 UTC m=+22.682372885"
Sep 11 17:19:05.125039 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:19:05.124966    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-139.ec2.internal" podStartSLOduration=21.124953048 podStartE2EDuration="21.124953048s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 17:18:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 17:19:05.124351236 +0000 UTC m=+22.697571341" watchObservedRunningTime="2026-09-11 17:19:05.124953048 +0000 UTC m=+22.698173160"
Sep 11 17:19:05.191268 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:19:05.191228    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-nltl2" podStartSLOduration=1.620996902 podStartE2EDuration="22.191210649s" totalImagesPullingTime="20.570213747s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 17:18:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 17:19:05.191099607 +0000 UTC m=+22.764319721" watchObservedRunningTime="2026-09-11 17:19:05.191210649 +0000 UTC m=+22.764430763"
Sep 11 17:19:05.208292 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:19:05.208251    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-9v9dt" podStartSLOduration=1.7601646130000002 podStartE2EDuration="22.208235903s" totalImagesPullingTime="20.44807129s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 17:18:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 17:19:05.207776322 +0000 UTC m=+22.780996435" watchObservedRunningTime="2026-09-11 17:19:05.208235903 +0000 UTC m=+22.781456014"
Sep 11 17:19:05.230190 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:19:05.230151    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-d4gm9" podStartSLOduration=1.8629239499999999 podStartE2EDuration="22.230137308s" totalImagesPullingTime="20.367213358s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 17:18:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 17:19:05.230058116 +0000 UTC m=+22.803278229" watchObservedRunningTime="2026-09-11 17:19:05.230137308 +0000 UTC m=+22.803357420"
Sep 11 17:19:05.248357 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:19:05.248255    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-k5j4d" podStartSLOduration=1.90155199 podStartE2EDuration="22.248235572s" totalImagesPullingTime="20.346683582s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 17:18:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 17:19:05.247983313 +0000 UTC m=+22.821203425" watchObservedRunningTime="2026-09-11 17:19:05.248235572 +0000 UTC m=+22.821455684"
Sep 11 17:19:05.920587 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:19:05.920490    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xfn47"
Sep 11 17:19:05.920587 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:19:05.920518    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bl4vr"
Sep 11 17:19:05.920835 ip-10-0-141-139 kubenswrapper[2577]: E0911 17:19:05.920614    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xfn47" podUID="1e5d9e0c-7bc0-4fcf-8845-8873df5b1bee"
Sep 11 17:19:05.920835 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:19:05.920630    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fsh8r"
Sep 11 17:19:05.920835 ip-10-0-141-139 kubenswrapper[2577]: E0911 17:19:05.920738    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bl4vr" podUID="5c1ac293-cc19-4446-8244-a8ca7fc8a8f4"
Sep 11 17:19:05.920945 ip-10-0-141-139 kubenswrapper[2577]: E0911 17:19:05.920833    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-fsh8r" podUID="73dd9a34-fac4-44c9-96d3-b2c8968ed271"
Sep 11 17:19:06.109816 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:19:06.109784    2577 generic.go:356] "Generic (PLEG): container finished" podID="a1ce372a-94c9-4089-8d8b-e4dee9b7e782" containerID="0be1c794661e39179ec6ad6f9174617ec67bafba82ecadf40f26ff359eb2bca2" exitCode=0
Sep 11 17:19:06.110304 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:19:06.109860    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-6shpl" event={"ID":"a1ce372a-94c9-4089-8d8b-e4dee9b7e782","Type":"ContainerDied","Data":"0be1c794661e39179ec6ad6f9174617ec67bafba82ecadf40f26ff359eb2bca2"}
Sep 11 17:19:06.113677 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:19:06.113444    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-sn5kw" event={"ID":"bb06c328-82b1-4690-aa4a-6584fb149952","Type":"ContainerStarted","Data":"b2ad076d41eb4551cc068e7cc7c874910d2c4fec053619cb7626325807f4f4aa"}
Sep 11 17:19:06.113778 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:19:06.113724    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-sn5kw" event={"ID":"bb06c328-82b1-4690-aa4a-6584fb149952","Type":"ContainerStarted","Data":"f8d87c4ab4205ed1eb806a70a56592b93de1b29271734b69493198d88e62e3ba"}
Sep 11 17:19:06.113835 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:19:06.113777    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-sn5kw" event={"ID":"bb06c328-82b1-4690-aa4a-6584fb149952","Type":"ContainerStarted","Data":"6a7e61b57fec2bf5102ef07c41182a2b451f6827f5e87e13be951b9ee338c539"}
Sep 11 17:19:06.115338 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:19:06.115274    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-139.ec2.internal" event={"ID":"628f3dc115ebc1acc6ed7da71915e638","Type":"ContainerStarted","Data":"0e25d1472793687bea150d1e88131f5b2145d66518069817d1d195606300880e"}
Sep 11 17:19:06.146055 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:19:06.146001    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-139.ec2.internal" podStartSLOduration=22.145986127 podStartE2EDuration="22.145986127s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 17:18:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 17:19:06.145405155 +0000 UTC m=+23.718625266" watchObservedRunningTime="2026-09-11 17:19:06.145986127 +0000 UTC m=+23.719206239"
Sep 11 17:19:06.925722 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:19:06.925694    2577 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Sep 11 17:19:07.119371 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:19:07.119289    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-vmxwz" event={"ID":"d64f20b5-ec23-4fdd-b8cc-ec63c443faa3","Type":"ContainerStarted","Data":"00e03a3953ca98b104640d1d62a69b062bfa1387e42ad02ac4e0ce46001a49af"}
Sep 11 17:19:07.121983 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:19:07.121945    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hxl5l" event={"ID":"daeb9a26-f9e6-4d68-bc7d-f31b5f10d621","Type":"ContainerStarted","Data":"fce647c9b4ea5dbebf1498fe385565fe8bb780d06d26cfec1403771c4c4ce097"}
Sep 11 17:19:07.135988 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:19:07.135942    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-vmxwz" podStartSLOduration=3.729179834 podStartE2EDuration="24.135924848s" totalImagesPullingTime="20.406745014s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 17:18:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 17:19:07.135767577 +0000 UTC m=+24.708987700" watchObservedRunningTime="2026-09-11 17:19:07.135924848 +0000 UTC m=+24.709144959"
Sep 11 17:19:07.561229 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:19:07.559837    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-g6gcb"
Sep 11 17:19:07.561229 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:19:07.561028    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-g6gcb"
Sep 11 17:19:07.861053 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:19:07.860882    2577 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-09-11T17:19:06.925715326Z","UUID":"66d163ce-c2fc-4819-a20c-01579e78a0ca","Handler":null,"Name":"","Endpoint":""}
Sep 11 17:19:07.862852 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:19:07.862827    2577 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Sep 11 17:19:07.862992 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:19:07.862879    2577 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Sep 11 17:19:07.920225 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:19:07.920176    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bl4vr"
Sep 11 17:19:07.920420 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:19:07.920187    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fsh8r"
Sep 11 17:19:07.920420 ip-10-0-141-139 kubenswrapper[2577]: E0911 17:19:07.920331    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bl4vr" podUID="5c1ac293-cc19-4446-8244-a8ca7fc8a8f4"
Sep 11 17:19:07.920420 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:19:07.920356    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xfn47"
Sep 11 17:19:07.920627 ip-10-0-141-139 kubenswrapper[2577]: E0911 17:19:07.920519    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-fsh8r" podUID="73dd9a34-fac4-44c9-96d3-b2c8968ed271"
Sep 11 17:19:07.920627 ip-10-0-141-139 kubenswrapper[2577]: E0911 17:19:07.920587    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xfn47" podUID="1e5d9e0c-7bc0-4fcf-8845-8873df5b1bee"
Sep 11 17:19:08.128090 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:19:08.127936    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-sn5kw" event={"ID":"bb06c328-82b1-4690-aa4a-6584fb149952","Type":"ContainerStarted","Data":"e5494b052372780ad32a2f0ff2d06ea9c9232457cccaa884ba98ea7e0078ffd8"}
Sep 11 17:19:09.133376 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:19:09.133339    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hxl5l" event={"ID":"daeb9a26-f9e6-4d68-bc7d-f31b5f10d621","Type":"ContainerStarted","Data":"bafb0fb3882e24af484265393033579721788bb524a05c9598ba5db72ecf7e14"}
Sep 11 17:19:09.152905 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:19:09.152852    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hxl5l" podStartSLOduration=6.639878972 podStartE2EDuration="26.1528359s" totalImagesPullingTime="19.512956928s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 17:18:43 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 17:19:09.152367484 +0000 UTC m=+26.725587599" watchObservedRunningTime="2026-09-11 17:19:09.1528359 +0000 UTC m=+26.726056012"
Sep 11 17:19:09.920533 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:19:09.920438    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fsh8r"
Sep 11 17:19:09.920681 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:19:09.920563    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bl4vr"
Sep 11 17:19:09.920681 ip-10-0-141-139 kubenswrapper[2577]: E0911 17:19:09.920575    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-fsh8r" podUID="73dd9a34-fac4-44c9-96d3-b2c8968ed271"
Sep 11 17:19:09.920681 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:19:09.920615    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xfn47"
Sep 11 17:19:09.920808 ip-10-0-141-139 kubenswrapper[2577]: E0911 17:19:09.920733    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bl4vr" podUID="5c1ac293-cc19-4446-8244-a8ca7fc8a8f4"
Sep 11 17:19:09.920860 ip-10-0-141-139 kubenswrapper[2577]: E0911 17:19:09.920819    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xfn47" podUID="1e5d9e0c-7bc0-4fcf-8845-8873df5b1bee"
Sep 11 17:19:11.919997 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:19:11.919807    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fsh8r"
Sep 11 17:19:11.920446 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:19:11.919813    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bl4vr"
Sep 11 17:19:11.920446 ip-10-0-141-139 kubenswrapper[2577]: E0911 17:19:11.920066    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-fsh8r" podUID="73dd9a34-fac4-44c9-96d3-b2c8968ed271"
Sep 11 17:19:11.920446 ip-10-0-141-139 kubenswrapper[2577]: E0911 17:19:11.920159    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bl4vr" podUID="5c1ac293-cc19-4446-8244-a8ca7fc8a8f4"
Sep 11 17:19:11.920446 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:19:11.919862    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xfn47"
Sep 11 17:19:11.920446 ip-10-0-141-139 kubenswrapper[2577]: E0911 17:19:11.920243    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xfn47" podUID="1e5d9e0c-7bc0-4fcf-8845-8873df5b1bee"
Sep 11 17:19:12.141894 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:19:12.141857    2577 generic.go:356] "Generic (PLEG): container finished" podID="a1ce372a-94c9-4089-8d8b-e4dee9b7e782" containerID="9d8164c4b6fbde94d074509b659f550e38a31e74a03229cfa9d30b287dbd5c4d" exitCode=0
Sep 11 17:19:12.142119 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:19:12.141940    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-6shpl" event={"ID":"a1ce372a-94c9-4089-8d8b-e4dee9b7e782","Type":"ContainerDied","Data":"9d8164c4b6fbde94d074509b659f550e38a31e74a03229cfa9d30b287dbd5c4d"}
Sep 11 17:19:12.146084 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:19:12.145981    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-sn5kw" event={"ID":"bb06c328-82b1-4690-aa4a-6584fb149952","Type":"ContainerStarted","Data":"ba31d867e835199d7e208930a9e1b851681a40da9d14e54f93673974d9767f3e"}
Sep 11 17:19:12.146284 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:19:12.146238    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-sn5kw"
Sep 11 17:19:12.146386 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:19:12.146316    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-sn5kw"
Sep 11 17:19:12.146386 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:19:12.146372    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-sn5kw"
Sep 11 17:19:12.162668 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:19:12.162644    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-sn5kw"
Sep 11 17:19:12.162798 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:19:12.162751    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-sn5kw"
Sep 11 17:19:12.223049 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:19:12.222999    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-sn5kw" podStartSLOduration=8.684920736 podStartE2EDuration="29.222982748s" totalImagesPullingTime="20.538062012s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 17:18:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 17:19:12.221584766 +0000 UTC m=+29.794804878" watchObservedRunningTime="2026-09-11 17:19:12.222982748 +0000 UTC m=+29.796202859"
Sep 11 17:19:12.358982 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:19:12.358945    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-g6gcb"
Sep 11 17:19:12.359189 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:19:12.359072    2577 prober_manager.go:312] "Failed to trigger a manual run" probe="Readiness"
Sep 11 17:19:12.359870 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:19:12.359847    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-g6gcb"
Sep 11 17:19:13.120294 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:19:13.120110    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-9v9dt_66d2b8f8-a4f7-4811-8034-2523351499ee/dns-node-resolver/0.log"
Sep 11 17:19:13.124574 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:19:13.124546    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-bl4vr"]
Sep 11 17:19:13.124708 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:19:13.124667    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bl4vr"
Sep 11 17:19:13.124781 ip-10-0-141-139 kubenswrapper[2577]: E0911 17:19:13.124756    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bl4vr" podUID="5c1ac293-cc19-4446-8244-a8ca7fc8a8f4"
Sep 11 17:19:13.135811 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:19:13.135703    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-xfn47"]
Sep 11 17:19:13.135930 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:19:13.135818    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xfn47"
Sep 11 17:19:13.135930 ip-10-0-141-139 kubenswrapper[2577]: E0911 17:19:13.135909    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xfn47" podUID="1e5d9e0c-7bc0-4fcf-8845-8873df5b1bee"
Sep 11 17:19:13.136329 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:19:13.136307    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-fsh8r"]
Sep 11 17:19:13.136421 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:19:13.136404    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fsh8r"
Sep 11 17:19:13.136544 ip-10-0-141-139 kubenswrapper[2577]: E0911 17:19:13.136526    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-fsh8r" podUID="73dd9a34-fac4-44c9-96d3-b2c8968ed271"
Sep 11 17:19:14.152336 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:19:14.152292    2577 generic.go:356] "Generic (PLEG): container finished" podID="a1ce372a-94c9-4089-8d8b-e4dee9b7e782" containerID="9f508906fa7f6b8019c2ad9c3ba3ea10359f20dd6f228fc85e2e69b6142e6cbe" exitCode=0
Sep 11 17:19:14.152722 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:19:14.152363    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-6shpl" event={"ID":"a1ce372a-94c9-4089-8d8b-e4dee9b7e782","Type":"ContainerDied","Data":"9f508906fa7f6b8019c2ad9c3ba3ea10359f20dd6f228fc85e2e69b6142e6cbe"}
Sep 11 17:19:14.473260 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:19:14.473235    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-k5j4d_c5878f27-a359-47d1-b223-08f4bd829380/node-ca/0.log"
Sep 11 17:19:14.920677 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:19:14.920644    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xfn47"
Sep 11 17:19:14.920677 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:19:14.920679    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fsh8r"
Sep 11 17:19:14.920908 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:19:14.920644    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bl4vr"
Sep 11 17:19:14.920908 ip-10-0-141-139 kubenswrapper[2577]: E0911 17:19:14.920769    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xfn47" podUID="1e5d9e0c-7bc0-4fcf-8845-8873df5b1bee"
Sep 11 17:19:14.920908 ip-10-0-141-139 kubenswrapper[2577]: E0911 17:19:14.920861    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bl4vr" podUID="5c1ac293-cc19-4446-8244-a8ca7fc8a8f4"
Sep 11 17:19:14.920999 ip-10-0-141-139 kubenswrapper[2577]: E0911 17:19:14.920952    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-fsh8r" podUID="73dd9a34-fac4-44c9-96d3-b2c8968ed271"
Sep 11 17:19:15.652295 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:19:15.652257    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/73dd9a34-fac4-44c9-96d3-b2c8968ed271-original-pull-secret\") pod \"global-pull-secret-syncer-fsh8r\" (UID: \"73dd9a34-fac4-44c9-96d3-b2c8968ed271\") " pod="kube-system/global-pull-secret-syncer-fsh8r"
Sep 11 17:19:15.652764 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:19:15.652305    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5c1ac293-cc19-4446-8244-a8ca7fc8a8f4-metrics-certs\") pod \"network-metrics-daemon-bl4vr\" (UID: \"5c1ac293-cc19-4446-8244-a8ca7fc8a8f4\") " pod="openshift-multus/network-metrics-daemon-bl4vr"
Sep 11 17:19:15.652764 ip-10-0-141-139 kubenswrapper[2577]: E0911 17:19:15.652404    2577 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 11 17:19:15.652764 ip-10-0-141-139 kubenswrapper[2577]: E0911 17:19:15.652415    2577 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 11 17:19:15.652764 ip-10-0-141-139 kubenswrapper[2577]: E0911 17:19:15.652472    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5c1ac293-cc19-4446-8244-a8ca7fc8a8f4-metrics-certs podName:5c1ac293-cc19-4446-8244-a8ca7fc8a8f4 nodeName:}" failed. No retries permitted until 2026-09-11 17:19:47.65245309 +0000 UTC m=+65.225673199 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/5c1ac293-cc19-4446-8244-a8ca7fc8a8f4-metrics-certs") pod "network-metrics-daemon-bl4vr" (UID: "5c1ac293-cc19-4446-8244-a8ca7fc8a8f4") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 11 17:19:15.652764 ip-10-0-141-139 kubenswrapper[2577]: E0911 17:19:15.652534    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/73dd9a34-fac4-44c9-96d3-b2c8968ed271-original-pull-secret podName:73dd9a34-fac4-44c9-96d3-b2c8968ed271 nodeName:}" failed. No retries permitted until 2026-09-11 17:19:47.652482956 +0000 UTC m=+65.225703060 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/73dd9a34-fac4-44c9-96d3-b2c8968ed271-original-pull-secret") pod "global-pull-secret-syncer-fsh8r" (UID: "73dd9a34-fac4-44c9-96d3-b2c8968ed271") : object "kube-system"/"original-pull-secret" not registered
Sep 11 17:19:15.752982 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:19:15.752891    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h2trs\" (UniqueName: \"kubernetes.io/projected/1e5d9e0c-7bc0-4fcf-8845-8873df5b1bee-kube-api-access-h2trs\") pod \"network-check-target-xfn47\" (UID: \"1e5d9e0c-7bc0-4fcf-8845-8873df5b1bee\") " pod="openshift-network-diagnostics/network-check-target-xfn47"
Sep 11 17:19:15.753109 ip-10-0-141-139 kubenswrapper[2577]: E0911 17:19:15.753065    2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 11 17:19:15.753109 ip-10-0-141-139 kubenswrapper[2577]: E0911 17:19:15.753087    2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 11 17:19:15.753109 ip-10-0-141-139 kubenswrapper[2577]: E0911 17:19:15.753098    2577 projected.go:196] Error preparing data for projected volume kube-api-access-h2trs for pod openshift-network-diagnostics/network-check-target-xfn47: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 11 17:19:15.753323 ip-10-0-141-139 kubenswrapper[2577]: E0911 17:19:15.753158    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/1e5d9e0c-7bc0-4fcf-8845-8873df5b1bee-kube-api-access-h2trs podName:1e5d9e0c-7bc0-4fcf-8845-8873df5b1bee nodeName:}" failed. No retries permitted until 2026-09-11 17:19:47.75314131 +0000 UTC m=+65.326361408 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-h2trs" (UniqueName: "kubernetes.io/projected/1e5d9e0c-7bc0-4fcf-8845-8873df5b1bee-kube-api-access-h2trs") pod "network-check-target-xfn47" (UID: "1e5d9e0c-7bc0-4fcf-8845-8873df5b1bee") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 11 17:19:16.159591 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:19:16.159490    2577 generic.go:356] "Generic (PLEG): container finished" podID="a1ce372a-94c9-4089-8d8b-e4dee9b7e782" containerID="7a9130c001d69974bebee313156cf8638f406af54ef87103cee3ee89ca33306a" exitCode=0
Sep 11 17:19:16.159591 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:19:16.159548    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-6shpl" event={"ID":"a1ce372a-94c9-4089-8d8b-e4dee9b7e782","Type":"ContainerDied","Data":"7a9130c001d69974bebee313156cf8638f406af54ef87103cee3ee89ca33306a"}
Sep 11 17:19:16.919877 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:19:16.919834    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bl4vr"
Sep 11 17:19:16.919877 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:19:16.919870    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xfn47"
Sep 11 17:19:16.920330 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:19:16.919973    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fsh8r"
Sep 11 17:19:16.920330 ip-10-0-141-139 kubenswrapper[2577]: E0911 17:19:16.919979    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bl4vr" podUID="5c1ac293-cc19-4446-8244-a8ca7fc8a8f4"
Sep 11 17:19:16.920330 ip-10-0-141-139 kubenswrapper[2577]: E0911 17:19:16.920034    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xfn47" podUID="1e5d9e0c-7bc0-4fcf-8845-8873df5b1bee"
Sep 11 17:19:16.920330 ip-10-0-141-139 kubenswrapper[2577]: E0911 17:19:16.920112    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-fsh8r" podUID="73dd9a34-fac4-44c9-96d3-b2c8968ed271"
Sep 11 17:19:18.920284 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:19:18.920204    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xfn47"
Sep 11 17:19:18.920284 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:19:18.920246    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bl4vr"
Sep 11 17:19:18.920284 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:19:18.920251    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fsh8r"
Sep 11 17:19:18.920993 ip-10-0-141-139 kubenswrapper[2577]: E0911 17:19:18.920342    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xfn47" podUID="1e5d9e0c-7bc0-4fcf-8845-8873df5b1bee"
Sep 11 17:19:18.920993 ip-10-0-141-139 kubenswrapper[2577]: E0911 17:19:18.920436    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bl4vr" podUID="5c1ac293-cc19-4446-8244-a8ca7fc8a8f4"
Sep 11 17:19:18.920993 ip-10-0-141-139 kubenswrapper[2577]: E0911 17:19:18.920544    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-fsh8r" podUID="73dd9a34-fac4-44c9-96d3-b2c8968ed271"
Sep 11 17:19:20.920030 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:19:20.919998    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bl4vr"
Sep 11 17:19:20.920466 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:19:20.920001    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xfn47"
Sep 11 17:19:20.920466 ip-10-0-141-139 kubenswrapper[2577]: E0911 17:19:20.920112    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bl4vr" podUID="5c1ac293-cc19-4446-8244-a8ca7fc8a8f4"
Sep 11 17:19:20.920466 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:19:20.920001    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fsh8r"
Sep 11 17:19:20.920466 ip-10-0-141-139 kubenswrapper[2577]: E0911 17:19:20.920190    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xfn47" podUID="1e5d9e0c-7bc0-4fcf-8845-8873df5b1bee"
Sep 11 17:19:20.920466 ip-10-0-141-139 kubenswrapper[2577]: E0911 17:19:20.920307    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-fsh8r" podUID="73dd9a34-fac4-44c9-96d3-b2c8968ed271"
Sep 11 17:19:22.177691 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:19:22.177471    2577 generic.go:356] "Generic (PLEG): container finished" podID="a1ce372a-94c9-4089-8d8b-e4dee9b7e782" containerID="4257086408aec483718a6fe85662b247f2ade91e952716003794bae09920d578" exitCode=0
Sep 11 17:19:22.178263 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:19:22.177533    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-6shpl" event={"ID":"a1ce372a-94c9-4089-8d8b-e4dee9b7e782","Type":"ContainerDied","Data":"4257086408aec483718a6fe85662b247f2ade91e952716003794bae09920d578"}
Sep 11 17:19:22.920656 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:19:22.920574    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bl4vr"
Sep 11 17:19:22.920804 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:19:22.920658    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xfn47"
Sep 11 17:19:22.920804 ip-10-0-141-139 kubenswrapper[2577]: E0911 17:19:22.920671    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bl4vr" podUID="5c1ac293-cc19-4446-8244-a8ca7fc8a8f4"
Sep 11 17:19:22.920804 ip-10-0-141-139 kubenswrapper[2577]: E0911 17:19:22.920778    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xfn47" podUID="1e5d9e0c-7bc0-4fcf-8845-8873df5b1bee"
Sep 11 17:19:22.920914 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:19:22.920814    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fsh8r"
Sep 11 17:19:22.920914 ip-10-0-141-139 kubenswrapper[2577]: E0911 17:19:22.920884    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-fsh8r" podUID="73dd9a34-fac4-44c9-96d3-b2c8968ed271"
Sep 11 17:19:23.183130 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:19:23.183046    2577 generic.go:356] "Generic (PLEG): container finished" podID="a1ce372a-94c9-4089-8d8b-e4dee9b7e782" containerID="6adad7f78a291dda561f8badd0f7db51cedf615d9ddc83d7c55316f34969c2e5" exitCode=0
Sep 11 17:19:23.183130 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:19:23.183111    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-6shpl" event={"ID":"a1ce372a-94c9-4089-8d8b-e4dee9b7e782","Type":"ContainerDied","Data":"6adad7f78a291dda561f8badd0f7db51cedf615d9ddc83d7c55316f34969c2e5"}
Sep 11 17:19:24.189140 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:19:24.189103    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-6shpl" event={"ID":"a1ce372a-94c9-4089-8d8b-e4dee9b7e782","Type":"ContainerStarted","Data":"6a7edef21a533a07ee6a3db28afe5843567da0a974aa5e65b4f1fb2697e84a98"}
Sep 11 17:19:24.240015 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:19:24.239962    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-6shpl" podStartSLOduration=7.678687659 podStartE2EDuration="41.239943506s" totalImagesPullingTime="33.561255847s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 17:18:43 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 17:19:24.237803086 +0000 UTC m=+41.811023210" watchObservedRunningTime="2026-09-11 17:19:24.239943506 +0000 UTC m=+41.813163618"
Sep 11 17:19:24.920426 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:19:24.920385    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xfn47"
Sep 11 17:19:24.920647 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:19:24.920429    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fsh8r"
Sep 11 17:19:24.920647 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:19:24.920385    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bl4vr"
Sep 11 17:19:24.920647 ip-10-0-141-139 kubenswrapper[2577]: E0911 17:19:24.920537    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xfn47" podUID="1e5d9e0c-7bc0-4fcf-8845-8873df5b1bee"
Sep 11 17:19:24.920647 ip-10-0-141-139 kubenswrapper[2577]: E0911 17:19:24.920617    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bl4vr" podUID="5c1ac293-cc19-4446-8244-a8ca7fc8a8f4"
Sep 11 17:19:24.920863 ip-10-0-141-139 kubenswrapper[2577]: E0911 17:19:24.920707    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-fsh8r" podUID="73dd9a34-fac4-44c9-96d3-b2c8968ed271"
Sep 11 17:19:26.920705 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:19:26.920674    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bl4vr"
Sep 11 17:19:26.920705 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:19:26.920692    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fsh8r"
Sep 11 17:19:26.921116 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:19:26.920689    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xfn47"
Sep 11 17:19:26.921116 ip-10-0-141-139 kubenswrapper[2577]: E0911 17:19:26.920805    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bl4vr" podUID="5c1ac293-cc19-4446-8244-a8ca7fc8a8f4"
Sep 11 17:19:26.921116 ip-10-0-141-139 kubenswrapper[2577]: E0911 17:19:26.920946    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-fsh8r" podUID="73dd9a34-fac4-44c9-96d3-b2c8968ed271"
Sep 11 17:19:26.921116 ip-10-0-141-139 kubenswrapper[2577]: E0911 17:19:26.921032    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xfn47" podUID="1e5d9e0c-7bc0-4fcf-8845-8873df5b1bee"
Sep 11 17:19:28.920744 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:19:28.920706    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xfn47"
Sep 11 17:19:28.921207 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:19:28.920706    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bl4vr"
Sep 11 17:19:28.921207 ip-10-0-141-139 kubenswrapper[2577]: E0911 17:19:28.920830    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xfn47" podUID="1e5d9e0c-7bc0-4fcf-8845-8873df5b1bee"
Sep 11 17:19:28.921207 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:19:28.920706    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fsh8r"
Sep 11 17:19:28.921207 ip-10-0-141-139 kubenswrapper[2577]: E0911 17:19:28.920913    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bl4vr" podUID="5c1ac293-cc19-4446-8244-a8ca7fc8a8f4"
Sep 11 17:19:28.921207 ip-10-0-141-139 kubenswrapper[2577]: E0911 17:19:28.920958    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-fsh8r" podUID="73dd9a34-fac4-44c9-96d3-b2c8968ed271"
Sep 11 17:19:30.920907 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:19:30.920863    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bl4vr"
Sep 11 17:19:30.921396 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:19:30.920877    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xfn47"
Sep 11 17:19:30.921396 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:19:30.920999    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fsh8r"
Sep 11 17:19:30.921396 ip-10-0-141-139 kubenswrapper[2577]: E0911 17:19:30.921001    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bl4vr" podUID="5c1ac293-cc19-4446-8244-a8ca7fc8a8f4"
Sep 11 17:19:30.921396 ip-10-0-141-139 kubenswrapper[2577]: E0911 17:19:30.921073    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xfn47" podUID="1e5d9e0c-7bc0-4fcf-8845-8873df5b1bee"
Sep 11 17:19:30.921396 ip-10-0-141-139 kubenswrapper[2577]: E0911 17:19:30.921157    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-fsh8r" podUID="73dd9a34-fac4-44c9-96d3-b2c8968ed271"
Sep 11 17:19:32.920826 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:19:32.920791    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bl4vr"
Sep 11 17:19:32.921260 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:19:32.920874    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fsh8r"
Sep 11 17:19:32.921260 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:19:32.920903    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xfn47"
Sep 11 17:19:32.921260 ip-10-0-141-139 kubenswrapper[2577]: E0911 17:19:32.920925    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bl4vr" podUID="5c1ac293-cc19-4446-8244-a8ca7fc8a8f4"
Sep 11 17:19:32.921260 ip-10-0-141-139 kubenswrapper[2577]: E0911 17:19:32.920978    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xfn47" podUID="1e5d9e0c-7bc0-4fcf-8845-8873df5b1bee"
Sep 11 17:19:32.921260 ip-10-0-141-139 kubenswrapper[2577]: E0911 17:19:32.921051    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-fsh8r" podUID="73dd9a34-fac4-44c9-96d3-b2c8968ed271"
Sep 11 17:19:34.919813 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:19:34.919771    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fsh8r"
Sep 11 17:19:34.919813 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:19:34.919793    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bl4vr"
Sep 11 17:19:34.919813 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:19:34.919807    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xfn47"
Sep 11 17:19:34.920368 ip-10-0-141-139 kubenswrapper[2577]: E0911 17:19:34.919906    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-fsh8r" podUID="73dd9a34-fac4-44c9-96d3-b2c8968ed271"
Sep 11 17:19:34.920368 ip-10-0-141-139 kubenswrapper[2577]: E0911 17:19:34.920026    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bl4vr" podUID="5c1ac293-cc19-4446-8244-a8ca7fc8a8f4"
Sep 11 17:19:34.920368 ip-10-0-141-139 kubenswrapper[2577]: E0911 17:19:34.920100    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xfn47" podUID="1e5d9e0c-7bc0-4fcf-8845-8873df5b1bee"
Sep 11 17:19:36.920613 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:19:36.920572    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bl4vr"
Sep 11 17:19:36.921154 ip-10-0-141-139 kubenswrapper[2577]: E0911 17:19:36.920705    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bl4vr" podUID="5c1ac293-cc19-4446-8244-a8ca7fc8a8f4"
Sep 11 17:19:36.921154 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:19:36.920731    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xfn47"
Sep 11 17:19:36.921154 ip-10-0-141-139 kubenswrapper[2577]: E0911 17:19:36.920821    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xfn47" podUID="1e5d9e0c-7bc0-4fcf-8845-8873df5b1bee"
Sep 11 17:19:36.921154 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:19:36.920854    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fsh8r"
Sep 11 17:19:36.921154 ip-10-0-141-139 kubenswrapper[2577]: E0911 17:19:36.920936    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-fsh8r" podUID="73dd9a34-fac4-44c9-96d3-b2c8968ed271"
Sep 11 17:19:38.699691 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:19:38.699656    2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-139.ec2.internal" event="NodeReady"
Sep 11 17:19:38.700105 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:19:38.699790    2577 kubelet_node_status.go:489] "Fast updating node status as it just became ready"
Sep 11 17:19:38.848432 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:19:38.848399    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-tt7ph"]
Sep 11 17:19:38.852627 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:19:38.852599    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-vnvj4"]
Sep 11 17:19:38.852764 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:19:38.852741    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-tt7ph"
Sep 11 17:19:38.854696 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:19:38.854673    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-vnvj4"
Sep 11 17:19:38.855957 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:19:38.855941    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Sep 11 17:19:38.856050 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:19:38.856009    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-wkt9z\""
Sep 11 17:19:38.859490 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:19:38.859455    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Sep 11 17:19:38.859597 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:19:38.859569    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Sep 11 17:19:38.859658 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:19:38.859627    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Sep 11 17:19:38.859741 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:19:38.859724    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Sep 11 17:19:38.859806 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:19:38.859774    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Sep 11 17:19:38.859862 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:19:38.859841    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Sep 11 17:19:38.860828 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:19:38.860811    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-p99b8\""
Sep 11 17:19:38.871917 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:19:38.871896    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-tt7ph"]
Sep 11 17:19:38.911183 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:19:38.911143    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-vnvj4"]
Sep 11 17:19:38.920709 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:19:38.920680    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xfn47"
Sep 11 17:19:38.920709 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:19:38.920703    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fsh8r"
Sep 11 17:19:38.920872 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:19:38.920715    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bl4vr"
Sep 11 17:19:38.924151 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:19:38.924128    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-g6qq7"]
Sep 11 17:19:38.925313 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:19:38.925295    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Sep 11 17:19:38.926056 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:19:38.926028    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-g6qq7"
Sep 11 17:19:38.929183 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:19:38.929167    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Sep 11 17:19:38.929266 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:19:38.929169    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Sep 11 17:19:38.930035 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:19:38.930017    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Sep 11 17:19:38.930323 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:19:38.930289    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-vthbj\""
Sep 11 17:19:38.930435 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:19:38.930343    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Sep 11 17:19:38.930622 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:19:38.930609    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-gdccs\""
Sep 11 17:19:38.930718 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:19:38.930704    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Sep 11 17:19:38.935582 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:19:38.935566    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-bj8s9\""
Sep 11 17:19:38.977479 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:19:38.977452    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-g6qq7"]
Sep 11 17:19:39.018151 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:19:39.018114    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/f68e4b95-ae13-4835-976a-a1af09fc70cf-data-volume\") pod \"insights-runtime-extractor-vnvj4\" (UID: \"f68e4b95-ae13-4835-976a-a1af09fc70cf\") " pod="openshift-insights/insights-runtime-extractor-vnvj4"
Sep 11 17:19:39.018151 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:19:39.018148    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/f68e4b95-ae13-4835-976a-a1af09fc70cf-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-vnvj4\" (UID: \"f68e4b95-ae13-4835-976a-a1af09fc70cf\") " pod="openshift-insights/insights-runtime-extractor-vnvj4"
Sep 11 17:19:39.018382 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:19:39.018167    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/d6d8f0b3-5098-4746-8e75-aee84eafa17b-metrics-tls\") pod \"dns-default-g6qq7\" (UID: \"d6d8f0b3-5098-4746-8e75-aee84eafa17b\") " pod="openshift-dns/dns-default-g6qq7"
Sep 11 17:19:39.018382 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:19:39.018243    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rfhtg\" (UniqueName: \"kubernetes.io/projected/894d4606-cd1f-45ae-a9fa-a3b9b5e00975-kube-api-access-rfhtg\") pod \"ingress-canary-tt7ph\" (UID: \"894d4606-cd1f-45ae-a9fa-a3b9b5e00975\") " pod="openshift-ingress-canary/ingress-canary-tt7ph"
Sep 11 17:19:39.018382 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:19:39.018281    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/f68e4b95-ae13-4835-976a-a1af09fc70cf-crio-socket\") pod \"insights-runtime-extractor-vnvj4\" (UID: \"f68e4b95-ae13-4835-976a-a1af09fc70cf\") " pod="openshift-insights/insights-runtime-extractor-vnvj4"
Sep 11 17:19:39.018382 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:19:39.018300    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6wwzv\" (UniqueName: \"kubernetes.io/projected/d6d8f0b3-5098-4746-8e75-aee84eafa17b-kube-api-access-6wwzv\") pod \"dns-default-g6qq7\" (UID: \"d6d8f0b3-5098-4746-8e75-aee84eafa17b\") " pod="openshift-dns/dns-default-g6qq7"
Sep 11 17:19:39.018382 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:19:39.018348    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/d6d8f0b3-5098-4746-8e75-aee84eafa17b-config-volume\") pod \"dns-default-g6qq7\" (UID: \"d6d8f0b3-5098-4746-8e75-aee84eafa17b\") " pod="openshift-dns/dns-default-g6qq7"
Sep 11 17:19:39.018687 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:19:39.018386    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d6d8f0b3-5098-4746-8e75-aee84eafa17b-tmp-dir\") pod \"dns-default-g6qq7\" (UID: \"d6d8f0b3-5098-4746-8e75-aee84eafa17b\") " pod="openshift-dns/dns-default-g6qq7"
Sep 11 17:19:39.018687 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:19:39.018455    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/f68e4b95-ae13-4835-976a-a1af09fc70cf-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-vnvj4\" (UID: \"f68e4b95-ae13-4835-976a-a1af09fc70cf\") " pod="openshift-insights/insights-runtime-extractor-vnvj4"
Sep 11 17:19:39.018687 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:19:39.018529    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b9txf\" (UniqueName: \"kubernetes.io/projected/f68e4b95-ae13-4835-976a-a1af09fc70cf-kube-api-access-b9txf\") pod \"insights-runtime-extractor-vnvj4\" (UID: \"f68e4b95-ae13-4835-976a-a1af09fc70cf\") " pod="openshift-insights/insights-runtime-extractor-vnvj4"
Sep 11 17:19:39.018687 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:19:39.018555    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/894d4606-cd1f-45ae-a9fa-a3b9b5e00975-cert\") pod \"ingress-canary-tt7ph\" (UID: \"894d4606-cd1f-45ae-a9fa-a3b9b5e00975\") " pod="openshift-ingress-canary/ingress-canary-tt7ph"
Sep 11 17:19:39.119377 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:19:39.119334    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/f68e4b95-ae13-4835-976a-a1af09fc70cf-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-vnvj4\" (UID: \"f68e4b95-ae13-4835-976a-a1af09fc70cf\") " pod="openshift-insights/insights-runtime-extractor-vnvj4"
Sep 11 17:19:39.119377 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:19:39.119373    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/d6d8f0b3-5098-4746-8e75-aee84eafa17b-metrics-tls\") pod \"dns-default-g6qq7\" (UID: \"d6d8f0b3-5098-4746-8e75-aee84eafa17b\") " pod="openshift-dns/dns-default-g6qq7"
Sep 11 17:19:39.119652 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:19:39.119550    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rfhtg\" (UniqueName: \"kubernetes.io/projected/894d4606-cd1f-45ae-a9fa-a3b9b5e00975-kube-api-access-rfhtg\") pod \"ingress-canary-tt7ph\" (UID: \"894d4606-cd1f-45ae-a9fa-a3b9b5e00975\") " pod="openshift-ingress-canary/ingress-canary-tt7ph"
Sep 11 17:19:39.119652 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:19:39.119598    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/f68e4b95-ae13-4835-976a-a1af09fc70cf-crio-socket\") pod \"insights-runtime-extractor-vnvj4\" (UID: \"f68e4b95-ae13-4835-976a-a1af09fc70cf\") " pod="openshift-insights/insights-runtime-extractor-vnvj4"
Sep 11 17:19:39.119652 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:19:39.119624    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6wwzv\" (UniqueName: \"kubernetes.io/projected/d6d8f0b3-5098-4746-8e75-aee84eafa17b-kube-api-access-6wwzv\") pod \"dns-default-g6qq7\" (UID: \"d6d8f0b3-5098-4746-8e75-aee84eafa17b\") " pod="openshift-dns/dns-default-g6qq7"
Sep 11 17:19:39.119652 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:19:39.119649    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/d6d8f0b3-5098-4746-8e75-aee84eafa17b-config-volume\") pod \"dns-default-g6qq7\" (UID: \"d6d8f0b3-5098-4746-8e75-aee84eafa17b\") " pod="openshift-dns/dns-default-g6qq7"
Sep 11 17:19:39.119852 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:19:39.119671    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d6d8f0b3-5098-4746-8e75-aee84eafa17b-tmp-dir\") pod \"dns-default-g6qq7\" (UID: \"d6d8f0b3-5098-4746-8e75-aee84eafa17b\") " pod="openshift-dns/dns-default-g6qq7"
Sep 11 17:19:39.119852 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:19:39.119688    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/f68e4b95-ae13-4835-976a-a1af09fc70cf-crio-socket\") pod \"insights-runtime-extractor-vnvj4\" (UID: \"f68e4b95-ae13-4835-976a-a1af09fc70cf\") " pod="openshift-insights/insights-runtime-extractor-vnvj4"
Sep 11 17:19:39.119852 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:19:39.119764    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/f68e4b95-ae13-4835-976a-a1af09fc70cf-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-vnvj4\" (UID: \"f68e4b95-ae13-4835-976a-a1af09fc70cf\") " pod="openshift-insights/insights-runtime-extractor-vnvj4"
Sep 11 17:19:39.119852 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:19:39.119834    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b9txf\" (UniqueName: \"kubernetes.io/projected/f68e4b95-ae13-4835-976a-a1af09fc70cf-kube-api-access-b9txf\") pod \"insights-runtime-extractor-vnvj4\" (UID: \"f68e4b95-ae13-4835-976a-a1af09fc70cf\") " pod="openshift-insights/insights-runtime-extractor-vnvj4"
Sep 11 17:19:39.120448 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:19:39.120044    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d6d8f0b3-5098-4746-8e75-aee84eafa17b-tmp-dir\") pod \"dns-default-g6qq7\" (UID: \"d6d8f0b3-5098-4746-8e75-aee84eafa17b\") " pod="openshift-dns/dns-default-g6qq7"
Sep 11 17:19:39.120448 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:19:39.120068    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/894d4606-cd1f-45ae-a9fa-a3b9b5e00975-cert\") pod \"ingress-canary-tt7ph\" (UID: \"894d4606-cd1f-45ae-a9fa-a3b9b5e00975\") " pod="openshift-ingress-canary/ingress-canary-tt7ph"
Sep 11 17:19:39.120448 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:19:39.120112    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/f68e4b95-ae13-4835-976a-a1af09fc70cf-data-volume\") pod \"insights-runtime-extractor-vnvj4\" (UID: \"f68e4b95-ae13-4835-976a-a1af09fc70cf\") " pod="openshift-insights/insights-runtime-extractor-vnvj4"
Sep 11 17:19:39.120448 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:19:39.120267    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/d6d8f0b3-5098-4746-8e75-aee84eafa17b-config-volume\") pod \"dns-default-g6qq7\" (UID: \"d6d8f0b3-5098-4746-8e75-aee84eafa17b\") " pod="openshift-dns/dns-default-g6qq7"
Sep 11 17:19:39.120448 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:19:39.120410    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/f68e4b95-ae13-4835-976a-a1af09fc70cf-data-volume\") pod \"insights-runtime-extractor-vnvj4\" (UID: \"f68e4b95-ae13-4835-976a-a1af09fc70cf\") " pod="openshift-insights/insights-runtime-extractor-vnvj4"
Sep 11 17:19:39.120448 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:19:39.120427    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/f68e4b95-ae13-4835-976a-a1af09fc70cf-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-vnvj4\" (UID: \"f68e4b95-ae13-4835-976a-a1af09fc70cf\") " pod="openshift-insights/insights-runtime-extractor-vnvj4"
Sep 11 17:19:39.123579 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:19:39.123559    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/f68e4b95-ae13-4835-976a-a1af09fc70cf-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-vnvj4\" (UID: \"f68e4b95-ae13-4835-976a-a1af09fc70cf\") " pod="openshift-insights/insights-runtime-extractor-vnvj4"
Sep 11 17:19:39.123664 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:19:39.123607    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/d6d8f0b3-5098-4746-8e75-aee84eafa17b-metrics-tls\") pod \"dns-default-g6qq7\" (UID: \"d6d8f0b3-5098-4746-8e75-aee84eafa17b\") " pod="openshift-dns/dns-default-g6qq7"
Sep 11 17:19:39.123722 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:19:39.123705    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/894d4606-cd1f-45ae-a9fa-a3b9b5e00975-cert\") pod \"ingress-canary-tt7ph\" (UID: \"894d4606-cd1f-45ae-a9fa-a3b9b5e00975\") " pod="openshift-ingress-canary/ingress-canary-tt7ph"
Sep 11 17:19:39.138172 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:19:39.138141    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-b9txf\" (UniqueName: \"kubernetes.io/projected/f68e4b95-ae13-4835-976a-a1af09fc70cf-kube-api-access-b9txf\") pod \"insights-runtime-extractor-vnvj4\" (UID: \"f68e4b95-ae13-4835-976a-a1af09fc70cf\") " pod="openshift-insights/insights-runtime-extractor-vnvj4"
Sep 11 17:19:39.138635 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:19:39.138614    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6wwzv\" (UniqueName: \"kubernetes.io/projected/d6d8f0b3-5098-4746-8e75-aee84eafa17b-kube-api-access-6wwzv\") pod \"dns-default-g6qq7\" (UID: \"d6d8f0b3-5098-4746-8e75-aee84eafa17b\") " pod="openshift-dns/dns-default-g6qq7"
Sep 11 17:19:39.139823 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:19:39.139806    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rfhtg\" (UniqueName: \"kubernetes.io/projected/894d4606-cd1f-45ae-a9fa-a3b9b5e00975-kube-api-access-rfhtg\") pod \"ingress-canary-tt7ph\" (UID: \"894d4606-cd1f-45ae-a9fa-a3b9b5e00975\") " pod="openshift-ingress-canary/ingress-canary-tt7ph"
Sep 11 17:19:39.162681 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:19:39.162652    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-tt7ph"
Sep 11 17:19:39.167487 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:19:39.167468    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-vnvj4"
Sep 11 17:19:39.247399 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:19:39.247372    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-g6qq7"
Sep 11 17:19:39.457604 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:19:39.457579    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-vnvj4"]
Sep 11 17:19:39.461326 ip-10-0-141-139 kubenswrapper[2577]: W0911 17:19:39.461294    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf68e4b95_ae13_4835_976a_a1af09fc70cf.slice/crio-b11b0684024c9febfdb34d346aa75824f4527b0fcfc61d3505fd62f6a0b24f4b WatchSource:0}: Error finding container b11b0684024c9febfdb34d346aa75824f4527b0fcfc61d3505fd62f6a0b24f4b: Status 404 returned error can't find the container with id b11b0684024c9febfdb34d346aa75824f4527b0fcfc61d3505fd62f6a0b24f4b
Sep 11 17:19:39.462576 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:19:39.462555    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-tt7ph"]
Sep 11 17:19:39.465184 ip-10-0-141-139 kubenswrapper[2577]: W0911 17:19:39.465163    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod894d4606_cd1f_45ae_a9fa_a3b9b5e00975.slice/crio-87feaf0f211ac390e734921180331f8377b1a74d71cea845d46bbd626e9ed5cd WatchSource:0}: Error finding container 87feaf0f211ac390e734921180331f8377b1a74d71cea845d46bbd626e9ed5cd: Status 404 returned error can't find the container with id 87feaf0f211ac390e734921180331f8377b1a74d71cea845d46bbd626e9ed5cd
Sep 11 17:19:39.538199 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:19:39.538171    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-g6qq7"]
Sep 11 17:19:39.541153 ip-10-0-141-139 kubenswrapper[2577]: W0911 17:19:39.541128    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd6d8f0b3_5098_4746_8e75_aee84eafa17b.slice/crio-09da3cce36cf46be3abcf6fafa5d873673bbf4b938b18df35a0a6e8158e532d3 WatchSource:0}: Error finding container 09da3cce36cf46be3abcf6fafa5d873673bbf4b938b18df35a0a6e8158e532d3: Status 404 returned error can't find the container with id 09da3cce36cf46be3abcf6fafa5d873673bbf4b938b18df35a0a6e8158e532d3
Sep 11 17:19:40.231555 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:19:40.231518    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-vnvj4" event={"ID":"f68e4b95-ae13-4835-976a-a1af09fc70cf","Type":"ContainerStarted","Data":"76f66bedf37e1ceb7a9d9d243bd52645247e3558086e3038223928c2fc2e77b6"}
Sep 11 17:19:40.232006 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:19:40.231603    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-vnvj4" event={"ID":"f68e4b95-ae13-4835-976a-a1af09fc70cf","Type":"ContainerStarted","Data":"89ed41e64d684a962fdef676c13be18eb258b89d930574c9be574f671c73c903"}
Sep 11 17:19:40.232006 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:19:40.231623    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-vnvj4" event={"ID":"f68e4b95-ae13-4835-976a-a1af09fc70cf","Type":"ContainerStarted","Data":"b11b0684024c9febfdb34d346aa75824f4527b0fcfc61d3505fd62f6a0b24f4b"}
Sep 11 17:19:40.232783 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:19:40.232750    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-tt7ph" event={"ID":"894d4606-cd1f-45ae-a9fa-a3b9b5e00975","Type":"ContainerStarted","Data":"87feaf0f211ac390e734921180331f8377b1a74d71cea845d46bbd626e9ed5cd"}
Sep 11 17:19:40.233803 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:19:40.233782    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-g6qq7" event={"ID":"d6d8f0b3-5098-4746-8e75-aee84eafa17b","Type":"ContainerStarted","Data":"09da3cce36cf46be3abcf6fafa5d873673bbf4b938b18df35a0a6e8158e532d3"}
Sep 11 17:19:40.377359 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:19:40.377269    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-5c74f887d6-ttmcp"]
Sep 11 17:19:40.379252 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:19:40.379225    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5c74f887d6-ttmcp"
Sep 11 17:19:40.384459 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:19:40.384433    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\""
Sep 11 17:19:40.384964 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:19:40.384931    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\""
Sep 11 17:19:40.385067 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:19:40.384982    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\""
Sep 11 17:19:40.385067 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:19:40.384997    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\""
Sep 11 17:19:40.385185 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:19:40.385153    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-tk8qn\""
Sep 11 17:19:40.385253 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:19:40.385232    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\""
Sep 11 17:19:40.385374 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:19:40.385279    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\""
Sep 11 17:19:40.385374 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:19:40.385234    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\""
Sep 11 17:19:40.398693 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:19:40.398667    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-5c74f887d6-ttmcp"]
Sep 11 17:19:40.531031 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:19:40.530991    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-84khb\" (UniqueName: \"kubernetes.io/projected/067ed90b-10ca-413b-9c86-3b48b474d237-kube-api-access-84khb\") pod \"console-5c74f887d6-ttmcp\" (UID: \"067ed90b-10ca-413b-9c86-3b48b474d237\") " pod="openshift-console/console-5c74f887d6-ttmcp"
Sep 11 17:19:40.531031 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:19:40.531031    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/067ed90b-10ca-413b-9c86-3b48b474d237-service-ca\") pod \"console-5c74f887d6-ttmcp\" (UID: \"067ed90b-10ca-413b-9c86-3b48b474d237\") " pod="openshift-console/console-5c74f887d6-ttmcp"
Sep 11 17:19:40.531281 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:19:40.531048    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/067ed90b-10ca-413b-9c86-3b48b474d237-console-config\") pod \"console-5c74f887d6-ttmcp\" (UID: \"067ed90b-10ca-413b-9c86-3b48b474d237\") " pod="openshift-console/console-5c74f887d6-ttmcp"
Sep 11 17:19:40.531281 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:19:40.531202    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/067ed90b-10ca-413b-9c86-3b48b474d237-console-serving-cert\") pod \"console-5c74f887d6-ttmcp\" (UID: \"067ed90b-10ca-413b-9c86-3b48b474d237\") " pod="openshift-console/console-5c74f887d6-ttmcp"
Sep 11 17:19:40.531281 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:19:40.531236    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/067ed90b-10ca-413b-9c86-3b48b474d237-console-oauth-config\") pod \"console-5c74f887d6-ttmcp\" (UID: \"067ed90b-10ca-413b-9c86-3b48b474d237\") " pod="openshift-console/console-5c74f887d6-ttmcp"
Sep 11 17:19:40.531281 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:19:40.531265    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/067ed90b-10ca-413b-9c86-3b48b474d237-oauth-serving-cert\") pod \"console-5c74f887d6-ttmcp\" (UID: \"067ed90b-10ca-413b-9c86-3b48b474d237\") " pod="openshift-console/console-5c74f887d6-ttmcp"
Sep 11 17:19:40.632449 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:19:40.632360    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/067ed90b-10ca-413b-9c86-3b48b474d237-oauth-serving-cert\") pod \"console-5c74f887d6-ttmcp\" (UID: \"067ed90b-10ca-413b-9c86-3b48b474d237\") " pod="openshift-console/console-5c74f887d6-ttmcp"
Sep 11 17:19:40.632449 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:19:40.632422    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-84khb\" (UniqueName: \"kubernetes.io/projected/067ed90b-10ca-413b-9c86-3b48b474d237-kube-api-access-84khb\") pod \"console-5c74f887d6-ttmcp\" (UID: \"067ed90b-10ca-413b-9c86-3b48b474d237\") " pod="openshift-console/console-5c74f887d6-ttmcp"
Sep 11 17:19:40.632680 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:19:40.632459    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/067ed90b-10ca-413b-9c86-3b48b474d237-service-ca\") pod \"console-5c74f887d6-ttmcp\" (UID: \"067ed90b-10ca-413b-9c86-3b48b474d237\") " pod="openshift-console/console-5c74f887d6-ttmcp"
Sep 11 17:19:40.632680 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:19:40.632484    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/067ed90b-10ca-413b-9c86-3b48b474d237-console-config\") pod \"console-5c74f887d6-ttmcp\" (UID: \"067ed90b-10ca-413b-9c86-3b48b474d237\") " pod="openshift-console/console-5c74f887d6-ttmcp"
Sep 11 17:19:40.632680 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:19:40.632575    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/067ed90b-10ca-413b-9c86-3b48b474d237-console-serving-cert\") pod \"console-5c74f887d6-ttmcp\" (UID: \"067ed90b-10ca-413b-9c86-3b48b474d237\") " pod="openshift-console/console-5c74f887d6-ttmcp"
Sep 11 17:19:40.632680 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:19:40.632598    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/067ed90b-10ca-413b-9c86-3b48b474d237-console-oauth-config\") pod \"console-5c74f887d6-ttmcp\" (UID: \"067ed90b-10ca-413b-9c86-3b48b474d237\") " pod="openshift-console/console-5c74f887d6-ttmcp"
Sep 11 17:19:40.633252 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:19:40.633221    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/067ed90b-10ca-413b-9c86-3b48b474d237-oauth-serving-cert\") pod \"console-5c74f887d6-ttmcp\" (UID: \"067ed90b-10ca-413b-9c86-3b48b474d237\") " pod="openshift-console/console-5c74f887d6-ttmcp"
Sep 11 17:19:40.633588 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:19:40.633567    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/067ed90b-10ca-413b-9c86-3b48b474d237-service-ca\") pod \"console-5c74f887d6-ttmcp\" (UID: \"067ed90b-10ca-413b-9c86-3b48b474d237\") " pod="openshift-console/console-5c74f887d6-ttmcp"
Sep 11 17:19:40.633813 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:19:40.633791    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/067ed90b-10ca-413b-9c86-3b48b474d237-console-config\") pod \"console-5c74f887d6-ttmcp\" (UID: \"067ed90b-10ca-413b-9c86-3b48b474d237\") " pod="openshift-console/console-5c74f887d6-ttmcp"
Sep 11 17:19:40.635475 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:19:40.635451    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/067ed90b-10ca-413b-9c86-3b48b474d237-console-serving-cert\") pod \"console-5c74f887d6-ttmcp\" (UID: \"067ed90b-10ca-413b-9c86-3b48b474d237\") " pod="openshift-console/console-5c74f887d6-ttmcp"
Sep 11 17:19:40.635594 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:19:40.635527    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/067ed90b-10ca-413b-9c86-3b48b474d237-console-oauth-config\") pod \"console-5c74f887d6-ttmcp\" (UID: \"067ed90b-10ca-413b-9c86-3b48b474d237\") " pod="openshift-console/console-5c74f887d6-ttmcp"
Sep 11 17:19:40.641488 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:19:40.641466    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-84khb\" (UniqueName: \"kubernetes.io/projected/067ed90b-10ca-413b-9c86-3b48b474d237-kube-api-access-84khb\") pod \"console-5c74f887d6-ttmcp\" (UID: \"067ed90b-10ca-413b-9c86-3b48b474d237\") " pod="openshift-console/console-5c74f887d6-ttmcp"
Sep 11 17:19:40.690312 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:19:40.690273    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5c74f887d6-ttmcp"
Sep 11 17:19:40.973874 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:19:40.973839    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-5c74f887d6-ttmcp"]
Sep 11 17:19:40.976858 ip-10-0-141-139 kubenswrapper[2577]: W0911 17:19:40.976828    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod067ed90b_10ca_413b_9c86_3b48b474d237.slice/crio-ab7a86735f9e94d087efc4c9704b0dd23a3986c9525be0dcd7e32ac5fd84723a WatchSource:0}: Error finding container ab7a86735f9e94d087efc4c9704b0dd23a3986c9525be0dcd7e32ac5fd84723a: Status 404 returned error can't find the container with id ab7a86735f9e94d087efc4c9704b0dd23a3986c9525be0dcd7e32ac5fd84723a
Sep 11 17:19:41.242092 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:19:41.241998    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5c74f887d6-ttmcp" event={"ID":"067ed90b-10ca-413b-9c86-3b48b474d237","Type":"ContainerStarted","Data":"ab7a86735f9e94d087efc4c9704b0dd23a3986c9525be0dcd7e32ac5fd84723a"}
Sep 11 17:19:42.081454 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:19:42.081418    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-68bcd7f987-j6qnv"]
Sep 11 17:19:42.083569 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:19:42.083541    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-68bcd7f987-j6qnv"
Sep 11 17:19:42.087661 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:19:42.087631    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-kube-rbac-proxy-config\""
Sep 11 17:19:42.087778 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:19:42.087680    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Sep 11 17:19:42.088092 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:19:42.088065    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-tls\""
Sep 11 17:19:42.088810 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:19:42.088778    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-dockercfg-8lqs4\""
Sep 11 17:19:42.088810 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:19:42.088802    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Sep 11 17:19:42.088949 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:19:42.088822    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Sep 11 17:19:42.097886 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:19:42.097826    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-68bcd7f987-j6qnv"]
Sep 11 17:19:42.245769 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:19:42.245735    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/f066e3ca-d446-4516-bc85-d3e0b841ba58-prometheus-operator-tls\") pod \"prometheus-operator-68bcd7f987-j6qnv\" (UID: \"f066e3ca-d446-4516-bc85-d3e0b841ba58\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-j6qnv"
Sep 11 17:19:42.245769 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:19:42.245778    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/f066e3ca-d446-4516-bc85-d3e0b841ba58-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-68bcd7f987-j6qnv\" (UID: \"f066e3ca-d446-4516-bc85-d3e0b841ba58\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-j6qnv"
Sep 11 17:19:42.246186 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:19:42.245810    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f066e3ca-d446-4516-bc85-d3e0b841ba58-metrics-client-ca\") pod \"prometheus-operator-68bcd7f987-j6qnv\" (UID: \"f066e3ca-d446-4516-bc85-d3e0b841ba58\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-j6qnv"
Sep 11 17:19:42.246186 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:19:42.245829    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nsc2r\" (UniqueName: \"kubernetes.io/projected/f066e3ca-d446-4516-bc85-d3e0b841ba58-kube-api-access-nsc2r\") pod \"prometheus-operator-68bcd7f987-j6qnv\" (UID: \"f066e3ca-d446-4516-bc85-d3e0b841ba58\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-j6qnv"
Sep 11 17:19:42.347298 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:19:42.347201    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/f066e3ca-d446-4516-bc85-d3e0b841ba58-prometheus-operator-tls\") pod \"prometheus-operator-68bcd7f987-j6qnv\" (UID: \"f066e3ca-d446-4516-bc85-d3e0b841ba58\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-j6qnv"
Sep 11 17:19:42.347298 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:19:42.347257    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/f066e3ca-d446-4516-bc85-d3e0b841ba58-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-68bcd7f987-j6qnv\" (UID: \"f066e3ca-d446-4516-bc85-d3e0b841ba58\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-j6qnv"
Sep 11 17:19:42.347564 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:19:42.347318    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f066e3ca-d446-4516-bc85-d3e0b841ba58-metrics-client-ca\") pod \"prometheus-operator-68bcd7f987-j6qnv\" (UID: \"f066e3ca-d446-4516-bc85-d3e0b841ba58\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-j6qnv"
Sep 11 17:19:42.347564 ip-10-0-141-139 kubenswrapper[2577]: E0911 17:19:42.347336    2577 secret.go:189] Couldn't get secret openshift-monitoring/prometheus-operator-tls: secret "prometheus-operator-tls" not found
Sep 11 17:19:42.347564 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:19:42.347343    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nsc2r\" (UniqueName: \"kubernetes.io/projected/f066e3ca-d446-4516-bc85-d3e0b841ba58-kube-api-access-nsc2r\") pod \"prometheus-operator-68bcd7f987-j6qnv\" (UID: \"f066e3ca-d446-4516-bc85-d3e0b841ba58\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-j6qnv"
Sep 11 17:19:42.347564 ip-10-0-141-139 kubenswrapper[2577]: E0911 17:19:42.347406    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f066e3ca-d446-4516-bc85-d3e0b841ba58-prometheus-operator-tls podName:f066e3ca-d446-4516-bc85-d3e0b841ba58 nodeName:}" failed. No retries permitted until 2026-09-11 17:19:42.847385332 +0000 UTC m=+60.420605442 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "prometheus-operator-tls" (UniqueName: "kubernetes.io/secret/f066e3ca-d446-4516-bc85-d3e0b841ba58-prometheus-operator-tls") pod "prometheus-operator-68bcd7f987-j6qnv" (UID: "f066e3ca-d446-4516-bc85-d3e0b841ba58") : secret "prometheus-operator-tls" not found
Sep 11 17:19:42.348644 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:19:42.348616    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f066e3ca-d446-4516-bc85-d3e0b841ba58-metrics-client-ca\") pod \"prometheus-operator-68bcd7f987-j6qnv\" (UID: \"f066e3ca-d446-4516-bc85-d3e0b841ba58\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-j6qnv"
Sep 11 17:19:42.350883 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:19:42.350856    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/f066e3ca-d446-4516-bc85-d3e0b841ba58-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-68bcd7f987-j6qnv\" (UID: \"f066e3ca-d446-4516-bc85-d3e0b841ba58\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-j6qnv"
Sep 11 17:19:42.359041 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:19:42.358988    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nsc2r\" (UniqueName: \"kubernetes.io/projected/f066e3ca-d446-4516-bc85-d3e0b841ba58-kube-api-access-nsc2r\") pod \"prometheus-operator-68bcd7f987-j6qnv\" (UID: \"f066e3ca-d446-4516-bc85-d3e0b841ba58\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-j6qnv"
Sep 11 17:19:42.851440 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:19:42.851408    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/f066e3ca-d446-4516-bc85-d3e0b841ba58-prometheus-operator-tls\") pod \"prometheus-operator-68bcd7f987-j6qnv\" (UID: \"f066e3ca-d446-4516-bc85-d3e0b841ba58\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-j6qnv"
Sep 11 17:19:42.854746 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:19:42.854706    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/f066e3ca-d446-4516-bc85-d3e0b841ba58-prometheus-operator-tls\") pod \"prometheus-operator-68bcd7f987-j6qnv\" (UID: \"f066e3ca-d446-4516-bc85-d3e0b841ba58\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-j6qnv"
Sep 11 17:19:42.996965 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:19:42.996928    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-dockercfg-8lqs4\""
Sep 11 17:19:43.004168 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:19:43.004130    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-68bcd7f987-j6qnv"
Sep 11 17:19:44.164231 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:19:44.164188    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-sn5kw"
Sep 11 17:19:45.015606 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:19:45.015518    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-68bcd7f987-j6qnv"]
Sep 11 17:19:45.018222 ip-10-0-141-139 kubenswrapper[2577]: W0911 17:19:45.018190    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf066e3ca_d446_4516_bc85_d3e0b841ba58.slice/crio-cc813b3d04c7d1e43e68c01ef2947483a3f762d4740dbc96bce390cea0dce20e WatchSource:0}: Error finding container cc813b3d04c7d1e43e68c01ef2947483a3f762d4740dbc96bce390cea0dce20e: Status 404 returned error can't find the container with id cc813b3d04c7d1e43e68c01ef2947483a3f762d4740dbc96bce390cea0dce20e
Sep 11 17:19:45.255351 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:19:45.255318    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-68bcd7f987-j6qnv" event={"ID":"f066e3ca-d446-4516-bc85-d3e0b841ba58","Type":"ContainerStarted","Data":"cc813b3d04c7d1e43e68c01ef2947483a3f762d4740dbc96bce390cea0dce20e"}
Sep 11 17:19:45.257064 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:19:45.257041    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-g6qq7" event={"ID":"d6d8f0b3-5098-4746-8e75-aee84eafa17b","Type":"ContainerStarted","Data":"6cc780806314c9d43eb5a5d384fdb043d0dc4938918f98443813db60af3b52b2"}
Sep 11 17:19:45.257129 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:19:45.257085    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-g6qq7" event={"ID":"d6d8f0b3-5098-4746-8e75-aee84eafa17b","Type":"ContainerStarted","Data":"e42a83f6941e992fe8710a4633e370d11f528df7900557bb4afb29650e7fb3f8"}
Sep 11 17:19:45.257184 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:19:45.257158    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-g6qq7"
Sep 11 17:19:45.259345 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:19:45.259325    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-vnvj4" event={"ID":"f68e4b95-ae13-4835-976a-a1af09fc70cf","Type":"ContainerStarted","Data":"440e4d062166253ea5c618c6cb390ef7749f1351e6c4d28bb8f0ff0495855245"}
Sep 11 17:19:45.260886 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:19:45.260864    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-tt7ph" event={"ID":"894d4606-cd1f-45ae-a9fa-a3b9b5e00975","Type":"ContainerStarted","Data":"48fedf853964a862db612d5b64a672982744a244f258271780e64567774bb728"}
Sep 11 17:19:45.262443 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:19:45.262420    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5c74f887d6-ttmcp" event={"ID":"067ed90b-10ca-413b-9c86-3b48b474d237","Type":"ContainerStarted","Data":"7613c98f189145c515d6414f488c1079ec43a3984dfc131fb51847463cabbd0a"}
Sep 11 17:19:45.284313 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:19:45.284211    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-g6qq7" podStartSLOduration=2.106804776 podStartE2EDuration="7.284196922s" totalImagesPullingTime="5.177392146s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 17:19:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 17:19:45.282020967 +0000 UTC m=+62.855241079" watchObservedRunningTime="2026-09-11 17:19:45.284196922 +0000 UTC m=+62.857417028"
Sep 11 17:19:45.325910 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:19:45.325858    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-tt7ph" podStartSLOduration=2.071977006 podStartE2EDuration="7.3258396s" totalImagesPullingTime="5.253862594s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 17:19:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 17:19:45.325184706 +0000 UTC m=+62.898404818" watchObservedRunningTime="2026-09-11 17:19:45.3258396 +0000 UTC m=+62.899059712"
Sep 11 17:19:45.365075 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:19:45.365020    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-vnvj4" podStartSLOduration=2.225592462 podStartE2EDuration="7.365003263s" totalImagesPullingTime="5.139410801s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 17:19:38 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 17:19:45.364761897 +0000 UTC m=+62.937982009" watchObservedRunningTime="2026-09-11 17:19:45.365003263 +0000 UTC m=+62.938223424"
Sep 11 17:19:45.401633 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:19:45.401584    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-5c74f887d6-ttmcp" podStartSLOduration=1.6166015759999999 podStartE2EDuration="5.401567191s" totalImagesPullingTime="3.784965615s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 17:19:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 17:19:45.400250713 +0000 UTC m=+62.973470825" watchObservedRunningTime="2026-09-11 17:19:45.401567191 +0000 UTC m=+62.974787333"
Sep 11 17:19:47.271525 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:19:47.271467    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-68bcd7f987-j6qnv" event={"ID":"f066e3ca-d446-4516-bc85-d3e0b841ba58","Type":"ContainerStarted","Data":"38391cd6c3231d9abca4fcfee116d732a0700294dc1d22ccaf54fe78408cc507"}
Sep 11 17:19:47.272015 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:19:47.271546    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-68bcd7f987-j6qnv" event={"ID":"f066e3ca-d446-4516-bc85-d3e0b841ba58","Type":"ContainerStarted","Data":"166e9807ae61020edd2bd0f2235c90abe0c754f839ba14d1713dda753560f1a2"}
Sep 11 17:19:47.691576 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:19:47.691463    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5c1ac293-cc19-4446-8244-a8ca7fc8a8f4-metrics-certs\") pod \"network-metrics-daemon-bl4vr\" (UID: \"5c1ac293-cc19-4446-8244-a8ca7fc8a8f4\") " pod="openshift-multus/network-metrics-daemon-bl4vr"
Sep 11 17:19:47.691730 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:19:47.691592    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/73dd9a34-fac4-44c9-96d3-b2c8968ed271-original-pull-secret\") pod \"global-pull-secret-syncer-fsh8r\" (UID: \"73dd9a34-fac4-44c9-96d3-b2c8968ed271\") " pod="kube-system/global-pull-secret-syncer-fsh8r"
Sep 11 17:19:47.694626 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:19:47.694603    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Sep 11 17:19:47.694626 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:19:47.694621    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Sep 11 17:19:47.704378 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:19:47.704344    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/73dd9a34-fac4-44c9-96d3-b2c8968ed271-original-pull-secret\") pod \"global-pull-secret-syncer-fsh8r\" (UID: \"73dd9a34-fac4-44c9-96d3-b2c8968ed271\") " pod="kube-system/global-pull-secret-syncer-fsh8r"
Sep 11 17:19:47.704473 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:19:47.704427    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5c1ac293-cc19-4446-8244-a8ca7fc8a8f4-metrics-certs\") pod \"network-metrics-daemon-bl4vr\" (UID: \"5c1ac293-cc19-4446-8244-a8ca7fc8a8f4\") " pod="openshift-multus/network-metrics-daemon-bl4vr"
Sep 11 17:19:47.791968 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:19:47.791917    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h2trs\" (UniqueName: \"kubernetes.io/projected/1e5d9e0c-7bc0-4fcf-8845-8873df5b1bee-kube-api-access-h2trs\") pod \"network-check-target-xfn47\" (UID: \"1e5d9e0c-7bc0-4fcf-8845-8873df5b1bee\") " pod="openshift-network-diagnostics/network-check-target-xfn47"
Sep 11 17:19:47.794696 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:19:47.794677    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Sep 11 17:19:47.805542 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:19:47.805521    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Sep 11 17:19:47.816754 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:19:47.816725    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-h2trs\" (UniqueName: \"kubernetes.io/projected/1e5d9e0c-7bc0-4fcf-8845-8873df5b1bee-kube-api-access-h2trs\") pod \"network-check-target-xfn47\" (UID: \"1e5d9e0c-7bc0-4fcf-8845-8873df5b1bee\") " pod="openshift-network-diagnostics/network-check-target-xfn47"
Sep 11 17:19:47.934548 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:19:47.934494    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-vthbj\""
Sep 11 17:19:47.936592 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:19:47.936574    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fsh8r"
Sep 11 17:19:47.941571 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:19:47.941549    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xfn47"
Sep 11 17:19:47.945119 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:19:47.945101    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-gdccs\""
Sep 11 17:19:47.953324 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:19:47.953297    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bl4vr"
Sep 11 17:19:48.222706 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:19:48.222648    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-68bcd7f987-j6qnv" podStartSLOduration=4.813801578 podStartE2EDuration="6.222624363s" totalImagesPullingTime="1.408822785s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 17:19:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 17:19:47.296476276 +0000 UTC m=+64.869696388" watchObservedRunningTime="2026-09-11 17:19:48.222624363 +0000 UTC m=+65.795844476"
Sep 11 17:19:48.223718 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:19:48.223697    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-fsh8r"]
Sep 11 17:19:48.224583 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:19:48.224562    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-xfn47"]
Sep 11 17:19:48.226450 ip-10-0-141-139 kubenswrapper[2577]: W0911 17:19:48.226427    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1e5d9e0c_7bc0_4fcf_8845_8873df5b1bee.slice/crio-6f42edce81afba27bdf9fa9e86d44335855fc96f6f199170ae3f862b5ea1f2ef WatchSource:0}: Error finding container 6f42edce81afba27bdf9fa9e86d44335855fc96f6f199170ae3f862b5ea1f2ef: Status 404 returned error can't find the container with id 6f42edce81afba27bdf9fa9e86d44335855fc96f6f199170ae3f862b5ea1f2ef
Sep 11 17:19:48.227099 ip-10-0-141-139 kubenswrapper[2577]: W0911 17:19:48.227038    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod73dd9a34_fac4_44c9_96d3_b2c8968ed271.slice/crio-9fc1167763ce9ca8d60ad8257cf1bee43f68c44311582ca62c8c5e35954aa58c WatchSource:0}: Error finding container 9fc1167763ce9ca8d60ad8257cf1bee43f68c44311582ca62c8c5e35954aa58c: Status 404 returned error can't find the container with id 9fc1167763ce9ca8d60ad8257cf1bee43f68c44311582ca62c8c5e35954aa58c
Sep 11 17:19:48.247915 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:19:48.247885    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-bl4vr"]
Sep 11 17:19:48.251054 ip-10-0-141-139 kubenswrapper[2577]: W0911 17:19:48.251033    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5c1ac293_cc19_4446_8244_a8ca7fc8a8f4.slice/crio-2ca55ec468fb1f283438570959da2230e023f07dc2ca4aa16f71ea0840ca70bc WatchSource:0}: Error finding container 2ca55ec468fb1f283438570959da2230e023f07dc2ca4aa16f71ea0840ca70bc: Status 404 returned error can't find the container with id 2ca55ec468fb1f283438570959da2230e023f07dc2ca4aa16f71ea0840ca70bc
Sep 11 17:19:48.276535 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:19:48.276464    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-bl4vr" event={"ID":"5c1ac293-cc19-4446-8244-a8ca7fc8a8f4","Type":"ContainerStarted","Data":"2ca55ec468fb1f283438570959da2230e023f07dc2ca4aa16f71ea0840ca70bc"}
Sep 11 17:19:48.278431 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:19:48.278403    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-xfn47" event={"ID":"1e5d9e0c-7bc0-4fcf-8845-8873df5b1bee","Type":"ContainerStarted","Data":"6f42edce81afba27bdf9fa9e86d44335855fc96f6f199170ae3f862b5ea1f2ef"}
Sep 11 17:19:48.279688 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:19:48.279659    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-fsh8r" event={"ID":"73dd9a34-fac4-44c9-96d3-b2c8968ed271","Type":"ContainerStarted","Data":"9fc1167763ce9ca8d60ad8257cf1bee43f68c44311582ca62c8c5e35954aa58c"}
Sep 11 17:19:48.556368 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:19:48.556285    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-57bf98d9fc-nrdtq"]
Sep 11 17:19:48.558904 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:19:48.558880    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-57bf98d9fc-nrdtq"
Sep 11 17:19:48.567364 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:19:48.567345    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\""
Sep 11 17:19:48.576705 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:19:48.576681    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-57bf98d9fc-nrdtq"]
Sep 11 17:19:48.698795 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:19:48.698756    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/85583e11-cdb9-4c56-9aab-aa6a53641fda-oauth-serving-cert\") pod \"console-57bf98d9fc-nrdtq\" (UID: \"85583e11-cdb9-4c56-9aab-aa6a53641fda\") " pod="openshift-console/console-57bf98d9fc-nrdtq"
Sep 11 17:19:48.698795 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:19:48.698798    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t9gsg\" (UniqueName: \"kubernetes.io/projected/85583e11-cdb9-4c56-9aab-aa6a53641fda-kube-api-access-t9gsg\") pod \"console-57bf98d9fc-nrdtq\" (UID: \"85583e11-cdb9-4c56-9aab-aa6a53641fda\") " pod="openshift-console/console-57bf98d9fc-nrdtq"
Sep 11 17:19:48.699042 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:19:48.698826    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/85583e11-cdb9-4c56-9aab-aa6a53641fda-console-serving-cert\") pod \"console-57bf98d9fc-nrdtq\" (UID: \"85583e11-cdb9-4c56-9aab-aa6a53641fda\") " pod="openshift-console/console-57bf98d9fc-nrdtq"
Sep 11 17:19:48.699042 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:19:48.698881    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/85583e11-cdb9-4c56-9aab-aa6a53641fda-trusted-ca-bundle\") pod \"console-57bf98d9fc-nrdtq\" (UID: \"85583e11-cdb9-4c56-9aab-aa6a53641fda\") " pod="openshift-console/console-57bf98d9fc-nrdtq"
Sep 11 17:19:48.699149 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:19:48.699037    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/85583e11-cdb9-4c56-9aab-aa6a53641fda-console-oauth-config\") pod \"console-57bf98d9fc-nrdtq\" (UID: \"85583e11-cdb9-4c56-9aab-aa6a53641fda\") " pod="openshift-console/console-57bf98d9fc-nrdtq"
Sep 11 17:19:48.699149 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:19:48.699065    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/85583e11-cdb9-4c56-9aab-aa6a53641fda-service-ca\") pod \"console-57bf98d9fc-nrdtq\" (UID: \"85583e11-cdb9-4c56-9aab-aa6a53641fda\") " pod="openshift-console/console-57bf98d9fc-nrdtq"
Sep 11 17:19:48.699149 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:19:48.699102    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/85583e11-cdb9-4c56-9aab-aa6a53641fda-console-config\") pod \"console-57bf98d9fc-nrdtq\" (UID: \"85583e11-cdb9-4c56-9aab-aa6a53641fda\") " pod="openshift-console/console-57bf98d9fc-nrdtq"
Sep 11 17:19:48.800066 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:19:48.800020    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/85583e11-cdb9-4c56-9aab-aa6a53641fda-console-oauth-config\") pod \"console-57bf98d9fc-nrdtq\" (UID: \"85583e11-cdb9-4c56-9aab-aa6a53641fda\") " pod="openshift-console/console-57bf98d9fc-nrdtq"
Sep 11 17:19:48.800280 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:19:48.800078    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/85583e11-cdb9-4c56-9aab-aa6a53641fda-service-ca\") pod \"console-57bf98d9fc-nrdtq\" (UID: \"85583e11-cdb9-4c56-9aab-aa6a53641fda\") " pod="openshift-console/console-57bf98d9fc-nrdtq"
Sep 11 17:19:48.800280 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:19:48.800114    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/85583e11-cdb9-4c56-9aab-aa6a53641fda-console-config\") pod \"console-57bf98d9fc-nrdtq\" (UID: \"85583e11-cdb9-4c56-9aab-aa6a53641fda\") " pod="openshift-console/console-57bf98d9fc-nrdtq"
Sep 11 17:19:48.800280 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:19:48.800146    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/85583e11-cdb9-4c56-9aab-aa6a53641fda-oauth-serving-cert\") pod \"console-57bf98d9fc-nrdtq\" (UID: \"85583e11-cdb9-4c56-9aab-aa6a53641fda\") " pod="openshift-console/console-57bf98d9fc-nrdtq"
Sep 11 17:19:48.800280 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:19:48.800173    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t9gsg\" (UniqueName: \"kubernetes.io/projected/85583e11-cdb9-4c56-9aab-aa6a53641fda-kube-api-access-t9gsg\") pod \"console-57bf98d9fc-nrdtq\" (UID: \"85583e11-cdb9-4c56-9aab-aa6a53641fda\") " pod="openshift-console/console-57bf98d9fc-nrdtq"
Sep 11 17:19:48.800280 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:19:48.800200    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/85583e11-cdb9-4c56-9aab-aa6a53641fda-console-serving-cert\") pod \"console-57bf98d9fc-nrdtq\" (UID: \"85583e11-cdb9-4c56-9aab-aa6a53641fda\") " pod="openshift-console/console-57bf98d9fc-nrdtq"
Sep 11 17:19:48.800280 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:19:48.800232    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/85583e11-cdb9-4c56-9aab-aa6a53641fda-trusted-ca-bundle\") pod \"console-57bf98d9fc-nrdtq\" (UID: \"85583e11-cdb9-4c56-9aab-aa6a53641fda\") " pod="openshift-console/console-57bf98d9fc-nrdtq"
Sep 11 17:19:48.801205 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:19:48.801174    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/85583e11-cdb9-4c56-9aab-aa6a53641fda-service-ca\") pod \"console-57bf98d9fc-nrdtq\" (UID: \"85583e11-cdb9-4c56-9aab-aa6a53641fda\") " pod="openshift-console/console-57bf98d9fc-nrdtq"
Sep 11 17:19:48.801474 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:19:48.801305    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/85583e11-cdb9-4c56-9aab-aa6a53641fda-oauth-serving-cert\") pod \"console-57bf98d9fc-nrdtq\" (UID: \"85583e11-cdb9-4c56-9aab-aa6a53641fda\") " pod="openshift-console/console-57bf98d9fc-nrdtq"
Sep 11 17:19:48.801612 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:19:48.801396    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/85583e11-cdb9-4c56-9aab-aa6a53641fda-console-config\") pod \"console-57bf98d9fc-nrdtq\" (UID: \"85583e11-cdb9-4c56-9aab-aa6a53641fda\") " pod="openshift-console/console-57bf98d9fc-nrdtq"
Sep 11 17:19:48.801686 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:19:48.801664    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/85583e11-cdb9-4c56-9aab-aa6a53641fda-trusted-ca-bundle\") pod \"console-57bf98d9fc-nrdtq\" (UID: \"85583e11-cdb9-4c56-9aab-aa6a53641fda\") " pod="openshift-console/console-57bf98d9fc-nrdtq"
Sep 11 17:19:48.803153 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:19:48.803124    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/85583e11-cdb9-4c56-9aab-aa6a53641fda-console-serving-cert\") pod \"console-57bf98d9fc-nrdtq\" (UID: \"85583e11-cdb9-4c56-9aab-aa6a53641fda\") " pod="openshift-console/console-57bf98d9fc-nrdtq"
Sep 11 17:19:48.803859 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:19:48.803839    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/85583e11-cdb9-4c56-9aab-aa6a53641fda-console-oauth-config\") pod \"console-57bf98d9fc-nrdtq\" (UID: \"85583e11-cdb9-4c56-9aab-aa6a53641fda\") " pod="openshift-console/console-57bf98d9fc-nrdtq"
Sep 11 17:19:48.812355 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:19:48.812283    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-t9gsg\" (UniqueName: \"kubernetes.io/projected/85583e11-cdb9-4c56-9aab-aa6a53641fda-kube-api-access-t9gsg\") pod \"console-57bf98d9fc-nrdtq\" (UID: \"85583e11-cdb9-4c56-9aab-aa6a53641fda\") " pod="openshift-console/console-57bf98d9fc-nrdtq"
Sep 11 17:19:48.869537 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:19:48.869481    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-57bf98d9fc-nrdtq"
Sep 11 17:19:49.173257 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:19:49.173168    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-57bf98d9fc-nrdtq"]
Sep 11 17:19:49.181547 ip-10-0-141-139 kubenswrapper[2577]: W0911 17:19:49.181493    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod85583e11_cdb9_4c56_9aab_aa6a53641fda.slice/crio-17645d3003be6f66d713a0e978aeaceadf23a05e21b9221d7e2fe27e7f02f615 WatchSource:0}: Error finding container 17645d3003be6f66d713a0e978aeaceadf23a05e21b9221d7e2fe27e7f02f615: Status 404 returned error can't find the container with id 17645d3003be6f66d713a0e978aeaceadf23a05e21b9221d7e2fe27e7f02f615
Sep 11 17:19:49.287066 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:19:49.286466    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-57bf98d9fc-nrdtq" event={"ID":"85583e11-cdb9-4c56-9aab-aa6a53641fda","Type":"ContainerStarted","Data":"7041f05901f1605ca2cfee51fdd897d5ae0124e0248fd3974ea8ddf8439362ec"}
Sep 11 17:19:49.287066 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:19:49.286549    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-57bf98d9fc-nrdtq" event={"ID":"85583e11-cdb9-4c56-9aab-aa6a53641fda","Type":"ContainerStarted","Data":"17645d3003be6f66d713a0e978aeaceadf23a05e21b9221d7e2fe27e7f02f615"}
Sep 11 17:19:49.335234 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:19:49.335023    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-57bf98d9fc-nrdtq" podStartSLOduration=1.334999576 podStartE2EDuration="1.334999576s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 17:19:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 17:19:49.330123564 +0000 UTC m=+66.903343680" watchObservedRunningTime="2026-09-11 17:19:49.334999576 +0000 UTC m=+66.908219692"
Sep 11 17:19:49.535888 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:19:49.535847    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/openshift-state-metrics-67dcf8c979-fwdgr"]
Sep 11 17:19:49.540792 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:19:49.540757    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-fwdgr"
Sep 11 17:19:49.544241 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:19:49.543496    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-dockercfg-72cp8\""
Sep 11 17:19:49.544241 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:19:49.543562    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-kube-rbac-proxy-config\""
Sep 11 17:19:49.544241 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:19:49.543832    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-tls\""
Sep 11 17:19:49.564655 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:19:49.564621    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-67dcf8c979-fwdgr"]
Sep 11 17:19:49.611314 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:19:49.611278    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/f1b075b6-9bae-41d6-b0c0-1c8b81c3207a-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-67dcf8c979-fwdgr\" (UID: \"f1b075b6-9bae-41d6-b0c0-1c8b81c3207a\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-fwdgr"
Sep 11 17:19:49.611536 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:19:49.611335    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/f1b075b6-9bae-41d6-b0c0-1c8b81c3207a-openshift-state-metrics-tls\") pod \"openshift-state-metrics-67dcf8c979-fwdgr\" (UID: \"f1b075b6-9bae-41d6-b0c0-1c8b81c3207a\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-fwdgr"
Sep 11 17:19:49.611536 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:19:49.611388    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-42szj\" (UniqueName: \"kubernetes.io/projected/f1b075b6-9bae-41d6-b0c0-1c8b81c3207a-kube-api-access-42szj\") pod \"openshift-state-metrics-67dcf8c979-fwdgr\" (UID: \"f1b075b6-9bae-41d6-b0c0-1c8b81c3207a\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-fwdgr"
Sep 11 17:19:49.611536 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:19:49.611405    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f1b075b6-9bae-41d6-b0c0-1c8b81c3207a-metrics-client-ca\") pod \"openshift-state-metrics-67dcf8c979-fwdgr\" (UID: \"f1b075b6-9bae-41d6-b0c0-1c8b81c3207a\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-fwdgr"
Sep 11 17:19:49.629764 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:19:49.629731    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/kube-state-metrics-59fbdd77d-bfz6b"]
Sep 11 17:19:49.632492 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:19:49.632469    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-59fbdd77d-bfz6b"
Sep 11 17:19:49.635134 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:19:49.635107    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-tls\""
Sep 11 17:19:49.635491 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:19:49.635469    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-kube-rbac-proxy-config\""
Sep 11 17:19:49.635842 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:19:49.635825    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-custom-resource-state-configmap\""
Sep 11 17:19:49.636091 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:19:49.636075    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-dockercfg-szhnd\""
Sep 11 17:19:49.647753 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:19:49.647651    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-59fbdd77d-bfz6b"]
Sep 11 17:19:49.656800 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:19:49.656748    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-mthpt"]
Sep 11 17:19:49.659565 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:19:49.659543    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-mthpt"
Sep 11 17:19:49.663395 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:19:49.663372    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Sep 11 17:19:49.663522 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:19:49.663424    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-rqg8q\""
Sep 11 17:19:49.664206 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:19:49.664189    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Sep 11 17:19:49.666580 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:19:49.666558    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Sep 11 17:19:49.713005 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:19:49.712742    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e41207d9-a802-4537-a459-01deb5c8fb16-metrics-client-ca\") pod \"kube-state-metrics-59fbdd77d-bfz6b\" (UID: \"e41207d9-a802-4537-a459-01deb5c8fb16\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-bfz6b"
Sep 11 17:19:49.713005 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:19:49.712790    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/df65734a-e4e6-4b4e-9114-35af7b635044-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-mthpt\" (UID: \"df65734a-e4e6-4b4e-9114-35af7b635044\") " pod="openshift-monitoring/node-exporter-mthpt"
Sep 11 17:19:49.713005 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:19:49.712827    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/f1b075b6-9bae-41d6-b0c0-1c8b81c3207a-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-67dcf8c979-fwdgr\" (UID: \"f1b075b6-9bae-41d6-b0c0-1c8b81c3207a\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-fwdgr"
Sep 11 17:19:49.713005 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:19:49.712883    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/e41207d9-a802-4537-a459-01deb5c8fb16-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-59fbdd77d-bfz6b\" (UID: \"e41207d9-a802-4537-a459-01deb5c8fb16\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-bfz6b"
Sep 11 17:19:49.713005 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:19:49.712916    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mpfmc\" (UniqueName: \"kubernetes.io/projected/e41207d9-a802-4537-a459-01deb5c8fb16-kube-api-access-mpfmc\") pod \"kube-state-metrics-59fbdd77d-bfz6b\" (UID: \"e41207d9-a802-4537-a459-01deb5c8fb16\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-bfz6b"
Sep 11 17:19:49.713005 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:19:49.712939    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/df65734a-e4e6-4b4e-9114-35af7b635044-sys\") pod \"node-exporter-mthpt\" (UID: \"df65734a-e4e6-4b4e-9114-35af7b635044\") " pod="openshift-monitoring/node-exporter-mthpt"
Sep 11 17:19:49.713005 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:19:49.712969    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/e41207d9-a802-4537-a459-01deb5c8fb16-volume-directive-shadow\") pod \"kube-state-metrics-59fbdd77d-bfz6b\" (UID: \"e41207d9-a802-4537-a459-01deb5c8fb16\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-bfz6b"
Sep 11 17:19:49.717442 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:19:49.714967    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/e41207d9-a802-4537-a459-01deb5c8fb16-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-59fbdd77d-bfz6b\" (UID: \"e41207d9-a802-4537-a459-01deb5c8fb16\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-bfz6b"
Sep 11 17:19:49.717442 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:19:49.715029    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/df65734a-e4e6-4b4e-9114-35af7b635044-node-exporter-textfile\") pod \"node-exporter-mthpt\" (UID: \"df65734a-e4e6-4b4e-9114-35af7b635044\") " pod="openshift-monitoring/node-exporter-mthpt"
Sep 11 17:19:49.717442 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:19:49.715065    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/f1b075b6-9bae-41d6-b0c0-1c8b81c3207a-openshift-state-metrics-tls\") pod \"openshift-state-metrics-67dcf8c979-fwdgr\" (UID: \"f1b075b6-9bae-41d6-b0c0-1c8b81c3207a\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-fwdgr"
Sep 11 17:19:49.717442 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:19:49.715093    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/df65734a-e4e6-4b4e-9114-35af7b635044-metrics-client-ca\") pod \"node-exporter-mthpt\" (UID: \"df65734a-e4e6-4b4e-9114-35af7b635044\") " pod="openshift-monitoring/node-exporter-mthpt"
Sep 11 17:19:49.717442 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:19:49.715131    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/df65734a-e4e6-4b4e-9114-35af7b635044-root\") pod \"node-exporter-mthpt\" (UID: \"df65734a-e4e6-4b4e-9114-35af7b635044\") " pod="openshift-monitoring/node-exporter-mthpt"
Sep 11 17:19:49.717442 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:19:49.715175    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/df65734a-e4e6-4b4e-9114-35af7b635044-node-exporter-accelerators-collector-config\") pod \"node-exporter-mthpt\" (UID: \"df65734a-e4e6-4b4e-9114-35af7b635044\") " pod="openshift-monitoring/node-exporter-mthpt"
Sep 11 17:19:49.717442 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:19:49.715202    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n9m4c\" (UniqueName: \"kubernetes.io/projected/df65734a-e4e6-4b4e-9114-35af7b635044-kube-api-access-n9m4c\") pod \"node-exporter-mthpt\" (UID: \"df65734a-e4e6-4b4e-9114-35af7b635044\") " pod="openshift-monitoring/node-exporter-mthpt"
Sep 11 17:19:49.717442 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:19:49.715233    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-42szj\" (UniqueName: \"kubernetes.io/projected/f1b075b6-9bae-41d6-b0c0-1c8b81c3207a-kube-api-access-42szj\") pod \"openshift-state-metrics-67dcf8c979-fwdgr\" (UID: \"f1b075b6-9bae-41d6-b0c0-1c8b81c3207a\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-fwdgr"
Sep 11 17:19:49.717442 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:19:49.715260    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f1b075b6-9bae-41d6-b0c0-1c8b81c3207a-metrics-client-ca\") pod \"openshift-state-metrics-67dcf8c979-fwdgr\" (UID: \"f1b075b6-9bae-41d6-b0c0-1c8b81c3207a\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-fwdgr"
Sep 11 17:19:49.717442 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:19:49.715295    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/e41207d9-a802-4537-a459-01deb5c8fb16-kube-state-metrics-tls\") pod \"kube-state-metrics-59fbdd77d-bfz6b\" (UID: \"e41207d9-a802-4537-a459-01deb5c8fb16\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-bfz6b"
Sep 11 17:19:49.717442 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:19:49.715326    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/df65734a-e4e6-4b4e-9114-35af7b635044-node-exporter-tls\") pod \"node-exporter-mthpt\" (UID: \"df65734a-e4e6-4b4e-9114-35af7b635044\") " pod="openshift-monitoring/node-exporter-mthpt"
Sep 11 17:19:49.717442 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:19:49.715364    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/df65734a-e4e6-4b4e-9114-35af7b635044-node-exporter-wtmp\") pod \"node-exporter-mthpt\" (UID: \"df65734a-e4e6-4b4e-9114-35af7b635044\") " pod="openshift-monitoring/node-exporter-mthpt"
Sep 11 17:19:49.717442 ip-10-0-141-139 kubenswrapper[2577]: E0911 17:19:49.715534    2577 secret.go:189] Couldn't get secret openshift-monitoring/openshift-state-metrics-tls: secret "openshift-state-metrics-tls" not found
Sep 11 17:19:49.717442 ip-10-0-141-139 kubenswrapper[2577]: E0911 17:19:49.715597    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f1b075b6-9bae-41d6-b0c0-1c8b81c3207a-openshift-state-metrics-tls podName:f1b075b6-9bae-41d6-b0c0-1c8b81c3207a nodeName:}" failed. No retries permitted until 2026-09-11 17:19:50.215577063 +0000 UTC m=+67.788797168 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "openshift-state-metrics-tls" (UniqueName: "kubernetes.io/secret/f1b075b6-9bae-41d6-b0c0-1c8b81c3207a-openshift-state-metrics-tls") pod "openshift-state-metrics-67dcf8c979-fwdgr" (UID: "f1b075b6-9bae-41d6-b0c0-1c8b81c3207a") : secret "openshift-state-metrics-tls" not found
Sep 11 17:19:49.717442 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:19:49.717020    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f1b075b6-9bae-41d6-b0c0-1c8b81c3207a-metrics-client-ca\") pod \"openshift-state-metrics-67dcf8c979-fwdgr\" (UID: \"f1b075b6-9bae-41d6-b0c0-1c8b81c3207a\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-fwdgr"
Sep 11 17:19:49.718314 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:19:49.717433    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/f1b075b6-9bae-41d6-b0c0-1c8b81c3207a-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-67dcf8c979-fwdgr\" (UID: \"f1b075b6-9bae-41d6-b0c0-1c8b81c3207a\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-fwdgr"
Sep 11 17:19:49.727171 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:19:49.727143    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-42szj\" (UniqueName: \"kubernetes.io/projected/f1b075b6-9bae-41d6-b0c0-1c8b81c3207a-kube-api-access-42szj\") pod \"openshift-state-metrics-67dcf8c979-fwdgr\" (UID: \"f1b075b6-9bae-41d6-b0c0-1c8b81c3207a\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-fwdgr"
Sep 11 17:19:49.816736 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:19:49.816696    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mpfmc\" (UniqueName: \"kubernetes.io/projected/e41207d9-a802-4537-a459-01deb5c8fb16-kube-api-access-mpfmc\") pod \"kube-state-metrics-59fbdd77d-bfz6b\" (UID: \"e41207d9-a802-4537-a459-01deb5c8fb16\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-bfz6b"
Sep 11 17:19:49.816898 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:19:49.816744    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/df65734a-e4e6-4b4e-9114-35af7b635044-sys\") pod \"node-exporter-mthpt\" (UID: \"df65734a-e4e6-4b4e-9114-35af7b635044\") " pod="openshift-monitoring/node-exporter-mthpt"
Sep 11 17:19:49.816898 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:19:49.816776    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/e41207d9-a802-4537-a459-01deb5c8fb16-volume-directive-shadow\") pod \"kube-state-metrics-59fbdd77d-bfz6b\" (UID: \"e41207d9-a802-4537-a459-01deb5c8fb16\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-bfz6b"
Sep 11 17:19:49.816898 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:19:49.816808    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/e41207d9-a802-4537-a459-01deb5c8fb16-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-59fbdd77d-bfz6b\" (UID: \"e41207d9-a802-4537-a459-01deb5c8fb16\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-bfz6b"
Sep 11 17:19:49.816898 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:19:49.816838    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/df65734a-e4e6-4b4e-9114-35af7b635044-node-exporter-textfile\") pod \"node-exporter-mthpt\" (UID: \"df65734a-e4e6-4b4e-9114-35af7b635044\") " pod="openshift-monitoring/node-exporter-mthpt"
Sep 11 17:19:49.816898 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:19:49.816879    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/df65734a-e4e6-4b4e-9114-35af7b635044-metrics-client-ca\") pod \"node-exporter-mthpt\" (UID: \"df65734a-e4e6-4b4e-9114-35af7b635044\") " pod="openshift-monitoring/node-exporter-mthpt"
Sep 11 17:19:49.817144 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:19:49.816913    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/df65734a-e4e6-4b4e-9114-35af7b635044-root\") pod \"node-exporter-mthpt\" (UID: \"df65734a-e4e6-4b4e-9114-35af7b635044\") " pod="openshift-monitoring/node-exporter-mthpt"
Sep 11 17:19:49.817144 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:19:49.816951    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/df65734a-e4e6-4b4e-9114-35af7b635044-node-exporter-accelerators-collector-config\") pod \"node-exporter-mthpt\" (UID: \"df65734a-e4e6-4b4e-9114-35af7b635044\") " pod="openshift-monitoring/node-exporter-mthpt"
Sep 11 17:19:49.817144 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:19:49.816978    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n9m4c\" (UniqueName: \"kubernetes.io/projected/df65734a-e4e6-4b4e-9114-35af7b635044-kube-api-access-n9m4c\") pod \"node-exporter-mthpt\" (UID: \"df65734a-e4e6-4b4e-9114-35af7b635044\") " pod="openshift-monitoring/node-exporter-mthpt"
Sep 11 17:19:49.817144 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:19:49.817016    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/e41207d9-a802-4537-a459-01deb5c8fb16-kube-state-metrics-tls\") pod \"kube-state-metrics-59fbdd77d-bfz6b\" (UID: \"e41207d9-a802-4537-a459-01deb5c8fb16\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-bfz6b"
Sep 11 17:19:49.817144 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:19:49.817041    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/df65734a-e4e6-4b4e-9114-35af7b635044-node-exporter-tls\") pod \"node-exporter-mthpt\" (UID: \"df65734a-e4e6-4b4e-9114-35af7b635044\") " pod="openshift-monitoring/node-exporter-mthpt"
Sep 11 17:19:49.817144 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:19:49.817074    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/df65734a-e4e6-4b4e-9114-35af7b635044-node-exporter-wtmp\") pod \"node-exporter-mthpt\" (UID: \"df65734a-e4e6-4b4e-9114-35af7b635044\") " pod="openshift-monitoring/node-exporter-mthpt"
Sep 11 17:19:49.817144 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:19:49.817110    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e41207d9-a802-4537-a459-01deb5c8fb16-metrics-client-ca\") pod \"kube-state-metrics-59fbdd77d-bfz6b\" (UID: \"e41207d9-a802-4537-a459-01deb5c8fb16\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-bfz6b"
Sep 11 17:19:49.817144 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:19:49.817142    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/df65734a-e4e6-4b4e-9114-35af7b635044-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-mthpt\" (UID: \"df65734a-e4e6-4b4e-9114-35af7b635044\") " pod="openshift-monitoring/node-exporter-mthpt"
Sep 11 17:19:49.817490 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:19:49.817178    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/e41207d9-a802-4537-a459-01deb5c8fb16-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-59fbdd77d-bfz6b\" (UID: \"e41207d9-a802-4537-a459-01deb5c8fb16\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-bfz6b"
Sep 11 17:19:49.817783 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:19:49.817758    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/df65734a-e4e6-4b4e-9114-35af7b635044-root\") pod \"node-exporter-mthpt\" (UID: \"df65734a-e4e6-4b4e-9114-35af7b635044\") " pod="openshift-monitoring/node-exporter-mthpt"
Sep 11 17:19:49.818107 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:19:49.818044    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/e41207d9-a802-4537-a459-01deb5c8fb16-volume-directive-shadow\") pod \"kube-state-metrics-59fbdd77d-bfz6b\" (UID: \"e41207d9-a802-4537-a459-01deb5c8fb16\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-bfz6b"
Sep 11 17:19:49.818304 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:19:49.818271    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/df65734a-e4e6-4b4e-9114-35af7b635044-sys\") pod \"node-exporter-mthpt\" (UID: \"df65734a-e4e6-4b4e-9114-35af7b635044\") " pod="openshift-monitoring/node-exporter-mthpt"
Sep 11 17:19:49.818413 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:19:49.818391    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/e41207d9-a802-4537-a459-01deb5c8fb16-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-59fbdd77d-bfz6b\" (UID: \"e41207d9-a802-4537-a459-01deb5c8fb16\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-bfz6b"
Sep 11 17:19:49.818542 ip-10-0-141-139 kubenswrapper[2577]: E0911 17:19:49.818526    2577 secret.go:189] Couldn't get secret openshift-monitoring/kube-state-metrics-tls: secret "kube-state-metrics-tls" not found
Sep 11 17:19:49.818620 ip-10-0-141-139 kubenswrapper[2577]: E0911 17:19:49.818588    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e41207d9-a802-4537-a459-01deb5c8fb16-kube-state-metrics-tls podName:e41207d9-a802-4537-a459-01deb5c8fb16 nodeName:}" failed. No retries permitted until 2026-09-11 17:19:50.318569621 +0000 UTC m=+67.891789710 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-state-metrics-tls" (UniqueName: "kubernetes.io/secret/e41207d9-a802-4537-a459-01deb5c8fb16-kube-state-metrics-tls") pod "kube-state-metrics-59fbdd77d-bfz6b" (UID: "e41207d9-a802-4537-a459-01deb5c8fb16") : secret "kube-state-metrics-tls" not found
Sep 11 17:19:49.818685 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:19:49.818654    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/df65734a-e4e6-4b4e-9114-35af7b635044-node-exporter-textfile\") pod \"node-exporter-mthpt\" (UID: \"df65734a-e4e6-4b4e-9114-35af7b635044\") " pod="openshift-monitoring/node-exporter-mthpt"
Sep 11 17:19:49.818747 ip-10-0-141-139 kubenswrapper[2577]: E0911 17:19:49.818730    2577 secret.go:189] Couldn't get secret openshift-monitoring/node-exporter-tls: secret "node-exporter-tls" not found
Sep 11 17:19:49.818803 ip-10-0-141-139 kubenswrapper[2577]: E0911 17:19:49.818778    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/df65734a-e4e6-4b4e-9114-35af7b635044-node-exporter-tls podName:df65734a-e4e6-4b4e-9114-35af7b635044 nodeName:}" failed. No retries permitted until 2026-09-11 17:19:50.318762416 +0000 UTC m=+67.891982521 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "node-exporter-tls" (UniqueName: "kubernetes.io/secret/df65734a-e4e6-4b4e-9114-35af7b635044-node-exporter-tls") pod "node-exporter-mthpt" (UID: "df65734a-e4e6-4b4e-9114-35af7b635044") : secret "node-exporter-tls" not found
Sep 11 17:19:49.818873 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:19:49.818803    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/df65734a-e4e6-4b4e-9114-35af7b635044-node-exporter-wtmp\") pod \"node-exporter-mthpt\" (UID: \"df65734a-e4e6-4b4e-9114-35af7b635044\") " pod="openshift-monitoring/node-exporter-mthpt"
Sep 11 17:19:49.819566 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:19:49.819375    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e41207d9-a802-4537-a459-01deb5c8fb16-metrics-client-ca\") pod \"kube-state-metrics-59fbdd77d-bfz6b\" (UID: \"e41207d9-a802-4537-a459-01deb5c8fb16\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-bfz6b"
Sep 11 17:19:49.819566 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:19:49.819495    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/df65734a-e4e6-4b4e-9114-35af7b635044-metrics-client-ca\") pod \"node-exporter-mthpt\" (UID: \"df65734a-e4e6-4b4e-9114-35af7b635044\") " pod="openshift-monitoring/node-exporter-mthpt"
Sep 11 17:19:49.819963 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:19:49.819935    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/df65734a-e4e6-4b4e-9114-35af7b635044-node-exporter-accelerators-collector-config\") pod \"node-exporter-mthpt\" (UID: \"df65734a-e4e6-4b4e-9114-35af7b635044\") " pod="openshift-monitoring/node-exporter-mthpt"
Sep 11 17:19:49.823006 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:19:49.822982    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/e41207d9-a802-4537-a459-01deb5c8fb16-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-59fbdd77d-bfz6b\" (UID: \"e41207d9-a802-4537-a459-01deb5c8fb16\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-bfz6b"
Sep 11 17:19:49.823964 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:19:49.823940    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/df65734a-e4e6-4b4e-9114-35af7b635044-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-mthpt\" (UID: \"df65734a-e4e6-4b4e-9114-35af7b635044\") " pod="openshift-monitoring/node-exporter-mthpt"
Sep 11 17:19:49.830005 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:19:49.829954    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mpfmc\" (UniqueName: \"kubernetes.io/projected/e41207d9-a802-4537-a459-01deb5c8fb16-kube-api-access-mpfmc\") pod \"kube-state-metrics-59fbdd77d-bfz6b\" (UID: \"e41207d9-a802-4537-a459-01deb5c8fb16\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-bfz6b"
Sep 11 17:19:49.832027 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:19:49.832000    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-n9m4c\" (UniqueName: \"kubernetes.io/projected/df65734a-e4e6-4b4e-9114-35af7b635044-kube-api-access-n9m4c\") pod \"node-exporter-mthpt\" (UID: \"df65734a-e4e6-4b4e-9114-35af7b635044\") " pod="openshift-monitoring/node-exporter-mthpt"
Sep 11 17:19:50.221472 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:19:50.221430    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/f1b075b6-9bae-41d6-b0c0-1c8b81c3207a-openshift-state-metrics-tls\") pod \"openshift-state-metrics-67dcf8c979-fwdgr\" (UID: \"f1b075b6-9bae-41d6-b0c0-1c8b81c3207a\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-fwdgr"
Sep 11 17:19:50.221686 ip-10-0-141-139 kubenswrapper[2577]: E0911 17:19:50.221665    2577 secret.go:189] Couldn't get secret openshift-monitoring/openshift-state-metrics-tls: secret "openshift-state-metrics-tls" not found
Sep 11 17:19:50.221754 ip-10-0-141-139 kubenswrapper[2577]: E0911 17:19:50.221728    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f1b075b6-9bae-41d6-b0c0-1c8b81c3207a-openshift-state-metrics-tls podName:f1b075b6-9bae-41d6-b0c0-1c8b81c3207a nodeName:}" failed. No retries permitted until 2026-09-11 17:19:51.221709757 +0000 UTC m=+68.794929849 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "openshift-state-metrics-tls" (UniqueName: "kubernetes.io/secret/f1b075b6-9bae-41d6-b0c0-1c8b81c3207a-openshift-state-metrics-tls") pod "openshift-state-metrics-67dcf8c979-fwdgr" (UID: "f1b075b6-9bae-41d6-b0c0-1c8b81c3207a") : secret "openshift-state-metrics-tls" not found
Sep 11 17:19:50.294714 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:19:50.294679    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-bl4vr" event={"ID":"5c1ac293-cc19-4446-8244-a8ca7fc8a8f4","Type":"ContainerStarted","Data":"d5186e047b6d287cb301f8c4a6a56df238c552d72c671b6c21c4ddd382f18bfa"}
Sep 11 17:19:50.295121 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:19:50.294797    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-bl4vr" event={"ID":"5c1ac293-cc19-4446-8244-a8ca7fc8a8f4","Type":"ContainerStarted","Data":"9e8b08d9a58e4c0990743216a7d928c1b7c88eae62c2165f4ecae3935463c10f"}
Sep 11 17:19:50.322824 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:19:50.322777    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/e41207d9-a802-4537-a459-01deb5c8fb16-kube-state-metrics-tls\") pod \"kube-state-metrics-59fbdd77d-bfz6b\" (UID: \"e41207d9-a802-4537-a459-01deb5c8fb16\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-bfz6b"
Sep 11 17:19:50.322824 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:19:50.322825    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/df65734a-e4e6-4b4e-9114-35af7b635044-node-exporter-tls\") pod \"node-exporter-mthpt\" (UID: \"df65734a-e4e6-4b4e-9114-35af7b635044\") " pod="openshift-monitoring/node-exporter-mthpt"
Sep 11 17:19:50.326983 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:19:50.326921    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/df65734a-e4e6-4b4e-9114-35af7b635044-node-exporter-tls\") pod \"node-exporter-mthpt\" (UID: \"df65734a-e4e6-4b4e-9114-35af7b635044\") " pod="openshift-monitoring/node-exporter-mthpt"
Sep 11 17:19:50.327428 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:19:50.327376    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/e41207d9-a802-4537-a459-01deb5c8fb16-kube-state-metrics-tls\") pod \"kube-state-metrics-59fbdd77d-bfz6b\" (UID: \"e41207d9-a802-4537-a459-01deb5c8fb16\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-bfz6b"
Sep 11 17:19:50.544912 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:19:50.544829    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-59fbdd77d-bfz6b"
Sep 11 17:19:50.573185 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:19:50.573149    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-mthpt"
Sep 11 17:19:50.691436 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:19:50.691401    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-5c74f887d6-ttmcp"
Sep 11 17:19:50.691638 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:19:50.691474    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-5c74f887d6-ttmcp"
Sep 11 17:19:50.698556 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:19:50.698519    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-5c74f887d6-ttmcp"
Sep 11 17:19:50.740526 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:19:50.737119    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-bl4vr" podStartSLOduration=65.340140589 podStartE2EDuration="1m6.737094635s" totalImagesPullingTime="1.396954046s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 17:18:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 17:19:50.327814755 +0000 UTC m=+67.901034866" watchObservedRunningTime="2026-09-11 17:19:50.737094635 +0000 UTC m=+68.310314749"
Sep 11 17:19:50.767038 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:19:50.766999    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 11 17:19:50.770771 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:19:50.770742    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Sep 11 17:19:50.776843 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:19:50.776812    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\""
Sep 11 17:19:50.776983 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:19:50.776823    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\""
Sep 11 17:19:50.777059 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:19:50.777045    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\""
Sep 11 17:19:50.777123 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:19:50.777079    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\""
Sep 11 17:19:50.777179 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:19:50.777146    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\""
Sep 11 17:19:50.777228 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:19:50.777195    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\""
Sep 11 17:19:50.777283 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:19:50.777261    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-w6gsg\""
Sep 11 17:19:50.777335 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:19:50.777311    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\""
Sep 11 17:19:50.777756 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:19:50.777527    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\""
Sep 11 17:19:50.777756 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:19:50.777534    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\""
Sep 11 17:19:50.819547 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:19:50.819443    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 11 17:19:50.844323 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:19:50.843141    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/4c942d42-091f-4c68-8faa-f6276bd26c15-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"4c942d42-091f-4c68-8faa-f6276bd26c15\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 17:19:50.844323 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:19:50.843199    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/4c942d42-091f-4c68-8faa-f6276bd26c15-config-volume\") pod \"alertmanager-main-0\" (UID: \"4c942d42-091f-4c68-8faa-f6276bd26c15\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 17:19:50.844323 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:19:50.843233    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/4c942d42-091f-4c68-8faa-f6276bd26c15-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"4c942d42-091f-4c68-8faa-f6276bd26c15\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 17:19:50.844323 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:19:50.843267    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/4c942d42-091f-4c68-8faa-f6276bd26c15-web-config\") pod \"alertmanager-main-0\" (UID: \"4c942d42-091f-4c68-8faa-f6276bd26c15\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 17:19:50.844323 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:19:50.843297    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/4c942d42-091f-4c68-8faa-f6276bd26c15-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"4c942d42-091f-4c68-8faa-f6276bd26c15\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 17:19:50.844323 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:19:50.843327    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/4c942d42-091f-4c68-8faa-f6276bd26c15-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"4c942d42-091f-4c68-8faa-f6276bd26c15\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 17:19:50.844323 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:19:50.843359    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/4c942d42-091f-4c68-8faa-f6276bd26c15-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"4c942d42-091f-4c68-8faa-f6276bd26c15\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 17:19:50.844323 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:19:50.843392    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/4c942d42-091f-4c68-8faa-f6276bd26c15-tls-assets\") pod \"alertmanager-main-0\" (UID: \"4c942d42-091f-4c68-8faa-f6276bd26c15\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 17:19:50.844323 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:19:50.843421    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gh4hv\" (UniqueName: \"kubernetes.io/projected/4c942d42-091f-4c68-8faa-f6276bd26c15-kube-api-access-gh4hv\") pod \"alertmanager-main-0\" (UID: \"4c942d42-091f-4c68-8faa-f6276bd26c15\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 17:19:50.844323 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:19:50.843458    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/4c942d42-091f-4c68-8faa-f6276bd26c15-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"4c942d42-091f-4c68-8faa-f6276bd26c15\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 17:19:50.844323 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:19:50.843579    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/4c942d42-091f-4c68-8faa-f6276bd26c15-config-out\") pod \"alertmanager-main-0\" (UID: \"4c942d42-091f-4c68-8faa-f6276bd26c15\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 17:19:50.844323 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:19:50.843612    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/4c942d42-091f-4c68-8faa-f6276bd26c15-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"4c942d42-091f-4c68-8faa-f6276bd26c15\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 17:19:50.844323 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:19:50.843641    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4c942d42-091f-4c68-8faa-f6276bd26c15-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"4c942d42-091f-4c68-8faa-f6276bd26c15\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 17:19:50.868660 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:19:50.868246    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-59fbdd77d-bfz6b"]
Sep 11 17:19:50.944660 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:19:50.944626    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/4c942d42-091f-4c68-8faa-f6276bd26c15-config-out\") pod \"alertmanager-main-0\" (UID: \"4c942d42-091f-4c68-8faa-f6276bd26c15\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 17:19:50.944660 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:19:50.944669    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/4c942d42-091f-4c68-8faa-f6276bd26c15-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"4c942d42-091f-4c68-8faa-f6276bd26c15\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 17:19:50.944908 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:19:50.944700    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4c942d42-091f-4c68-8faa-f6276bd26c15-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"4c942d42-091f-4c68-8faa-f6276bd26c15\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 17:19:50.945116 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:19:50.945087    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/4c942d42-091f-4c68-8faa-f6276bd26c15-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"4c942d42-091f-4c68-8faa-f6276bd26c15\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 17:19:50.945116 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:19:50.945102    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/4c942d42-091f-4c68-8faa-f6276bd26c15-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"4c942d42-091f-4c68-8faa-f6276bd26c15\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 17:19:50.945296 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:19:50.945133    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/4c942d42-091f-4c68-8faa-f6276bd26c15-config-volume\") pod \"alertmanager-main-0\" (UID: \"4c942d42-091f-4c68-8faa-f6276bd26c15\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 17:19:50.945596 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:19:50.945569    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/4c942d42-091f-4c68-8faa-f6276bd26c15-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"4c942d42-091f-4c68-8faa-f6276bd26c15\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 17:19:50.945712 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:19:50.945606    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4c942d42-091f-4c68-8faa-f6276bd26c15-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"4c942d42-091f-4c68-8faa-f6276bd26c15\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 17:19:50.945712 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:19:50.945619    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/4c942d42-091f-4c68-8faa-f6276bd26c15-web-config\") pod \"alertmanager-main-0\" (UID: \"4c942d42-091f-4c68-8faa-f6276bd26c15\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 17:19:50.945712 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:19:50.945670    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/4c942d42-091f-4c68-8faa-f6276bd26c15-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"4c942d42-091f-4c68-8faa-f6276bd26c15\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 17:19:50.945712 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:19:50.945698    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/4c942d42-091f-4c68-8faa-f6276bd26c15-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"4c942d42-091f-4c68-8faa-f6276bd26c15\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 17:19:50.945919 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:19:50.945741    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/4c942d42-091f-4c68-8faa-f6276bd26c15-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"4c942d42-091f-4c68-8faa-f6276bd26c15\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 17:19:50.945919 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:19:50.945814    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/4c942d42-091f-4c68-8faa-f6276bd26c15-tls-assets\") pod \"alertmanager-main-0\" (UID: \"4c942d42-091f-4c68-8faa-f6276bd26c15\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 17:19:50.945919 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:19:50.945844    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gh4hv\" (UniqueName: \"kubernetes.io/projected/4c942d42-091f-4c68-8faa-f6276bd26c15-kube-api-access-gh4hv\") pod \"alertmanager-main-0\" (UID: \"4c942d42-091f-4c68-8faa-f6276bd26c15\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 17:19:50.945919 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:19:50.945905    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/4c942d42-091f-4c68-8faa-f6276bd26c15-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"4c942d42-091f-4c68-8faa-f6276bd26c15\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 17:19:50.946535 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:19:50.946487    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/4c942d42-091f-4c68-8faa-f6276bd26c15-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"4c942d42-091f-4c68-8faa-f6276bd26c15\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 17:19:50.947617 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:19:50.947579    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/4c942d42-091f-4c68-8faa-f6276bd26c15-config-out\") pod \"alertmanager-main-0\" (UID: \"4c942d42-091f-4c68-8faa-f6276bd26c15\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 17:19:50.949310 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:19:50.949266    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/4c942d42-091f-4c68-8faa-f6276bd26c15-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"4c942d42-091f-4c68-8faa-f6276bd26c15\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 17:19:50.949993 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:19:50.949960    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/4c942d42-091f-4c68-8faa-f6276bd26c15-config-volume\") pod \"alertmanager-main-0\" (UID: \"4c942d42-091f-4c68-8faa-f6276bd26c15\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 17:19:50.950581 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:19:50.950557    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/4c942d42-091f-4c68-8faa-f6276bd26c15-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"4c942d42-091f-4c68-8faa-f6276bd26c15\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 17:19:50.950691 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:19:50.950589    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/4c942d42-091f-4c68-8faa-f6276bd26c15-web-config\") pod \"alertmanager-main-0\" (UID: \"4c942d42-091f-4c68-8faa-f6276bd26c15\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 17:19:50.950776 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:19:50.950749    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/4c942d42-091f-4c68-8faa-f6276bd26c15-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"4c942d42-091f-4c68-8faa-f6276bd26c15\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 17:19:50.951018 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:19:50.950975    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/4c942d42-091f-4c68-8faa-f6276bd26c15-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"4c942d42-091f-4c68-8faa-f6276bd26c15\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 17:19:50.951108 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:19:50.951044    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/4c942d42-091f-4c68-8faa-f6276bd26c15-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"4c942d42-091f-4c68-8faa-f6276bd26c15\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 17:19:50.951298 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:19:50.951275    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/4c942d42-091f-4c68-8faa-f6276bd26c15-tls-assets\") pod \"alertmanager-main-0\" (UID: \"4c942d42-091f-4c68-8faa-f6276bd26c15\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 17:19:50.962685 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:19:50.962657    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gh4hv\" (UniqueName: \"kubernetes.io/projected/4c942d42-091f-4c68-8faa-f6276bd26c15-kube-api-access-gh4hv\") pod \"alertmanager-main-0\" (UID: \"4c942d42-091f-4c68-8faa-f6276bd26c15\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 17:19:51.083950 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:19:51.083852    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Sep 11 17:19:51.248538 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:19:51.248479    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/f1b075b6-9bae-41d6-b0c0-1c8b81c3207a-openshift-state-metrics-tls\") pod \"openshift-state-metrics-67dcf8c979-fwdgr\" (UID: \"f1b075b6-9bae-41d6-b0c0-1c8b81c3207a\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-fwdgr"
Sep 11 17:19:51.251472 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:19:51.251436    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/f1b075b6-9bae-41d6-b0c0-1c8b81c3207a-openshift-state-metrics-tls\") pod \"openshift-state-metrics-67dcf8c979-fwdgr\" (UID: \"f1b075b6-9bae-41d6-b0c0-1c8b81c3207a\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-fwdgr"
Sep 11 17:19:51.304068 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:19:51.304035    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-5c74f887d6-ttmcp"
Sep 11 17:19:51.353951 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:19:51.353867    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-fwdgr"
Sep 11 17:19:51.639791 ip-10-0-141-139 kubenswrapper[2577]: W0911 17:19:51.639730    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode41207d9_a802_4537_a459_01deb5c8fb16.slice/crio-6cc6150422de0188492ff609cbb03a8e68305e523732136384e638e093f46ff1 WatchSource:0}: Error finding container 6cc6150422de0188492ff609cbb03a8e68305e523732136384e638e093f46ff1: Status 404 returned error can't find the container with id 6cc6150422de0188492ff609cbb03a8e68305e523732136384e638e093f46ff1
Sep 11 17:19:52.305086 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:19:52.305037    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-59fbdd77d-bfz6b" event={"ID":"e41207d9-a802-4537-a459-01deb5c8fb16","Type":"ContainerStarted","Data":"6cc6150422de0188492ff609cbb03a8e68305e523732136384e638e093f46ff1"}
Sep 11 17:19:52.796700 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:19:52.796665    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/thanos-querier-7775c558d6-95wpp"]
Sep 11 17:19:52.799596 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:19:52.799569    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-7775c558d6-95wpp"
Sep 11 17:19:52.811137 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:19:52.811111    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-metrics\""
Sep 11 17:19:52.811279 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:19:52.811193    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-rules\""
Sep 11 17:19:52.811279 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:19:52.811223    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-web\""
Sep 11 17:19:52.811486 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:19:52.811462    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-tls\""
Sep 11 17:19:52.811616 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:19:52.811465    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-dockercfg-m8dr2\""
Sep 11 17:19:52.811616 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:19:52.811535    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy\""
Sep 11 17:19:52.814889 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:19:52.814867    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-grpc-tls-ff7e3vkehtdcm\""
Sep 11 17:19:52.855667 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:19:52.855633    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-7775c558d6-95wpp"]
Sep 11 17:19:52.869323 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:19:52.869289    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/44e112d2-e31f-4dfc-abc3-17f9c80ae9b4-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-7775c558d6-95wpp\" (UID: \"44e112d2-e31f-4dfc-abc3-17f9c80ae9b4\") " pod="openshift-monitoring/thanos-querier-7775c558d6-95wpp"
Sep 11 17:19:52.869531 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:19:52.869339    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/44e112d2-e31f-4dfc-abc3-17f9c80ae9b4-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-7775c558d6-95wpp\" (UID: \"44e112d2-e31f-4dfc-abc3-17f9c80ae9b4\") " pod="openshift-monitoring/thanos-querier-7775c558d6-95wpp"
Sep 11 17:19:52.869531 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:19:52.869376    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/44e112d2-e31f-4dfc-abc3-17f9c80ae9b4-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-7775c558d6-95wpp\" (UID: \"44e112d2-e31f-4dfc-abc3-17f9c80ae9b4\") " pod="openshift-monitoring/thanos-querier-7775c558d6-95wpp"
Sep 11 17:19:52.869531 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:19:52.869408    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/44e112d2-e31f-4dfc-abc3-17f9c80ae9b4-metrics-client-ca\") pod \"thanos-querier-7775c558d6-95wpp\" (UID: \"44e112d2-e31f-4dfc-abc3-17f9c80ae9b4\") " pod="openshift-monitoring/thanos-querier-7775c558d6-95wpp"
Sep 11 17:19:52.869531 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:19:52.869442    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6d6wm\" (UniqueName: \"kubernetes.io/projected/44e112d2-e31f-4dfc-abc3-17f9c80ae9b4-kube-api-access-6d6wm\") pod \"thanos-querier-7775c558d6-95wpp\" (UID: \"44e112d2-e31f-4dfc-abc3-17f9c80ae9b4\") " pod="openshift-monitoring/thanos-querier-7775c558d6-95wpp"
Sep 11 17:19:52.869531 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:19:52.869477    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/44e112d2-e31f-4dfc-abc3-17f9c80ae9b4-secret-thanos-querier-tls\") pod \"thanos-querier-7775c558d6-95wpp\" (UID: \"44e112d2-e31f-4dfc-abc3-17f9c80ae9b4\") " pod="openshift-monitoring/thanos-querier-7775c558d6-95wpp"
Sep 11 17:19:52.869531 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:19:52.869529    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/44e112d2-e31f-4dfc-abc3-17f9c80ae9b4-secret-grpc-tls\") pod \"thanos-querier-7775c558d6-95wpp\" (UID: \"44e112d2-e31f-4dfc-abc3-17f9c80ae9b4\") " pod="openshift-monitoring/thanos-querier-7775c558d6-95wpp"
Sep 11 17:19:52.869816 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:19:52.869594    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/44e112d2-e31f-4dfc-abc3-17f9c80ae9b4-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-7775c558d6-95wpp\" (UID: \"44e112d2-e31f-4dfc-abc3-17f9c80ae9b4\") " pod="openshift-monitoring/thanos-querier-7775c558d6-95wpp"
Sep 11 17:19:52.970088 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:19:52.970045    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/44e112d2-e31f-4dfc-abc3-17f9c80ae9b4-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-7775c558d6-95wpp\" (UID: \"44e112d2-e31f-4dfc-abc3-17f9c80ae9b4\") " pod="openshift-monitoring/thanos-querier-7775c558d6-95wpp"
Sep 11 17:19:52.970270 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:19:52.970110    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/44e112d2-e31f-4dfc-abc3-17f9c80ae9b4-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-7775c558d6-95wpp\" (UID: \"44e112d2-e31f-4dfc-abc3-17f9c80ae9b4\") " pod="openshift-monitoring/thanos-querier-7775c558d6-95wpp"
Sep 11 17:19:52.970270 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:19:52.970141    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/44e112d2-e31f-4dfc-abc3-17f9c80ae9b4-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-7775c558d6-95wpp\" (UID: \"44e112d2-e31f-4dfc-abc3-17f9c80ae9b4\") " pod="openshift-monitoring/thanos-querier-7775c558d6-95wpp"
Sep 11 17:19:52.970270 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:19:52.970187    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/44e112d2-e31f-4dfc-abc3-17f9c80ae9b4-metrics-client-ca\") pod \"thanos-querier-7775c558d6-95wpp\" (UID: \"44e112d2-e31f-4dfc-abc3-17f9c80ae9b4\") " pod="openshift-monitoring/thanos-querier-7775c558d6-95wpp"
Sep 11 17:19:52.970270 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:19:52.970219    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6d6wm\" (UniqueName: \"kubernetes.io/projected/44e112d2-e31f-4dfc-abc3-17f9c80ae9b4-kube-api-access-6d6wm\") pod \"thanos-querier-7775c558d6-95wpp\" (UID: \"44e112d2-e31f-4dfc-abc3-17f9c80ae9b4\") " pod="openshift-monitoring/thanos-querier-7775c558d6-95wpp"
Sep 11 17:19:52.970270 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:19:52.970256    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/44e112d2-e31f-4dfc-abc3-17f9c80ae9b4-secret-thanos-querier-tls\") pod \"thanos-querier-7775c558d6-95wpp\" (UID: \"44e112d2-e31f-4dfc-abc3-17f9c80ae9b4\") " pod="openshift-monitoring/thanos-querier-7775c558d6-95wpp"
Sep 11 17:19:52.970599 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:19:52.970281    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/44e112d2-e31f-4dfc-abc3-17f9c80ae9b4-secret-grpc-tls\") pod \"thanos-querier-7775c558d6-95wpp\" (UID: \"44e112d2-e31f-4dfc-abc3-17f9c80ae9b4\") " pod="openshift-monitoring/thanos-querier-7775c558d6-95wpp"
Sep 11 17:19:52.970599 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:19:52.970313    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/44e112d2-e31f-4dfc-abc3-17f9c80ae9b4-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-7775c558d6-95wpp\" (UID: \"44e112d2-e31f-4dfc-abc3-17f9c80ae9b4\") " pod="openshift-monitoring/thanos-querier-7775c558d6-95wpp"
Sep 11 17:19:52.971604 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:19:52.971206    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/44e112d2-e31f-4dfc-abc3-17f9c80ae9b4-metrics-client-ca\") pod \"thanos-querier-7775c558d6-95wpp\" (UID: \"44e112d2-e31f-4dfc-abc3-17f9c80ae9b4\") " pod="openshift-monitoring/thanos-querier-7775c558d6-95wpp"
Sep 11 17:19:52.973736 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:19:52.973711    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/44e112d2-e31f-4dfc-abc3-17f9c80ae9b4-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-7775c558d6-95wpp\" (UID: \"44e112d2-e31f-4dfc-abc3-17f9c80ae9b4\") " pod="openshift-monitoring/thanos-querier-7775c558d6-95wpp"
Sep 11 17:19:52.973832 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:19:52.973783    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/44e112d2-e31f-4dfc-abc3-17f9c80ae9b4-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-7775c558d6-95wpp\" (UID: \"44e112d2-e31f-4dfc-abc3-17f9c80ae9b4\") " pod="openshift-monitoring/thanos-querier-7775c558d6-95wpp"
Sep 11 17:19:52.973900 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:19:52.973880    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/44e112d2-e31f-4dfc-abc3-17f9c80ae9b4-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-7775c558d6-95wpp\" (UID: \"44e112d2-e31f-4dfc-abc3-17f9c80ae9b4\") " pod="openshift-monitoring/thanos-querier-7775c558d6-95wpp"
Sep 11 17:19:52.974336 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:19:52.974312    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/44e112d2-e31f-4dfc-abc3-17f9c80ae9b4-secret-grpc-tls\") pod \"thanos-querier-7775c558d6-95wpp\" (UID: \"44e112d2-e31f-4dfc-abc3-17f9c80ae9b4\") " pod="openshift-monitoring/thanos-querier-7775c558d6-95wpp"
Sep 11 17:19:52.974949 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:19:52.974925    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/44e112d2-e31f-4dfc-abc3-17f9c80ae9b4-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-7775c558d6-95wpp\" (UID: \"44e112d2-e31f-4dfc-abc3-17f9c80ae9b4\") " pod="openshift-monitoring/thanos-querier-7775c558d6-95wpp"
Sep 11 17:19:52.975374 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:19:52.975334    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/44e112d2-e31f-4dfc-abc3-17f9c80ae9b4-secret-thanos-querier-tls\") pod \"thanos-querier-7775c558d6-95wpp\" (UID: \"44e112d2-e31f-4dfc-abc3-17f9c80ae9b4\") " pod="openshift-monitoring/thanos-querier-7775c558d6-95wpp"
Sep 11 17:19:52.985813 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:19:52.985782    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6d6wm\" (UniqueName: \"kubernetes.io/projected/44e112d2-e31f-4dfc-abc3-17f9c80ae9b4-kube-api-access-6d6wm\") pod \"thanos-querier-7775c558d6-95wpp\" (UID: \"44e112d2-e31f-4dfc-abc3-17f9c80ae9b4\") " pod="openshift-monitoring/thanos-querier-7775c558d6-95wpp"
Sep 11 17:19:53.095265 ip-10-0-141-139 kubenswrapper[2577]: W0911 17:19:53.095224    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddf65734a_e4e6_4b4e_9114_35af7b635044.slice/crio-2edcd56e041a9b59ed34185fb94a74291a2a79752b604c86361a569c491801bd WatchSource:0}: Error finding container 2edcd56e041a9b59ed34185fb94a74291a2a79752b604c86361a569c491801bd: Status 404 returned error can't find the container with id 2edcd56e041a9b59ed34185fb94a74291a2a79752b604c86361a569c491801bd
Sep 11 17:19:53.110296 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:19:53.110131    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-7775c558d6-95wpp"
Sep 11 17:19:53.312142 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:19:53.312092    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-fsh8r" event={"ID":"73dd9a34-fac4-44c9-96d3-b2c8968ed271","Type":"ContainerStarted","Data":"690c1a50e3a8acdf92652aff1c10d30d2d1aad8e00cb123564234738d6262b9a"}
Sep 11 17:19:53.314344 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:19:53.314276    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-xfn47" event={"ID":"1e5d9e0c-7bc0-4fcf-8845-8873df5b1bee","Type":"ContainerStarted","Data":"fedbed0a5962de41e356672d39e3136362b59bb566267977ee448b31a4559fe3"}
Sep 11 17:19:53.314582 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:19:53.314562    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-xfn47"
Sep 11 17:19:53.316095 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:19:53.316072    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-mthpt" event={"ID":"df65734a-e4e6-4b4e-9114-35af7b635044","Type":"ContainerStarted","Data":"2edcd56e041a9b59ed34185fb94a74291a2a79752b604c86361a569c491801bd"}
Sep 11 17:19:53.341219 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:19:53.341107    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-fsh8r" podStartSLOduration=64.460263383 podStartE2EDuration="1m9.341088838s" totalImagesPullingTime="4.880825455s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 17:18:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 17:19:53.34089531 +0000 UTC m=+70.914115423" watchObservedRunningTime="2026-09-11 17:19:53.341088838 +0000 UTC m=+70.914308942"
Sep 11 17:19:53.379825 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:19:53.379770    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-xfn47" podStartSLOduration=64.512699891 podStartE2EDuration="1m9.379752176s" totalImagesPullingTime="4.867052285s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 17:18:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 17:19:53.378662365 +0000 UTC m=+70.951882478" watchObservedRunningTime="2026-09-11 17:19:53.379752176 +0000 UTC m=+70.952972288"
Sep 11 17:19:53.386565 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:19:53.386525    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-67dcf8c979-fwdgr"]
Sep 11 17:19:53.390038 ip-10-0-141-139 kubenswrapper[2577]: W0911 17:19:53.390012    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf1b075b6_9bae_41d6_b0c0_1c8b81c3207a.slice/crio-01f2958ea2ed05286d0197de41e13b07163cc2a9696a51b7235eaeec4f660dcb WatchSource:0}: Error finding container 01f2958ea2ed05286d0197de41e13b07163cc2a9696a51b7235eaeec4f660dcb: Status 404 returned error can't find the container with id 01f2958ea2ed05286d0197de41e13b07163cc2a9696a51b7235eaeec4f660dcb
Sep 11 17:19:53.428164 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:19:53.428137    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 11 17:19:53.435853 ip-10-0-141-139 kubenswrapper[2577]: W0911 17:19:53.435819    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4c942d42_091f_4c68_8faa_f6276bd26c15.slice/crio-1351eb8bcfd974cd6b2c8b6505b18ceee7c4925274a6385fd484e5d4892645bf WatchSource:0}: Error finding container 1351eb8bcfd974cd6b2c8b6505b18ceee7c4925274a6385fd484e5d4892645bf: Status 404 returned error can't find the container with id 1351eb8bcfd974cd6b2c8b6505b18ceee7c4925274a6385fd484e5d4892645bf
Sep 11 17:19:53.443277 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:19:53.442874    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-7775c558d6-95wpp"]
Sep 11 17:19:53.445852 ip-10-0-141-139 kubenswrapper[2577]: W0911 17:19:53.445821    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod44e112d2_e31f_4dfc_abc3_17f9c80ae9b4.slice/crio-29a68b4651e963e37fab9e507e58d1bf0d0f7cd153bd7cb2104e478749724cb8 WatchSource:0}: Error finding container 29a68b4651e963e37fab9e507e58d1bf0d0f7cd153bd7cb2104e478749724cb8: Status 404 returned error can't find the container with id 29a68b4651e963e37fab9e507e58d1bf0d0f7cd153bd7cb2104e478749724cb8
Sep 11 17:19:53.993267 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:19:53.991926    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/metrics-server-7f9b5d9fd4-wnwpt"]
Sep 11 17:19:53.994517 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:19:53.994469    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-7f9b5d9fd4-wnwpt"
Sep 11 17:19:53.998323 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:19:53.998170    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-server-audit-profiles\""
Sep 11 17:19:53.998323 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:19:53.998238    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-dockercfg-j2fs8\""
Sep 11 17:19:53.999273 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:19:53.999249    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\""
Sep 11 17:19:53.999400 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:19:53.999378    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-client-certs\""
Sep 11 17:19:53.999562 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:19:53.999532    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-a8rq9oo9vadnt\""
Sep 11 17:19:54.000299 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:19:54.000282    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-tls\""
Sep 11 17:19:54.025029 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:19:54.024995    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-7f9b5d9fd4-wnwpt"]
Sep 11 17:19:54.081108 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:19:54.081070    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/7b9f4b24-4fe6-4fc8-8917-08f335a75844-audit-log\") pod \"metrics-server-7f9b5d9fd4-wnwpt\" (UID: \"7b9f4b24-4fe6-4fc8-8917-08f335a75844\") " pod="openshift-monitoring/metrics-server-7f9b5d9fd4-wnwpt"
Sep 11 17:19:54.081305 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:19:54.081131    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b99lr\" (UniqueName: \"kubernetes.io/projected/7b9f4b24-4fe6-4fc8-8917-08f335a75844-kube-api-access-b99lr\") pod \"metrics-server-7f9b5d9fd4-wnwpt\" (UID: \"7b9f4b24-4fe6-4fc8-8917-08f335a75844\") " pod="openshift-monitoring/metrics-server-7f9b5d9fd4-wnwpt"
Sep 11 17:19:54.081305 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:19:54.081223    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/7b9f4b24-4fe6-4fc8-8917-08f335a75844-client-ca-bundle\") pod \"metrics-server-7f9b5d9fd4-wnwpt\" (UID: \"7b9f4b24-4fe6-4fc8-8917-08f335a75844\") " pod="openshift-monitoring/metrics-server-7f9b5d9fd4-wnwpt"
Sep 11 17:19:54.081305 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:19:54.081253    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/7b9f4b24-4fe6-4fc8-8917-08f335a75844-secret-metrics-server-client-certs\") pod \"metrics-server-7f9b5d9fd4-wnwpt\" (UID: \"7b9f4b24-4fe6-4fc8-8917-08f335a75844\") " pod="openshift-monitoring/metrics-server-7f9b5d9fd4-wnwpt"
Sep 11 17:19:54.081305 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:19:54.081276    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/7b9f4b24-4fe6-4fc8-8917-08f335a75844-secret-metrics-server-tls\") pod \"metrics-server-7f9b5d9fd4-wnwpt\" (UID: \"7b9f4b24-4fe6-4fc8-8917-08f335a75844\") " pod="openshift-monitoring/metrics-server-7f9b5d9fd4-wnwpt"
Sep 11 17:19:54.081445 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:19:54.081320    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7b9f4b24-4fe6-4fc8-8917-08f335a75844-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-7f9b5d9fd4-wnwpt\" (UID: \"7b9f4b24-4fe6-4fc8-8917-08f335a75844\") " pod="openshift-monitoring/metrics-server-7f9b5d9fd4-wnwpt"
Sep 11 17:19:54.081445 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:19:54.081394    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/7b9f4b24-4fe6-4fc8-8917-08f335a75844-metrics-server-audit-profiles\") pod \"metrics-server-7f9b5d9fd4-wnwpt\" (UID: \"7b9f4b24-4fe6-4fc8-8917-08f335a75844\") " pod="openshift-monitoring/metrics-server-7f9b5d9fd4-wnwpt"
Sep 11 17:19:54.182159 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:19:54.182119    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/7b9f4b24-4fe6-4fc8-8917-08f335a75844-secret-metrics-server-tls\") pod \"metrics-server-7f9b5d9fd4-wnwpt\" (UID: \"7b9f4b24-4fe6-4fc8-8917-08f335a75844\") " pod="openshift-monitoring/metrics-server-7f9b5d9fd4-wnwpt"
Sep 11 17:19:54.182350 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:19:54.182171    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7b9f4b24-4fe6-4fc8-8917-08f335a75844-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-7f9b5d9fd4-wnwpt\" (UID: \"7b9f4b24-4fe6-4fc8-8917-08f335a75844\") " pod="openshift-monitoring/metrics-server-7f9b5d9fd4-wnwpt"
Sep 11 17:19:54.182350 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:19:54.182208    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/7b9f4b24-4fe6-4fc8-8917-08f335a75844-metrics-server-audit-profiles\") pod \"metrics-server-7f9b5d9fd4-wnwpt\" (UID: \"7b9f4b24-4fe6-4fc8-8917-08f335a75844\") " pod="openshift-monitoring/metrics-server-7f9b5d9fd4-wnwpt"
Sep 11 17:19:54.182350 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:19:54.182270    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/7b9f4b24-4fe6-4fc8-8917-08f335a75844-audit-log\") pod \"metrics-server-7f9b5d9fd4-wnwpt\" (UID: \"7b9f4b24-4fe6-4fc8-8917-08f335a75844\") " pod="openshift-monitoring/metrics-server-7f9b5d9fd4-wnwpt"
Sep 11 17:19:54.182350 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:19:54.182318    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b99lr\" (UniqueName: \"kubernetes.io/projected/7b9f4b24-4fe6-4fc8-8917-08f335a75844-kube-api-access-b99lr\") pod \"metrics-server-7f9b5d9fd4-wnwpt\" (UID: \"7b9f4b24-4fe6-4fc8-8917-08f335a75844\") " pod="openshift-monitoring/metrics-server-7f9b5d9fd4-wnwpt"
Sep 11 17:19:54.182587 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:19:54.182382    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/7b9f4b24-4fe6-4fc8-8917-08f335a75844-client-ca-bundle\") pod \"metrics-server-7f9b5d9fd4-wnwpt\" (UID: \"7b9f4b24-4fe6-4fc8-8917-08f335a75844\") " pod="openshift-monitoring/metrics-server-7f9b5d9fd4-wnwpt"
Sep 11 17:19:54.182587 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:19:54.182412    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/7b9f4b24-4fe6-4fc8-8917-08f335a75844-secret-metrics-server-client-certs\") pod \"metrics-server-7f9b5d9fd4-wnwpt\" (UID: \"7b9f4b24-4fe6-4fc8-8917-08f335a75844\") " pod="openshift-monitoring/metrics-server-7f9b5d9fd4-wnwpt"
Sep 11 17:19:54.182744 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:19:54.182720    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/7b9f4b24-4fe6-4fc8-8917-08f335a75844-audit-log\") pod \"metrics-server-7f9b5d9fd4-wnwpt\" (UID: \"7b9f4b24-4fe6-4fc8-8917-08f335a75844\") " pod="openshift-monitoring/metrics-server-7f9b5d9fd4-wnwpt"
Sep 11 17:19:54.183306 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:19:54.183246    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7b9f4b24-4fe6-4fc8-8917-08f335a75844-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-7f9b5d9fd4-wnwpt\" (UID: \"7b9f4b24-4fe6-4fc8-8917-08f335a75844\") " pod="openshift-monitoring/metrics-server-7f9b5d9fd4-wnwpt"
Sep 11 17:19:54.183689 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:19:54.183662    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/7b9f4b24-4fe6-4fc8-8917-08f335a75844-metrics-server-audit-profiles\") pod \"metrics-server-7f9b5d9fd4-wnwpt\" (UID: \"7b9f4b24-4fe6-4fc8-8917-08f335a75844\") " pod="openshift-monitoring/metrics-server-7f9b5d9fd4-wnwpt"
Sep 11 17:19:54.185422 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:19:54.185389    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/7b9f4b24-4fe6-4fc8-8917-08f335a75844-client-ca-bundle\") pod \"metrics-server-7f9b5d9fd4-wnwpt\" (UID: \"7b9f4b24-4fe6-4fc8-8917-08f335a75844\") " pod="openshift-monitoring/metrics-server-7f9b5d9fd4-wnwpt"
Sep 11 17:19:54.185548 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:19:54.185428    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/7b9f4b24-4fe6-4fc8-8917-08f335a75844-secret-metrics-server-client-certs\") pod \"metrics-server-7f9b5d9fd4-wnwpt\" (UID: \"7b9f4b24-4fe6-4fc8-8917-08f335a75844\") " pod="openshift-monitoring/metrics-server-7f9b5d9fd4-wnwpt"
Sep 11 17:19:54.185548 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:19:54.185427    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/7b9f4b24-4fe6-4fc8-8917-08f335a75844-secret-metrics-server-tls\") pod \"metrics-server-7f9b5d9fd4-wnwpt\" (UID: \"7b9f4b24-4fe6-4fc8-8917-08f335a75844\") " pod="openshift-monitoring/metrics-server-7f9b5d9fd4-wnwpt"
Sep 11 17:19:54.197381 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:19:54.197352    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-b99lr\" (UniqueName: \"kubernetes.io/projected/7b9f4b24-4fe6-4fc8-8917-08f335a75844-kube-api-access-b99lr\") pod \"metrics-server-7f9b5d9fd4-wnwpt\" (UID: \"7b9f4b24-4fe6-4fc8-8917-08f335a75844\") " pod="openshift-monitoring/metrics-server-7f9b5d9fd4-wnwpt"
Sep 11 17:19:54.298824 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:19:54.297838    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/monitoring-plugin-5d85778b9d-cqvpv"]
Sep 11 17:19:54.300165 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:19:54.300146    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-5d85778b9d-cqvpv"
Sep 11 17:19:54.306276 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:19:54.306234    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-7f9b5d9fd4-wnwpt"
Sep 11 17:19:54.310327 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:19:54.310303    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-cert\""
Sep 11 17:19:54.310595 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:19:54.310493    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-dockercfg-gbgp2\""
Sep 11 17:19:54.328301 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:19:54.328264    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-7775c558d6-95wpp" event={"ID":"44e112d2-e31f-4dfc-abc3-17f9c80ae9b4","Type":"ContainerStarted","Data":"29a68b4651e963e37fab9e507e58d1bf0d0f7cd153bd7cb2104e478749724cb8"}
Sep 11 17:19:54.330954 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:19:54.330921    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-fwdgr" event={"ID":"f1b075b6-9bae-41d6-b0c0-1c8b81c3207a","Type":"ContainerStarted","Data":"72a4ae0b1a3f4026378940798e04f149c5101c11471c5e9eeff092f06fcf48fe"}
Sep 11 17:19:54.331069 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:19:54.330985    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-fwdgr" event={"ID":"f1b075b6-9bae-41d6-b0c0-1c8b81c3207a","Type":"ContainerStarted","Data":"c5b11a2fad9f1c0cef24c8274566f8cbee0f04689e9c62cdd5d256acd1d615cb"}
Sep 11 17:19:54.331069 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:19:54.331014    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-fwdgr" event={"ID":"f1b075b6-9bae-41d6-b0c0-1c8b81c3207a","Type":"ContainerStarted","Data":"01f2958ea2ed05286d0197de41e13b07163cc2a9696a51b7235eaeec4f660dcb"}
Sep 11 17:19:54.332990 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:19:54.332916    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"4c942d42-091f-4c68-8faa-f6276bd26c15","Type":"ContainerStarted","Data":"1351eb8bcfd974cd6b2c8b6505b18ceee7c4925274a6385fd484e5d4892645bf"}
Sep 11 17:19:54.336165 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:19:54.336140    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-5d85778b9d-cqvpv"]
Sep 11 17:19:54.385922 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:19:54.384910    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xd295\" (UniqueName: \"kubernetes.io/projected/071e2825-99c5-4ccf-9443-e0cdf8f52df3-kube-api-access-xd295\") pod \"monitoring-plugin-5d85778b9d-cqvpv\" (UID: \"071e2825-99c5-4ccf-9443-e0cdf8f52df3\") " pod="openshift-monitoring/monitoring-plugin-5d85778b9d-cqvpv"
Sep 11 17:19:54.385922 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:19:54.384958    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/071e2825-99c5-4ccf-9443-e0cdf8f52df3-monitoring-plugin-cert\") pod \"monitoring-plugin-5d85778b9d-cqvpv\" (UID: \"071e2825-99c5-4ccf-9443-e0cdf8f52df3\") " pod="openshift-monitoring/monitoring-plugin-5d85778b9d-cqvpv"
Sep 11 17:19:54.486324 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:19:54.486282    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xd295\" (UniqueName: \"kubernetes.io/projected/071e2825-99c5-4ccf-9443-e0cdf8f52df3-kube-api-access-xd295\") pod \"monitoring-plugin-5d85778b9d-cqvpv\" (UID: \"071e2825-99c5-4ccf-9443-e0cdf8f52df3\") " pod="openshift-monitoring/monitoring-plugin-5d85778b9d-cqvpv"
Sep 11 17:19:54.486495 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:19:54.486336    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/071e2825-99c5-4ccf-9443-e0cdf8f52df3-monitoring-plugin-cert\") pod \"monitoring-plugin-5d85778b9d-cqvpv\" (UID: \"071e2825-99c5-4ccf-9443-e0cdf8f52df3\") " pod="openshift-monitoring/monitoring-plugin-5d85778b9d-cqvpv"
Sep 11 17:19:54.486495 ip-10-0-141-139 kubenswrapper[2577]: E0911 17:19:54.486479    2577 secret.go:189] Couldn't get secret openshift-monitoring/monitoring-plugin-cert: secret "monitoring-plugin-cert" not found
Sep 11 17:19:54.486750 ip-10-0-141-139 kubenswrapper[2577]: E0911 17:19:54.486726    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/071e2825-99c5-4ccf-9443-e0cdf8f52df3-monitoring-plugin-cert podName:071e2825-99c5-4ccf-9443-e0cdf8f52df3 nodeName:}" failed. No retries permitted until 2026-09-11 17:19:54.986691835 +0000 UTC m=+72.559911943 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "monitoring-plugin-cert" (UniqueName: "kubernetes.io/secret/071e2825-99c5-4ccf-9443-e0cdf8f52df3-monitoring-plugin-cert") pod "monitoring-plugin-5d85778b9d-cqvpv" (UID: "071e2825-99c5-4ccf-9443-e0cdf8f52df3") : secret "monitoring-plugin-cert" not found
Sep 11 17:19:54.556879 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:19:54.556847    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xd295\" (UniqueName: \"kubernetes.io/projected/071e2825-99c5-4ccf-9443-e0cdf8f52df3-kube-api-access-xd295\") pod \"monitoring-plugin-5d85778b9d-cqvpv\" (UID: \"071e2825-99c5-4ccf-9443-e0cdf8f52df3\") " pod="openshift-monitoring/monitoring-plugin-5d85778b9d-cqvpv"
Sep 11 17:19:54.626409 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:19:54.626360    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-7f9b5d9fd4-wnwpt"]
Sep 11 17:19:54.631231 ip-10-0-141-139 kubenswrapper[2577]: W0911 17:19:54.631195    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7b9f4b24_4fe6_4fc8_8917_08f335a75844.slice/crio-4eafd8b434fa465bd2ea0ac66ccfbe7de3f94949af06636ee4d168c1fef2a464 WatchSource:0}: Error finding container 4eafd8b434fa465bd2ea0ac66ccfbe7de3f94949af06636ee4d168c1fef2a464: Status 404 returned error can't find the container with id 4eafd8b434fa465bd2ea0ac66ccfbe7de3f94949af06636ee4d168c1fef2a464
Sep 11 17:19:54.841157 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:19:54.840374    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-57bf98d9fc-nrdtq"]
Sep 11 17:19:54.888232 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:19:54.888181    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-7874849c67-ftnmh"]
Sep 11 17:19:54.905924 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:19:54.905882    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7874849c67-ftnmh"
Sep 11 17:19:54.917662 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:19:54.917629    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-7874849c67-ftnmh"]
Sep 11 17:19:54.930378 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:19:54.930340    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/telemeter-client-76875c59d5-d247c"]
Sep 11 17:19:54.960175 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:19:54.958394    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-76875c59d5-d247c"]
Sep 11 17:19:54.960175 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:19:54.958533    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-76875c59d5-d247c"
Sep 11 17:19:54.962744 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:19:54.961713    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-kube-rbac-proxy-config\""
Sep 11 17:19:54.962744 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:19:54.961971    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-client-serving-certs-ca-bundle\""
Sep 11 17:19:54.962744 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:19:54.962010    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client\""
Sep 11 17:19:54.962744 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:19:54.962228    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"federate-client-certs\""
Sep 11 17:19:54.962744 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:19:54.962401    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-dockercfg-fnr64\""
Sep 11 17:19:54.962744 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:19:54.962576    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-tls\""
Sep 11 17:19:54.969468 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:19:54.969212    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-trusted-ca-bundle-8i12ta5c71j38\""
Sep 11 17:19:54.990105 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:19:54.990070    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/addf52ab-b262-4b3a-be3d-5ad1e772090f-secret-telemeter-client\") pod \"telemeter-client-76875c59d5-d247c\" (UID: \"addf52ab-b262-4b3a-be3d-5ad1e772090f\") " pod="openshift-monitoring/telemeter-client-76875c59d5-d247c"
Sep 11 17:19:54.990249 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:19:54.990127    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/addf52ab-b262-4b3a-be3d-5ad1e772090f-metrics-client-ca\") pod \"telemeter-client-76875c59d5-d247c\" (UID: \"addf52ab-b262-4b3a-be3d-5ad1e772090f\") " pod="openshift-monitoring/telemeter-client-76875c59d5-d247c"
Sep 11 17:19:54.990249 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:19:54.990188    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/c2dc1ae2-696a-49f6-85ae-071961ef87fc-oauth-serving-cert\") pod \"console-7874849c67-ftnmh\" (UID: \"c2dc1ae2-696a-49f6-85ae-071961ef87fc\") " pod="openshift-console/console-7874849c67-ftnmh"
Sep 11 17:19:54.990375 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:19:54.990289    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/071e2825-99c5-4ccf-9443-e0cdf8f52df3-monitoring-plugin-cert\") pod \"monitoring-plugin-5d85778b9d-cqvpv\" (UID: \"071e2825-99c5-4ccf-9443-e0cdf8f52df3\") " pod="openshift-monitoring/monitoring-plugin-5d85778b9d-cqvpv"
Sep 11 17:19:54.990375 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:19:54.990350    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/c2dc1ae2-696a-49f6-85ae-071961ef87fc-console-serving-cert\") pod \"console-7874849c67-ftnmh\" (UID: \"c2dc1ae2-696a-49f6-85ae-071961ef87fc\") " pod="openshift-console/console-7874849c67-ftnmh"
Sep 11 17:19:54.990474 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:19:54.990395    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/addf52ab-b262-4b3a-be3d-5ad1e772090f-federate-client-tls\") pod \"telemeter-client-76875c59d5-d247c\" (UID: \"addf52ab-b262-4b3a-be3d-5ad1e772090f\") " pod="openshift-monitoring/telemeter-client-76875c59d5-d247c"
Sep 11 17:19:54.990474 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:19:54.990423    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/addf52ab-b262-4b3a-be3d-5ad1e772090f-telemeter-client-tls\") pod \"telemeter-client-76875c59d5-d247c\" (UID: \"addf52ab-b262-4b3a-be3d-5ad1e772090f\") " pod="openshift-monitoring/telemeter-client-76875c59d5-d247c"
Sep 11 17:19:54.990474 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:19:54.990457    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cv8zd\" (UniqueName: \"kubernetes.io/projected/addf52ab-b262-4b3a-be3d-5ad1e772090f-kube-api-access-cv8zd\") pod \"telemeter-client-76875c59d5-d247c\" (UID: \"addf52ab-b262-4b3a-be3d-5ad1e772090f\") " pod="openshift-monitoring/telemeter-client-76875c59d5-d247c"
Sep 11 17:19:54.990679 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:19:54.990482    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-svgxp\" (UniqueName: \"kubernetes.io/projected/c2dc1ae2-696a-49f6-85ae-071961ef87fc-kube-api-access-svgxp\") pod \"console-7874849c67-ftnmh\" (UID: \"c2dc1ae2-696a-49f6-85ae-071961ef87fc\") " pod="openshift-console/console-7874849c67-ftnmh"
Sep 11 17:19:54.990679 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:19:54.990526    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/c2dc1ae2-696a-49f6-85ae-071961ef87fc-console-config\") pod \"console-7874849c67-ftnmh\" (UID: \"c2dc1ae2-696a-49f6-85ae-071961ef87fc\") " pod="openshift-console/console-7874849c67-ftnmh"
Sep 11 17:19:54.990679 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:19:54.990563    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/addf52ab-b262-4b3a-be3d-5ad1e772090f-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-76875c59d5-d247c\" (UID: \"addf52ab-b262-4b3a-be3d-5ad1e772090f\") " pod="openshift-monitoring/telemeter-client-76875c59d5-d247c"
Sep 11 17:19:54.990679 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:19:54.990601    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/addf52ab-b262-4b3a-be3d-5ad1e772090f-telemeter-trusted-ca-bundle\") pod \"telemeter-client-76875c59d5-d247c\" (UID: \"addf52ab-b262-4b3a-be3d-5ad1e772090f\") " pod="openshift-monitoring/telemeter-client-76875c59d5-d247c"
Sep 11 17:19:54.992809 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:19:54.992604    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/c2dc1ae2-696a-49f6-85ae-071961ef87fc-console-oauth-config\") pod \"console-7874849c67-ftnmh\" (UID: \"c2dc1ae2-696a-49f6-85ae-071961ef87fc\") " pod="openshift-console/console-7874849c67-ftnmh"
Sep 11 17:19:54.992809 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:19:54.992665    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/addf52ab-b262-4b3a-be3d-5ad1e772090f-serving-certs-ca-bundle\") pod \"telemeter-client-76875c59d5-d247c\" (UID: \"addf52ab-b262-4b3a-be3d-5ad1e772090f\") " pod="openshift-monitoring/telemeter-client-76875c59d5-d247c"
Sep 11 17:19:54.992809 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:19:54.992695    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/c2dc1ae2-696a-49f6-85ae-071961ef87fc-service-ca\") pod \"console-7874849c67-ftnmh\" (UID: \"c2dc1ae2-696a-49f6-85ae-071961ef87fc\") " pod="openshift-console/console-7874849c67-ftnmh"
Sep 11 17:19:54.992809 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:19:54.992723    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c2dc1ae2-696a-49f6-85ae-071961ef87fc-trusted-ca-bundle\") pod \"console-7874849c67-ftnmh\" (UID: \"c2dc1ae2-696a-49f6-85ae-071961ef87fc\") " pod="openshift-console/console-7874849c67-ftnmh"
Sep 11 17:19:54.994339 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:19:54.994296    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/071e2825-99c5-4ccf-9443-e0cdf8f52df3-monitoring-plugin-cert\") pod \"monitoring-plugin-5d85778b9d-cqvpv\" (UID: \"071e2825-99c5-4ccf-9443-e0cdf8f52df3\") " pod="openshift-monitoring/monitoring-plugin-5d85778b9d-cqvpv"
Sep 11 17:19:55.093948 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:19:55.093860    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/c2dc1ae2-696a-49f6-85ae-071961ef87fc-console-serving-cert\") pod \"console-7874849c67-ftnmh\" (UID: \"c2dc1ae2-696a-49f6-85ae-071961ef87fc\") " pod="openshift-console/console-7874849c67-ftnmh"
Sep 11 17:19:55.093948 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:19:55.093922    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/addf52ab-b262-4b3a-be3d-5ad1e772090f-federate-client-tls\") pod \"telemeter-client-76875c59d5-d247c\" (UID: \"addf52ab-b262-4b3a-be3d-5ad1e772090f\") " pod="openshift-monitoring/telemeter-client-76875c59d5-d247c"
Sep 11 17:19:55.094179 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:19:55.093951    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/addf52ab-b262-4b3a-be3d-5ad1e772090f-telemeter-client-tls\") pod \"telemeter-client-76875c59d5-d247c\" (UID: \"addf52ab-b262-4b3a-be3d-5ad1e772090f\") " pod="openshift-monitoring/telemeter-client-76875c59d5-d247c"
Sep 11 17:19:55.094179 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:19:55.093980    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cv8zd\" (UniqueName: \"kubernetes.io/projected/addf52ab-b262-4b3a-be3d-5ad1e772090f-kube-api-access-cv8zd\") pod \"telemeter-client-76875c59d5-d247c\" (UID: \"addf52ab-b262-4b3a-be3d-5ad1e772090f\") " pod="openshift-monitoring/telemeter-client-76875c59d5-d247c"
Sep 11 17:19:55.094179 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:19:55.094014    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-svgxp\" (UniqueName: \"kubernetes.io/projected/c2dc1ae2-696a-49f6-85ae-071961ef87fc-kube-api-access-svgxp\") pod \"console-7874849c67-ftnmh\" (UID: \"c2dc1ae2-696a-49f6-85ae-071961ef87fc\") " pod="openshift-console/console-7874849c67-ftnmh"
Sep 11 17:19:55.094179 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:19:55.094044    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/c2dc1ae2-696a-49f6-85ae-071961ef87fc-console-config\") pod \"console-7874849c67-ftnmh\" (UID: \"c2dc1ae2-696a-49f6-85ae-071961ef87fc\") " pod="openshift-console/console-7874849c67-ftnmh"
Sep 11 17:19:55.094179 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:19:55.094090    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/addf52ab-b262-4b3a-be3d-5ad1e772090f-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-76875c59d5-d247c\" (UID: \"addf52ab-b262-4b3a-be3d-5ad1e772090f\") " pod="openshift-monitoring/telemeter-client-76875c59d5-d247c"
Sep 11 17:19:55.094179 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:19:55.094118    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/addf52ab-b262-4b3a-be3d-5ad1e772090f-telemeter-trusted-ca-bundle\") pod \"telemeter-client-76875c59d5-d247c\" (UID: \"addf52ab-b262-4b3a-be3d-5ad1e772090f\") " pod="openshift-monitoring/telemeter-client-76875c59d5-d247c"
Sep 11 17:19:55.094179 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:19:55.094141    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/c2dc1ae2-696a-49f6-85ae-071961ef87fc-console-oauth-config\") pod \"console-7874849c67-ftnmh\" (UID: \"c2dc1ae2-696a-49f6-85ae-071961ef87fc\") " pod="openshift-console/console-7874849c67-ftnmh"
Sep 11 17:19:55.094179 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:19:55.094168    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/addf52ab-b262-4b3a-be3d-5ad1e772090f-serving-certs-ca-bundle\") pod \"telemeter-client-76875c59d5-d247c\" (UID: \"addf52ab-b262-4b3a-be3d-5ad1e772090f\") " pod="openshift-monitoring/telemeter-client-76875c59d5-d247c"
Sep 11 17:19:55.094607 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:19:55.094199    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/c2dc1ae2-696a-49f6-85ae-071961ef87fc-service-ca\") pod \"console-7874849c67-ftnmh\" (UID: \"c2dc1ae2-696a-49f6-85ae-071961ef87fc\") " pod="openshift-console/console-7874849c67-ftnmh"
Sep 11 17:19:55.094607 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:19:55.094225    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c2dc1ae2-696a-49f6-85ae-071961ef87fc-trusted-ca-bundle\") pod \"console-7874849c67-ftnmh\" (UID: \"c2dc1ae2-696a-49f6-85ae-071961ef87fc\") " pod="openshift-console/console-7874849c67-ftnmh"
Sep 11 17:19:55.094607 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:19:55.094295    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/addf52ab-b262-4b3a-be3d-5ad1e772090f-secret-telemeter-client\") pod \"telemeter-client-76875c59d5-d247c\" (UID: \"addf52ab-b262-4b3a-be3d-5ad1e772090f\") " pod="openshift-monitoring/telemeter-client-76875c59d5-d247c"
Sep 11 17:19:55.094607 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:19:55.094338    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/addf52ab-b262-4b3a-be3d-5ad1e772090f-metrics-client-ca\") pod \"telemeter-client-76875c59d5-d247c\" (UID: \"addf52ab-b262-4b3a-be3d-5ad1e772090f\") " pod="openshift-monitoring/telemeter-client-76875c59d5-d247c"
Sep 11 17:19:55.094607 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:19:55.094363    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/c2dc1ae2-696a-49f6-85ae-071961ef87fc-oauth-serving-cert\") pod \"console-7874849c67-ftnmh\" (UID: \"c2dc1ae2-696a-49f6-85ae-071961ef87fc\") " pod="openshift-console/console-7874849c67-ftnmh"
Sep 11 17:19:55.096792 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:19:55.095142    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/c2dc1ae2-696a-49f6-85ae-071961ef87fc-oauth-serving-cert\") pod \"console-7874849c67-ftnmh\" (UID: \"c2dc1ae2-696a-49f6-85ae-071961ef87fc\") " pod="openshift-console/console-7874849c67-ftnmh"
Sep 11 17:19:55.096792 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:19:55.095409    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/c2dc1ae2-696a-49f6-85ae-071961ef87fc-console-config\") pod \"console-7874849c67-ftnmh\" (UID: \"c2dc1ae2-696a-49f6-85ae-071961ef87fc\") " pod="openshift-console/console-7874849c67-ftnmh"
Sep 11 17:19:55.096792 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:19:55.095990    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/c2dc1ae2-696a-49f6-85ae-071961ef87fc-service-ca\") pod \"console-7874849c67-ftnmh\" (UID: \"c2dc1ae2-696a-49f6-85ae-071961ef87fc\") " pod="openshift-console/console-7874849c67-ftnmh"
Sep 11 17:19:55.096792 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:19:55.096481    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/addf52ab-b262-4b3a-be3d-5ad1e772090f-serving-certs-ca-bundle\") pod \"telemeter-client-76875c59d5-d247c\" (UID: \"addf52ab-b262-4b3a-be3d-5ad1e772090f\") " pod="openshift-monitoring/telemeter-client-76875c59d5-d247c"
Sep 11 17:19:55.096792 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:19:55.096747    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/addf52ab-b262-4b3a-be3d-5ad1e772090f-metrics-client-ca\") pod \"telemeter-client-76875c59d5-d247c\" (UID: \"addf52ab-b262-4b3a-be3d-5ad1e772090f\") " pod="openshift-monitoring/telemeter-client-76875c59d5-d247c"
Sep 11 17:19:55.097144 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:19:55.096890    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c2dc1ae2-696a-49f6-85ae-071961ef87fc-trusted-ca-bundle\") pod \"console-7874849c67-ftnmh\" (UID: \"c2dc1ae2-696a-49f6-85ae-071961ef87fc\") " pod="openshift-console/console-7874849c67-ftnmh"
Sep 11 17:19:55.097886 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:19:55.097845    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/addf52ab-b262-4b3a-be3d-5ad1e772090f-telemeter-trusted-ca-bundle\") pod \"telemeter-client-76875c59d5-d247c\" (UID: \"addf52ab-b262-4b3a-be3d-5ad1e772090f\") " pod="openshift-monitoring/telemeter-client-76875c59d5-d247c"
Sep 11 17:19:55.100637 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:19:55.100593    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/c2dc1ae2-696a-49f6-85ae-071961ef87fc-console-serving-cert\") pod \"console-7874849c67-ftnmh\" (UID: \"c2dc1ae2-696a-49f6-85ae-071961ef87fc\") " pod="openshift-console/console-7874849c67-ftnmh"
Sep 11 17:19:55.101067 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:19:55.101045    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/addf52ab-b262-4b3a-be3d-5ad1e772090f-secret-telemeter-client\") pod \"telemeter-client-76875c59d5-d247c\" (UID: \"addf52ab-b262-4b3a-be3d-5ad1e772090f\") " pod="openshift-monitoring/telemeter-client-76875c59d5-d247c"
Sep 11 17:19:55.105378 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:19:55.105338    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-svgxp\" (UniqueName: \"kubernetes.io/projected/c2dc1ae2-696a-49f6-85ae-071961ef87fc-kube-api-access-svgxp\") pod \"console-7874849c67-ftnmh\" (UID: \"c2dc1ae2-696a-49f6-85ae-071961ef87fc\") " pod="openshift-console/console-7874849c67-ftnmh"
Sep 11 17:19:55.105702 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:19:55.105679    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cv8zd\" (UniqueName: \"kubernetes.io/projected/addf52ab-b262-4b3a-be3d-5ad1e772090f-kube-api-access-cv8zd\") pod \"telemeter-client-76875c59d5-d247c\" (UID: \"addf52ab-b262-4b3a-be3d-5ad1e772090f\") " pod="openshift-monitoring/telemeter-client-76875c59d5-d247c"
Sep 11 17:19:55.118285 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:19:55.118227    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/addf52ab-b262-4b3a-be3d-5ad1e772090f-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-76875c59d5-d247c\" (UID: \"addf52ab-b262-4b3a-be3d-5ad1e772090f\") " pod="openshift-monitoring/telemeter-client-76875c59d5-d247c"
Sep 11 17:19:55.118285 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:19:55.118247    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/addf52ab-b262-4b3a-be3d-5ad1e772090f-federate-client-tls\") pod \"telemeter-client-76875c59d5-d247c\" (UID: \"addf52ab-b262-4b3a-be3d-5ad1e772090f\") " pod="openshift-monitoring/telemeter-client-76875c59d5-d247c"
Sep 11 17:19:55.118457 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:19:55.118341    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/addf52ab-b262-4b3a-be3d-5ad1e772090f-telemeter-client-tls\") pod \"telemeter-client-76875c59d5-d247c\" (UID: \"addf52ab-b262-4b3a-be3d-5ad1e772090f\") " pod="openshift-monitoring/telemeter-client-76875c59d5-d247c"
Sep 11 17:19:55.120106 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:19:55.120056    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/c2dc1ae2-696a-49f6-85ae-071961ef87fc-console-oauth-config\") pod \"console-7874849c67-ftnmh\" (UID: \"c2dc1ae2-696a-49f6-85ae-071961ef87fc\") " pod="openshift-console/console-7874849c67-ftnmh"
Sep 11 17:19:55.210830 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:19:55.210790    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-5d85778b9d-cqvpv"
Sep 11 17:19:55.222218 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:19:55.222180    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7874849c67-ftnmh"
Sep 11 17:19:55.269838 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:19:55.269809    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-g6qq7"
Sep 11 17:19:55.295267 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:19:55.295209    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-76875c59d5-d247c"
Sep 11 17:19:55.339918 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:19:55.339886    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-59fbdd77d-bfz6b" event={"ID":"e41207d9-a802-4537-a459-01deb5c8fb16","Type":"ContainerStarted","Data":"fa8274f0a63ea8cf3d787048dbf800e7ab4aed8f9a6a2599228c5fb92eab7382"}
Sep 11 17:19:55.340331 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:19:55.339955    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-59fbdd77d-bfz6b" event={"ID":"e41207d9-a802-4537-a459-01deb5c8fb16","Type":"ContainerStarted","Data":"a3d417d72f3da0ece341631283afcce85cd6ef7d011769a6b96ba5eddfc60756"}
Sep 11 17:19:55.340331 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:19:55.339989    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-59fbdd77d-bfz6b" event={"ID":"e41207d9-a802-4537-a459-01deb5c8fb16","Type":"ContainerStarted","Data":"16d0a054ea765c6593a50eba3a8681fbb493892870c5ed99303775f57479da85"}
Sep 11 17:19:55.341795 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:19:55.341767    2577 generic.go:356] "Generic (PLEG): container finished" podID="df65734a-e4e6-4b4e-9114-35af7b635044" containerID="f167035ac25f82b6a8a23d6e8f6871a4c1c32345b9d61446a5fe646fc8794d0b" exitCode=0
Sep 11 17:19:55.341897 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:19:55.341841    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-mthpt" event={"ID":"df65734a-e4e6-4b4e-9114-35af7b635044","Type":"ContainerDied","Data":"f167035ac25f82b6a8a23d6e8f6871a4c1c32345b9d61446a5fe646fc8794d0b"}
Sep 11 17:19:55.343098 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:19:55.343075    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-7f9b5d9fd4-wnwpt" event={"ID":"7b9f4b24-4fe6-4fc8-8917-08f335a75844","Type":"ContainerStarted","Data":"4eafd8b434fa465bd2ea0ac66ccfbe7de3f94949af06636ee4d168c1fef2a464"}
Sep 11 17:19:55.362556 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:19:55.362431    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/kube-state-metrics-59fbdd77d-bfz6b" podStartSLOduration=3.7114538169999998 podStartE2EDuration="6.362410675s" totalImagesPullingTime="2.650956858s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 17:19:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 17:19:55.360754805 +0000 UTC m=+72.933974920" watchObservedRunningTime="2026-09-11 17:19:55.362410675 +0000 UTC m=+72.935630789"
Sep 11 17:19:56.084730 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:19:56.084693    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 11 17:19:56.093573 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:19:56.093542    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 17:19:56.100858 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:19:56.100822    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\""
Sep 11 17:19:56.106765 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:19:56.106742    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\""
Sep 11 17:19:56.106765 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:19:56.106763    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\""
Sep 11 17:19:56.108102 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:19:56.108057    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\""
Sep 11 17:19:56.108102 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:19:56.108093    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\""
Sep 11 17:19:56.108306 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:19:56.108141    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\""
Sep 11 17:19:56.108457 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:19:56.108439    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\""
Sep 11 17:19:56.108551 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:19:56.108530    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\""
Sep 11 17:19:56.108885 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:19:56.108866    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-827rr\""
Sep 11 17:19:56.109081 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:19:56.109064    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\""
Sep 11 17:19:56.109242 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:19:56.109226    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\""
Sep 11 17:19:56.109381 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:19:56.109368    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\""
Sep 11 17:19:56.109640 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:19:56.109618    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-7jkijutllc784\""
Sep 11 17:19:56.111298 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:19:56.109840    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\""
Sep 11 17:19:56.111298 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:19:56.109961    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\""
Sep 11 17:19:56.111298 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:19:56.110971    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\""
Sep 11 17:19:56.122528 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:19:56.121610    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 11 17:19:56.207219 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:19:56.207179    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/5fd69637-9132-4804-8a78-c24a626114ac-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"5fd69637-9132-4804-8a78-c24a626114ac\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 17:19:56.207469 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:19:56.207241    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/5fd69637-9132-4804-8a78-c24a626114ac-config\") pod \"prometheus-k8s-0\" (UID: \"5fd69637-9132-4804-8a78-c24a626114ac\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 17:19:56.207469 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:19:56.207278    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5fd69637-9132-4804-8a78-c24a626114ac-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"5fd69637-9132-4804-8a78-c24a626114ac\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 17:19:56.207469 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:19:56.207311    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rcfgn\" (UniqueName: \"kubernetes.io/projected/5fd69637-9132-4804-8a78-c24a626114ac-kube-api-access-rcfgn\") pod \"prometheus-k8s-0\" (UID: \"5fd69637-9132-4804-8a78-c24a626114ac\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 17:19:56.207469 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:19:56.207347    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/5fd69637-9132-4804-8a78-c24a626114ac-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"5fd69637-9132-4804-8a78-c24a626114ac\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 17:19:56.207469 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:19:56.207372    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/5fd69637-9132-4804-8a78-c24a626114ac-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"5fd69637-9132-4804-8a78-c24a626114ac\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 17:19:56.207469 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:19:56.207395    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/5fd69637-9132-4804-8a78-c24a626114ac-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"5fd69637-9132-4804-8a78-c24a626114ac\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 17:19:56.207804 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:19:56.207466    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/5fd69637-9132-4804-8a78-c24a626114ac-config-out\") pod \"prometheus-k8s-0\" (UID: \"5fd69637-9132-4804-8a78-c24a626114ac\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 17:19:56.207804 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:19:56.207495    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/5fd69637-9132-4804-8a78-c24a626114ac-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"5fd69637-9132-4804-8a78-c24a626114ac\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 17:19:56.207804 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:19:56.207538    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/5fd69637-9132-4804-8a78-c24a626114ac-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"5fd69637-9132-4804-8a78-c24a626114ac\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 17:19:56.207804 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:19:56.207574    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/5fd69637-9132-4804-8a78-c24a626114ac-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"5fd69637-9132-4804-8a78-c24a626114ac\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 17:19:56.207804 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:19:56.207605    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/5fd69637-9132-4804-8a78-c24a626114ac-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"5fd69637-9132-4804-8a78-c24a626114ac\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 17:19:56.207804 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:19:56.207633    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5fd69637-9132-4804-8a78-c24a626114ac-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"5fd69637-9132-4804-8a78-c24a626114ac\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 17:19:56.207804 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:19:56.207667    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/5fd69637-9132-4804-8a78-c24a626114ac-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"5fd69637-9132-4804-8a78-c24a626114ac\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 17:19:56.207804 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:19:56.207685    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/5fd69637-9132-4804-8a78-c24a626114ac-web-config\") pod \"prometheus-k8s-0\" (UID: \"5fd69637-9132-4804-8a78-c24a626114ac\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 17:19:56.207804 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:19:56.207761    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/5fd69637-9132-4804-8a78-c24a626114ac-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"5fd69637-9132-4804-8a78-c24a626114ac\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 17:19:56.208136 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:19:56.207811    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/5fd69637-9132-4804-8a78-c24a626114ac-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"5fd69637-9132-4804-8a78-c24a626114ac\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 17:19:56.208136 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:19:56.207834    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/5fd69637-9132-4804-8a78-c24a626114ac-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"5fd69637-9132-4804-8a78-c24a626114ac\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 17:19:56.208136 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:19:56.207850    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5fd69637-9132-4804-8a78-c24a626114ac-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"5fd69637-9132-4804-8a78-c24a626114ac\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 17:19:56.208136 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:19:56.207867    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/5fd69637-9132-4804-8a78-c24a626114ac-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"5fd69637-9132-4804-8a78-c24a626114ac\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 17:19:56.311684 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:19:56.309429    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/5fd69637-9132-4804-8a78-c24a626114ac-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"5fd69637-9132-4804-8a78-c24a626114ac\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 17:19:56.311684 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:19:56.309497    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/5fd69637-9132-4804-8a78-c24a626114ac-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"5fd69637-9132-4804-8a78-c24a626114ac\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 17:19:56.311684 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:19:56.309556    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/5fd69637-9132-4804-8a78-c24a626114ac-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"5fd69637-9132-4804-8a78-c24a626114ac\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 17:19:56.311684 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:19:56.309581    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5fd69637-9132-4804-8a78-c24a626114ac-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"5fd69637-9132-4804-8a78-c24a626114ac\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 17:19:56.311684 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:19:56.309607    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/5fd69637-9132-4804-8a78-c24a626114ac-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"5fd69637-9132-4804-8a78-c24a626114ac\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 17:19:56.311684 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:19:56.309633    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/5fd69637-9132-4804-8a78-c24a626114ac-web-config\") pod \"prometheus-k8s-0\" (UID: \"5fd69637-9132-4804-8a78-c24a626114ac\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 17:19:56.311684 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:19:56.309668    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/5fd69637-9132-4804-8a78-c24a626114ac-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"5fd69637-9132-4804-8a78-c24a626114ac\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 17:19:56.311684 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:19:56.309704    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/5fd69637-9132-4804-8a78-c24a626114ac-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"5fd69637-9132-4804-8a78-c24a626114ac\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 17:19:56.311684 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:19:56.309735    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/5fd69637-9132-4804-8a78-c24a626114ac-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"5fd69637-9132-4804-8a78-c24a626114ac\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 17:19:56.311684 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:19:56.309756    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5fd69637-9132-4804-8a78-c24a626114ac-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"5fd69637-9132-4804-8a78-c24a626114ac\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 17:19:56.311684 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:19:56.309782    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/5fd69637-9132-4804-8a78-c24a626114ac-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"5fd69637-9132-4804-8a78-c24a626114ac\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 17:19:56.311684 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:19:56.309832    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/5fd69637-9132-4804-8a78-c24a626114ac-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"5fd69637-9132-4804-8a78-c24a626114ac\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 17:19:56.311684 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:19:56.309866    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/5fd69637-9132-4804-8a78-c24a626114ac-config\") pod \"prometheus-k8s-0\" (UID: \"5fd69637-9132-4804-8a78-c24a626114ac\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 17:19:56.311684 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:19:56.309901    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5fd69637-9132-4804-8a78-c24a626114ac-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"5fd69637-9132-4804-8a78-c24a626114ac\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 17:19:56.311684 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:19:56.309929    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rcfgn\" (UniqueName: \"kubernetes.io/projected/5fd69637-9132-4804-8a78-c24a626114ac-kube-api-access-rcfgn\") pod \"prometheus-k8s-0\" (UID: \"5fd69637-9132-4804-8a78-c24a626114ac\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 17:19:56.311684 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:19:56.309973    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/5fd69637-9132-4804-8a78-c24a626114ac-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"5fd69637-9132-4804-8a78-c24a626114ac\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 17:19:56.311684 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:19:56.310001    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/5fd69637-9132-4804-8a78-c24a626114ac-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"5fd69637-9132-4804-8a78-c24a626114ac\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 17:19:56.314343 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:19:56.310024    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/5fd69637-9132-4804-8a78-c24a626114ac-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"5fd69637-9132-4804-8a78-c24a626114ac\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 17:19:56.314343 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:19:56.310081    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/5fd69637-9132-4804-8a78-c24a626114ac-config-out\") pod \"prometheus-k8s-0\" (UID: \"5fd69637-9132-4804-8a78-c24a626114ac\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 17:19:56.314343 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:19:56.310105    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/5fd69637-9132-4804-8a78-c24a626114ac-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"5fd69637-9132-4804-8a78-c24a626114ac\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 17:19:56.314343 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:19:56.312279    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/5fd69637-9132-4804-8a78-c24a626114ac-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"5fd69637-9132-4804-8a78-c24a626114ac\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 17:19:56.314343 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:19:56.313104    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5fd69637-9132-4804-8a78-c24a626114ac-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"5fd69637-9132-4804-8a78-c24a626114ac\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 17:19:56.314343 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:19:56.313379    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5fd69637-9132-4804-8a78-c24a626114ac-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"5fd69637-9132-4804-8a78-c24a626114ac\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 17:19:56.314343 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:19:56.313814    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/5fd69637-9132-4804-8a78-c24a626114ac-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"5fd69637-9132-4804-8a78-c24a626114ac\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 17:19:56.314343 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:19:56.314044    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/5fd69637-9132-4804-8a78-c24a626114ac-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"5fd69637-9132-4804-8a78-c24a626114ac\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 17:19:56.315039 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:19:56.314854    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/5fd69637-9132-4804-8a78-c24a626114ac-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"5fd69637-9132-4804-8a78-c24a626114ac\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 17:19:56.315039 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:19:56.314987    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/5fd69637-9132-4804-8a78-c24a626114ac-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"5fd69637-9132-4804-8a78-c24a626114ac\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 17:19:56.316172 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:19:56.315182    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/5fd69637-9132-4804-8a78-c24a626114ac-web-config\") pod \"prometheus-k8s-0\" (UID: \"5fd69637-9132-4804-8a78-c24a626114ac\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 17:19:56.316172 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:19:56.315230    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/5fd69637-9132-4804-8a78-c24a626114ac-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"5fd69637-9132-4804-8a78-c24a626114ac\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 17:19:56.316321 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:19:56.316166    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5fd69637-9132-4804-8a78-c24a626114ac-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"5fd69637-9132-4804-8a78-c24a626114ac\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 17:19:56.316790 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:19:56.316743    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/5fd69637-9132-4804-8a78-c24a626114ac-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"5fd69637-9132-4804-8a78-c24a626114ac\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 17:19:56.317402 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:19:56.317322    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/5fd69637-9132-4804-8a78-c24a626114ac-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"5fd69637-9132-4804-8a78-c24a626114ac\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 17:19:56.317783 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:19:56.317759    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/5fd69637-9132-4804-8a78-c24a626114ac-config\") pod \"prometheus-k8s-0\" (UID: \"5fd69637-9132-4804-8a78-c24a626114ac\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 17:19:56.318822 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:19:56.318354    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/5fd69637-9132-4804-8a78-c24a626114ac-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"5fd69637-9132-4804-8a78-c24a626114ac\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 17:19:56.319043 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:19:56.319022    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/5fd69637-9132-4804-8a78-c24a626114ac-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"5fd69637-9132-4804-8a78-c24a626114ac\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 17:19:56.319908 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:19:56.319868    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/5fd69637-9132-4804-8a78-c24a626114ac-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"5fd69637-9132-4804-8a78-c24a626114ac\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 17:19:56.320474 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:19:56.320437    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/5fd69637-9132-4804-8a78-c24a626114ac-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"5fd69637-9132-4804-8a78-c24a626114ac\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 17:19:56.321198 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:19:56.321180    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rcfgn\" (UniqueName: \"kubernetes.io/projected/5fd69637-9132-4804-8a78-c24a626114ac-kube-api-access-rcfgn\") pod \"prometheus-k8s-0\" (UID: \"5fd69637-9132-4804-8a78-c24a626114ac\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 17:19:56.322573 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:19:56.322531    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/5fd69637-9132-4804-8a78-c24a626114ac-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"5fd69637-9132-4804-8a78-c24a626114ac\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 17:19:56.324345 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:19:56.324301    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/5fd69637-9132-4804-8a78-c24a626114ac-config-out\") pod \"prometheus-k8s-0\" (UID: \"5fd69637-9132-4804-8a78-c24a626114ac\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 17:19:56.411307 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:19:56.411218    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 17:19:56.808735 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:19:56.808630    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-5d85778b9d-cqvpv"]
Sep 11 17:19:56.809107 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:19:56.809052    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-7874849c67-ftnmh"]
Sep 11 17:19:56.811597 ip-10-0-141-139 kubenswrapper[2577]: W0911 17:19:56.811570    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod071e2825_99c5_4ccf_9443_e0cdf8f52df3.slice/crio-f9c821e73349e00a32a8d8354f52969848b83f3d36f37c2ec9f17d1e8cdc8f53 WatchSource:0}: Error finding container f9c821e73349e00a32a8d8354f52969848b83f3d36f37c2ec9f17d1e8cdc8f53: Status 404 returned error can't find the container with id f9c821e73349e00a32a8d8354f52969848b83f3d36f37c2ec9f17d1e8cdc8f53
Sep 11 17:19:56.818816 ip-10-0-141-139 kubenswrapper[2577]: W0911 17:19:56.817551    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc2dc1ae2_696a_49f6_85ae_071961ef87fc.slice/crio-bf71f9b6c4c7ea9969e6dd2415f2b232171852fa564c3bc82c170bcdfc4c674b WatchSource:0}: Error finding container bf71f9b6c4c7ea9969e6dd2415f2b232171852fa564c3bc82c170bcdfc4c674b: Status 404 returned error can't find the container with id bf71f9b6c4c7ea9969e6dd2415f2b232171852fa564c3bc82c170bcdfc4c674b
Sep 11 17:19:56.833018 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:19:56.832924    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-76875c59d5-d247c"]
Sep 11 17:19:56.842167 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:19:56.842142    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 11 17:19:57.357488 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:19:57.357306    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-7775c558d6-95wpp" event={"ID":"44e112d2-e31f-4dfc-abc3-17f9c80ae9b4","Type":"ContainerStarted","Data":"b016ef8ecc5d651f9dd73c86c41e1d6948bb2ab32bfcba661e8c1c7996d10c8a"}
Sep 11 17:19:57.357488 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:19:57.357389    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-7775c558d6-95wpp" event={"ID":"44e112d2-e31f-4dfc-abc3-17f9c80ae9b4","Type":"ContainerStarted","Data":"71d9144aba07ce3a4aa08625edc843b4876b985489d5079953a379f9cc769ca1"}
Sep 11 17:19:57.357488 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:19:57.357434    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-7775c558d6-95wpp" event={"ID":"44e112d2-e31f-4dfc-abc3-17f9c80ae9b4","Type":"ContainerStarted","Data":"e48cd99d810956aee2be394c310907a15d4c517752266059ed9686049db64e81"}
Sep 11 17:19:57.360073 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:19:57.360039    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-fwdgr" event={"ID":"f1b075b6-9bae-41d6-b0c0-1c8b81c3207a","Type":"ContainerStarted","Data":"ea55fdf4ab75fde8948453b5dfdcf615b16b67b2e0b0a003783c6f1f4c267435"}
Sep 11 17:19:57.362767 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:19:57.362743    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-mthpt" event={"ID":"df65734a-e4e6-4b4e-9114-35af7b635044","Type":"ContainerStarted","Data":"40de3b0d08e6ed5db493452535632cda8e8bac293ff213f2c21b27bc621f1dee"}
Sep 11 17:19:57.362899 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:19:57.362788    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-mthpt" event={"ID":"df65734a-e4e6-4b4e-9114-35af7b635044","Type":"ContainerStarted","Data":"3986c4a0e508b84134606692642a9251fdbaa3330f42cdeeb080674bbb688d25"}
Sep 11 17:19:57.364097 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:19:57.364073    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-76875c59d5-d247c" event={"ID":"addf52ab-b262-4b3a-be3d-5ad1e772090f","Type":"ContainerStarted","Data":"361fb708f550dd282ec8fe53774ced1e7483211f31f39f994c17b0663e79464a"}
Sep 11 17:19:57.365997 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:19:57.365976    2577 generic.go:356] "Generic (PLEG): container finished" podID="4c942d42-091f-4c68-8faa-f6276bd26c15" containerID="706f26f8e9e50b678349bb0d141ec51a0d860f198c61f17ef4d33144981cf609" exitCode=0
Sep 11 17:19:57.366098 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:19:57.366042    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"4c942d42-091f-4c68-8faa-f6276bd26c15","Type":"ContainerDied","Data":"706f26f8e9e50b678349bb0d141ec51a0d860f198c61f17ef4d33144981cf609"}
Sep 11 17:19:57.367555 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:19:57.367533    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-5d85778b9d-cqvpv" event={"ID":"071e2825-99c5-4ccf-9443-e0cdf8f52df3","Type":"ContainerStarted","Data":"f9c821e73349e00a32a8d8354f52969848b83f3d36f37c2ec9f17d1e8cdc8f53"}
Sep 11 17:19:57.369312 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:19:57.369238    2577 generic.go:356] "Generic (PLEG): container finished" podID="5fd69637-9132-4804-8a78-c24a626114ac" containerID="1a6761dccea5c2043447fe170447e2694f286770a6b87a3291619d817bc0333e" exitCode=0
Sep 11 17:19:57.369365 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:19:57.369333    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"5fd69637-9132-4804-8a78-c24a626114ac","Type":"ContainerDied","Data":"1a6761dccea5c2043447fe170447e2694f286770a6b87a3291619d817bc0333e"}
Sep 11 17:19:57.369414 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:19:57.369391    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"5fd69637-9132-4804-8a78-c24a626114ac","Type":"ContainerStarted","Data":"3facfe5208fdc61e6436842dfdee7e771f3cff11368ffde44922965ff1dd6be3"}
Sep 11 17:19:57.371417 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:19:57.371381    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7874849c67-ftnmh" event={"ID":"c2dc1ae2-696a-49f6-85ae-071961ef87fc","Type":"ContainerStarted","Data":"edcd0f3e37f3fc1fb5dfafbfc38c28e84e03196dbdfed1e9f3fe7e0f5ad240f6"}
Sep 11 17:19:57.371525 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:19:57.371419    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7874849c67-ftnmh" event={"ID":"c2dc1ae2-696a-49f6-85ae-071961ef87fc","Type":"ContainerStarted","Data":"bf71f9b6c4c7ea9969e6dd2415f2b232171852fa564c3bc82c170bcdfc4c674b"}
Sep 11 17:19:57.382480 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:19:57.382430    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-fwdgr" podStartSLOduration=5.389844491 podStartE2EDuration="8.382416269s" totalImagesPullingTime="2.992571778s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 17:19:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 17:19:57.381283758 +0000 UTC m=+74.954503909" watchObservedRunningTime="2026-09-11 17:19:57.382416269 +0000 UTC m=+74.955636380"
Sep 11 17:19:57.470049 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:19:57.469979    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-7874849c67-ftnmh" podStartSLOduration=3.469959915 podStartE2EDuration="3.469959915s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 17:19:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 17:19:57.468468884 +0000 UTC m=+75.041689011" watchObservedRunningTime="2026-09-11 17:19:57.469959915 +0000 UTC m=+75.043180027"
Sep 11 17:19:57.500971 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:19:57.500906    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-mthpt" podStartSLOduration=7.214156695 podStartE2EDuration="8.500882276s" totalImagesPullingTime="1.19541125s" totalInitContainerRuntime="91.314331ms" isStatefulPod=true podCreationTimestamp="2026-09-11 17:19:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 17:19:57.498164723 +0000 UTC m=+75.071384837" watchObservedRunningTime="2026-09-11 17:19:57.500882276 +0000 UTC m=+75.074102393"
Sep 11 17:19:58.380206 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:19:58.379700    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-7f9b5d9fd4-wnwpt" event={"ID":"7b9f4b24-4fe6-4fc8-8917-08f335a75844","Type":"ContainerStarted","Data":"b2c0315b7e4c2bf6810d25ba1d0c9be722244ef189f918d2526ef070fcc9cc7f"}
Sep 11 17:19:58.401748 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:19:58.401688    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/metrics-server-7f9b5d9fd4-wnwpt" podStartSLOduration=2.374278695 podStartE2EDuration="5.401666545s" totalImagesPullingTime="3.02738785s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 17:19:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 17:19:58.400711561 +0000 UTC m=+75.973931673" watchObservedRunningTime="2026-09-11 17:19:58.401666545 +0000 UTC m=+75.974886659"
Sep 11 17:19:58.870481 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:19:58.870393    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-57bf98d9fc-nrdtq"
Sep 11 17:20:00.391779 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:20:00.391737    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-7775c558d6-95wpp" event={"ID":"44e112d2-e31f-4dfc-abc3-17f9c80ae9b4","Type":"ContainerStarted","Data":"8cbbe93f6644eb1e7c939c8a3909ff3ed4f53338566979f4b09008b8f54714cd"}
Sep 11 17:20:00.392191 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:20:00.391866    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-7775c558d6-95wpp" event={"ID":"44e112d2-e31f-4dfc-abc3-17f9c80ae9b4","Type":"ContainerStarted","Data":"0f4a86174c09970a861c6956c71bedcab46ed084e6972f7f8b8191963ae7a6b1"}
Sep 11 17:20:00.392191 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:20:00.391918    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/thanos-querier-7775c558d6-95wpp"
Sep 11 17:20:00.392191 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:20:00.391961    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-7775c558d6-95wpp" event={"ID":"44e112d2-e31f-4dfc-abc3-17f9c80ae9b4","Type":"ContainerStarted","Data":"55b396cfcbe4ac1df302d1f4cd08dfb4c81778a31d585f17ad7755b1edb5a186"}
Sep 11 17:20:00.394015 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:20:00.393979    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-76875c59d5-d247c" event={"ID":"addf52ab-b262-4b3a-be3d-5ad1e772090f","Type":"ContainerStarted","Data":"f994167ecfadf61576ed339634dad1663baccea8e4e94bfa757cbbce7f05f956"}
Sep 11 17:20:00.394151 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:20:00.394027    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-76875c59d5-d247c" event={"ID":"addf52ab-b262-4b3a-be3d-5ad1e772090f","Type":"ContainerStarted","Data":"ee45c55b73c2d02191e61e0551b6d656630c095febcd60d17bf7dd7bf7f98076"}
Sep 11 17:20:00.394151 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:20:00.394057    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-76875c59d5-d247c" event={"ID":"addf52ab-b262-4b3a-be3d-5ad1e772090f","Type":"ContainerStarted","Data":"3973199e6f34863adb5180f520580af3d3208966fa3b0128423b9da5f32dbf0a"}
Sep 11 17:20:00.397317 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:20:00.397290    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"4c942d42-091f-4c68-8faa-f6276bd26c15","Type":"ContainerStarted","Data":"2a24c94179b5571ff0b18b99cde019ee8ca1d8efad41302ae3b2d8a6261c2e91"}
Sep 11 17:20:00.397428 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:20:00.397359    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"4c942d42-091f-4c68-8faa-f6276bd26c15","Type":"ContainerStarted","Data":"b1f982011d7acf52740c0f356f0e3a683b4ea5cd14e5f448bf5a894760045ae3"}
Sep 11 17:20:00.397428 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:20:00.397402    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"4c942d42-091f-4c68-8faa-f6276bd26c15","Type":"ContainerStarted","Data":"dfa205ad9943339ff6910fb2a0c2264eb3a5e31ef821c80704d257e26679e2c5"}
Sep 11 17:20:00.397539 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:20:00.397443    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"4c942d42-091f-4c68-8faa-f6276bd26c15","Type":"ContainerStarted","Data":"690666babfe741372f4d6e8b575c8bfd942f8fb501d16771706ef198dc4c302d"}
Sep 11 17:20:00.397539 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:20:00.397472    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"4c942d42-091f-4c68-8faa-f6276bd26c15","Type":"ContainerStarted","Data":"96e5540b18bfe38148c3a743a19f590d6495d387fff0d72121d52aece670802d"}
Sep 11 17:20:00.397539 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:20:00.397495    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"4c942d42-091f-4c68-8faa-f6276bd26c15","Type":"ContainerStarted","Data":"4adc38e198f4130856a225a3b2b4e8554d30cc92b333b82133b06156ead5d81b"}
Sep 11 17:20:00.398890 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:20:00.398870    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-5d85778b9d-cqvpv" event={"ID":"071e2825-99c5-4ccf-9443-e0cdf8f52df3","Type":"ContainerStarted","Data":"48df1af8d0815f290918ce26712d4e26d886154ff3ac8a28514b8702d1c1f428"}
Sep 11 17:20:00.399043 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:20:00.399027    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/monitoring-plugin-5d85778b9d-cqvpv"
Sep 11 17:20:00.404208 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:20:00.404187    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/monitoring-plugin-5d85778b9d-cqvpv"
Sep 11 17:20:00.427102 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:20:00.427054    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/thanos-querier-7775c558d6-95wpp" podStartSLOduration=2.431223121 podStartE2EDuration="8.427040024s" totalImagesPullingTime="5.995816903s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 17:19:52 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 17:20:00.424360723 +0000 UTC m=+77.997581019" watchObservedRunningTime="2026-09-11 17:20:00.427040024 +0000 UTC m=+78.000260135"
Sep 11 17:20:00.444488 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:20:00.444437    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/monitoring-plugin-5d85778b9d-cqvpv" podStartSLOduration=3.452793206 podStartE2EDuration="6.444422677s" totalImagesPullingTime="2.991629471s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 17:19:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 17:20:00.442747032 +0000 UTC m=+78.015967144" watchObservedRunningTime="2026-09-11 17:20:00.444422677 +0000 UTC m=+78.017642789"
Sep 11 17:20:00.475174 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:20:00.475114    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=4.88728559 podStartE2EDuration="10.475090671s" totalImagesPullingTime="5.562766727s" totalInitContainerRuntime="25.038354ms" isStatefulPod=false podCreationTimestamp="2026-09-11 17:19:50 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 17:20:00.473889536 +0000 UTC m=+78.047109648" watchObservedRunningTime="2026-09-11 17:20:00.475090671 +0000 UTC m=+78.048310784"
Sep 11 17:20:00.504682 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:20:00.504615    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/telemeter-client-76875c59d5-d247c" podStartSLOduration=3.533616714 podStartE2EDuration="6.504590895s" totalImagesPullingTime="2.970974181s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 17:19:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 17:20:00.50364443 +0000 UTC m=+78.076864544" watchObservedRunningTime="2026-09-11 17:20:00.504590895 +0000 UTC m=+78.077811007"
Sep 11 17:20:00.631135 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:20:00.631054    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-7874849c67-ftnmh"]
Sep 11 17:20:01.414581 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:20:01.414552    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/thanos-querier-7775c558d6-95wpp"
Sep 11 17:20:03.420797 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:20:03.420726    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"5fd69637-9132-4804-8a78-c24a626114ac","Type":"ContainerStarted","Data":"abb93082e2cd607833152e12ebb5876b1f5c585dcde9887119928424b818a4f4"}
Sep 11 17:20:03.421402 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:20:03.421335    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"5fd69637-9132-4804-8a78-c24a626114ac","Type":"ContainerStarted","Data":"c8e220cba977043454c3a697b3d5103a838307813ec5e12dbf9d5a0dca863f00"}
Sep 11 17:20:04.428550 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:20:04.428497    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"5fd69637-9132-4804-8a78-c24a626114ac","Type":"ContainerStarted","Data":"8bcb4a57ee489421690089d725dbb3151aa118dd33caa6c76f908d98ea27983b"}
Sep 11 17:20:04.429036 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:20:04.428590    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"5fd69637-9132-4804-8a78-c24a626114ac","Type":"ContainerStarted","Data":"f4527bcc824da4ce1f425452b2656cd58345ded9ffdfc095445d688c156713d8"}
Sep 11 17:20:04.429036 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:20:04.428635    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"5fd69637-9132-4804-8a78-c24a626114ac","Type":"ContainerStarted","Data":"eba339879615deda7f50d8d1dd1ad1077d4ea017f76db24e87fb5a36968f79bb"}
Sep 11 17:20:04.429036 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:20:04.428676    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"5fd69637-9132-4804-8a78-c24a626114ac","Type":"ContainerStarted","Data":"4461e9c87e7a5a370709ae9cbd84d391d356580798d90b66d29dc05f917568e5"}
Sep 11 17:20:04.471989 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:20:04.471939    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=2.6099771 podStartE2EDuration="8.471919778s" totalImagesPullingTime="5.839895316s" totalInitContainerRuntime="22.047362ms" isStatefulPod=false podCreationTimestamp="2026-09-11 17:19:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 17:20:04.469189149 +0000 UTC m=+82.042409261" watchObservedRunningTime="2026-09-11 17:20:04.471919778 +0000 UTC m=+82.045139894"
Sep 11 17:20:05.222974 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:20:05.222940    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-7874849c67-ftnmh"
Sep 11 17:20:06.413849 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:20:06.413817    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 17:20:11.534761 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:20:11.534718    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-5c74f887d6-ttmcp"]
Sep 11 17:20:14.307368 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:20:14.307328    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/metrics-server-7f9b5d9fd4-wnwpt"
Sep 11 17:20:14.307806 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:20:14.307417    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/metrics-server-7f9b5d9fd4-wnwpt"
Sep 11 17:20:19.863345 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:20:19.863279    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-57bf98d9fc-nrdtq" podUID="85583e11-cdb9-4c56-9aab-aa6a53641fda" containerName="console" containerID="cri-o://7041f05901f1605ca2cfee51fdd897d5ae0124e0248fd3974ea8ddf8439362ec" gracePeriod=15
Sep 11 17:20:20.107417 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:20:20.107390    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-57bf98d9fc-nrdtq"
Sep 11 17:20:20.249944 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:20:20.249903    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/85583e11-cdb9-4c56-9aab-aa6a53641fda-console-oauth-config\") pod \"85583e11-cdb9-4c56-9aab-aa6a53641fda\" (UID: \"85583e11-cdb9-4c56-9aab-aa6a53641fda\") "
Sep 11 17:20:20.250121 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:20:20.249965    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/85583e11-cdb9-4c56-9aab-aa6a53641fda-oauth-serving-cert\") pod \"85583e11-cdb9-4c56-9aab-aa6a53641fda\" (UID: \"85583e11-cdb9-4c56-9aab-aa6a53641fda\") "
Sep 11 17:20:20.250121 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:20:20.250005    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/85583e11-cdb9-4c56-9aab-aa6a53641fda-trusted-ca-bundle\") pod \"85583e11-cdb9-4c56-9aab-aa6a53641fda\" (UID: \"85583e11-cdb9-4c56-9aab-aa6a53641fda\") "
Sep 11 17:20:20.250121 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:20:20.250075    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/85583e11-cdb9-4c56-9aab-aa6a53641fda-console-serving-cert\") pod \"85583e11-cdb9-4c56-9aab-aa6a53641fda\" (UID: \"85583e11-cdb9-4c56-9aab-aa6a53641fda\") "
Sep 11 17:20:20.250291 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:20:20.250138    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/85583e11-cdb9-4c56-9aab-aa6a53641fda-console-config\") pod \"85583e11-cdb9-4c56-9aab-aa6a53641fda\" (UID: \"85583e11-cdb9-4c56-9aab-aa6a53641fda\") "
Sep 11 17:20:20.250291 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:20:20.250169    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-t9gsg\" (UniqueName: \"kubernetes.io/projected/85583e11-cdb9-4c56-9aab-aa6a53641fda-kube-api-access-t9gsg\") pod \"85583e11-cdb9-4c56-9aab-aa6a53641fda\" (UID: \"85583e11-cdb9-4c56-9aab-aa6a53641fda\") "
Sep 11 17:20:20.250291 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:20:20.250216    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/85583e11-cdb9-4c56-9aab-aa6a53641fda-service-ca\") pod \"85583e11-cdb9-4c56-9aab-aa6a53641fda\" (UID: \"85583e11-cdb9-4c56-9aab-aa6a53641fda\") "
Sep 11 17:20:20.250745 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:20:20.250471    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/85583e11-cdb9-4c56-9aab-aa6a53641fda-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "85583e11-cdb9-4c56-9aab-aa6a53641fda" (UID: "85583e11-cdb9-4c56-9aab-aa6a53641fda"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 17:20:20.250745 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:20:20.250637    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/85583e11-cdb9-4c56-9aab-aa6a53641fda-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "85583e11-cdb9-4c56-9aab-aa6a53641fda" (UID: "85583e11-cdb9-4c56-9aab-aa6a53641fda"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 17:20:20.250907 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:20:20.250631    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/85583e11-cdb9-4c56-9aab-aa6a53641fda-console-config" (OuterVolumeSpecName: "console-config") pod "85583e11-cdb9-4c56-9aab-aa6a53641fda" (UID: "85583e11-cdb9-4c56-9aab-aa6a53641fda"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 17:20:20.251005 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:20:20.250983    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/85583e11-cdb9-4c56-9aab-aa6a53641fda-service-ca" (OuterVolumeSpecName: "service-ca") pod "85583e11-cdb9-4c56-9aab-aa6a53641fda" (UID: "85583e11-cdb9-4c56-9aab-aa6a53641fda"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 17:20:20.252600 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:20:20.252578    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/85583e11-cdb9-4c56-9aab-aa6a53641fda-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "85583e11-cdb9-4c56-9aab-aa6a53641fda" (UID: "85583e11-cdb9-4c56-9aab-aa6a53641fda"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 17:20:20.252704 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:20:20.252684    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/85583e11-cdb9-4c56-9aab-aa6a53641fda-kube-api-access-t9gsg" (OuterVolumeSpecName: "kube-api-access-t9gsg") pod "85583e11-cdb9-4c56-9aab-aa6a53641fda" (UID: "85583e11-cdb9-4c56-9aab-aa6a53641fda"). InnerVolumeSpecName "kube-api-access-t9gsg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 17:20:20.252763 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:20:20.252705    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/85583e11-cdb9-4c56-9aab-aa6a53641fda-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "85583e11-cdb9-4c56-9aab-aa6a53641fda" (UID: "85583e11-cdb9-4c56-9aab-aa6a53641fda"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 17:20:20.350964 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:20:20.350919    2577 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/85583e11-cdb9-4c56-9aab-aa6a53641fda-console-serving-cert\") on node \"ip-10-0-141-139.ec2.internal\" DevicePath \"\""
Sep 11 17:20:20.350964 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:20:20.350953    2577 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/85583e11-cdb9-4c56-9aab-aa6a53641fda-console-config\") on node \"ip-10-0-141-139.ec2.internal\" DevicePath \"\""
Sep 11 17:20:20.350964 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:20:20.350963    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t9gsg\" (UniqueName: \"kubernetes.io/projected/85583e11-cdb9-4c56-9aab-aa6a53641fda-kube-api-access-t9gsg\") on node \"ip-10-0-141-139.ec2.internal\" DevicePath \"\""
Sep 11 17:20:20.350964 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:20:20.350971    2577 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/85583e11-cdb9-4c56-9aab-aa6a53641fda-service-ca\") on node \"ip-10-0-141-139.ec2.internal\" DevicePath \"\""
Sep 11 17:20:20.351238 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:20:20.350980    2577 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/85583e11-cdb9-4c56-9aab-aa6a53641fda-console-oauth-config\") on node \"ip-10-0-141-139.ec2.internal\" DevicePath \"\""
Sep 11 17:20:20.351238 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:20:20.350989    2577 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/85583e11-cdb9-4c56-9aab-aa6a53641fda-oauth-serving-cert\") on node \"ip-10-0-141-139.ec2.internal\" DevicePath \"\""
Sep 11 17:20:20.351238 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:20:20.350998    2577 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/85583e11-cdb9-4c56-9aab-aa6a53641fda-trusted-ca-bundle\") on node \"ip-10-0-141-139.ec2.internal\" DevicePath \"\""
Sep 11 17:20:20.503307 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:20:20.503213    2577 generic.go:356] "Generic (PLEG): container finished" podID="85583e11-cdb9-4c56-9aab-aa6a53641fda" containerID="7041f05901f1605ca2cfee51fdd897d5ae0124e0248fd3974ea8ddf8439362ec" exitCode=0
Sep 11 17:20:20.503307 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:20:20.503263    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-57bf98d9fc-nrdtq"
Sep 11 17:20:20.503489 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:20:20.503298    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-57bf98d9fc-nrdtq" event={"ID":"85583e11-cdb9-4c56-9aab-aa6a53641fda","Type":"ContainerDied","Data":"7041f05901f1605ca2cfee51fdd897d5ae0124e0248fd3974ea8ddf8439362ec"}
Sep 11 17:20:20.503489 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:20:20.503351    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-57bf98d9fc-nrdtq" event={"ID":"85583e11-cdb9-4c56-9aab-aa6a53641fda","Type":"ContainerDied","Data":"17645d3003be6f66d713a0e978aeaceadf23a05e21b9221d7e2fe27e7f02f615"}
Sep 11 17:20:20.503489 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:20:20.503371    2577 scope.go:117] "RemoveContainer" containerID="7041f05901f1605ca2cfee51fdd897d5ae0124e0248fd3974ea8ddf8439362ec"
Sep 11 17:20:20.512637 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:20:20.512618    2577 scope.go:117] "RemoveContainer" containerID="7041f05901f1605ca2cfee51fdd897d5ae0124e0248fd3974ea8ddf8439362ec"
Sep 11 17:20:20.512958 ip-10-0-141-139 kubenswrapper[2577]: E0911 17:20:20.512940    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7041f05901f1605ca2cfee51fdd897d5ae0124e0248fd3974ea8ddf8439362ec\": container with ID starting with 7041f05901f1605ca2cfee51fdd897d5ae0124e0248fd3974ea8ddf8439362ec not found: ID does not exist" containerID="7041f05901f1605ca2cfee51fdd897d5ae0124e0248fd3974ea8ddf8439362ec"
Sep 11 17:20:20.513002 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:20:20.512970    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7041f05901f1605ca2cfee51fdd897d5ae0124e0248fd3974ea8ddf8439362ec"} err="failed to get container status \"7041f05901f1605ca2cfee51fdd897d5ae0124e0248fd3974ea8ddf8439362ec\": rpc error: code = NotFound desc = could not find container \"7041f05901f1605ca2cfee51fdd897d5ae0124e0248fd3974ea8ddf8439362ec\": container with ID starting with 7041f05901f1605ca2cfee51fdd897d5ae0124e0248fd3974ea8ddf8439362ec not found: ID does not exist"
Sep 11 17:20:20.531104 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:20:20.531071    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-57bf98d9fc-nrdtq"]
Sep 11 17:20:20.535714 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:20:20.535685    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-57bf98d9fc-nrdtq"]
Sep 11 17:20:20.925372 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:20:20.925298    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="85583e11-cdb9-4c56-9aab-aa6a53641fda" path="/var/lib/kubelet/pods/85583e11-cdb9-4c56-9aab-aa6a53641fda/volumes"
Sep 11 17:20:25.346269 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:20:25.346238    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-xfn47"
Sep 11 17:20:25.653877 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:20:25.653788    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-7874849c67-ftnmh" podUID="c2dc1ae2-696a-49f6-85ae-071961ef87fc" containerName="console" containerID="cri-o://edcd0f3e37f3fc1fb5dfafbfc38c28e84e03196dbdfed1e9f3fe7e0f5ad240f6" gracePeriod=15
Sep 11 17:20:25.931839 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:20:25.931755    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7874849c67-ftnmh"
Sep 11 17:20:26.002470 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:20:26.002436    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c2dc1ae2-696a-49f6-85ae-071961ef87fc-trusted-ca-bundle\") pod \"c2dc1ae2-696a-49f6-85ae-071961ef87fc\" (UID: \"c2dc1ae2-696a-49f6-85ae-071961ef87fc\") "
Sep 11 17:20:26.002684 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:20:26.002486    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/c2dc1ae2-696a-49f6-85ae-071961ef87fc-console-serving-cert\") pod \"c2dc1ae2-696a-49f6-85ae-071961ef87fc\" (UID: \"c2dc1ae2-696a-49f6-85ae-071961ef87fc\") "
Sep 11 17:20:26.002684 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:20:26.002540    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/c2dc1ae2-696a-49f6-85ae-071961ef87fc-oauth-serving-cert\") pod \"c2dc1ae2-696a-49f6-85ae-071961ef87fc\" (UID: \"c2dc1ae2-696a-49f6-85ae-071961ef87fc\") "
Sep 11 17:20:26.002684 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:20:26.002563    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/c2dc1ae2-696a-49f6-85ae-071961ef87fc-console-oauth-config\") pod \"c2dc1ae2-696a-49f6-85ae-071961ef87fc\" (UID: \"c2dc1ae2-696a-49f6-85ae-071961ef87fc\") "
Sep 11 17:20:26.002684 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:20:26.002589    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/c2dc1ae2-696a-49f6-85ae-071961ef87fc-service-ca\") pod \"c2dc1ae2-696a-49f6-85ae-071961ef87fc\" (UID: \"c2dc1ae2-696a-49f6-85ae-071961ef87fc\") "
Sep 11 17:20:26.002684 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:20:26.002621    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-svgxp\" (UniqueName: \"kubernetes.io/projected/c2dc1ae2-696a-49f6-85ae-071961ef87fc-kube-api-access-svgxp\") pod \"c2dc1ae2-696a-49f6-85ae-071961ef87fc\" (UID: \"c2dc1ae2-696a-49f6-85ae-071961ef87fc\") "
Sep 11 17:20:26.002684 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:20:26.002655    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/c2dc1ae2-696a-49f6-85ae-071961ef87fc-console-config\") pod \"c2dc1ae2-696a-49f6-85ae-071961ef87fc\" (UID: \"c2dc1ae2-696a-49f6-85ae-071961ef87fc\") "
Sep 11 17:20:26.003064 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:20:26.003032    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c2dc1ae2-696a-49f6-85ae-071961ef87fc-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "c2dc1ae2-696a-49f6-85ae-071961ef87fc" (UID: "c2dc1ae2-696a-49f6-85ae-071961ef87fc"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 17:20:26.003143 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:20:26.003105    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c2dc1ae2-696a-49f6-85ae-071961ef87fc-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "c2dc1ae2-696a-49f6-85ae-071961ef87fc" (UID: "c2dc1ae2-696a-49f6-85ae-071961ef87fc"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 17:20:26.003143 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:20:26.003118    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c2dc1ae2-696a-49f6-85ae-071961ef87fc-service-ca" (OuterVolumeSpecName: "service-ca") pod "c2dc1ae2-696a-49f6-85ae-071961ef87fc" (UID: "c2dc1ae2-696a-49f6-85ae-071961ef87fc"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 17:20:26.003242 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:20:26.003147    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c2dc1ae2-696a-49f6-85ae-071961ef87fc-console-config" (OuterVolumeSpecName: "console-config") pod "c2dc1ae2-696a-49f6-85ae-071961ef87fc" (UID: "c2dc1ae2-696a-49f6-85ae-071961ef87fc"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 17:20:26.004969 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:20:26.004938    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c2dc1ae2-696a-49f6-85ae-071961ef87fc-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "c2dc1ae2-696a-49f6-85ae-071961ef87fc" (UID: "c2dc1ae2-696a-49f6-85ae-071961ef87fc"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 17:20:26.005070 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:20:26.004991    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c2dc1ae2-696a-49f6-85ae-071961ef87fc-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "c2dc1ae2-696a-49f6-85ae-071961ef87fc" (UID: "c2dc1ae2-696a-49f6-85ae-071961ef87fc"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 17:20:26.005070 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:20:26.005012    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c2dc1ae2-696a-49f6-85ae-071961ef87fc-kube-api-access-svgxp" (OuterVolumeSpecName: "kube-api-access-svgxp") pod "c2dc1ae2-696a-49f6-85ae-071961ef87fc" (UID: "c2dc1ae2-696a-49f6-85ae-071961ef87fc"). InnerVolumeSpecName "kube-api-access-svgxp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 17:20:26.103493 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:20:26.103453    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-svgxp\" (UniqueName: \"kubernetes.io/projected/c2dc1ae2-696a-49f6-85ae-071961ef87fc-kube-api-access-svgxp\") on node \"ip-10-0-141-139.ec2.internal\" DevicePath \"\""
Sep 11 17:20:26.103493 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:20:26.103488    2577 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/c2dc1ae2-696a-49f6-85ae-071961ef87fc-console-config\") on node \"ip-10-0-141-139.ec2.internal\" DevicePath \"\""
Sep 11 17:20:26.103493 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:20:26.103522    2577 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c2dc1ae2-696a-49f6-85ae-071961ef87fc-trusted-ca-bundle\") on node \"ip-10-0-141-139.ec2.internal\" DevicePath \"\""
Sep 11 17:20:26.103732 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:20:26.103536    2577 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/c2dc1ae2-696a-49f6-85ae-071961ef87fc-console-serving-cert\") on node \"ip-10-0-141-139.ec2.internal\" DevicePath \"\""
Sep 11 17:20:26.103732 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:20:26.103545    2577 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/c2dc1ae2-696a-49f6-85ae-071961ef87fc-oauth-serving-cert\") on node \"ip-10-0-141-139.ec2.internal\" DevicePath \"\""
Sep 11 17:20:26.103732 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:20:26.103552    2577 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/c2dc1ae2-696a-49f6-85ae-071961ef87fc-console-oauth-config\") on node \"ip-10-0-141-139.ec2.internal\" DevicePath \"\""
Sep 11 17:20:26.103732 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:20:26.103561    2577 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/c2dc1ae2-696a-49f6-85ae-071961ef87fc-service-ca\") on node \"ip-10-0-141-139.ec2.internal\" DevicePath \"\""
Sep 11 17:20:26.531599 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:20:26.531554    2577 generic.go:356] "Generic (PLEG): container finished" podID="c2dc1ae2-696a-49f6-85ae-071961ef87fc" containerID="edcd0f3e37f3fc1fb5dfafbfc38c28e84e03196dbdfed1e9f3fe7e0f5ad240f6" exitCode=0
Sep 11 17:20:26.531599 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:20:26.531600    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7874849c67-ftnmh"
Sep 11 17:20:26.532084 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:20:26.531635    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7874849c67-ftnmh" event={"ID":"c2dc1ae2-696a-49f6-85ae-071961ef87fc","Type":"ContainerDied","Data":"edcd0f3e37f3fc1fb5dfafbfc38c28e84e03196dbdfed1e9f3fe7e0f5ad240f6"}
Sep 11 17:20:26.532084 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:20:26.531686    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7874849c67-ftnmh" event={"ID":"c2dc1ae2-696a-49f6-85ae-071961ef87fc","Type":"ContainerDied","Data":"bf71f9b6c4c7ea9969e6dd2415f2b232171852fa564c3bc82c170bcdfc4c674b"}
Sep 11 17:20:26.532084 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:20:26.531706    2577 scope.go:117] "RemoveContainer" containerID="edcd0f3e37f3fc1fb5dfafbfc38c28e84e03196dbdfed1e9f3fe7e0f5ad240f6"
Sep 11 17:20:26.540823 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:20:26.540805    2577 scope.go:117] "RemoveContainer" containerID="edcd0f3e37f3fc1fb5dfafbfc38c28e84e03196dbdfed1e9f3fe7e0f5ad240f6"
Sep 11 17:20:26.541087 ip-10-0-141-139 kubenswrapper[2577]: E0911 17:20:26.541065    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"edcd0f3e37f3fc1fb5dfafbfc38c28e84e03196dbdfed1e9f3fe7e0f5ad240f6\": container with ID starting with edcd0f3e37f3fc1fb5dfafbfc38c28e84e03196dbdfed1e9f3fe7e0f5ad240f6 not found: ID does not exist" containerID="edcd0f3e37f3fc1fb5dfafbfc38c28e84e03196dbdfed1e9f3fe7e0f5ad240f6"
Sep 11 17:20:26.541152 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:20:26.541101    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"edcd0f3e37f3fc1fb5dfafbfc38c28e84e03196dbdfed1e9f3fe7e0f5ad240f6"} err="failed to get container status \"edcd0f3e37f3fc1fb5dfafbfc38c28e84e03196dbdfed1e9f3fe7e0f5ad240f6\": rpc error: code = NotFound desc = could not find container \"edcd0f3e37f3fc1fb5dfafbfc38c28e84e03196dbdfed1e9f3fe7e0f5ad240f6\": container with ID starting with edcd0f3e37f3fc1fb5dfafbfc38c28e84e03196dbdfed1e9f3fe7e0f5ad240f6 not found: ID does not exist"
Sep 11 17:20:26.553418 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:20:26.553394    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-7874849c67-ftnmh"]
Sep 11 17:20:26.558246 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:20:26.558223    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-7874849c67-ftnmh"]
Sep 11 17:20:26.925143 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:20:26.925063    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c2dc1ae2-696a-49f6-85ae-071961ef87fc" path="/var/lib/kubelet/pods/c2dc1ae2-696a-49f6-85ae-071961ef87fc/volumes"
Sep 11 17:20:34.312147 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:20:34.312115    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/metrics-server-7f9b5d9fd4-wnwpt"
Sep 11 17:20:34.316276 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:20:34.316246    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/metrics-server-7f9b5d9fd4-wnwpt"
Sep 11 17:20:36.554804 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:20:36.554761    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-5c74f887d6-ttmcp" podUID="067ed90b-10ca-413b-9c86-3b48b474d237" containerName="console" containerID="cri-o://7613c98f189145c515d6414f488c1079ec43a3984dfc131fb51847463cabbd0a" gracePeriod=15
Sep 11 17:20:36.804215 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:20:36.804185    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5c74f887d6-ttmcp"
Sep 11 17:20:36.902150 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:20:36.902057    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/067ed90b-10ca-413b-9c86-3b48b474d237-console-serving-cert\") pod \"067ed90b-10ca-413b-9c86-3b48b474d237\" (UID: \"067ed90b-10ca-413b-9c86-3b48b474d237\") "
Sep 11 17:20:36.902150 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:20:36.902099    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/067ed90b-10ca-413b-9c86-3b48b474d237-service-ca\") pod \"067ed90b-10ca-413b-9c86-3b48b474d237\" (UID: \"067ed90b-10ca-413b-9c86-3b48b474d237\") "
Sep 11 17:20:36.902150 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:20:36.902142    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/067ed90b-10ca-413b-9c86-3b48b474d237-console-oauth-config\") pod \"067ed90b-10ca-413b-9c86-3b48b474d237\" (UID: \"067ed90b-10ca-413b-9c86-3b48b474d237\") "
Sep 11 17:20:36.902402 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:20:36.902199    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/067ed90b-10ca-413b-9c86-3b48b474d237-console-config\") pod \"067ed90b-10ca-413b-9c86-3b48b474d237\" (UID: \"067ed90b-10ca-413b-9c86-3b48b474d237\") "
Sep 11 17:20:36.902402 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:20:36.902231    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-84khb\" (UniqueName: \"kubernetes.io/projected/067ed90b-10ca-413b-9c86-3b48b474d237-kube-api-access-84khb\") pod \"067ed90b-10ca-413b-9c86-3b48b474d237\" (UID: \"067ed90b-10ca-413b-9c86-3b48b474d237\") "
Sep 11 17:20:36.902402 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:20:36.902254    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/067ed90b-10ca-413b-9c86-3b48b474d237-oauth-serving-cert\") pod \"067ed90b-10ca-413b-9c86-3b48b474d237\" (UID: \"067ed90b-10ca-413b-9c86-3b48b474d237\") "
Sep 11 17:20:36.902674 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:20:36.902643    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/067ed90b-10ca-413b-9c86-3b48b474d237-service-ca" (OuterVolumeSpecName: "service-ca") pod "067ed90b-10ca-413b-9c86-3b48b474d237" (UID: "067ed90b-10ca-413b-9c86-3b48b474d237"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 17:20:36.902754 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:20:36.902729    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/067ed90b-10ca-413b-9c86-3b48b474d237-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "067ed90b-10ca-413b-9c86-3b48b474d237" (UID: "067ed90b-10ca-413b-9c86-3b48b474d237"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 17:20:36.902811 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:20:36.902727    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/067ed90b-10ca-413b-9c86-3b48b474d237-console-config" (OuterVolumeSpecName: "console-config") pod "067ed90b-10ca-413b-9c86-3b48b474d237" (UID: "067ed90b-10ca-413b-9c86-3b48b474d237"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 17:20:36.904719 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:20:36.904695    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/067ed90b-10ca-413b-9c86-3b48b474d237-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "067ed90b-10ca-413b-9c86-3b48b474d237" (UID: "067ed90b-10ca-413b-9c86-3b48b474d237"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 17:20:36.904938 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:20:36.904907    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/067ed90b-10ca-413b-9c86-3b48b474d237-kube-api-access-84khb" (OuterVolumeSpecName: "kube-api-access-84khb") pod "067ed90b-10ca-413b-9c86-3b48b474d237" (UID: "067ed90b-10ca-413b-9c86-3b48b474d237"). InnerVolumeSpecName "kube-api-access-84khb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 17:20:36.904989 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:20:36.904969    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/067ed90b-10ca-413b-9c86-3b48b474d237-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "067ed90b-10ca-413b-9c86-3b48b474d237" (UID: "067ed90b-10ca-413b-9c86-3b48b474d237"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 17:20:37.003836 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:20:37.003793    2577 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/067ed90b-10ca-413b-9c86-3b48b474d237-console-serving-cert\") on node \"ip-10-0-141-139.ec2.internal\" DevicePath \"\""
Sep 11 17:20:37.003836 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:20:37.003826    2577 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/067ed90b-10ca-413b-9c86-3b48b474d237-service-ca\") on node \"ip-10-0-141-139.ec2.internal\" DevicePath \"\""
Sep 11 17:20:37.003836 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:20:37.003835    2577 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/067ed90b-10ca-413b-9c86-3b48b474d237-console-oauth-config\") on node \"ip-10-0-141-139.ec2.internal\" DevicePath \"\""
Sep 11 17:20:37.003836 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:20:37.003843    2577 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/067ed90b-10ca-413b-9c86-3b48b474d237-console-config\") on node \"ip-10-0-141-139.ec2.internal\" DevicePath \"\""
Sep 11 17:20:37.004087 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:20:37.003853    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-84khb\" (UniqueName: \"kubernetes.io/projected/067ed90b-10ca-413b-9c86-3b48b474d237-kube-api-access-84khb\") on node \"ip-10-0-141-139.ec2.internal\" DevicePath \"\""
Sep 11 17:20:37.004087 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:20:37.003862    2577 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/067ed90b-10ca-413b-9c86-3b48b474d237-oauth-serving-cert\") on node \"ip-10-0-141-139.ec2.internal\" DevicePath \"\""
Sep 11 17:20:37.583166 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:20:37.583129    2577 generic.go:356] "Generic (PLEG): container finished" podID="067ed90b-10ca-413b-9c86-3b48b474d237" containerID="7613c98f189145c515d6414f488c1079ec43a3984dfc131fb51847463cabbd0a" exitCode=0
Sep 11 17:20:37.583166 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:20:37.583170    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5c74f887d6-ttmcp" event={"ID":"067ed90b-10ca-413b-9c86-3b48b474d237","Type":"ContainerDied","Data":"7613c98f189145c515d6414f488c1079ec43a3984dfc131fb51847463cabbd0a"}
Sep 11 17:20:37.583714 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:20:37.583175    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5c74f887d6-ttmcp"
Sep 11 17:20:37.583714 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:20:37.583212    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5c74f887d6-ttmcp" event={"ID":"067ed90b-10ca-413b-9c86-3b48b474d237","Type":"ContainerDied","Data":"ab7a86735f9e94d087efc4c9704b0dd23a3986c9525be0dcd7e32ac5fd84723a"}
Sep 11 17:20:37.583714 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:20:37.583232    2577 scope.go:117] "RemoveContainer" containerID="7613c98f189145c515d6414f488c1079ec43a3984dfc131fb51847463cabbd0a"
Sep 11 17:20:37.591763 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:20:37.591741    2577 scope.go:117] "RemoveContainer" containerID="7613c98f189145c515d6414f488c1079ec43a3984dfc131fb51847463cabbd0a"
Sep 11 17:20:37.592064 ip-10-0-141-139 kubenswrapper[2577]: E0911 17:20:37.592045    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7613c98f189145c515d6414f488c1079ec43a3984dfc131fb51847463cabbd0a\": container with ID starting with 7613c98f189145c515d6414f488c1079ec43a3984dfc131fb51847463cabbd0a not found: ID does not exist" containerID="7613c98f189145c515d6414f488c1079ec43a3984dfc131fb51847463cabbd0a"
Sep 11 17:20:37.592120 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:20:37.592076    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7613c98f189145c515d6414f488c1079ec43a3984dfc131fb51847463cabbd0a"} err="failed to get container status \"7613c98f189145c515d6414f488c1079ec43a3984dfc131fb51847463cabbd0a\": rpc error: code = NotFound desc = could not find container \"7613c98f189145c515d6414f488c1079ec43a3984dfc131fb51847463cabbd0a\": container with ID starting with 7613c98f189145c515d6414f488c1079ec43a3984dfc131fb51847463cabbd0a not found: ID does not exist"
Sep 11 17:20:37.604616 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:20:37.604589    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-5c74f887d6-ttmcp"]
Sep 11 17:20:37.607358 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:20:37.607335    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-5c74f887d6-ttmcp"]
Sep 11 17:20:38.925816 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:20:38.925782    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="067ed90b-10ca-413b-9c86-3b48b474d237" path="/var/lib/kubelet/pods/067ed90b-10ca-413b-9c86-3b48b474d237/volumes"
Sep 11 17:20:56.412286 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:20:56.412253    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 17:20:56.432580 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:20:56.432550    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 17:20:56.681555 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:20:56.681458    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 17:21:10.385942 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:21:10.385901    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 11 17:21:10.725245 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:21:10.725182    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="4c942d42-091f-4c68-8faa-f6276bd26c15" containerName="alertmanager" containerID="cri-o://4adc38e198f4130856a225a3b2b4e8554d30cc92b333b82133b06156ead5d81b" gracePeriod=120
Sep 11 17:21:10.725442 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:21:10.725254    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="4c942d42-091f-4c68-8faa-f6276bd26c15" containerName="kube-rbac-proxy" containerID="cri-o://dfa205ad9943339ff6910fb2a0c2264eb3a5e31ef821c80704d257e26679e2c5" gracePeriod=120
Sep 11 17:21:10.725442 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:21:10.725247    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="4c942d42-091f-4c68-8faa-f6276bd26c15" containerName="kube-rbac-proxy-metric" containerID="cri-o://b1f982011d7acf52740c0f356f0e3a683b4ea5cd14e5f448bf5a894760045ae3" gracePeriod=120
Sep 11 17:21:10.725442 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:21:10.725311    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="4c942d42-091f-4c68-8faa-f6276bd26c15" containerName="prom-label-proxy" containerID="cri-o://2a24c94179b5571ff0b18b99cde019ee8ca1d8efad41302ae3b2d8a6261c2e91" gracePeriod=120
Sep 11 17:21:10.725442 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:21:10.725249    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="4c942d42-091f-4c68-8faa-f6276bd26c15" containerName="kube-rbac-proxy-web" containerID="cri-o://690666babfe741372f4d6e8b575c8bfd942f8fb501d16771706ef198dc4c302d" gracePeriod=120
Sep 11 17:21:10.725442 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:21:10.725320    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="4c942d42-091f-4c68-8faa-f6276bd26c15" containerName="config-reloader" containerID="cri-o://96e5540b18bfe38148c3a743a19f590d6495d387fff0d72121d52aece670802d" gracePeriod=120
Sep 11 17:21:11.732681 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:21:11.732641    2577 generic.go:356] "Generic (PLEG): container finished" podID="4c942d42-091f-4c68-8faa-f6276bd26c15" containerID="2a24c94179b5571ff0b18b99cde019ee8ca1d8efad41302ae3b2d8a6261c2e91" exitCode=0
Sep 11 17:21:11.732681 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:21:11.732672    2577 generic.go:356] "Generic (PLEG): container finished" podID="4c942d42-091f-4c68-8faa-f6276bd26c15" containerID="dfa205ad9943339ff6910fb2a0c2264eb3a5e31ef821c80704d257e26679e2c5" exitCode=0
Sep 11 17:21:11.732681 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:21:11.732679    2577 generic.go:356] "Generic (PLEG): container finished" podID="4c942d42-091f-4c68-8faa-f6276bd26c15" containerID="96e5540b18bfe38148c3a743a19f590d6495d387fff0d72121d52aece670802d" exitCode=0
Sep 11 17:21:11.732681 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:21:11.732685    2577 generic.go:356] "Generic (PLEG): container finished" podID="4c942d42-091f-4c68-8faa-f6276bd26c15" containerID="4adc38e198f4130856a225a3b2b4e8554d30cc92b333b82133b06156ead5d81b" exitCode=0
Sep 11 17:21:11.733199 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:21:11.732713    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"4c942d42-091f-4c68-8faa-f6276bd26c15","Type":"ContainerDied","Data":"2a24c94179b5571ff0b18b99cde019ee8ca1d8efad41302ae3b2d8a6261c2e91"}
Sep 11 17:21:11.733199 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:21:11.732801    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"4c942d42-091f-4c68-8faa-f6276bd26c15","Type":"ContainerDied","Data":"dfa205ad9943339ff6910fb2a0c2264eb3a5e31ef821c80704d257e26679e2c5"}
Sep 11 17:21:11.733199 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:21:11.732850    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"4c942d42-091f-4c68-8faa-f6276bd26c15","Type":"ContainerDied","Data":"96e5540b18bfe38148c3a743a19f590d6495d387fff0d72121d52aece670802d"}
Sep 11 17:21:11.733199 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:21:11.732889    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"4c942d42-091f-4c68-8faa-f6276bd26c15","Type":"ContainerDied","Data":"4adc38e198f4130856a225a3b2b4e8554d30cc92b333b82133b06156ead5d81b"}
Sep 11 17:21:11.962396 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:21:11.962370    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Sep 11 17:21:12.017760 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:21:12.017674    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/4c942d42-091f-4c68-8faa-f6276bd26c15-secret-alertmanager-kube-rbac-proxy\") pod \"4c942d42-091f-4c68-8faa-f6276bd26c15\" (UID: \"4c942d42-091f-4c68-8faa-f6276bd26c15\") "
Sep 11 17:21:12.017760 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:21:12.017743    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/4c942d42-091f-4c68-8faa-f6276bd26c15-cluster-tls-config\") pod \"4c942d42-091f-4c68-8faa-f6276bd26c15\" (UID: \"4c942d42-091f-4c68-8faa-f6276bd26c15\") "
Sep 11 17:21:12.017969 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:21:12.017936    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/4c942d42-091f-4c68-8faa-f6276bd26c15-secret-alertmanager-kube-rbac-proxy-metric\") pod \"4c942d42-091f-4c68-8faa-f6276bd26c15\" (UID: \"4c942d42-091f-4c68-8faa-f6276bd26c15\") "
Sep 11 17:21:12.018050 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:21:12.017995    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gh4hv\" (UniqueName: \"kubernetes.io/projected/4c942d42-091f-4c68-8faa-f6276bd26c15-kube-api-access-gh4hv\") pod \"4c942d42-091f-4c68-8faa-f6276bd26c15\" (UID: \"4c942d42-091f-4c68-8faa-f6276bd26c15\") "
Sep 11 17:21:12.018050 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:21:12.018034    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/4c942d42-091f-4c68-8faa-f6276bd26c15-metrics-client-ca\") pod \"4c942d42-091f-4c68-8faa-f6276bd26c15\" (UID: \"4c942d42-091f-4c68-8faa-f6276bd26c15\") "
Sep 11 17:21:12.018199 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:21:12.018086    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/4c942d42-091f-4c68-8faa-f6276bd26c15-secret-alertmanager-kube-rbac-proxy-web\") pod \"4c942d42-091f-4c68-8faa-f6276bd26c15\" (UID: \"4c942d42-091f-4c68-8faa-f6276bd26c15\") "
Sep 11 17:21:12.018317 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:21:12.018294    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/4c942d42-091f-4c68-8faa-f6276bd26c15-config-out\") pod \"4c942d42-091f-4c68-8faa-f6276bd26c15\" (UID: \"4c942d42-091f-4c68-8faa-f6276bd26c15\") "
Sep 11 17:21:12.019224 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:21:12.018516    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/4c942d42-091f-4c68-8faa-f6276bd26c15-secret-alertmanager-main-tls\") pod \"4c942d42-091f-4c68-8faa-f6276bd26c15\" (UID: \"4c942d42-091f-4c68-8faa-f6276bd26c15\") "
Sep 11 17:21:12.019224 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:21:12.018580    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/4c942d42-091f-4c68-8faa-f6276bd26c15-config-volume\") pod \"4c942d42-091f-4c68-8faa-f6276bd26c15\" (UID: \"4c942d42-091f-4c68-8faa-f6276bd26c15\") "
Sep 11 17:21:12.019224 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:21:12.018605    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/4c942d42-091f-4c68-8faa-f6276bd26c15-tls-assets\") pod \"4c942d42-091f-4c68-8faa-f6276bd26c15\" (UID: \"4c942d42-091f-4c68-8faa-f6276bd26c15\") "
Sep 11 17:21:12.019224 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:21:12.018638    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/4c942d42-091f-4c68-8faa-f6276bd26c15-alertmanager-main-db\") pod \"4c942d42-091f-4c68-8faa-f6276bd26c15\" (UID: \"4c942d42-091f-4c68-8faa-f6276bd26c15\") "
Sep 11 17:21:12.019224 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:21:12.018665    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4c942d42-091f-4c68-8faa-f6276bd26c15-alertmanager-trusted-ca-bundle\") pod \"4c942d42-091f-4c68-8faa-f6276bd26c15\" (UID: \"4c942d42-091f-4c68-8faa-f6276bd26c15\") "
Sep 11 17:21:12.019224 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:21:12.018697    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/4c942d42-091f-4c68-8faa-f6276bd26c15-web-config\") pod \"4c942d42-091f-4c68-8faa-f6276bd26c15\" (UID: \"4c942d42-091f-4c68-8faa-f6276bd26c15\") "
Sep 11 17:21:12.019224 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:21:12.018986    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4c942d42-091f-4c68-8faa-f6276bd26c15-metrics-client-ca" (OuterVolumeSpecName: "metrics-client-ca") pod "4c942d42-091f-4c68-8faa-f6276bd26c15" (UID: "4c942d42-091f-4c68-8faa-f6276bd26c15"). InnerVolumeSpecName "metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 17:21:12.019668 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:21:12.019586    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4c942d42-091f-4c68-8faa-f6276bd26c15-alertmanager-main-db" (OuterVolumeSpecName: "alertmanager-main-db") pod "4c942d42-091f-4c68-8faa-f6276bd26c15" (UID: "4c942d42-091f-4c68-8faa-f6276bd26c15"). InnerVolumeSpecName "alertmanager-main-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 17:21:12.020232 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:21:12.020201    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4c942d42-091f-4c68-8faa-f6276bd26c15-alertmanager-trusted-ca-bundle" (OuterVolumeSpecName: "alertmanager-trusted-ca-bundle") pod "4c942d42-091f-4c68-8faa-f6276bd26c15" (UID: "4c942d42-091f-4c68-8faa-f6276bd26c15"). InnerVolumeSpecName "alertmanager-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 17:21:12.021040 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:21:12.020984    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4c942d42-091f-4c68-8faa-f6276bd26c15-secret-alertmanager-kube-rbac-proxy" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy") pod "4c942d42-091f-4c68-8faa-f6276bd26c15" (UID: "4c942d42-091f-4c68-8faa-f6276bd26c15"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 17:21:12.021457 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:21:12.021408    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4c942d42-091f-4c68-8faa-f6276bd26c15-secret-alertmanager-kube-rbac-proxy-web" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy-web") pod "4c942d42-091f-4c68-8faa-f6276bd26c15" (UID: "4c942d42-091f-4c68-8faa-f6276bd26c15"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 17:21:12.021679 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:21:12.021585    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4c942d42-091f-4c68-8faa-f6276bd26c15-config-out" (OuterVolumeSpecName: "config-out") pod "4c942d42-091f-4c68-8faa-f6276bd26c15" (UID: "4c942d42-091f-4c68-8faa-f6276bd26c15"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 17:21:12.022356 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:21:12.022321    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4c942d42-091f-4c68-8faa-f6276bd26c15-kube-api-access-gh4hv" (OuterVolumeSpecName: "kube-api-access-gh4hv") pod "4c942d42-091f-4c68-8faa-f6276bd26c15" (UID: "4c942d42-091f-4c68-8faa-f6276bd26c15"). InnerVolumeSpecName "kube-api-access-gh4hv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 17:21:12.023576 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:21:12.023541    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4c942d42-091f-4c68-8faa-f6276bd26c15-config-volume" (OuterVolumeSpecName: "config-volume") pod "4c942d42-091f-4c68-8faa-f6276bd26c15" (UID: "4c942d42-091f-4c68-8faa-f6276bd26c15"). InnerVolumeSpecName "config-volume". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 17:21:12.024336 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:21:12.024308    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4c942d42-091f-4c68-8faa-f6276bd26c15-secret-alertmanager-main-tls" (OuterVolumeSpecName: "secret-alertmanager-main-tls") pod "4c942d42-091f-4c68-8faa-f6276bd26c15" (UID: "4c942d42-091f-4c68-8faa-f6276bd26c15"). InnerVolumeSpecName "secret-alertmanager-main-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 17:21:12.024990 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:21:12.024961    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4c942d42-091f-4c68-8faa-f6276bd26c15-tls-assets" (OuterVolumeSpecName: "tls-assets") pod "4c942d42-091f-4c68-8faa-f6276bd26c15" (UID: "4c942d42-091f-4c68-8faa-f6276bd26c15"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 17:21:12.025148 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:21:12.025116    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4c942d42-091f-4c68-8faa-f6276bd26c15-secret-alertmanager-kube-rbac-proxy-metric" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy-metric") pod "4c942d42-091f-4c68-8faa-f6276bd26c15" (UID: "4c942d42-091f-4c68-8faa-f6276bd26c15"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-metric". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 17:21:12.028735 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:21:12.028710    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4c942d42-091f-4c68-8faa-f6276bd26c15-cluster-tls-config" (OuterVolumeSpecName: "cluster-tls-config") pod "4c942d42-091f-4c68-8faa-f6276bd26c15" (UID: "4c942d42-091f-4c68-8faa-f6276bd26c15"). InnerVolumeSpecName "cluster-tls-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 17:21:12.035731 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:21:12.035704    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4c942d42-091f-4c68-8faa-f6276bd26c15-web-config" (OuterVolumeSpecName: "web-config") pod "4c942d42-091f-4c68-8faa-f6276bd26c15" (UID: "4c942d42-091f-4c68-8faa-f6276bd26c15"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 17:21:12.119652 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:21:12.119616    2577 reconciler_common.go:299] "Volume detached for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/4c942d42-091f-4c68-8faa-f6276bd26c15-cluster-tls-config\") on node \"ip-10-0-141-139.ec2.internal\" DevicePath \"\""
Sep 11 17:21:12.119652 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:21:12.119648    2577 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/4c942d42-091f-4c68-8faa-f6276bd26c15-secret-alertmanager-kube-rbac-proxy-metric\") on node \"ip-10-0-141-139.ec2.internal\" DevicePath \"\""
Sep 11 17:21:12.119933 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:21:12.119663    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gh4hv\" (UniqueName: \"kubernetes.io/projected/4c942d42-091f-4c68-8faa-f6276bd26c15-kube-api-access-gh4hv\") on node \"ip-10-0-141-139.ec2.internal\" DevicePath \"\""
Sep 11 17:21:12.119933 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:21:12.119677    2577 reconciler_common.go:299] "Volume detached for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/4c942d42-091f-4c68-8faa-f6276bd26c15-metrics-client-ca\") on node \"ip-10-0-141-139.ec2.internal\" DevicePath \"\""
Sep 11 17:21:12.119933 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:21:12.119689    2577 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/4c942d42-091f-4c68-8faa-f6276bd26c15-secret-alertmanager-kube-rbac-proxy-web\") on node \"ip-10-0-141-139.ec2.internal\" DevicePath \"\""
Sep 11 17:21:12.119933 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:21:12.119702    2577 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/4c942d42-091f-4c68-8faa-f6276bd26c15-config-out\") on node \"ip-10-0-141-139.ec2.internal\" DevicePath \"\""
Sep 11 17:21:12.119933 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:21:12.119714    2577 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/4c942d42-091f-4c68-8faa-f6276bd26c15-secret-alertmanager-main-tls\") on node \"ip-10-0-141-139.ec2.internal\" DevicePath \"\""
Sep 11 17:21:12.119933 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:21:12.119726    2577 reconciler_common.go:299] "Volume detached for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/4c942d42-091f-4c68-8faa-f6276bd26c15-config-volume\") on node \"ip-10-0-141-139.ec2.internal\" DevicePath \"\""
Sep 11 17:21:12.119933 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:21:12.119737    2577 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/4c942d42-091f-4c68-8faa-f6276bd26c15-tls-assets\") on node \"ip-10-0-141-139.ec2.internal\" DevicePath \"\""
Sep 11 17:21:12.119933 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:21:12.119748    2577 reconciler_common.go:299] "Volume detached for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/4c942d42-091f-4c68-8faa-f6276bd26c15-alertmanager-main-db\") on node \"ip-10-0-141-139.ec2.internal\" DevicePath \"\""
Sep 11 17:21:12.119933 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:21:12.119759    2577 reconciler_common.go:299] "Volume detached for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4c942d42-091f-4c68-8faa-f6276bd26c15-alertmanager-trusted-ca-bundle\") on node \"ip-10-0-141-139.ec2.internal\" DevicePath \"\""
Sep 11 17:21:12.119933 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:21:12.119771    2577 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/4c942d42-091f-4c68-8faa-f6276bd26c15-web-config\") on node \"ip-10-0-141-139.ec2.internal\" DevicePath \"\""
Sep 11 17:21:12.119933 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:21:12.119802    2577 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/4c942d42-091f-4c68-8faa-f6276bd26c15-secret-alertmanager-kube-rbac-proxy\") on node \"ip-10-0-141-139.ec2.internal\" DevicePath \"\""
Sep 11 17:21:12.744644 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:21:12.744611    2577 generic.go:356] "Generic (PLEG): container finished" podID="4c942d42-091f-4c68-8faa-f6276bd26c15" containerID="b1f982011d7acf52740c0f356f0e3a683b4ea5cd14e5f448bf5a894760045ae3" exitCode=0
Sep 11 17:21:12.744644 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:21:12.744637    2577 generic.go:356] "Generic (PLEG): container finished" podID="4c942d42-091f-4c68-8faa-f6276bd26c15" containerID="690666babfe741372f4d6e8b575c8bfd942f8fb501d16771706ef198dc4c302d" exitCode=0
Sep 11 17:21:12.745186 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:21:12.744694    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Sep 11 17:21:12.745186 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:21:12.744695    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"4c942d42-091f-4c68-8faa-f6276bd26c15","Type":"ContainerDied","Data":"b1f982011d7acf52740c0f356f0e3a683b4ea5cd14e5f448bf5a894760045ae3"}
Sep 11 17:21:12.745186 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:21:12.744825    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"4c942d42-091f-4c68-8faa-f6276bd26c15","Type":"ContainerDied","Data":"690666babfe741372f4d6e8b575c8bfd942f8fb501d16771706ef198dc4c302d"}
Sep 11 17:21:12.745186 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:21:12.744868    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"4c942d42-091f-4c68-8faa-f6276bd26c15","Type":"ContainerDied","Data":"1351eb8bcfd974cd6b2c8b6505b18ceee7c4925274a6385fd484e5d4892645bf"}
Sep 11 17:21:12.745186 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:21:12.744886    2577 scope.go:117] "RemoveContainer" containerID="2a24c94179b5571ff0b18b99cde019ee8ca1d8efad41302ae3b2d8a6261c2e91"
Sep 11 17:21:12.753327 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:21:12.753305    2577 scope.go:117] "RemoveContainer" containerID="b1f982011d7acf52740c0f356f0e3a683b4ea5cd14e5f448bf5a894760045ae3"
Sep 11 17:21:12.760723 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:21:12.760703    2577 scope.go:117] "RemoveContainer" containerID="dfa205ad9943339ff6910fb2a0c2264eb3a5e31ef821c80704d257e26679e2c5"
Sep 11 17:21:12.768278 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:21:12.768257    2577 scope.go:117] "RemoveContainer" containerID="690666babfe741372f4d6e8b575c8bfd942f8fb501d16771706ef198dc4c302d"
Sep 11 17:21:12.769580 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:21:12.769561    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 11 17:21:12.776165 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:21:12.776147    2577 scope.go:117] "RemoveContainer" containerID="96e5540b18bfe38148c3a743a19f590d6495d387fff0d72121d52aece670802d"
Sep 11 17:21:12.776967 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:21:12.776941    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 11 17:21:12.783542 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:21:12.783493    2577 scope.go:117] "RemoveContainer" containerID="4adc38e198f4130856a225a3b2b4e8554d30cc92b333b82133b06156ead5d81b"
Sep 11 17:21:12.791481 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:21:12.791462    2577 scope.go:117] "RemoveContainer" containerID="706f26f8e9e50b678349bb0d141ec51a0d860f198c61f17ef4d33144981cf609"
Sep 11 17:21:12.803032 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:21:12.803010    2577 scope.go:117] "RemoveContainer" containerID="2a24c94179b5571ff0b18b99cde019ee8ca1d8efad41302ae3b2d8a6261c2e91"
Sep 11 17:21:12.803286 ip-10-0-141-139 kubenswrapper[2577]: E0911 17:21:12.803271    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2a24c94179b5571ff0b18b99cde019ee8ca1d8efad41302ae3b2d8a6261c2e91\": container with ID starting with 2a24c94179b5571ff0b18b99cde019ee8ca1d8efad41302ae3b2d8a6261c2e91 not found: ID does not exist" containerID="2a24c94179b5571ff0b18b99cde019ee8ca1d8efad41302ae3b2d8a6261c2e91"
Sep 11 17:21:12.803333 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:21:12.803296    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2a24c94179b5571ff0b18b99cde019ee8ca1d8efad41302ae3b2d8a6261c2e91"} err="failed to get container status \"2a24c94179b5571ff0b18b99cde019ee8ca1d8efad41302ae3b2d8a6261c2e91\": rpc error: code = NotFound desc = could not find container \"2a24c94179b5571ff0b18b99cde019ee8ca1d8efad41302ae3b2d8a6261c2e91\": container with ID starting with 2a24c94179b5571ff0b18b99cde019ee8ca1d8efad41302ae3b2d8a6261c2e91 not found: ID does not exist"
Sep 11 17:21:12.803333 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:21:12.803315    2577 scope.go:117] "RemoveContainer" containerID="b1f982011d7acf52740c0f356f0e3a683b4ea5cd14e5f448bf5a894760045ae3"
Sep 11 17:21:12.803591 ip-10-0-141-139 kubenswrapper[2577]: E0911 17:21:12.803573    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b1f982011d7acf52740c0f356f0e3a683b4ea5cd14e5f448bf5a894760045ae3\": container with ID starting with b1f982011d7acf52740c0f356f0e3a683b4ea5cd14e5f448bf5a894760045ae3 not found: ID does not exist" containerID="b1f982011d7acf52740c0f356f0e3a683b4ea5cd14e5f448bf5a894760045ae3"
Sep 11 17:21:12.803681 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:21:12.803597    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b1f982011d7acf52740c0f356f0e3a683b4ea5cd14e5f448bf5a894760045ae3"} err="failed to get container status \"b1f982011d7acf52740c0f356f0e3a683b4ea5cd14e5f448bf5a894760045ae3\": rpc error: code = NotFound desc = could not find container \"b1f982011d7acf52740c0f356f0e3a683b4ea5cd14e5f448bf5a894760045ae3\": container with ID starting with b1f982011d7acf52740c0f356f0e3a683b4ea5cd14e5f448bf5a894760045ae3 not found: ID does not exist"
Sep 11 17:21:12.803681 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:21:12.803613    2577 scope.go:117] "RemoveContainer" containerID="dfa205ad9943339ff6910fb2a0c2264eb3a5e31ef821c80704d257e26679e2c5"
Sep 11 17:21:12.803828 ip-10-0-141-139 kubenswrapper[2577]: E0911 17:21:12.803812    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dfa205ad9943339ff6910fb2a0c2264eb3a5e31ef821c80704d257e26679e2c5\": container with ID starting with dfa205ad9943339ff6910fb2a0c2264eb3a5e31ef821c80704d257e26679e2c5 not found: ID does not exist" containerID="dfa205ad9943339ff6910fb2a0c2264eb3a5e31ef821c80704d257e26679e2c5"
Sep 11 17:21:12.803864 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:21:12.803833    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dfa205ad9943339ff6910fb2a0c2264eb3a5e31ef821c80704d257e26679e2c5"} err="failed to get container status \"dfa205ad9943339ff6910fb2a0c2264eb3a5e31ef821c80704d257e26679e2c5\": rpc error: code = NotFound desc = could not find container \"dfa205ad9943339ff6910fb2a0c2264eb3a5e31ef821c80704d257e26679e2c5\": container with ID starting with dfa205ad9943339ff6910fb2a0c2264eb3a5e31ef821c80704d257e26679e2c5 not found: ID does not exist"
Sep 11 17:21:12.803864 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:21:12.803845    2577 scope.go:117] "RemoveContainer" containerID="690666babfe741372f4d6e8b575c8bfd942f8fb501d16771706ef198dc4c302d"
Sep 11 17:21:12.804043 ip-10-0-141-139 kubenswrapper[2577]: E0911 17:21:12.804027    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"690666babfe741372f4d6e8b575c8bfd942f8fb501d16771706ef198dc4c302d\": container with ID starting with 690666babfe741372f4d6e8b575c8bfd942f8fb501d16771706ef198dc4c302d not found: ID does not exist" containerID="690666babfe741372f4d6e8b575c8bfd942f8fb501d16771706ef198dc4c302d"
Sep 11 17:21:12.804081 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:21:12.804046    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"690666babfe741372f4d6e8b575c8bfd942f8fb501d16771706ef198dc4c302d"} err="failed to get container status \"690666babfe741372f4d6e8b575c8bfd942f8fb501d16771706ef198dc4c302d\": rpc error: code = NotFound desc = could not find container \"690666babfe741372f4d6e8b575c8bfd942f8fb501d16771706ef198dc4c302d\": container with ID starting with 690666babfe741372f4d6e8b575c8bfd942f8fb501d16771706ef198dc4c302d not found: ID does not exist"
Sep 11 17:21:12.804081 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:21:12.804058    2577 scope.go:117] "RemoveContainer" containerID="96e5540b18bfe38148c3a743a19f590d6495d387fff0d72121d52aece670802d"
Sep 11 17:21:12.804342 ip-10-0-141-139 kubenswrapper[2577]: E0911 17:21:12.804322    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"96e5540b18bfe38148c3a743a19f590d6495d387fff0d72121d52aece670802d\": container with ID starting with 96e5540b18bfe38148c3a743a19f590d6495d387fff0d72121d52aece670802d not found: ID does not exist" containerID="96e5540b18bfe38148c3a743a19f590d6495d387fff0d72121d52aece670802d"
Sep 11 17:21:12.804385 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:21:12.804348    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"96e5540b18bfe38148c3a743a19f590d6495d387fff0d72121d52aece670802d"} err="failed to get container status \"96e5540b18bfe38148c3a743a19f590d6495d387fff0d72121d52aece670802d\": rpc error: code = NotFound desc = could not find container \"96e5540b18bfe38148c3a743a19f590d6495d387fff0d72121d52aece670802d\": container with ID starting with 96e5540b18bfe38148c3a743a19f590d6495d387fff0d72121d52aece670802d not found: ID does not exist"
Sep 11 17:21:12.804385 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:21:12.804364    2577 scope.go:117] "RemoveContainer" containerID="4adc38e198f4130856a225a3b2b4e8554d30cc92b333b82133b06156ead5d81b"
Sep 11 17:21:12.804646 ip-10-0-141-139 kubenswrapper[2577]: E0911 17:21:12.804620    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4adc38e198f4130856a225a3b2b4e8554d30cc92b333b82133b06156ead5d81b\": container with ID starting with 4adc38e198f4130856a225a3b2b4e8554d30cc92b333b82133b06156ead5d81b not found: ID does not exist" containerID="4adc38e198f4130856a225a3b2b4e8554d30cc92b333b82133b06156ead5d81b"
Sep 11 17:21:12.804725 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:21:12.804653    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4adc38e198f4130856a225a3b2b4e8554d30cc92b333b82133b06156ead5d81b"} err="failed to get container status \"4adc38e198f4130856a225a3b2b4e8554d30cc92b333b82133b06156ead5d81b\": rpc error: code = NotFound desc = could not find container \"4adc38e198f4130856a225a3b2b4e8554d30cc92b333b82133b06156ead5d81b\": container with ID starting with 4adc38e198f4130856a225a3b2b4e8554d30cc92b333b82133b06156ead5d81b not found: ID does not exist"
Sep 11 17:21:12.804725 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:21:12.804669    2577 scope.go:117] "RemoveContainer" containerID="706f26f8e9e50b678349bb0d141ec51a0d860f198c61f17ef4d33144981cf609"
Sep 11 17:21:12.804985 ip-10-0-141-139 kubenswrapper[2577]: E0911 17:21:12.804964    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"706f26f8e9e50b678349bb0d141ec51a0d860f198c61f17ef4d33144981cf609\": container with ID starting with 706f26f8e9e50b678349bb0d141ec51a0d860f198c61f17ef4d33144981cf609 not found: ID does not exist" containerID="706f26f8e9e50b678349bb0d141ec51a0d860f198c61f17ef4d33144981cf609"
Sep 11 17:21:12.805078 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:21:12.804991    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"706f26f8e9e50b678349bb0d141ec51a0d860f198c61f17ef4d33144981cf609"} err="failed to get container status \"706f26f8e9e50b678349bb0d141ec51a0d860f198c61f17ef4d33144981cf609\": rpc error: code = NotFound desc = could not find container \"706f26f8e9e50b678349bb0d141ec51a0d860f198c61f17ef4d33144981cf609\": container with ID starting with 706f26f8e9e50b678349bb0d141ec51a0d860f198c61f17ef4d33144981cf609 not found: ID does not exist"
Sep 11 17:21:12.805078 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:21:12.805008    2577 scope.go:117] "RemoveContainer" containerID="2a24c94179b5571ff0b18b99cde019ee8ca1d8efad41302ae3b2d8a6261c2e91"
Sep 11 17:21:12.806065 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:21:12.805342    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2a24c94179b5571ff0b18b99cde019ee8ca1d8efad41302ae3b2d8a6261c2e91"} err="failed to get container status \"2a24c94179b5571ff0b18b99cde019ee8ca1d8efad41302ae3b2d8a6261c2e91\": rpc error: code = NotFound desc = could not find container \"2a24c94179b5571ff0b18b99cde019ee8ca1d8efad41302ae3b2d8a6261c2e91\": container with ID starting with 2a24c94179b5571ff0b18b99cde019ee8ca1d8efad41302ae3b2d8a6261c2e91 not found: ID does not exist"
Sep 11 17:21:12.806065 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:21:12.805367    2577 scope.go:117] "RemoveContainer" containerID="b1f982011d7acf52740c0f356f0e3a683b4ea5cd14e5f448bf5a894760045ae3"
Sep 11 17:21:12.806065 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:21:12.805742    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b1f982011d7acf52740c0f356f0e3a683b4ea5cd14e5f448bf5a894760045ae3"} err="failed to get container status \"b1f982011d7acf52740c0f356f0e3a683b4ea5cd14e5f448bf5a894760045ae3\": rpc error: code = NotFound desc = could not find container \"b1f982011d7acf52740c0f356f0e3a683b4ea5cd14e5f448bf5a894760045ae3\": container with ID starting with b1f982011d7acf52740c0f356f0e3a683b4ea5cd14e5f448bf5a894760045ae3 not found: ID does not exist"
Sep 11 17:21:12.806065 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:21:12.805766    2577 scope.go:117] "RemoveContainer" containerID="dfa205ad9943339ff6910fb2a0c2264eb3a5e31ef821c80704d257e26679e2c5"
Sep 11 17:21:12.806316 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:21:12.806081    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dfa205ad9943339ff6910fb2a0c2264eb3a5e31ef821c80704d257e26679e2c5"} err="failed to get container status \"dfa205ad9943339ff6910fb2a0c2264eb3a5e31ef821c80704d257e26679e2c5\": rpc error: code = NotFound desc = could not find container \"dfa205ad9943339ff6910fb2a0c2264eb3a5e31ef821c80704d257e26679e2c5\": container with ID starting with dfa205ad9943339ff6910fb2a0c2264eb3a5e31ef821c80704d257e26679e2c5 not found: ID does not exist"
Sep 11 17:21:12.806316 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:21:12.806104    2577 scope.go:117] "RemoveContainer" containerID="690666babfe741372f4d6e8b575c8bfd942f8fb501d16771706ef198dc4c302d"
Sep 11 17:21:12.806466 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:21:12.806402    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"690666babfe741372f4d6e8b575c8bfd942f8fb501d16771706ef198dc4c302d"} err="failed to get container status \"690666babfe741372f4d6e8b575c8bfd942f8fb501d16771706ef198dc4c302d\": rpc error: code = NotFound desc = could not find container \"690666babfe741372f4d6e8b575c8bfd942f8fb501d16771706ef198dc4c302d\": container with ID starting with 690666babfe741372f4d6e8b575c8bfd942f8fb501d16771706ef198dc4c302d not found: ID does not exist"
Sep 11 17:21:12.806466 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:21:12.806422    2577 scope.go:117] "RemoveContainer" containerID="96e5540b18bfe38148c3a743a19f590d6495d387fff0d72121d52aece670802d"
Sep 11 17:21:12.806740 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:21:12.806716    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"96e5540b18bfe38148c3a743a19f590d6495d387fff0d72121d52aece670802d"} err="failed to get container status \"96e5540b18bfe38148c3a743a19f590d6495d387fff0d72121d52aece670802d\": rpc error: code = NotFound desc = could not find container \"96e5540b18bfe38148c3a743a19f590d6495d387fff0d72121d52aece670802d\": container with ID starting with 96e5540b18bfe38148c3a743a19f590d6495d387fff0d72121d52aece670802d not found: ID does not exist"
Sep 11 17:21:12.806801 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:21:12.806743    2577 scope.go:117] "RemoveContainer" containerID="4adc38e198f4130856a225a3b2b4e8554d30cc92b333b82133b06156ead5d81b"
Sep 11 17:21:12.806978 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:21:12.806959    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4adc38e198f4130856a225a3b2b4e8554d30cc92b333b82133b06156ead5d81b"} err="failed to get container status \"4adc38e198f4130856a225a3b2b4e8554d30cc92b333b82133b06156ead5d81b\": rpc error: code = NotFound desc = could not find container \"4adc38e198f4130856a225a3b2b4e8554d30cc92b333b82133b06156ead5d81b\": container with ID starting with 4adc38e198f4130856a225a3b2b4e8554d30cc92b333b82133b06156ead5d81b not found: ID does not exist"
Sep 11 17:21:12.807055 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:21:12.806980    2577 scope.go:117] "RemoveContainer" containerID="706f26f8e9e50b678349bb0d141ec51a0d860f198c61f17ef4d33144981cf609"
Sep 11 17:21:12.807227 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:21:12.807201    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"706f26f8e9e50b678349bb0d141ec51a0d860f198c61f17ef4d33144981cf609"} err="failed to get container status \"706f26f8e9e50b678349bb0d141ec51a0d860f198c61f17ef4d33144981cf609\": rpc error: code = NotFound desc = could not find container \"706f26f8e9e50b678349bb0d141ec51a0d860f198c61f17ef4d33144981cf609\": container with ID starting with 706f26f8e9e50b678349bb0d141ec51a0d860f198c61f17ef4d33144981cf609 not found: ID does not exist"
Sep 11 17:21:12.807957 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:21:12.807937    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 11 17:21:12.808250 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:21:12.808235    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4c942d42-091f-4c68-8faa-f6276bd26c15" containerName="init-config-reloader"
Sep 11 17:21:12.808250 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:21:12.808251    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="4c942d42-091f-4c68-8faa-f6276bd26c15" containerName="init-config-reloader"
Sep 11 17:21:12.808337 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:21:12.808258    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4c942d42-091f-4c68-8faa-f6276bd26c15" containerName="alertmanager"
Sep 11 17:21:12.808337 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:21:12.808264    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="4c942d42-091f-4c68-8faa-f6276bd26c15" containerName="alertmanager"
Sep 11 17:21:12.808337 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:21:12.808271    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4c942d42-091f-4c68-8faa-f6276bd26c15" containerName="kube-rbac-proxy-web"
Sep 11 17:21:12.808337 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:21:12.808276    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="4c942d42-091f-4c68-8faa-f6276bd26c15" containerName="kube-rbac-proxy-web"
Sep 11 17:21:12.808337 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:21:12.808283    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="85583e11-cdb9-4c56-9aab-aa6a53641fda" containerName="console"
Sep 11 17:21:12.808337 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:21:12.808289    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="85583e11-cdb9-4c56-9aab-aa6a53641fda" containerName="console"
Sep 11 17:21:12.808337 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:21:12.808302    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c2dc1ae2-696a-49f6-85ae-071961ef87fc" containerName="console"
Sep 11 17:21:12.808337 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:21:12.808308    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="c2dc1ae2-696a-49f6-85ae-071961ef87fc" containerName="console"
Sep 11 17:21:12.808337 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:21:12.808317    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4c942d42-091f-4c68-8faa-f6276bd26c15" containerName="kube-rbac-proxy"
Sep 11 17:21:12.808337 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:21:12.808322    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="4c942d42-091f-4c68-8faa-f6276bd26c15" containerName="kube-rbac-proxy"
Sep 11 17:21:12.808337 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:21:12.808334    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="067ed90b-10ca-413b-9c86-3b48b474d237" containerName="console"
Sep 11 17:21:12.808337 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:21:12.808339    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="067ed90b-10ca-413b-9c86-3b48b474d237" containerName="console"
Sep 11 17:21:12.808740 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:21:12.808346    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4c942d42-091f-4c68-8faa-f6276bd26c15" containerName="config-reloader"
Sep 11 17:21:12.808740 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:21:12.808351    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="4c942d42-091f-4c68-8faa-f6276bd26c15" containerName="config-reloader"
Sep 11 17:21:12.808740 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:21:12.808356    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4c942d42-091f-4c68-8faa-f6276bd26c15" containerName="prom-label-proxy"
Sep 11 17:21:12.808740 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:21:12.808361    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="4c942d42-091f-4c68-8faa-f6276bd26c15" containerName="prom-label-proxy"
Sep 11 17:21:12.808740 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:21:12.808375    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4c942d42-091f-4c68-8faa-f6276bd26c15" containerName="kube-rbac-proxy-metric"
Sep 11 17:21:12.808740 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:21:12.808381    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="4c942d42-091f-4c68-8faa-f6276bd26c15" containerName="kube-rbac-proxy-metric"
Sep 11 17:21:12.808740 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:21:12.808437    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="4c942d42-091f-4c68-8faa-f6276bd26c15" containerName="alertmanager"
Sep 11 17:21:12.808740 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:21:12.808447    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="067ed90b-10ca-413b-9c86-3b48b474d237" containerName="console"
Sep 11 17:21:12.808740 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:21:12.808455    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="85583e11-cdb9-4c56-9aab-aa6a53641fda" containerName="console"
Sep 11 17:21:12.808740 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:21:12.808460    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="4c942d42-091f-4c68-8faa-f6276bd26c15" containerName="prom-label-proxy"
Sep 11 17:21:12.808740 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:21:12.808465    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="4c942d42-091f-4c68-8faa-f6276bd26c15" containerName="kube-rbac-proxy"
Sep 11 17:21:12.808740 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:21:12.808470    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="c2dc1ae2-696a-49f6-85ae-071961ef87fc" containerName="console"
Sep 11 17:21:12.808740 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:21:12.808476    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="4c942d42-091f-4c68-8faa-f6276bd26c15" containerName="kube-rbac-proxy-metric"
Sep 11 17:21:12.808740 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:21:12.808481    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="4c942d42-091f-4c68-8faa-f6276bd26c15" containerName="kube-rbac-proxy-web"
Sep 11 17:21:12.808740 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:21:12.808486    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="4c942d42-091f-4c68-8faa-f6276bd26c15" containerName="config-reloader"
Sep 11 17:21:12.812367 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:21:12.812352    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Sep 11 17:21:12.814868 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:21:12.814811    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\""
Sep 11 17:21:12.814991 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:21:12.814875    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\""
Sep 11 17:21:12.814991 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:21:12.814918    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\""
Sep 11 17:21:12.815182 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:21:12.815153    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\""
Sep 11 17:21:12.815323 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:21:12.815186    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\""
Sep 11 17:21:12.815323 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:21:12.815197    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-w6gsg\""
Sep 11 17:21:12.815441 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:21:12.815318    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\""
Sep 11 17:21:12.815441 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:21:12.815414    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\""
Sep 11 17:21:12.815697 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:21:12.815680    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\""
Sep 11 17:21:12.820751 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:21:12.820731    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\""
Sep 11 17:21:12.826011 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:21:12.825989    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/cc806362-6764-4116-b742-c516e267f9cc-config-volume\") pod \"alertmanager-main-0\" (UID: \"cc806362-6764-4116-b742-c516e267f9cc\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 17:21:12.826106 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:21:12.826028    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/cc806362-6764-4116-b742-c516e267f9cc-web-config\") pod \"alertmanager-main-0\" (UID: \"cc806362-6764-4116-b742-c516e267f9cc\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 17:21:12.826106 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:21:12.826049    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ljdnw\" (UniqueName: \"kubernetes.io/projected/cc806362-6764-4116-b742-c516e267f9cc-kube-api-access-ljdnw\") pod \"alertmanager-main-0\" (UID: \"cc806362-6764-4116-b742-c516e267f9cc\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 17:21:12.826106 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:21:12.826076    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/cc806362-6764-4116-b742-c516e267f9cc-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"cc806362-6764-4116-b742-c516e267f9cc\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 17:21:12.826224 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:21:12.826117    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/cc806362-6764-4116-b742-c516e267f9cc-config-out\") pod \"alertmanager-main-0\" (UID: \"cc806362-6764-4116-b742-c516e267f9cc\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 17:21:12.826224 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:21:12.826152    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/cc806362-6764-4116-b742-c516e267f9cc-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"cc806362-6764-4116-b742-c516e267f9cc\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 17:21:12.826224 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:21:12.826170    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/cc806362-6764-4116-b742-c516e267f9cc-tls-assets\") pod \"alertmanager-main-0\" (UID: \"cc806362-6764-4116-b742-c516e267f9cc\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 17:21:12.826224 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:21:12.826192    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/cc806362-6764-4116-b742-c516e267f9cc-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"cc806362-6764-4116-b742-c516e267f9cc\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 17:21:12.826224 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:21:12.826220    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cc806362-6764-4116-b742-c516e267f9cc-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"cc806362-6764-4116-b742-c516e267f9cc\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 17:21:12.826401 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:21:12.826241    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/cc806362-6764-4116-b742-c516e267f9cc-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"cc806362-6764-4116-b742-c516e267f9cc\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 17:21:12.826401 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:21:12.826257    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/cc806362-6764-4116-b742-c516e267f9cc-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"cc806362-6764-4116-b742-c516e267f9cc\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 17:21:12.826401 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:21:12.826309    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/cc806362-6764-4116-b742-c516e267f9cc-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"cc806362-6764-4116-b742-c516e267f9cc\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 17:21:12.826401 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:21:12.826336    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/cc806362-6764-4116-b742-c516e267f9cc-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"cc806362-6764-4116-b742-c516e267f9cc\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 17:21:12.829059 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:21:12.829038    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 11 17:21:12.925741 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:21:12.925709    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4c942d42-091f-4c68-8faa-f6276bd26c15" path="/var/lib/kubelet/pods/4c942d42-091f-4c68-8faa-f6276bd26c15/volumes"
Sep 11 17:21:12.926993 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:21:12.926973    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/cc806362-6764-4116-b742-c516e267f9cc-web-config\") pod \"alertmanager-main-0\" (UID: \"cc806362-6764-4116-b742-c516e267f9cc\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 17:21:12.927044 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:21:12.927001    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ljdnw\" (UniqueName: \"kubernetes.io/projected/cc806362-6764-4116-b742-c516e267f9cc-kube-api-access-ljdnw\") pod \"alertmanager-main-0\" (UID: \"cc806362-6764-4116-b742-c516e267f9cc\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 17:21:12.927044 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:21:12.927020    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/cc806362-6764-4116-b742-c516e267f9cc-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"cc806362-6764-4116-b742-c516e267f9cc\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 17:21:12.927113 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:21:12.927082    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/cc806362-6764-4116-b742-c516e267f9cc-config-out\") pod \"alertmanager-main-0\" (UID: \"cc806362-6764-4116-b742-c516e267f9cc\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 17:21:12.927252 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:21:12.927233    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/cc806362-6764-4116-b742-c516e267f9cc-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"cc806362-6764-4116-b742-c516e267f9cc\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 17:21:12.927317 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:21:12.927266    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/cc806362-6764-4116-b742-c516e267f9cc-tls-assets\") pod \"alertmanager-main-0\" (UID: \"cc806362-6764-4116-b742-c516e267f9cc\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 17:21:12.927317 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:21:12.927299    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/cc806362-6764-4116-b742-c516e267f9cc-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"cc806362-6764-4116-b742-c516e267f9cc\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 17:21:12.927400 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:21:12.927327    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cc806362-6764-4116-b742-c516e267f9cc-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"cc806362-6764-4116-b742-c516e267f9cc\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 17:21:12.927400 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:21:12.927356    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/cc806362-6764-4116-b742-c516e267f9cc-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"cc806362-6764-4116-b742-c516e267f9cc\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 17:21:12.927400 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:21:12.927382    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/cc806362-6764-4116-b742-c516e267f9cc-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"cc806362-6764-4116-b742-c516e267f9cc\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 17:21:12.927576 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:21:12.927436    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/cc806362-6764-4116-b742-c516e267f9cc-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"cc806362-6764-4116-b742-c516e267f9cc\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 17:21:12.927576 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:21:12.927481    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/cc806362-6764-4116-b742-c516e267f9cc-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"cc806362-6764-4116-b742-c516e267f9cc\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 17:21:12.927576 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:21:12.927553    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/cc806362-6764-4116-b742-c516e267f9cc-config-volume\") pod \"alertmanager-main-0\" (UID: \"cc806362-6764-4116-b742-c516e267f9cc\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 17:21:12.927918 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:21:12.927895    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/cc806362-6764-4116-b742-c516e267f9cc-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"cc806362-6764-4116-b742-c516e267f9cc\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 17:21:12.929001 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:21:12.928971    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cc806362-6764-4116-b742-c516e267f9cc-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"cc806362-6764-4116-b742-c516e267f9cc\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 17:21:12.929116 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:21:12.929030    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/cc806362-6764-4116-b742-c516e267f9cc-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"cc806362-6764-4116-b742-c516e267f9cc\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 17:21:12.930300 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:21:12.930272    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/cc806362-6764-4116-b742-c516e267f9cc-config-out\") pod \"alertmanager-main-0\" (UID: \"cc806362-6764-4116-b742-c516e267f9cc\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 17:21:12.930657 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:21:12.930616    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/cc806362-6764-4116-b742-c516e267f9cc-tls-assets\") pod \"alertmanager-main-0\" (UID: \"cc806362-6764-4116-b742-c516e267f9cc\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 17:21:12.930960 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:21:12.930894    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/cc806362-6764-4116-b742-c516e267f9cc-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"cc806362-6764-4116-b742-c516e267f9cc\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 17:21:12.931185 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:21:12.931156    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/cc806362-6764-4116-b742-c516e267f9cc-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"cc806362-6764-4116-b742-c516e267f9cc\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 17:21:12.931355 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:21:12.931334    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/cc806362-6764-4116-b742-c516e267f9cc-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"cc806362-6764-4116-b742-c516e267f9cc\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 17:21:12.931404 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:21:12.931337    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/cc806362-6764-4116-b742-c516e267f9cc-web-config\") pod \"alertmanager-main-0\" (UID: \"cc806362-6764-4116-b742-c516e267f9cc\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 17:21:12.931814 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:21:12.931793    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/cc806362-6764-4116-b742-c516e267f9cc-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"cc806362-6764-4116-b742-c516e267f9cc\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 17:21:12.931907 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:21:12.931802    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/cc806362-6764-4116-b742-c516e267f9cc-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"cc806362-6764-4116-b742-c516e267f9cc\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 17:21:12.932126 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:21:12.932112    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/cc806362-6764-4116-b742-c516e267f9cc-config-volume\") pod \"alertmanager-main-0\" (UID: \"cc806362-6764-4116-b742-c516e267f9cc\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 17:21:12.935401 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:21:12.935379    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ljdnw\" (UniqueName: \"kubernetes.io/projected/cc806362-6764-4116-b742-c516e267f9cc-kube-api-access-ljdnw\") pod \"alertmanager-main-0\" (UID: \"cc806362-6764-4116-b742-c516e267f9cc\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 17:21:13.122428 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:21:13.122358    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Sep 11 17:21:13.407980 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:21:13.407945    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 11 17:21:13.409038 ip-10-0-141-139 kubenswrapper[2577]: W0911 17:21:13.409013    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcc806362_6764_4116_b742_c516e267f9cc.slice/crio-c40a1ef19d9c826d159679df05a59df9ff848be8697431c9c79733b86e12c07e WatchSource:0}: Error finding container c40a1ef19d9c826d159679df05a59df9ff848be8697431c9c79733b86e12c07e: Status 404 returned error can't find the container with id c40a1ef19d9c826d159679df05a59df9ff848be8697431c9c79733b86e12c07e
Sep 11 17:21:13.751059 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:21:13.751027    2577 generic.go:356] "Generic (PLEG): container finished" podID="cc806362-6764-4116-b742-c516e267f9cc" containerID="ac52ea6b308db741a39f9976109e20ace71c28f39b3790c2791bf142caa45366" exitCode=0
Sep 11 17:21:13.751440 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:21:13.751118    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"cc806362-6764-4116-b742-c516e267f9cc","Type":"ContainerDied","Data":"ac52ea6b308db741a39f9976109e20ace71c28f39b3790c2791bf142caa45366"}
Sep 11 17:21:13.751440 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:21:13.751194    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"cc806362-6764-4116-b742-c516e267f9cc","Type":"ContainerStarted","Data":"c40a1ef19d9c826d159679df05a59df9ff848be8697431c9c79733b86e12c07e"}
Sep 11 17:21:14.567033 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:21:14.566999    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 11 17:21:14.567313 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:21:14.567286    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="5fd69637-9132-4804-8a78-c24a626114ac" containerName="prometheus" containerID="cri-o://c8e220cba977043454c3a697b3d5103a838307813ec5e12dbf9d5a0dca863f00" gracePeriod=600
Sep 11 17:21:14.567388 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:21:14.567325    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="5fd69637-9132-4804-8a78-c24a626114ac" containerName="kube-rbac-proxy" containerID="cri-o://f4527bcc824da4ce1f425452b2656cd58345ded9ffdfc095445d688c156713d8" gracePeriod=600
Sep 11 17:21:14.567388 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:21:14.567365    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="5fd69637-9132-4804-8a78-c24a626114ac" containerName="kube-rbac-proxy-thanos" containerID="cri-o://8bcb4a57ee489421690089d725dbb3151aa118dd33caa6c76f908d98ea27983b" gracePeriod=600
Sep 11 17:21:14.567488 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:21:14.567342    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="5fd69637-9132-4804-8a78-c24a626114ac" containerName="config-reloader" containerID="cri-o://abb93082e2cd607833152e12ebb5876b1f5c585dcde9887119928424b818a4f4" gracePeriod=600
Sep 11 17:21:14.567488 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:21:14.567335    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="5fd69637-9132-4804-8a78-c24a626114ac" containerName="thanos-sidecar" containerID="cri-o://4461e9c87e7a5a370709ae9cbd84d391d356580798d90b66d29dc05f917568e5" gracePeriod=600
Sep 11 17:21:14.567488 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:21:14.567366    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="5fd69637-9132-4804-8a78-c24a626114ac" containerName="kube-rbac-proxy-web" containerID="cri-o://eba339879615deda7f50d8d1dd1ad1077d4ea017f76db24e87fb5a36968f79bb" gracePeriod=600
Sep 11 17:21:14.762552 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:21:14.762496    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"cc806362-6764-4116-b742-c516e267f9cc","Type":"ContainerStarted","Data":"a859a6b8b152d12e7888472408bfcbcfd36dcc69afac3f0c5f0bfdebd121cea3"}
Sep 11 17:21:14.762945 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:21:14.762602    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"cc806362-6764-4116-b742-c516e267f9cc","Type":"ContainerStarted","Data":"3468d239fb418a6bb06dd5be0e225abc192901d8418bbb239f836c8b30af3578"}
Sep 11 17:21:14.762945 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:21:14.762643    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"cc806362-6764-4116-b742-c516e267f9cc","Type":"ContainerStarted","Data":"55118e4c650d69ab4e9ded19213ac29984702d0054584e6f35f98792bb3ee949"}
Sep 11 17:21:14.762945 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:21:14.762684    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"cc806362-6764-4116-b742-c516e267f9cc","Type":"ContainerStarted","Data":"1f000040a5f70ba544b4c056c533bf6e7902942b812723cdeb3a16a62f7ecf94"}
Sep 11 17:21:14.762945 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:21:14.762722    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"cc806362-6764-4116-b742-c516e267f9cc","Type":"ContainerStarted","Data":"10ef5bd830a70a68e373dddea09334dce152b3d9ffc038c65808afd7ff1cff4e"}
Sep 11 17:21:14.762945 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:21:14.762759    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"cc806362-6764-4116-b742-c516e267f9cc","Type":"ContainerStarted","Data":"d46c55e06819eb04965d85f6605b95e95c742173973c409a56dcde19b7399e88"}
Sep 11 17:21:14.768287 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:21:14.768252    2577 generic.go:356] "Generic (PLEG): container finished" podID="5fd69637-9132-4804-8a78-c24a626114ac" containerID="8bcb4a57ee489421690089d725dbb3151aa118dd33caa6c76f908d98ea27983b" exitCode=0
Sep 11 17:21:14.768287 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:21:14.768284    2577 generic.go:356] "Generic (PLEG): container finished" podID="5fd69637-9132-4804-8a78-c24a626114ac" containerID="f4527bcc824da4ce1f425452b2656cd58345ded9ffdfc095445d688c156713d8" exitCode=0
Sep 11 17:21:14.768287 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:21:14.768296    2577 generic.go:356] "Generic (PLEG): container finished" podID="5fd69637-9132-4804-8a78-c24a626114ac" containerID="eba339879615deda7f50d8d1dd1ad1077d4ea017f76db24e87fb5a36968f79bb" exitCode=0
Sep 11 17:21:14.768641 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:21:14.768306    2577 generic.go:356] "Generic (PLEG): container finished" podID="5fd69637-9132-4804-8a78-c24a626114ac" containerID="4461e9c87e7a5a370709ae9cbd84d391d356580798d90b66d29dc05f917568e5" exitCode=0
Sep 11 17:21:14.768641 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:21:14.768318    2577 generic.go:356] "Generic (PLEG): container finished" podID="5fd69637-9132-4804-8a78-c24a626114ac" containerID="abb93082e2cd607833152e12ebb5876b1f5c585dcde9887119928424b818a4f4" exitCode=0
Sep 11 17:21:14.768641 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:21:14.768332    2577 generic.go:356] "Generic (PLEG): container finished" podID="5fd69637-9132-4804-8a78-c24a626114ac" containerID="c8e220cba977043454c3a697b3d5103a838307813ec5e12dbf9d5a0dca863f00" exitCode=0
Sep 11 17:21:14.768641 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:21:14.768337    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"5fd69637-9132-4804-8a78-c24a626114ac","Type":"ContainerDied","Data":"8bcb4a57ee489421690089d725dbb3151aa118dd33caa6c76f908d98ea27983b"}
Sep 11 17:21:14.768641 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:21:14.768458    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"5fd69637-9132-4804-8a78-c24a626114ac","Type":"ContainerDied","Data":"f4527bcc824da4ce1f425452b2656cd58345ded9ffdfc095445d688c156713d8"}
Sep 11 17:21:14.768641 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:21:14.768552    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"5fd69637-9132-4804-8a78-c24a626114ac","Type":"ContainerDied","Data":"eba339879615deda7f50d8d1dd1ad1077d4ea017f76db24e87fb5a36968f79bb"}
Sep 11 17:21:14.768641 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:21:14.768598    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"5fd69637-9132-4804-8a78-c24a626114ac","Type":"ContainerDied","Data":"4461e9c87e7a5a370709ae9cbd84d391d356580798d90b66d29dc05f917568e5"}
Sep 11 17:21:14.768641 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:21:14.768641    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"5fd69637-9132-4804-8a78-c24a626114ac","Type":"ContainerDied","Data":"abb93082e2cd607833152e12ebb5876b1f5c585dcde9887119928424b818a4f4"}
Sep 11 17:21:14.769036 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:21:14.768679    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"5fd69637-9132-4804-8a78-c24a626114ac","Type":"ContainerDied","Data":"c8e220cba977043454c3a697b3d5103a838307813ec5e12dbf9d5a0dca863f00"}
Sep 11 17:21:14.792605 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:21:14.792543    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=2.777929651 podStartE2EDuration="2.792517622s" totalImagesPullingTime="0s" totalInitContainerRuntime="14.587971ms" isStatefulPod=false podCreationTimestamp="2026-09-11 17:21:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 17:21:14.790378236 +0000 UTC m=+152.363598349" watchObservedRunningTime="2026-09-11 17:21:14.792517622 +0000 UTC m=+152.365737726"
Sep 11 17:21:14.836862 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:21:14.836832    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 17:21:14.945773 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:21:14.945743    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rcfgn\" (UniqueName: \"kubernetes.io/projected/5fd69637-9132-4804-8a78-c24a626114ac-kube-api-access-rcfgn\") pod \"5fd69637-9132-4804-8a78-c24a626114ac\" (UID: \"5fd69637-9132-4804-8a78-c24a626114ac\") "
Sep 11 17:21:14.945949 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:21:14.945783    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5fd69637-9132-4804-8a78-c24a626114ac-configmap-serving-certs-ca-bundle\") pod \"5fd69637-9132-4804-8a78-c24a626114ac\" (UID: \"5fd69637-9132-4804-8a78-c24a626114ac\") "
Sep 11 17:21:14.945949 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:21:14.945802    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/5fd69637-9132-4804-8a78-c24a626114ac-prometheus-k8s-rulefiles-0\") pod \"5fd69637-9132-4804-8a78-c24a626114ac\" (UID: \"5fd69637-9132-4804-8a78-c24a626114ac\") "
Sep 11 17:21:14.945949 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:21:14.945841    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/5fd69637-9132-4804-8a78-c24a626114ac-prometheus-k8s-db\") pod \"5fd69637-9132-4804-8a78-c24a626114ac\" (UID: \"5fd69637-9132-4804-8a78-c24a626114ac\") "
Sep 11 17:21:14.946118 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:21:14.945950    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/5fd69637-9132-4804-8a78-c24a626114ac-config-out\") pod \"5fd69637-9132-4804-8a78-c24a626114ac\" (UID: \"5fd69637-9132-4804-8a78-c24a626114ac\") "
Sep 11 17:21:14.946118 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:21:14.946016    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/5fd69637-9132-4804-8a78-c24a626114ac-configmap-metrics-client-ca\") pod \"5fd69637-9132-4804-8a78-c24a626114ac\" (UID: \"5fd69637-9132-4804-8a78-c24a626114ac\") "
Sep 11 17:21:14.946118 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:21:14.946049    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5fd69637-9132-4804-8a78-c24a626114ac-configmap-kubelet-serving-ca-bundle\") pod \"5fd69637-9132-4804-8a78-c24a626114ac\" (UID: \"5fd69637-9132-4804-8a78-c24a626114ac\") "
Sep 11 17:21:14.946118 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:21:14.946083    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/5fd69637-9132-4804-8a78-c24a626114ac-prometheus-k8s-rulefiles-1\") pod \"5fd69637-9132-4804-8a78-c24a626114ac\" (UID: \"5fd69637-9132-4804-8a78-c24a626114ac\") "
Sep 11 17:21:14.946118 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:21:14.946107    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/5fd69637-9132-4804-8a78-c24a626114ac-tls-assets\") pod \"5fd69637-9132-4804-8a78-c24a626114ac\" (UID: \"5fd69637-9132-4804-8a78-c24a626114ac\") "
Sep 11 17:21:14.946352 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:21:14.946153    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/5fd69637-9132-4804-8a78-c24a626114ac-secret-grpc-tls\") pod \"5fd69637-9132-4804-8a78-c24a626114ac\" (UID: \"5fd69637-9132-4804-8a78-c24a626114ac\") "
Sep 11 17:21:14.946352 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:21:14.946181    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/5fd69637-9132-4804-8a78-c24a626114ac-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"5fd69637-9132-4804-8a78-c24a626114ac\" (UID: \"5fd69637-9132-4804-8a78-c24a626114ac\") "
Sep 11 17:21:14.946352 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:21:14.946184    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5fd69637-9132-4804-8a78-c24a626114ac-configmap-serving-certs-ca-bundle" (OuterVolumeSpecName: "configmap-serving-certs-ca-bundle") pod "5fd69637-9132-4804-8a78-c24a626114ac" (UID: "5fd69637-9132-4804-8a78-c24a626114ac"). InnerVolumeSpecName "configmap-serving-certs-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 17:21:14.946352 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:21:14.946204    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/5fd69637-9132-4804-8a78-c24a626114ac-secret-metrics-client-certs\") pod \"5fd69637-9132-4804-8a78-c24a626114ac\" (UID: \"5fd69637-9132-4804-8a78-c24a626114ac\") "
Sep 11 17:21:14.946352 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:21:14.946228    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/5fd69637-9132-4804-8a78-c24a626114ac-secret-kube-rbac-proxy\") pod \"5fd69637-9132-4804-8a78-c24a626114ac\" (UID: \"5fd69637-9132-4804-8a78-c24a626114ac\") "
Sep 11 17:21:14.946352 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:21:14.946268    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/5fd69637-9132-4804-8a78-c24a626114ac-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"5fd69637-9132-4804-8a78-c24a626114ac\" (UID: \"5fd69637-9132-4804-8a78-c24a626114ac\") "
Sep 11 17:21:14.946352 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:21:14.946298    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/5fd69637-9132-4804-8a78-c24a626114ac-prometheus-k8s-rulefiles-2\") pod \"5fd69637-9132-4804-8a78-c24a626114ac\" (UID: \"5fd69637-9132-4804-8a78-c24a626114ac\") "
Sep 11 17:21:14.946352 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:21:14.946325    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5fd69637-9132-4804-8a78-c24a626114ac-prometheus-trusted-ca-bundle\") pod \"5fd69637-9132-4804-8a78-c24a626114ac\" (UID: \"5fd69637-9132-4804-8a78-c24a626114ac\") "
Sep 11 17:21:14.947369 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:21:14.946365    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/5fd69637-9132-4804-8a78-c24a626114ac-secret-prometheus-k8s-tls\") pod \"5fd69637-9132-4804-8a78-c24a626114ac\" (UID: \"5fd69637-9132-4804-8a78-c24a626114ac\") "
Sep 11 17:21:14.947369 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:21:14.946408    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/5fd69637-9132-4804-8a78-c24a626114ac-web-config\") pod \"5fd69637-9132-4804-8a78-c24a626114ac\" (UID: \"5fd69637-9132-4804-8a78-c24a626114ac\") "
Sep 11 17:21:14.947369 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:21:14.946435    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/5fd69637-9132-4804-8a78-c24a626114ac-thanos-prometheus-http-client-file\") pod \"5fd69637-9132-4804-8a78-c24a626114ac\" (UID: \"5fd69637-9132-4804-8a78-c24a626114ac\") "
Sep 11 17:21:14.947369 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:21:14.946436    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5fd69637-9132-4804-8a78-c24a626114ac-configmap-metrics-client-ca" (OuterVolumeSpecName: "configmap-metrics-client-ca") pod "5fd69637-9132-4804-8a78-c24a626114ac" (UID: "5fd69637-9132-4804-8a78-c24a626114ac"). InnerVolumeSpecName "configmap-metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 17:21:14.947369 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:21:14.946495    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/5fd69637-9132-4804-8a78-c24a626114ac-config\") pod \"5fd69637-9132-4804-8a78-c24a626114ac\" (UID: \"5fd69637-9132-4804-8a78-c24a626114ac\") "
Sep 11 17:21:14.947369 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:21:14.946811    2577 reconciler_common.go:299] "Volume detached for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5fd69637-9132-4804-8a78-c24a626114ac-configmap-serving-certs-ca-bundle\") on node \"ip-10-0-141-139.ec2.internal\" DevicePath \"\""
Sep 11 17:21:14.947369 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:21:14.946834    2577 reconciler_common.go:299] "Volume detached for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/5fd69637-9132-4804-8a78-c24a626114ac-configmap-metrics-client-ca\") on node \"ip-10-0-141-139.ec2.internal\" DevicePath \"\""
Sep 11 17:21:14.947837 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:21:14.947428    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5fd69637-9132-4804-8a78-c24a626114ac-prometheus-k8s-rulefiles-0" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-0") pod "5fd69637-9132-4804-8a78-c24a626114ac" (UID: "5fd69637-9132-4804-8a78-c24a626114ac"). InnerVolumeSpecName "prometheus-k8s-rulefiles-0". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 17:21:14.947837 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:21:14.947548    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5fd69637-9132-4804-8a78-c24a626114ac-configmap-kubelet-serving-ca-bundle" (OuterVolumeSpecName: "configmap-kubelet-serving-ca-bundle") pod "5fd69637-9132-4804-8a78-c24a626114ac" (UID: "5fd69637-9132-4804-8a78-c24a626114ac"). InnerVolumeSpecName "configmap-kubelet-serving-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 17:21:14.947837 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:21:14.947771    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5fd69637-9132-4804-8a78-c24a626114ac-prometheus-k8s-rulefiles-1" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-1") pod "5fd69637-9132-4804-8a78-c24a626114ac" (UID: "5fd69637-9132-4804-8a78-c24a626114ac"). InnerVolumeSpecName "prometheus-k8s-rulefiles-1". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 17:21:14.948184 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:21:14.948153    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5fd69637-9132-4804-8a78-c24a626114ac-prometheus-k8s-db" (OuterVolumeSpecName: "prometheus-k8s-db") pod "5fd69637-9132-4804-8a78-c24a626114ac" (UID: "5fd69637-9132-4804-8a78-c24a626114ac"). InnerVolumeSpecName "prometheus-k8s-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 17:21:14.949162 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:21:14.948876    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5fd69637-9132-4804-8a78-c24a626114ac-prometheus-trusted-ca-bundle" (OuterVolumeSpecName: "prometheus-trusted-ca-bundle") pod "5fd69637-9132-4804-8a78-c24a626114ac" (UID: "5fd69637-9132-4804-8a78-c24a626114ac"). InnerVolumeSpecName "prometheus-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 17:21:14.950771 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:21:14.949942    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5fd69637-9132-4804-8a78-c24a626114ac-prometheus-k8s-rulefiles-2" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-2") pod "5fd69637-9132-4804-8a78-c24a626114ac" (UID: "5fd69637-9132-4804-8a78-c24a626114ac"). InnerVolumeSpecName "prometheus-k8s-rulefiles-2". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 17:21:14.950771 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:21:14.950050    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5fd69637-9132-4804-8a78-c24a626114ac-tls-assets" (OuterVolumeSpecName: "tls-assets") pod "5fd69637-9132-4804-8a78-c24a626114ac" (UID: "5fd69637-9132-4804-8a78-c24a626114ac"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 17:21:14.950771 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:21:14.950526    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5fd69637-9132-4804-8a78-c24a626114ac-kube-api-access-rcfgn" (OuterVolumeSpecName: "kube-api-access-rcfgn") pod "5fd69637-9132-4804-8a78-c24a626114ac" (UID: "5fd69637-9132-4804-8a78-c24a626114ac"). InnerVolumeSpecName "kube-api-access-rcfgn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 17:21:14.950771 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:21:14.950748    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5fd69637-9132-4804-8a78-c24a626114ac-config-out" (OuterVolumeSpecName: "config-out") pod "5fd69637-9132-4804-8a78-c24a626114ac" (UID: "5fd69637-9132-4804-8a78-c24a626114ac"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 17:21:14.951247 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:21:14.951013    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5fd69637-9132-4804-8a78-c24a626114ac-thanos-prometheus-http-client-file" (OuterVolumeSpecName: "thanos-prometheus-http-client-file") pod "5fd69637-9132-4804-8a78-c24a626114ac" (UID: "5fd69637-9132-4804-8a78-c24a626114ac"). InnerVolumeSpecName "thanos-prometheus-http-client-file". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 17:21:14.951247 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:21:14.951147    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5fd69637-9132-4804-8a78-c24a626114ac-secret-grpc-tls" (OuterVolumeSpecName: "secret-grpc-tls") pod "5fd69637-9132-4804-8a78-c24a626114ac" (UID: "5fd69637-9132-4804-8a78-c24a626114ac"). InnerVolumeSpecName "secret-grpc-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 17:21:14.951363 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:21:14.951304    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5fd69637-9132-4804-8a78-c24a626114ac-config" (OuterVolumeSpecName: "config") pod "5fd69637-9132-4804-8a78-c24a626114ac" (UID: "5fd69637-9132-4804-8a78-c24a626114ac"). InnerVolumeSpecName "config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 17:21:14.952013 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:21:14.951978    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5fd69637-9132-4804-8a78-c24a626114ac-secret-prometheus-k8s-kube-rbac-proxy-web" (OuterVolumeSpecName: "secret-prometheus-k8s-kube-rbac-proxy-web") pod "5fd69637-9132-4804-8a78-c24a626114ac" (UID: "5fd69637-9132-4804-8a78-c24a626114ac"). InnerVolumeSpecName "secret-prometheus-k8s-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 17:21:14.952174 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:21:14.952139    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5fd69637-9132-4804-8a78-c24a626114ac-secret-prometheus-k8s-tls" (OuterVolumeSpecName: "secret-prometheus-k8s-tls") pod "5fd69637-9132-4804-8a78-c24a626114ac" (UID: "5fd69637-9132-4804-8a78-c24a626114ac"). InnerVolumeSpecName "secret-prometheus-k8s-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 17:21:14.954873 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:21:14.953253    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5fd69637-9132-4804-8a78-c24a626114ac-secret-metrics-client-certs" (OuterVolumeSpecName: "secret-metrics-client-certs") pod "5fd69637-9132-4804-8a78-c24a626114ac" (UID: "5fd69637-9132-4804-8a78-c24a626114ac"). InnerVolumeSpecName "secret-metrics-client-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 17:21:14.954873 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:21:14.953331    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5fd69637-9132-4804-8a78-c24a626114ac-secret-prometheus-k8s-thanos-sidecar-tls" (OuterVolumeSpecName: "secret-prometheus-k8s-thanos-sidecar-tls") pod "5fd69637-9132-4804-8a78-c24a626114ac" (UID: "5fd69637-9132-4804-8a78-c24a626114ac"). InnerVolumeSpecName "secret-prometheus-k8s-thanos-sidecar-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 17:21:14.955119 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:21:14.955093    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5fd69637-9132-4804-8a78-c24a626114ac-secret-kube-rbac-proxy" (OuterVolumeSpecName: "secret-kube-rbac-proxy") pod "5fd69637-9132-4804-8a78-c24a626114ac" (UID: "5fd69637-9132-4804-8a78-c24a626114ac"). InnerVolumeSpecName "secret-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 17:21:14.963212 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:21:14.963177    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5fd69637-9132-4804-8a78-c24a626114ac-web-config" (OuterVolumeSpecName: "web-config") pod "5fd69637-9132-4804-8a78-c24a626114ac" (UID: "5fd69637-9132-4804-8a78-c24a626114ac"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 17:21:15.047439 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:21:15.047398    2577 reconciler_common.go:299] "Volume detached for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/5fd69637-9132-4804-8a78-c24a626114ac-secret-grpc-tls\") on node \"ip-10-0-141-139.ec2.internal\" DevicePath \"\""
Sep 11 17:21:15.047439 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:21:15.047434    2577 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/5fd69637-9132-4804-8a78-c24a626114ac-secret-prometheus-k8s-thanos-sidecar-tls\") on node \"ip-10-0-141-139.ec2.internal\" DevicePath \"\""
Sep 11 17:21:15.047439 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:21:15.047444    2577 reconciler_common.go:299] "Volume detached for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/5fd69637-9132-4804-8a78-c24a626114ac-secret-metrics-client-certs\") on node \"ip-10-0-141-139.ec2.internal\" DevicePath \"\""
Sep 11 17:21:15.047703 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:21:15.047455    2577 reconciler_common.go:299] "Volume detached for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/5fd69637-9132-4804-8a78-c24a626114ac-secret-kube-rbac-proxy\") on node \"ip-10-0-141-139.ec2.internal\" DevicePath \"\""
Sep 11 17:21:15.047703 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:21:15.047465    2577 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/5fd69637-9132-4804-8a78-c24a626114ac-secret-prometheus-k8s-kube-rbac-proxy-web\") on node \"ip-10-0-141-139.ec2.internal\" DevicePath \"\""
Sep 11 17:21:15.047703 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:21:15.047473    2577 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/5fd69637-9132-4804-8a78-c24a626114ac-prometheus-k8s-rulefiles-2\") on node \"ip-10-0-141-139.ec2.internal\" DevicePath \"\""
Sep 11 17:21:15.047703 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:21:15.047482    2577 reconciler_common.go:299] "Volume detached for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5fd69637-9132-4804-8a78-c24a626114ac-prometheus-trusted-ca-bundle\") on node \"ip-10-0-141-139.ec2.internal\" DevicePath \"\""
Sep 11 17:21:15.047703 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:21:15.047490    2577 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/5fd69637-9132-4804-8a78-c24a626114ac-secret-prometheus-k8s-tls\") on node \"ip-10-0-141-139.ec2.internal\" DevicePath \"\""
Sep 11 17:21:15.047703 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:21:15.047529    2577 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/5fd69637-9132-4804-8a78-c24a626114ac-web-config\") on node \"ip-10-0-141-139.ec2.internal\" DevicePath \"\""
Sep 11 17:21:15.047703 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:21:15.047542    2577 reconciler_common.go:299] "Volume detached for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/5fd69637-9132-4804-8a78-c24a626114ac-thanos-prometheus-http-client-file\") on node \"ip-10-0-141-139.ec2.internal\" DevicePath \"\""
Sep 11 17:21:15.047703 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:21:15.047554    2577 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/secret/5fd69637-9132-4804-8a78-c24a626114ac-config\") on node \"ip-10-0-141-139.ec2.internal\" DevicePath \"\""
Sep 11 17:21:15.047703 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:21:15.047567    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rcfgn\" (UniqueName: \"kubernetes.io/projected/5fd69637-9132-4804-8a78-c24a626114ac-kube-api-access-rcfgn\") on node \"ip-10-0-141-139.ec2.internal\" DevicePath \"\""
Sep 11 17:21:15.047703 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:21:15.047581    2577 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/5fd69637-9132-4804-8a78-c24a626114ac-prometheus-k8s-rulefiles-0\") on node \"ip-10-0-141-139.ec2.internal\" DevicePath \"\""
Sep 11 17:21:15.047703 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:21:15.047592    2577 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/5fd69637-9132-4804-8a78-c24a626114ac-prometheus-k8s-db\") on node \"ip-10-0-141-139.ec2.internal\" DevicePath \"\""
Sep 11 17:21:15.047703 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:21:15.047600    2577 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/5fd69637-9132-4804-8a78-c24a626114ac-config-out\") on node \"ip-10-0-141-139.ec2.internal\" DevicePath \"\""
Sep 11 17:21:15.047703 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:21:15.047609    2577 reconciler_common.go:299] "Volume detached for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5fd69637-9132-4804-8a78-c24a626114ac-configmap-kubelet-serving-ca-bundle\") on node \"ip-10-0-141-139.ec2.internal\" DevicePath \"\""
Sep 11 17:21:15.047703 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:21:15.047617    2577 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/5fd69637-9132-4804-8a78-c24a626114ac-prometheus-k8s-rulefiles-1\") on node \"ip-10-0-141-139.ec2.internal\" DevicePath \"\""
Sep 11 17:21:15.047703 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:21:15.047625    2577 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/5fd69637-9132-4804-8a78-c24a626114ac-tls-assets\") on node \"ip-10-0-141-139.ec2.internal\" DevicePath \"\""
Sep 11 17:21:15.776225 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:21:15.776197    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 17:21:15.776225 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:21:15.776204    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"5fd69637-9132-4804-8a78-c24a626114ac","Type":"ContainerDied","Data":"3facfe5208fdc61e6436842dfdee7e771f3cff11368ffde44922965ff1dd6be3"}
Sep 11 17:21:15.776728 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:21:15.776300    2577 scope.go:117] "RemoveContainer" containerID="8bcb4a57ee489421690089d725dbb3151aa118dd33caa6c76f908d98ea27983b"
Sep 11 17:21:15.784772 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:21:15.784754    2577 scope.go:117] "RemoveContainer" containerID="f4527bcc824da4ce1f425452b2656cd58345ded9ffdfc095445d688c156713d8"
Sep 11 17:21:15.792884 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:21:15.792865    2577 scope.go:117] "RemoveContainer" containerID="eba339879615deda7f50d8d1dd1ad1077d4ea017f76db24e87fb5a36968f79bb"
Sep 11 17:21:15.800629 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:21:15.800608    2577 scope.go:117] "RemoveContainer" containerID="4461e9c87e7a5a370709ae9cbd84d391d356580798d90b66d29dc05f917568e5"
Sep 11 17:21:15.801584 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:21:15.801558    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 11 17:21:15.806919 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:21:15.806894    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 11 17:21:15.809097 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:21:15.809073    2577 scope.go:117] "RemoveContainer" containerID="abb93082e2cd607833152e12ebb5876b1f5c585dcde9887119928424b818a4f4"
Sep 11 17:21:15.816292 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:21:15.816274    2577 scope.go:117] "RemoveContainer" containerID="c8e220cba977043454c3a697b3d5103a838307813ec5e12dbf9d5a0dca863f00"
Sep 11 17:21:15.823825 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:21:15.823810    2577 scope.go:117] "RemoveContainer" containerID="1a6761dccea5c2043447fe170447e2694f286770a6b87a3291619d817bc0333e"
Sep 11 17:21:15.846518 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:21:15.846483    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 11 17:21:15.846810 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:21:15.846798    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5fd69637-9132-4804-8a78-c24a626114ac" containerName="prometheus"
Sep 11 17:21:15.846848 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:21:15.846812    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="5fd69637-9132-4804-8a78-c24a626114ac" containerName="prometheus"
Sep 11 17:21:15.846848 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:21:15.846823    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5fd69637-9132-4804-8a78-c24a626114ac" containerName="kube-rbac-proxy"
Sep 11 17:21:15.846848 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:21:15.846828    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="5fd69637-9132-4804-8a78-c24a626114ac" containerName="kube-rbac-proxy"
Sep 11 17:21:15.846848 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:21:15.846835    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5fd69637-9132-4804-8a78-c24a626114ac" containerName="init-config-reloader"
Sep 11 17:21:15.846848 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:21:15.846840    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="5fd69637-9132-4804-8a78-c24a626114ac" containerName="init-config-reloader"
Sep 11 17:21:15.846992 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:21:15.846855    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5fd69637-9132-4804-8a78-c24a626114ac" containerName="config-reloader"
Sep 11 17:21:15.846992 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:21:15.846860    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="5fd69637-9132-4804-8a78-c24a626114ac" containerName="config-reloader"
Sep 11 17:21:15.846992 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:21:15.846867    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5fd69637-9132-4804-8a78-c24a626114ac" containerName="kube-rbac-proxy-web"
Sep 11 17:21:15.846992 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:21:15.846872    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="5fd69637-9132-4804-8a78-c24a626114ac" containerName="kube-rbac-proxy-web"
Sep 11 17:21:15.846992 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:21:15.846880    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5fd69637-9132-4804-8a78-c24a626114ac" containerName="kube-rbac-proxy-thanos"
Sep 11 17:21:15.846992 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:21:15.846885    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="5fd69637-9132-4804-8a78-c24a626114ac" containerName="kube-rbac-proxy-thanos"
Sep 11 17:21:15.846992 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:21:15.846891    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5fd69637-9132-4804-8a78-c24a626114ac" containerName="thanos-sidecar"
Sep 11 17:21:15.846992 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:21:15.846895    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="5fd69637-9132-4804-8a78-c24a626114ac" containerName="thanos-sidecar"
Sep 11 17:21:15.846992 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:21:15.846943    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="5fd69637-9132-4804-8a78-c24a626114ac" containerName="config-reloader"
Sep 11 17:21:15.846992 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:21:15.846950    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="5fd69637-9132-4804-8a78-c24a626114ac" containerName="kube-rbac-proxy-thanos"
Sep 11 17:21:15.846992 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:21:15.846956    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="5fd69637-9132-4804-8a78-c24a626114ac" containerName="kube-rbac-proxy-web"
Sep 11 17:21:15.846992 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:21:15.846961    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="5fd69637-9132-4804-8a78-c24a626114ac" containerName="thanos-sidecar"
Sep 11 17:21:15.846992 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:21:15.846967    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="5fd69637-9132-4804-8a78-c24a626114ac" containerName="prometheus"
Sep 11 17:21:15.846992 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:21:15.846972    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="5fd69637-9132-4804-8a78-c24a626114ac" containerName="kube-rbac-proxy"
Sep 11 17:21:15.850729 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:21:15.850713    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 17:21:15.853944 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:21:15.853920    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\""
Sep 11 17:21:15.854061 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:21:15.853987    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\""
Sep 11 17:21:15.854322 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:21:15.854302    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\""
Sep 11 17:21:15.854478 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:21:15.854372    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\""
Sep 11 17:21:15.854478 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:21:15.854415    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\""
Sep 11 17:21:15.854478 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:21:15.854434    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-827rr\""
Sep 11 17:21:15.854478 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:21:15.854444    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\""
Sep 11 17:21:15.854478 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:21:15.854479    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\""
Sep 11 17:21:15.854766 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:21:15.854592    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-7jkijutllc784\""
Sep 11 17:21:15.854766 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:21:15.854745    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\""
Sep 11 17:21:15.854766 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:21:15.854747    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\""
Sep 11 17:21:15.854891 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:21:15.854875    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\""
Sep 11 17:21:15.855790 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:21:15.855527    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\""
Sep 11 17:21:15.855790 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:21:15.855530    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\""
Sep 11 17:21:15.866842 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:21:15.866697    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\""
Sep 11 17:21:15.868049 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:21:15.868025    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\""
Sep 11 17:21:15.869076 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:21:15.869056    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 11 17:21:15.954644 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:21:15.954608    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d669874a-e27f-4a06-af94-6cbb47fb7b57-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"d669874a-e27f-4a06-af94-6cbb47fb7b57\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 17:21:15.954644 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:21:15.954647    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/d669874a-e27f-4a06-af94-6cbb47fb7b57-config\") pod \"prometheus-k8s-0\" (UID: \"d669874a-e27f-4a06-af94-6cbb47fb7b57\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 17:21:15.954885 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:21:15.954666    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/d669874a-e27f-4a06-af94-6cbb47fb7b57-web-config\") pod \"prometheus-k8s-0\" (UID: \"d669874a-e27f-4a06-af94-6cbb47fb7b57\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 17:21:15.954885 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:21:15.954688    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/d669874a-e27f-4a06-af94-6cbb47fb7b57-config-out\") pod \"prometheus-k8s-0\" (UID: \"d669874a-e27f-4a06-af94-6cbb47fb7b57\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 17:21:15.954885 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:21:15.954744    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/d669874a-e27f-4a06-af94-6cbb47fb7b57-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"d669874a-e27f-4a06-af94-6cbb47fb7b57\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 17:21:15.954885 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:21:15.954829    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/d669874a-e27f-4a06-af94-6cbb47fb7b57-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"d669874a-e27f-4a06-af94-6cbb47fb7b57\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 17:21:15.954885 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:21:15.954858    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hlmpg\" (UniqueName: \"kubernetes.io/projected/d669874a-e27f-4a06-af94-6cbb47fb7b57-kube-api-access-hlmpg\") pod \"prometheus-k8s-0\" (UID: \"d669874a-e27f-4a06-af94-6cbb47fb7b57\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 17:21:15.955134 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:21:15.954893    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/d669874a-e27f-4a06-af94-6cbb47fb7b57-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"d669874a-e27f-4a06-af94-6cbb47fb7b57\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 17:21:15.955134 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:21:15.954916    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/d669874a-e27f-4a06-af94-6cbb47fb7b57-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"d669874a-e27f-4a06-af94-6cbb47fb7b57\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 17:21:15.955134 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:21:15.954942    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/d669874a-e27f-4a06-af94-6cbb47fb7b57-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"d669874a-e27f-4a06-af94-6cbb47fb7b57\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 17:21:15.955134 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:21:15.955021    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/d669874a-e27f-4a06-af94-6cbb47fb7b57-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"d669874a-e27f-4a06-af94-6cbb47fb7b57\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 17:21:15.955134 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:21:15.955045    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/d669874a-e27f-4a06-af94-6cbb47fb7b57-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"d669874a-e27f-4a06-af94-6cbb47fb7b57\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 17:21:15.955134 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:21:15.955069    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/d669874a-e27f-4a06-af94-6cbb47fb7b57-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"d669874a-e27f-4a06-af94-6cbb47fb7b57\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 17:21:15.955134 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:21:15.955105    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/d669874a-e27f-4a06-af94-6cbb47fb7b57-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"d669874a-e27f-4a06-af94-6cbb47fb7b57\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 17:21:15.955384 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:21:15.955141    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/d669874a-e27f-4a06-af94-6cbb47fb7b57-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"d669874a-e27f-4a06-af94-6cbb47fb7b57\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 17:21:15.955384 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:21:15.955165    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d669874a-e27f-4a06-af94-6cbb47fb7b57-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"d669874a-e27f-4a06-af94-6cbb47fb7b57\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 17:21:15.955384 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:21:15.955183    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/d669874a-e27f-4a06-af94-6cbb47fb7b57-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"d669874a-e27f-4a06-af94-6cbb47fb7b57\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 17:21:15.955384 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:21:15.955212    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/d669874a-e27f-4a06-af94-6cbb47fb7b57-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"d669874a-e27f-4a06-af94-6cbb47fb7b57\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 17:21:15.955384 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:21:15.955243    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d669874a-e27f-4a06-af94-6cbb47fb7b57-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"d669874a-e27f-4a06-af94-6cbb47fb7b57\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 17:21:15.955384 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:21:15.955271    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d669874a-e27f-4a06-af94-6cbb47fb7b57-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"d669874a-e27f-4a06-af94-6cbb47fb7b57\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 17:21:16.055919 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:21:16.055831    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/d669874a-e27f-4a06-af94-6cbb47fb7b57-config-out\") pod \"prometheus-k8s-0\" (UID: \"d669874a-e27f-4a06-af94-6cbb47fb7b57\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 17:21:16.055919 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:21:16.055868    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/d669874a-e27f-4a06-af94-6cbb47fb7b57-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"d669874a-e27f-4a06-af94-6cbb47fb7b57\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 17:21:16.055919 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:21:16.055898    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/d669874a-e27f-4a06-af94-6cbb47fb7b57-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"d669874a-e27f-4a06-af94-6cbb47fb7b57\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 17:21:16.055919 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:21:16.055917    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hlmpg\" (UniqueName: \"kubernetes.io/projected/d669874a-e27f-4a06-af94-6cbb47fb7b57-kube-api-access-hlmpg\") pod \"prometheus-k8s-0\" (UID: \"d669874a-e27f-4a06-af94-6cbb47fb7b57\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 17:21:16.056231 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:21:16.055946    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/d669874a-e27f-4a06-af94-6cbb47fb7b57-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"d669874a-e27f-4a06-af94-6cbb47fb7b57\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 17:21:16.056231 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:21:16.055971    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/d669874a-e27f-4a06-af94-6cbb47fb7b57-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"d669874a-e27f-4a06-af94-6cbb47fb7b57\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 17:21:16.056231 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:21:16.055996    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/d669874a-e27f-4a06-af94-6cbb47fb7b57-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"d669874a-e27f-4a06-af94-6cbb47fb7b57\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 17:21:16.056231 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:21:16.056043    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/d669874a-e27f-4a06-af94-6cbb47fb7b57-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"d669874a-e27f-4a06-af94-6cbb47fb7b57\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 17:21:16.056231 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:21:16.056066    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/d669874a-e27f-4a06-af94-6cbb47fb7b57-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"d669874a-e27f-4a06-af94-6cbb47fb7b57\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 17:21:16.056231 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:21:16.056086    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/d669874a-e27f-4a06-af94-6cbb47fb7b57-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"d669874a-e27f-4a06-af94-6cbb47fb7b57\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 17:21:16.056231 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:21:16.056111    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/d669874a-e27f-4a06-af94-6cbb47fb7b57-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"d669874a-e27f-4a06-af94-6cbb47fb7b57\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 17:21:16.056231 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:21:16.056133    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/d669874a-e27f-4a06-af94-6cbb47fb7b57-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"d669874a-e27f-4a06-af94-6cbb47fb7b57\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 17:21:16.056231 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:21:16.056157    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d669874a-e27f-4a06-af94-6cbb47fb7b57-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"d669874a-e27f-4a06-af94-6cbb47fb7b57\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 17:21:16.056231 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:21:16.056177    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/d669874a-e27f-4a06-af94-6cbb47fb7b57-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"d669874a-e27f-4a06-af94-6cbb47fb7b57\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 17:21:16.056231 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:21:16.056203    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/d669874a-e27f-4a06-af94-6cbb47fb7b57-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"d669874a-e27f-4a06-af94-6cbb47fb7b57\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 17:21:16.056231 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:21:16.056226    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d669874a-e27f-4a06-af94-6cbb47fb7b57-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"d669874a-e27f-4a06-af94-6cbb47fb7b57\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 17:21:16.057006 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:21:16.056248    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d669874a-e27f-4a06-af94-6cbb47fb7b57-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"d669874a-e27f-4a06-af94-6cbb47fb7b57\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 17:21:16.057006 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:21:16.056313    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d669874a-e27f-4a06-af94-6cbb47fb7b57-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"d669874a-e27f-4a06-af94-6cbb47fb7b57\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 17:21:16.057006 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:21:16.056342    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/d669874a-e27f-4a06-af94-6cbb47fb7b57-config\") pod \"prometheus-k8s-0\" (UID: \"d669874a-e27f-4a06-af94-6cbb47fb7b57\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 17:21:16.057006 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:21:16.056363    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/d669874a-e27f-4a06-af94-6cbb47fb7b57-web-config\") pod \"prometheus-k8s-0\" (UID: \"d669874a-e27f-4a06-af94-6cbb47fb7b57\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 17:21:16.057006 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:21:16.056392    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/d669874a-e27f-4a06-af94-6cbb47fb7b57-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"d669874a-e27f-4a06-af94-6cbb47fb7b57\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 17:21:16.057006 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:21:16.056831    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/d669874a-e27f-4a06-af94-6cbb47fb7b57-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"d669874a-e27f-4a06-af94-6cbb47fb7b57\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 17:21:16.057920 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:21:16.057891    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d669874a-e27f-4a06-af94-6cbb47fb7b57-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"d669874a-e27f-4a06-af94-6cbb47fb7b57\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 17:21:16.058058 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:21:16.058033    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d669874a-e27f-4a06-af94-6cbb47fb7b57-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"d669874a-e27f-4a06-af94-6cbb47fb7b57\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 17:21:16.058126 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:21:16.058104    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/d669874a-e27f-4a06-af94-6cbb47fb7b57-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"d669874a-e27f-4a06-af94-6cbb47fb7b57\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 17:21:16.058271 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:21:16.058245    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d669874a-e27f-4a06-af94-6cbb47fb7b57-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"d669874a-e27f-4a06-af94-6cbb47fb7b57\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 17:21:16.058404 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:21:16.058286    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d669874a-e27f-4a06-af94-6cbb47fb7b57-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"d669874a-e27f-4a06-af94-6cbb47fb7b57\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 17:21:16.060022 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:21:16.059962    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/d669874a-e27f-4a06-af94-6cbb47fb7b57-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"d669874a-e27f-4a06-af94-6cbb47fb7b57\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 17:21:16.060554 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:21:16.060530    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/d669874a-e27f-4a06-af94-6cbb47fb7b57-config-out\") pod \"prometheus-k8s-0\" (UID: \"d669874a-e27f-4a06-af94-6cbb47fb7b57\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 17:21:16.060913 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:21:16.060889    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/d669874a-e27f-4a06-af94-6cbb47fb7b57-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"d669874a-e27f-4a06-af94-6cbb47fb7b57\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 17:21:16.061045 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:21:16.061017    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/d669874a-e27f-4a06-af94-6cbb47fb7b57-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"d669874a-e27f-4a06-af94-6cbb47fb7b57\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 17:21:16.061152 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:21:16.061128    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/d669874a-e27f-4a06-af94-6cbb47fb7b57-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"d669874a-e27f-4a06-af94-6cbb47fb7b57\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 17:21:16.061293 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:21:16.061272    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/d669874a-e27f-4a06-af94-6cbb47fb7b57-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"d669874a-e27f-4a06-af94-6cbb47fb7b57\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 17:21:16.061534 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:21:16.061491    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/d669874a-e27f-4a06-af94-6cbb47fb7b57-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"d669874a-e27f-4a06-af94-6cbb47fb7b57\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 17:21:16.062237 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:21:16.062208    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/d669874a-e27f-4a06-af94-6cbb47fb7b57-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"d669874a-e27f-4a06-af94-6cbb47fb7b57\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 17:21:16.062317 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:21:16.062242    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/d669874a-e27f-4a06-af94-6cbb47fb7b57-web-config\") pod \"prometheus-k8s-0\" (UID: \"d669874a-e27f-4a06-af94-6cbb47fb7b57\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 17:21:16.062317 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:21:16.062303    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/d669874a-e27f-4a06-af94-6cbb47fb7b57-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"d669874a-e27f-4a06-af94-6cbb47fb7b57\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 17:21:16.062688 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:21:16.062665    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/d669874a-e27f-4a06-af94-6cbb47fb7b57-config\") pod \"prometheus-k8s-0\" (UID: \"d669874a-e27f-4a06-af94-6cbb47fb7b57\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 17:21:16.062896 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:21:16.062878    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/d669874a-e27f-4a06-af94-6cbb47fb7b57-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"d669874a-e27f-4a06-af94-6cbb47fb7b57\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 17:21:16.064969 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:21:16.064950    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hlmpg\" (UniqueName: \"kubernetes.io/projected/d669874a-e27f-4a06-af94-6cbb47fb7b57-kube-api-access-hlmpg\") pod \"prometheus-k8s-0\" (UID: \"d669874a-e27f-4a06-af94-6cbb47fb7b57\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 17:21:16.164342 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:21:16.164287    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 17:21:16.453460 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:21:16.453432    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 11 17:21:16.455885 ip-10-0-141-139 kubenswrapper[2577]: W0911 17:21:16.455857    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd669874a_e27f_4a06_af94_6cbb47fb7b57.slice/crio-66159edc0c9a415bd765e7778feacec4299e70c509ad2d83ad47893e3415c846 WatchSource:0}: Error finding container 66159edc0c9a415bd765e7778feacec4299e70c509ad2d83ad47893e3415c846: Status 404 returned error can't find the container with id 66159edc0c9a415bd765e7778feacec4299e70c509ad2d83ad47893e3415c846
Sep 11 17:21:16.782686 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:21:16.782652    2577 generic.go:356] "Generic (PLEG): container finished" podID="d669874a-e27f-4a06-af94-6cbb47fb7b57" containerID="baa4c78d0208224e30aaec710f1b2ec1a4d78163ef239742be37eeba3e24437b" exitCode=0
Sep 11 17:21:16.783162 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:21:16.782697    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"d669874a-e27f-4a06-af94-6cbb47fb7b57","Type":"ContainerDied","Data":"baa4c78d0208224e30aaec710f1b2ec1a4d78163ef239742be37eeba3e24437b"}
Sep 11 17:21:16.783162 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:21:16.782747    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"d669874a-e27f-4a06-af94-6cbb47fb7b57","Type":"ContainerStarted","Data":"66159edc0c9a415bd765e7778feacec4299e70c509ad2d83ad47893e3415c846"}
Sep 11 17:21:16.926799 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:21:16.926767    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5fd69637-9132-4804-8a78-c24a626114ac" path="/var/lib/kubelet/pods/5fd69637-9132-4804-8a78-c24a626114ac/volumes"
Sep 11 17:21:17.795367 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:21:17.795326    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"d669874a-e27f-4a06-af94-6cbb47fb7b57","Type":"ContainerStarted","Data":"d8d88cae469eaac44eea85c2ac6a0e9cb9c45c8df99500e353d954550c2bd273"}
Sep 11 17:21:17.795723 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:21:17.795421    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"d669874a-e27f-4a06-af94-6cbb47fb7b57","Type":"ContainerStarted","Data":"fe7ddbff1da54f7d496058a71702111df8e723ef2da16c2033038c320a314f31"}
Sep 11 17:21:17.795723 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:21:17.795450    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"d669874a-e27f-4a06-af94-6cbb47fb7b57","Type":"ContainerStarted","Data":"3b31a05e8a5b42292c558257b4434c038b60c1539a54b20aeee734836a5b8f27"}
Sep 11 17:21:17.795723 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:21:17.795489    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"d669874a-e27f-4a06-af94-6cbb47fb7b57","Type":"ContainerStarted","Data":"2d5e0f21548b902a84b3d8c3399b4835329a6b294ab89d28891809b151f30efa"}
Sep 11 17:21:17.795723 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:21:17.795536    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"d669874a-e27f-4a06-af94-6cbb47fb7b57","Type":"ContainerStarted","Data":"f2dba39c81b293b2decf34eca307dd7d742d70739ce9bda7e28529a486ef9d4c"}
Sep 11 17:21:17.795723 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:21:17.795571    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"d669874a-e27f-4a06-af94-6cbb47fb7b57","Type":"ContainerStarted","Data":"310d3e5e41c97bc077606823ad8be32caa1bfe6016fdafd7375d560237c758f5"}
Sep 11 17:21:17.834903 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:21:17.834844    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=2.813194175 podStartE2EDuration="2.8348277s" totalImagesPullingTime="0s" totalInitContainerRuntime="21.633525ms" isStatefulPod=false podCreationTimestamp="2026-09-11 17:21:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 17:21:17.830071004 +0000 UTC m=+155.403291116" watchObservedRunningTime="2026-09-11 17:21:17.8348277 +0000 UTC m=+155.408047812"
Sep 11 17:21:21.165305 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:21:21.165243    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 17:22:16.164872 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:22:16.164812    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 17:22:16.181301 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:22:16.181273    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 17:22:17.055888 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:22:17.055857    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 17:22:55.715346 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:22:55.715312    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-jmmxv"]
Sep 11 17:22:55.718704 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:22:55.718684    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-559cdd8c87-jmmxv"
Sep 11 17:22:55.721349 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:22:55.721324    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-cainjector-dockercfg-8nzql\""
Sep 11 17:22:55.722278 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:22:55.722244    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\""
Sep 11 17:22:55.722278 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:22:55.722258    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\""
Sep 11 17:22:55.728696 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:22:55.728675    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-jmmxv"]
Sep 11 17:22:55.843949 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:22:55.843907    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wdmc9\" (UniqueName: \"kubernetes.io/projected/48f064d9-c4be-46fe-bd6f-da7109379aa0-kube-api-access-wdmc9\") pod \"cert-manager-cainjector-559cdd8c87-jmmxv\" (UID: \"48f064d9-c4be-46fe-bd6f-da7109379aa0\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-jmmxv"
Sep 11 17:22:55.844142 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:22:55.843959    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/48f064d9-c4be-46fe-bd6f-da7109379aa0-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-jmmxv\" (UID: \"48f064d9-c4be-46fe-bd6f-da7109379aa0\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-jmmxv"
Sep 11 17:22:55.944756 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:22:55.944711    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wdmc9\" (UniqueName: \"kubernetes.io/projected/48f064d9-c4be-46fe-bd6f-da7109379aa0-kube-api-access-wdmc9\") pod \"cert-manager-cainjector-559cdd8c87-jmmxv\" (UID: \"48f064d9-c4be-46fe-bd6f-da7109379aa0\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-jmmxv"
Sep 11 17:22:55.944929 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:22:55.944765    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/48f064d9-c4be-46fe-bd6f-da7109379aa0-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-jmmxv\" (UID: \"48f064d9-c4be-46fe-bd6f-da7109379aa0\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-jmmxv"
Sep 11 17:22:55.953433 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:22:55.953405    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wdmc9\" (UniqueName: \"kubernetes.io/projected/48f064d9-c4be-46fe-bd6f-da7109379aa0-kube-api-access-wdmc9\") pod \"cert-manager-cainjector-559cdd8c87-jmmxv\" (UID: \"48f064d9-c4be-46fe-bd6f-da7109379aa0\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-jmmxv"
Sep 11 17:22:55.953591 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:22:55.953473    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/48f064d9-c4be-46fe-bd6f-da7109379aa0-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-jmmxv\" (UID: \"48f064d9-c4be-46fe-bd6f-da7109379aa0\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-jmmxv"
Sep 11 17:22:56.032304 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:22:56.032218    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-559cdd8c87-jmmxv"
Sep 11 17:22:56.308681 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:22:56.308597    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-jmmxv"]
Sep 11 17:22:56.311464 ip-10-0-141-139 kubenswrapper[2577]: W0911 17:22:56.311442    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod48f064d9_c4be_46fe_bd6f_da7109379aa0.slice/crio-aac0e1d05535580bade633c09cea82270f5f965e1a1f5b5aae2a1cf87da4e68c WatchSource:0}: Error finding container aac0e1d05535580bade633c09cea82270f5f965e1a1f5b5aae2a1cf87da4e68c: Status 404 returned error can't find the container with id aac0e1d05535580bade633c09cea82270f5f965e1a1f5b5aae2a1cf87da4e68c
Sep 11 17:22:57.206387 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:22:57.206344    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-559cdd8c87-jmmxv" event={"ID":"48f064d9-c4be-46fe-bd6f-da7109379aa0","Type":"ContainerStarted","Data":"aac0e1d05535580bade633c09cea82270f5f965e1a1f5b5aae2a1cf87da4e68c"}
Sep 11 17:23:00.220599 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:23:00.220557    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-559cdd8c87-jmmxv" event={"ID":"48f064d9-c4be-46fe-bd6f-da7109379aa0","Type":"ContainerStarted","Data":"cd44e438a172b95de329991d5b619917f4ad453e5f9b9f10fcaa93f43c2b37a4"}
Sep 11 17:23:00.240602 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:23:00.240546    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-cainjector-559cdd8c87-jmmxv" podStartSLOduration=2.130994044 podStartE2EDuration="5.240531835s" totalImagesPullingTime="3.109537791s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 17:22:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 17:23:00.239180782 +0000 UTC m=+257.812400894" watchObservedRunningTime="2026-09-11 17:23:00.240531835 +0000 UTC m=+257.813751946"
Sep 11 17:23:04.137494 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:23:04.137402    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-55b4d954d5-bj825"]
Sep 11 17:23:04.140814 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:23:04.140791    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-55b4d954d5-bj825"
Sep 11 17:23:04.143192 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:23:04.143176    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-dockercfg-2kwwv\""
Sep 11 17:23:04.150812 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:23:04.150787    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-55b4d954d5-bj825"]
Sep 11 17:23:04.218301 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:23:04.218268    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q8rq6\" (UniqueName: \"kubernetes.io/projected/0586929b-d704-415c-9213-7f000eb6b162-kube-api-access-q8rq6\") pod \"cert-manager-55b4d954d5-bj825\" (UID: \"0586929b-d704-415c-9213-7f000eb6b162\") " pod="cert-manager/cert-manager-55b4d954d5-bj825"
Sep 11 17:23:04.218533 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:23:04.218339    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/0586929b-d704-415c-9213-7f000eb6b162-bound-sa-token\") pod \"cert-manager-55b4d954d5-bj825\" (UID: \"0586929b-d704-415c-9213-7f000eb6b162\") " pod="cert-manager/cert-manager-55b4d954d5-bj825"
Sep 11 17:23:04.319055 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:23:04.319018    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/0586929b-d704-415c-9213-7f000eb6b162-bound-sa-token\") pod \"cert-manager-55b4d954d5-bj825\" (UID: \"0586929b-d704-415c-9213-7f000eb6b162\") " pod="cert-manager/cert-manager-55b4d954d5-bj825"
Sep 11 17:23:04.319254 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:23:04.319119    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q8rq6\" (UniqueName: \"kubernetes.io/projected/0586929b-d704-415c-9213-7f000eb6b162-kube-api-access-q8rq6\") pod \"cert-manager-55b4d954d5-bj825\" (UID: \"0586929b-d704-415c-9213-7f000eb6b162\") " pod="cert-manager/cert-manager-55b4d954d5-bj825"
Sep 11 17:23:04.327618 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:23:04.327592    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/0586929b-d704-415c-9213-7f000eb6b162-bound-sa-token\") pod \"cert-manager-55b4d954d5-bj825\" (UID: \"0586929b-d704-415c-9213-7f000eb6b162\") " pod="cert-manager/cert-manager-55b4d954d5-bj825"
Sep 11 17:23:04.327723 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:23:04.327648    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-q8rq6\" (UniqueName: \"kubernetes.io/projected/0586929b-d704-415c-9213-7f000eb6b162-kube-api-access-q8rq6\") pod \"cert-manager-55b4d954d5-bj825\" (UID: \"0586929b-d704-415c-9213-7f000eb6b162\") " pod="cert-manager/cert-manager-55b4d954d5-bj825"
Sep 11 17:23:04.450022 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:23:04.449922    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-55b4d954d5-bj825"
Sep 11 17:23:04.722327 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:23:04.722176    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-55b4d954d5-bj825"]
Sep 11 17:23:04.725153 ip-10-0-141-139 kubenswrapper[2577]: W0911 17:23:04.725124    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod0586929b_d704_415c_9213_7f000eb6b162.slice/crio-899b9fce7414116c5ff7f60ae69e16adcce703a1ad22ab437b80d16ed30eee76 WatchSource:0}: Error finding container 899b9fce7414116c5ff7f60ae69e16adcce703a1ad22ab437b80d16ed30eee76: Status 404 returned error can't find the container with id 899b9fce7414116c5ff7f60ae69e16adcce703a1ad22ab437b80d16ed30eee76
Sep 11 17:23:05.240697 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:23:05.240657    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-55b4d954d5-bj825" event={"ID":"0586929b-d704-415c-9213-7f000eb6b162","Type":"ContainerStarted","Data":"edae5d3811bb7818cb535b2d22b247d8f9f1cc54b7f7fd21f7fd53266c63f1e0"}
Sep 11 17:23:05.241132 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:23:05.240708    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-55b4d954d5-bj825" event={"ID":"0586929b-d704-415c-9213-7f000eb6b162","Type":"ContainerStarted","Data":"899b9fce7414116c5ff7f60ae69e16adcce703a1ad22ab437b80d16ed30eee76"}
Sep 11 17:23:05.262750 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:23:05.262697    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-55b4d954d5-bj825" podStartSLOduration=1.262680521 podStartE2EDuration="1.262680521s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 17:23:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 17:23:05.257227633 +0000 UTC m=+262.830447744" watchObservedRunningTime="2026-09-11 17:23:05.262680521 +0000 UTC m=+262.835900632"
Sep 11 17:23:29.954560 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:23:29.954527    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/lws-controller-manager-845d467d5c-xnvgd"]
Sep 11 17:23:29.960488 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:23:29.960465    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-845d467d5c-xnvgd"
Sep 11 17:23:29.964173 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:23:29.964151    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\""
Sep 11 17:23:29.964316 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:23:29.964252    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\""
Sep 11 17:23:29.964316 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:23:29.964269    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"lws-manager-config\""
Sep 11 17:23:29.965208 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:23:29.965190    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"webhook-server-cert\""
Sep 11 17:23:29.965297 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:23:29.965247    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"metrics-server-cert\""
Sep 11 17:23:29.965364 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:23:29.965342    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"lws-controller-manager-dockercfg-9nhbc\""
Sep 11 17:23:29.989352 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:23:29.989325    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-845d467d5c-xnvgd"]
Sep 11 17:23:30.033172 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:23:30.033135    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/e0102433-2481-47fa-ac24-665e4326f8e8-cert\") pod \"lws-controller-manager-845d467d5c-xnvgd\" (UID: \"e0102433-2481-47fa-ac24-665e4326f8e8\") " pod="openshift-lws-operator/lws-controller-manager-845d467d5c-xnvgd"
Sep 11 17:23:30.033172 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:23:30.033177    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/e0102433-2481-47fa-ac24-665e4326f8e8-metrics-cert\") pod \"lws-controller-manager-845d467d5c-xnvgd\" (UID: \"e0102433-2481-47fa-ac24-665e4326f8e8\") " pod="openshift-lws-operator/lws-controller-manager-845d467d5c-xnvgd"
Sep 11 17:23:30.033381 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:23:30.033242    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/e0102433-2481-47fa-ac24-665e4326f8e8-manager-config\") pod \"lws-controller-manager-845d467d5c-xnvgd\" (UID: \"e0102433-2481-47fa-ac24-665e4326f8e8\") " pod="openshift-lws-operator/lws-controller-manager-845d467d5c-xnvgd"
Sep 11 17:23:30.033381 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:23:30.033273    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mwcs5\" (UniqueName: \"kubernetes.io/projected/e0102433-2481-47fa-ac24-665e4326f8e8-kube-api-access-mwcs5\") pod \"lws-controller-manager-845d467d5c-xnvgd\" (UID: \"e0102433-2481-47fa-ac24-665e4326f8e8\") " pod="openshift-lws-operator/lws-controller-manager-845d467d5c-xnvgd"
Sep 11 17:23:30.134378 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:23:30.134334    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/e0102433-2481-47fa-ac24-665e4326f8e8-cert\") pod \"lws-controller-manager-845d467d5c-xnvgd\" (UID: \"e0102433-2481-47fa-ac24-665e4326f8e8\") " pod="openshift-lws-operator/lws-controller-manager-845d467d5c-xnvgd"
Sep 11 17:23:30.134624 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:23:30.134393    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/e0102433-2481-47fa-ac24-665e4326f8e8-metrics-cert\") pod \"lws-controller-manager-845d467d5c-xnvgd\" (UID: \"e0102433-2481-47fa-ac24-665e4326f8e8\") " pod="openshift-lws-operator/lws-controller-manager-845d467d5c-xnvgd"
Sep 11 17:23:30.134624 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:23:30.134446    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/e0102433-2481-47fa-ac24-665e4326f8e8-manager-config\") pod \"lws-controller-manager-845d467d5c-xnvgd\" (UID: \"e0102433-2481-47fa-ac24-665e4326f8e8\") " pod="openshift-lws-operator/lws-controller-manager-845d467d5c-xnvgd"
Sep 11 17:23:30.134624 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:23:30.134478    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mwcs5\" (UniqueName: \"kubernetes.io/projected/e0102433-2481-47fa-ac24-665e4326f8e8-kube-api-access-mwcs5\") pod \"lws-controller-manager-845d467d5c-xnvgd\" (UID: \"e0102433-2481-47fa-ac24-665e4326f8e8\") " pod="openshift-lws-operator/lws-controller-manager-845d467d5c-xnvgd"
Sep 11 17:23:30.135147 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:23:30.135126    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/e0102433-2481-47fa-ac24-665e4326f8e8-manager-config\") pod \"lws-controller-manager-845d467d5c-xnvgd\" (UID: \"e0102433-2481-47fa-ac24-665e4326f8e8\") " pod="openshift-lws-operator/lws-controller-manager-845d467d5c-xnvgd"
Sep 11 17:23:30.137151 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:23:30.137130    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/e0102433-2481-47fa-ac24-665e4326f8e8-metrics-cert\") pod \"lws-controller-manager-845d467d5c-xnvgd\" (UID: \"e0102433-2481-47fa-ac24-665e4326f8e8\") " pod="openshift-lws-operator/lws-controller-manager-845d467d5c-xnvgd"
Sep 11 17:23:30.137241 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:23:30.137185    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/e0102433-2481-47fa-ac24-665e4326f8e8-cert\") pod \"lws-controller-manager-845d467d5c-xnvgd\" (UID: \"e0102433-2481-47fa-ac24-665e4326f8e8\") " pod="openshift-lws-operator/lws-controller-manager-845d467d5c-xnvgd"
Sep 11 17:23:30.148825 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:23:30.148800    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mwcs5\" (UniqueName: \"kubernetes.io/projected/e0102433-2481-47fa-ac24-665e4326f8e8-kube-api-access-mwcs5\") pod \"lws-controller-manager-845d467d5c-xnvgd\" (UID: \"e0102433-2481-47fa-ac24-665e4326f8e8\") " pod="openshift-lws-operator/lws-controller-manager-845d467d5c-xnvgd"
Sep 11 17:23:30.270134 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:23:30.270104    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-845d467d5c-xnvgd"
Sep 11 17:23:30.646173 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:23:30.646142    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-845d467d5c-xnvgd"]
Sep 11 17:23:30.647639 ip-10-0-141-139 kubenswrapper[2577]: W0911 17:23:30.647615    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode0102433_2481_47fa_ac24_665e4326f8e8.slice/crio-84e6f8de063863d37bc3a21226a8842b04087a170ec89708217d716cec2799e0 WatchSource:0}: Error finding container 84e6f8de063863d37bc3a21226a8842b04087a170ec89708217d716cec2799e0: Status 404 returned error can't find the container with id 84e6f8de063863d37bc3a21226a8842b04087a170ec89708217d716cec2799e0
Sep 11 17:23:30.860394 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:23:30.860359    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-b887fdd9f-jc4h9"]
Sep 11 17:23:30.865516 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:23:30.865488    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-b887fdd9f-jc4h9"
Sep 11 17:23:30.878396 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:23:30.878355    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\""
Sep 11 17:23:30.878562 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:23:30.878471    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\""
Sep 11 17:23:30.878562 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:23:30.878490    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\""
Sep 11 17:23:30.878562 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:23:30.878528    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-vxtdm\""
Sep 11 17:23:30.878836 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:23:30.878822    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\""
Sep 11 17:23:30.888968 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:23:30.888942    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-b887fdd9f-jc4h9"]
Sep 11 17:23:31.044277 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:23:31.044239    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/65106181-ae0e-493b-b7d7-b6707f6f1bab-manifests\") pod \"opendatahub-operator-controller-manager-b887fdd9f-jc4h9\" (UID: \"65106181-ae0e-493b-b7d7-b6707f6f1bab\") " pod="opendatahub/opendatahub-operator-controller-manager-b887fdd9f-jc4h9"
Sep 11 17:23:31.044718 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:23:31.044292    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/65106181-ae0e-493b-b7d7-b6707f6f1bab-webhook-cert\") pod \"opendatahub-operator-controller-manager-b887fdd9f-jc4h9\" (UID: \"65106181-ae0e-493b-b7d7-b6707f6f1bab\") " pod="opendatahub/opendatahub-operator-controller-manager-b887fdd9f-jc4h9"
Sep 11 17:23:31.044718 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:23:31.044314    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/65106181-ae0e-493b-b7d7-b6707f6f1bab-apiservice-cert\") pod \"opendatahub-operator-controller-manager-b887fdd9f-jc4h9\" (UID: \"65106181-ae0e-493b-b7d7-b6707f6f1bab\") " pod="opendatahub/opendatahub-operator-controller-manager-b887fdd9f-jc4h9"
Sep 11 17:23:31.044718 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:23:31.044356    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-64hm5\" (UniqueName: \"kubernetes.io/projected/65106181-ae0e-493b-b7d7-b6707f6f1bab-kube-api-access-64hm5\") pod \"opendatahub-operator-controller-manager-b887fdd9f-jc4h9\" (UID: \"65106181-ae0e-493b-b7d7-b6707f6f1bab\") " pod="opendatahub/opendatahub-operator-controller-manager-b887fdd9f-jc4h9"
Sep 11 17:23:31.044718 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:23:31.044400    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/65106181-ae0e-493b-b7d7-b6707f6f1bab-tmp\") pod \"opendatahub-operator-controller-manager-b887fdd9f-jc4h9\" (UID: \"65106181-ae0e-493b-b7d7-b6707f6f1bab\") " pod="opendatahub/opendatahub-operator-controller-manager-b887fdd9f-jc4h9"
Sep 11 17:23:31.145279 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:23:31.145243    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/65106181-ae0e-493b-b7d7-b6707f6f1bab-manifests\") pod \"opendatahub-operator-controller-manager-b887fdd9f-jc4h9\" (UID: \"65106181-ae0e-493b-b7d7-b6707f6f1bab\") " pod="opendatahub/opendatahub-operator-controller-manager-b887fdd9f-jc4h9"
Sep 11 17:23:31.145475 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:23:31.145287    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/65106181-ae0e-493b-b7d7-b6707f6f1bab-webhook-cert\") pod \"opendatahub-operator-controller-manager-b887fdd9f-jc4h9\" (UID: \"65106181-ae0e-493b-b7d7-b6707f6f1bab\") " pod="opendatahub/opendatahub-operator-controller-manager-b887fdd9f-jc4h9"
Sep 11 17:23:31.145475 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:23:31.145307    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/65106181-ae0e-493b-b7d7-b6707f6f1bab-apiservice-cert\") pod \"opendatahub-operator-controller-manager-b887fdd9f-jc4h9\" (UID: \"65106181-ae0e-493b-b7d7-b6707f6f1bab\") " pod="opendatahub/opendatahub-operator-controller-manager-b887fdd9f-jc4h9"
Sep 11 17:23:31.145475 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:23:31.145326    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-64hm5\" (UniqueName: \"kubernetes.io/projected/65106181-ae0e-493b-b7d7-b6707f6f1bab-kube-api-access-64hm5\") pod \"opendatahub-operator-controller-manager-b887fdd9f-jc4h9\" (UID: \"65106181-ae0e-493b-b7d7-b6707f6f1bab\") " pod="opendatahub/opendatahub-operator-controller-manager-b887fdd9f-jc4h9"
Sep 11 17:23:31.145475 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:23:31.145353    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/65106181-ae0e-493b-b7d7-b6707f6f1bab-tmp\") pod \"opendatahub-operator-controller-manager-b887fdd9f-jc4h9\" (UID: \"65106181-ae0e-493b-b7d7-b6707f6f1bab\") " pod="opendatahub/opendatahub-operator-controller-manager-b887fdd9f-jc4h9"
Sep 11 17:23:31.145751 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:23:31.145729    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/65106181-ae0e-493b-b7d7-b6707f6f1bab-manifests\") pod \"opendatahub-operator-controller-manager-b887fdd9f-jc4h9\" (UID: \"65106181-ae0e-493b-b7d7-b6707f6f1bab\") " pod="opendatahub/opendatahub-operator-controller-manager-b887fdd9f-jc4h9"
Sep 11 17:23:31.147901 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:23:31.147868    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/65106181-ae0e-493b-b7d7-b6707f6f1bab-tmp\") pod \"opendatahub-operator-controller-manager-b887fdd9f-jc4h9\" (UID: \"65106181-ae0e-493b-b7d7-b6707f6f1bab\") " pod="opendatahub/opendatahub-operator-controller-manager-b887fdd9f-jc4h9"
Sep 11 17:23:31.148056 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:23:31.147986    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/65106181-ae0e-493b-b7d7-b6707f6f1bab-webhook-cert\") pod \"opendatahub-operator-controller-manager-b887fdd9f-jc4h9\" (UID: \"65106181-ae0e-493b-b7d7-b6707f6f1bab\") " pod="opendatahub/opendatahub-operator-controller-manager-b887fdd9f-jc4h9"
Sep 11 17:23:31.148137 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:23:31.148116    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/65106181-ae0e-493b-b7d7-b6707f6f1bab-apiservice-cert\") pod \"opendatahub-operator-controller-manager-b887fdd9f-jc4h9\" (UID: \"65106181-ae0e-493b-b7d7-b6707f6f1bab\") " pod="opendatahub/opendatahub-operator-controller-manager-b887fdd9f-jc4h9"
Sep 11 17:23:31.180591 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:23:31.180565    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-64hm5\" (UniqueName: \"kubernetes.io/projected/65106181-ae0e-493b-b7d7-b6707f6f1bab-kube-api-access-64hm5\") pod \"opendatahub-operator-controller-manager-b887fdd9f-jc4h9\" (UID: \"65106181-ae0e-493b-b7d7-b6707f6f1bab\") " pod="opendatahub/opendatahub-operator-controller-manager-b887fdd9f-jc4h9"
Sep 11 17:23:31.350914 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:23:31.350831    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-845d467d5c-xnvgd" event={"ID":"e0102433-2481-47fa-ac24-665e4326f8e8","Type":"ContainerStarted","Data":"84e6f8de063863d37bc3a21226a8842b04087a170ec89708217d716cec2799e0"}
Sep 11 17:23:31.477107 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:23:31.477072    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-b887fdd9f-jc4h9"
Sep 11 17:23:31.790004 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:23:31.789975    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-b887fdd9f-jc4h9"]
Sep 11 17:23:31.793516 ip-10-0-141-139 kubenswrapper[2577]: W0911 17:23:31.793464    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod65106181_ae0e_493b_b7d7_b6707f6f1bab.slice/crio-238028c899292591d10fa2f4a716593833cac2ee1a2ca333f94281e69eaf1787 WatchSource:0}: Error finding container 238028c899292591d10fa2f4a716593833cac2ee1a2ca333f94281e69eaf1787: Status 404 returned error can't find the container with id 238028c899292591d10fa2f4a716593833cac2ee1a2ca333f94281e69eaf1787
Sep 11 17:23:32.366321 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:23:32.366287    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-b887fdd9f-jc4h9" event={"ID":"65106181-ae0e-493b-b7d7-b6707f6f1bab","Type":"ContainerStarted","Data":"238028c899292591d10fa2f4a716593833cac2ee1a2ca333f94281e69eaf1787"}
Sep 11 17:23:34.379105 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:23:34.379060    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-845d467d5c-xnvgd" event={"ID":"e0102433-2481-47fa-ac24-665e4326f8e8","Type":"ContainerStarted","Data":"560fa0e443e5e8b2065876f249f6967d0f16771283ea299078c99f795285410d"}
Sep 11 17:23:34.379546 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:23:34.379176    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-lws-operator/lws-controller-manager-845d467d5c-xnvgd"
Sep 11 17:23:34.417056 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:23:34.416996    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/lws-controller-manager-845d467d5c-xnvgd" podStartSLOduration=2.7644833699999998 podStartE2EDuration="5.41697845s" totalImagesPullingTime="2.65249508s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 17:23:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 17:23:34.415855699 +0000 UTC m=+291.989075814" watchObservedRunningTime="2026-09-11 17:23:34.41697845 +0000 UTC m=+291.990198562"
Sep 11 17:23:36.395584 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:23:36.395544    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-b887fdd9f-jc4h9" event={"ID":"65106181-ae0e-493b-b7d7-b6707f6f1bab","Type":"ContainerStarted","Data":"c3cc26d7e6463dc978ce6a6e9b79cd0edc8e0287aa4b05dc993bfd209f286fd0"}
Sep 11 17:23:37.401479 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:23:37.401439    2577 generic.go:356] "Generic (PLEG): container finished" podID="65106181-ae0e-493b-b7d7-b6707f6f1bab" containerID="c3cc26d7e6463dc978ce6a6e9b79cd0edc8e0287aa4b05dc993bfd209f286fd0" exitCode=0
Sep 11 17:23:37.401479 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:23:37.401481    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-b887fdd9f-jc4h9" event={"ID":"65106181-ae0e-493b-b7d7-b6707f6f1bab","Type":"ContainerDied","Data":"c3cc26d7e6463dc978ce6a6e9b79cd0edc8e0287aa4b05dc993bfd209f286fd0"}
Sep 11 17:23:38.408242 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:23:38.408205    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-b887fdd9f-jc4h9" event={"ID":"65106181-ae0e-493b-b7d7-b6707f6f1bab","Type":"ContainerStarted","Data":"1aac073174544ef86fab8806a453dc18d768adcd2d77261c95e10d3250ea267f"}
Sep 11 17:23:38.408669 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:23:38.408346    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-b887fdd9f-jc4h9"
Sep 11 17:23:38.448768 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:23:38.448716    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-b887fdd9f-jc4h9" podStartSLOduration=3.098502524 podStartE2EDuration="8.44869846s" totalImagesPullingTime="4.267720868s" totalInitContainerRuntime="1.082475068s" isStatefulPod=false podCreationTimestamp="2026-09-11 17:23:30 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 17:23:38.448395895 +0000 UTC m=+296.021616006" watchObservedRunningTime="2026-09-11 17:23:38.44869846 +0000 UTC m=+296.021918571"
Sep 11 17:23:42.825989 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:23:42.825960    2577 kubelet.go:1648] "Image garbage collection succeeded"
Sep 11 17:23:45.388532 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:23:45.388474    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-lws-operator/lws-controller-manager-845d467d5c-xnvgd"
Sep 11 17:23:49.414401 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:23:49.414367    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-b887fdd9f-jc4h9"
Sep 11 17:23:54.846852 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:23:54.846814    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-7d6df6ff46-hkdnw"]
Sep 11 17:23:54.850194 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:23:54.850175    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-7d6df6ff46-hkdnw"
Sep 11 17:23:54.870939 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:23:54.870908    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
Sep 11 17:23:54.871048 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:23:54.870981    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
Sep 11 17:23:54.878699 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:23:54.878675    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\""
Sep 11 17:23:54.885250 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:23:54.885228    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-fr7vj\""
Sep 11 17:23:54.886793 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:23:54.886775    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-7d6df6ff46-hkdnw"]
Sep 11 17:23:54.889683 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:23:54.889656    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\""
Sep 11 17:23:54.946678 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:23:54.946643    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zf85f\" (UniqueName: \"kubernetes.io/projected/78758cdb-2c93-44c5-9039-ff181899b15f-kube-api-access-zf85f\") pod \"kube-auth-proxy-7d6df6ff46-hkdnw\" (UID: \"78758cdb-2c93-44c5-9039-ff181899b15f\") " pod="openshift-ingress/kube-auth-proxy-7d6df6ff46-hkdnw"
Sep 11 17:23:54.946874 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:23:54.946706    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/78758cdb-2c93-44c5-9039-ff181899b15f-tmp\") pod \"kube-auth-proxy-7d6df6ff46-hkdnw\" (UID: \"78758cdb-2c93-44c5-9039-ff181899b15f\") " pod="openshift-ingress/kube-auth-proxy-7d6df6ff46-hkdnw"
Sep 11 17:23:54.946874 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:23:54.946752    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/78758cdb-2c93-44c5-9039-ff181899b15f-tls-certs\") pod \"kube-auth-proxy-7d6df6ff46-hkdnw\" (UID: \"78758cdb-2c93-44c5-9039-ff181899b15f\") " pod="openshift-ingress/kube-auth-proxy-7d6df6ff46-hkdnw"
Sep 11 17:23:55.047764 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:23:55.047729    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zf85f\" (UniqueName: \"kubernetes.io/projected/78758cdb-2c93-44c5-9039-ff181899b15f-kube-api-access-zf85f\") pod \"kube-auth-proxy-7d6df6ff46-hkdnw\" (UID: \"78758cdb-2c93-44c5-9039-ff181899b15f\") " pod="openshift-ingress/kube-auth-proxy-7d6df6ff46-hkdnw"
Sep 11 17:23:55.047965 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:23:55.047782    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/78758cdb-2c93-44c5-9039-ff181899b15f-tmp\") pod \"kube-auth-proxy-7d6df6ff46-hkdnw\" (UID: \"78758cdb-2c93-44c5-9039-ff181899b15f\") " pod="openshift-ingress/kube-auth-proxy-7d6df6ff46-hkdnw"
Sep 11 17:23:55.047965 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:23:55.047818    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/78758cdb-2c93-44c5-9039-ff181899b15f-tls-certs\") pod \"kube-auth-proxy-7d6df6ff46-hkdnw\" (UID: \"78758cdb-2c93-44c5-9039-ff181899b15f\") " pod="openshift-ingress/kube-auth-proxy-7d6df6ff46-hkdnw"
Sep 11 17:23:55.050229 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:23:55.050201    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/78758cdb-2c93-44c5-9039-ff181899b15f-tmp\") pod \"kube-auth-proxy-7d6df6ff46-hkdnw\" (UID: \"78758cdb-2c93-44c5-9039-ff181899b15f\") " pod="openshift-ingress/kube-auth-proxy-7d6df6ff46-hkdnw"
Sep 11 17:23:55.050398 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:23:55.050381    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/78758cdb-2c93-44c5-9039-ff181899b15f-tls-certs\") pod \"kube-auth-proxy-7d6df6ff46-hkdnw\" (UID: \"78758cdb-2c93-44c5-9039-ff181899b15f\") " pod="openshift-ingress/kube-auth-proxy-7d6df6ff46-hkdnw"
Sep 11 17:23:55.073778 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:23:55.073748    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zf85f\" (UniqueName: \"kubernetes.io/projected/78758cdb-2c93-44c5-9039-ff181899b15f-kube-api-access-zf85f\") pod \"kube-auth-proxy-7d6df6ff46-hkdnw\" (UID: \"78758cdb-2c93-44c5-9039-ff181899b15f\") " pod="openshift-ingress/kube-auth-proxy-7d6df6ff46-hkdnw"
Sep 11 17:23:55.159639 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:23:55.159557    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-7d6df6ff46-hkdnw"
Sep 11 17:23:55.436951 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:23:55.436928    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-7d6df6ff46-hkdnw"]
Sep 11 17:23:55.439419 ip-10-0-141-139 kubenswrapper[2577]: W0911 17:23:55.439393    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod78758cdb_2c93_44c5_9039_ff181899b15f.slice/crio-11089c3b0f43ff5d3253fa31cbaa879090ec12264a9edaeb4f68730d2a7dd5c0 WatchSource:0}: Error finding container 11089c3b0f43ff5d3253fa31cbaa879090ec12264a9edaeb4f68730d2a7dd5c0: Status 404 returned error can't find the container with id 11089c3b0f43ff5d3253fa31cbaa879090ec12264a9edaeb4f68730d2a7dd5c0
Sep 11 17:23:55.441324 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:23:55.441308    2577 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 11 17:23:55.485473 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:23:55.485436    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-7d6df6ff46-hkdnw" event={"ID":"78758cdb-2c93-44c5-9039-ff181899b15f","Type":"ContainerStarted","Data":"11089c3b0f43ff5d3253fa31cbaa879090ec12264a9edaeb4f68730d2a7dd5c0"}
Sep 11 17:23:59.516297 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:23:59.516255    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-7d6df6ff46-hkdnw" event={"ID":"78758cdb-2c93-44c5-9039-ff181899b15f","Type":"ContainerStarted","Data":"c4a523213c4d2d2ce1305e62c709858ceb02162e7c590a6c9bd176fd530acf40"}
Sep 11 17:23:59.539254 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:23:59.539201    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-7d6df6ff46-hkdnw" podStartSLOduration=2.366129459 podStartE2EDuration="5.539185763s" totalImagesPullingTime="3.173056304s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 17:23:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 17:23:59.538836781 +0000 UTC m=+317.112056904" watchObservedRunningTime="2026-09-11 17:23:59.539185763 +0000 UTC m=+317.112405875"
Sep 11 17:25:32.036126 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:25:32.036090    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-console-plugin-fbdd4f445-z6f86"]
Sep 11 17:25:32.039188 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:25:32.039166    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-z6f86"
Sep 11 17:25:32.041896 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:25:32.041867    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\""
Sep 11 17:25:32.042710 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:25:32.042686    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"plugin-serving-cert\""
Sep 11 17:25:32.042885 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:25:32.042686    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-r5glb\""
Sep 11 17:25:32.042885 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:25:32.042729    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\""
Sep 11 17:25:32.043529 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:25:32.043490    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kuadrant-console-nginx-conf\""
Sep 11 17:25:32.049445 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:25:32.049418    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-fbdd4f445-z6f86"]
Sep 11 17:25:32.147002 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:25:32.146969    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/972596b5-0a5e-4145-be4e-dfb8cbda2dfe-plugin-serving-cert\") pod \"kuadrant-console-plugin-fbdd4f445-z6f86\" (UID: \"972596b5-0a5e-4145-be4e-dfb8cbda2dfe\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-z6f86"
Sep 11 17:25:32.147187 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:25:32.147018    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/972596b5-0a5e-4145-be4e-dfb8cbda2dfe-nginx-conf\") pod \"kuadrant-console-plugin-fbdd4f445-z6f86\" (UID: \"972596b5-0a5e-4145-be4e-dfb8cbda2dfe\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-z6f86"
Sep 11 17:25:32.147187 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:25:32.147090    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-frjl9\" (UniqueName: \"kubernetes.io/projected/972596b5-0a5e-4145-be4e-dfb8cbda2dfe-kube-api-access-frjl9\") pod \"kuadrant-console-plugin-fbdd4f445-z6f86\" (UID: \"972596b5-0a5e-4145-be4e-dfb8cbda2dfe\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-z6f86"
Sep 11 17:25:32.247677 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:25:32.247636    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/972596b5-0a5e-4145-be4e-dfb8cbda2dfe-plugin-serving-cert\") pod \"kuadrant-console-plugin-fbdd4f445-z6f86\" (UID: \"972596b5-0a5e-4145-be4e-dfb8cbda2dfe\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-z6f86"
Sep 11 17:25:32.247677 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:25:32.247684    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/972596b5-0a5e-4145-be4e-dfb8cbda2dfe-nginx-conf\") pod \"kuadrant-console-plugin-fbdd4f445-z6f86\" (UID: \"972596b5-0a5e-4145-be4e-dfb8cbda2dfe\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-z6f86"
Sep 11 17:25:32.247967 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:25:32.247712    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-frjl9\" (UniqueName: \"kubernetes.io/projected/972596b5-0a5e-4145-be4e-dfb8cbda2dfe-kube-api-access-frjl9\") pod \"kuadrant-console-plugin-fbdd4f445-z6f86\" (UID: \"972596b5-0a5e-4145-be4e-dfb8cbda2dfe\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-z6f86"
Sep 11 17:25:32.248377 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:25:32.248352    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/972596b5-0a5e-4145-be4e-dfb8cbda2dfe-nginx-conf\") pod \"kuadrant-console-plugin-fbdd4f445-z6f86\" (UID: \"972596b5-0a5e-4145-be4e-dfb8cbda2dfe\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-z6f86"
Sep 11 17:25:32.250369 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:25:32.250346    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/972596b5-0a5e-4145-be4e-dfb8cbda2dfe-plugin-serving-cert\") pod \"kuadrant-console-plugin-fbdd4f445-z6f86\" (UID: \"972596b5-0a5e-4145-be4e-dfb8cbda2dfe\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-z6f86"
Sep 11 17:25:32.260751 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:25:32.260722    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-frjl9\" (UniqueName: \"kubernetes.io/projected/972596b5-0a5e-4145-be4e-dfb8cbda2dfe-kube-api-access-frjl9\") pod \"kuadrant-console-plugin-fbdd4f445-z6f86\" (UID: \"972596b5-0a5e-4145-be4e-dfb8cbda2dfe\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-z6f86"
Sep 11 17:25:32.350826 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:25:32.350728    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-z6f86"
Sep 11 17:25:32.624050 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:25:32.623979    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-fbdd4f445-z6f86"]
Sep 11 17:25:32.626605 ip-10-0-141-139 kubenswrapper[2577]: W0911 17:25:32.626577    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod972596b5_0a5e_4145_be4e_dfb8cbda2dfe.slice/crio-9025cc9b79ac06f0058e5805b9d1cc38ffcf35358fdde32399eeba0f532f48fa WatchSource:0}: Error finding container 9025cc9b79ac06f0058e5805b9d1cc38ffcf35358fdde32399eeba0f532f48fa: Status 404 returned error can't find the container with id 9025cc9b79ac06f0058e5805b9d1cc38ffcf35358fdde32399eeba0f532f48fa
Sep 11 17:25:32.940390 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:25:32.940309    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-z6f86" event={"ID":"972596b5-0a5e-4145-be4e-dfb8cbda2dfe","Type":"ContainerStarted","Data":"9025cc9b79ac06f0058e5805b9d1cc38ffcf35358fdde32399eeba0f532f48fa"}
Sep 11 17:25:37.968167 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:25:37.968130    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-z6f86" event={"ID":"972596b5-0a5e-4145-be4e-dfb8cbda2dfe","Type":"ContainerStarted","Data":"5e453a85c77c09ed69240c0aa675bf22908efc4393e8ed90aed4d81763889115"}
Sep 11 17:25:37.986362 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:25:37.986298    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-z6f86" podStartSLOduration=1.373506564 podStartE2EDuration="5.986283056s" totalImagesPullingTime="4.612776492s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 17:25:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 17:25:37.983825236 +0000 UTC m=+415.557045348" watchObservedRunningTime="2026-09-11 17:25:37.986283056 +0000 UTC m=+415.559503168"
Sep 11 17:26:22.384108 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:26:22.384074    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-nkmkm"]
Sep 11 17:26:22.403530 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:26:22.403486    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-84994f7ffc-nkmkm"
Sep 11 17:26:22.403716 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:26:22.403638    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-nkmkm"]
Sep 11 17:26:22.405850 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:26:22.405824    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\""
Sep 11 17:26:22.413963 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:26:22.413940    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cbzn9\" (UniqueName: \"kubernetes.io/projected/a0d4403a-011b-4bf2-ae88-a27c8ee0fa86-kube-api-access-cbzn9\") pod \"limitador-limitador-84994f7ffc-nkmkm\" (UID: \"a0d4403a-011b-4bf2-ae88-a27c8ee0fa86\") " pod="kuadrant-system/limitador-limitador-84994f7ffc-nkmkm"
Sep 11 17:26:22.414067 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:26:22.413982    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/a0d4403a-011b-4bf2-ae88-a27c8ee0fa86-config-file\") pod \"limitador-limitador-84994f7ffc-nkmkm\" (UID: \"a0d4403a-011b-4bf2-ae88-a27c8ee0fa86\") " pod="kuadrant-system/limitador-limitador-84994f7ffc-nkmkm"
Sep 11 17:26:22.422357 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:26:22.422334    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-nkmkm"]
Sep 11 17:26:22.514739 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:26:22.514706    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cbzn9\" (UniqueName: \"kubernetes.io/projected/a0d4403a-011b-4bf2-ae88-a27c8ee0fa86-kube-api-access-cbzn9\") pod \"limitador-limitador-84994f7ffc-nkmkm\" (UID: \"a0d4403a-011b-4bf2-ae88-a27c8ee0fa86\") " pod="kuadrant-system/limitador-limitador-84994f7ffc-nkmkm"
Sep 11 17:26:22.514911 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:26:22.514754    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/a0d4403a-011b-4bf2-ae88-a27c8ee0fa86-config-file\") pod \"limitador-limitador-84994f7ffc-nkmkm\" (UID: \"a0d4403a-011b-4bf2-ae88-a27c8ee0fa86\") " pod="kuadrant-system/limitador-limitador-84994f7ffc-nkmkm"
Sep 11 17:26:22.515399 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:26:22.515379    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/a0d4403a-011b-4bf2-ae88-a27c8ee0fa86-config-file\") pod \"limitador-limitador-84994f7ffc-nkmkm\" (UID: \"a0d4403a-011b-4bf2-ae88-a27c8ee0fa86\") " pod="kuadrant-system/limitador-limitador-84994f7ffc-nkmkm"
Sep 11 17:26:22.528192 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:26:22.528157    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cbzn9\" (UniqueName: \"kubernetes.io/projected/a0d4403a-011b-4bf2-ae88-a27c8ee0fa86-kube-api-access-cbzn9\") pod \"limitador-limitador-84994f7ffc-nkmkm\" (UID: \"a0d4403a-011b-4bf2-ae88-a27c8ee0fa86\") " pod="kuadrant-system/limitador-limitador-84994f7ffc-nkmkm"
Sep 11 17:26:22.716410 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:26:22.716369    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-84994f7ffc-nkmkm"
Sep 11 17:26:22.779998 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:26:22.779889    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-tfwmr"]
Sep 11 17:26:22.784153 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:26:22.783801    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7bcfd8f9cd-tfwmr"
Sep 11 17:26:22.797919 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:26:22.797893    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-nx4ln\""
Sep 11 17:26:22.800314 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:26:22.800276    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-tfwmr"]
Sep 11 17:26:22.816832 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:26:22.816796    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8pcr2\" (UniqueName: \"kubernetes.io/projected/a24537f0-4662-40d2-8175-fdaf4d750d9f-kube-api-access-8pcr2\") pod \"authorino-7bcfd8f9cd-tfwmr\" (UID: \"a24537f0-4662-40d2-8175-fdaf4d750d9f\") " pod="kuadrant-system/authorino-7bcfd8f9cd-tfwmr"
Sep 11 17:26:22.917530 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:26:22.917411    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8pcr2\" (UniqueName: \"kubernetes.io/projected/a24537f0-4662-40d2-8175-fdaf4d750d9f-kube-api-access-8pcr2\") pod \"authorino-7bcfd8f9cd-tfwmr\" (UID: \"a24537f0-4662-40d2-8175-fdaf4d750d9f\") " pod="kuadrant-system/authorino-7bcfd8f9cd-tfwmr"
Sep 11 17:26:22.927244 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:26:22.927217    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8pcr2\" (UniqueName: \"kubernetes.io/projected/a24537f0-4662-40d2-8175-fdaf4d750d9f-kube-api-access-8pcr2\") pod \"authorino-7bcfd8f9cd-tfwmr\" (UID: \"a24537f0-4662-40d2-8175-fdaf4d750d9f\") " pod="kuadrant-system/authorino-7bcfd8f9cd-tfwmr"
Sep 11 17:26:22.990613 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:26:22.990581    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-nkmkm"]
Sep 11 17:26:22.993973 ip-10-0-141-139 kubenswrapper[2577]: W0911 17:26:22.993936    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda0d4403a_011b_4bf2_ae88_a27c8ee0fa86.slice/crio-e20fb1f32d0f3ab764f4d64f538731c710d71181500d02eda6c93947a6854ed9 WatchSource:0}: Error finding container e20fb1f32d0f3ab764f4d64f538731c710d71181500d02eda6c93947a6854ed9: Status 404 returned error can't find the container with id e20fb1f32d0f3ab764f4d64f538731c710d71181500d02eda6c93947a6854ed9
Sep 11 17:26:23.098545 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:26:23.098512    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7bcfd8f9cd-tfwmr"
Sep 11 17:26:23.181244 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:26:23.181167    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-84994f7ffc-nkmkm" event={"ID":"a0d4403a-011b-4bf2-ae88-a27c8ee0fa86","Type":"ContainerStarted","Data":"e20fb1f32d0f3ab764f4d64f538731c710d71181500d02eda6c93947a6854ed9"}
Sep 11 17:26:23.380297 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:26:23.380130    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-tfwmr"]
Sep 11 17:26:23.383293 ip-10-0-141-139 kubenswrapper[2577]: W0911 17:26:23.383266    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda24537f0_4662_40d2_8175_fdaf4d750d9f.slice/crio-e47c388d778b4b087acf8ae692184776b130e9b4340db531fecedce210e2f15f WatchSource:0}: Error finding container e47c388d778b4b087acf8ae692184776b130e9b4340db531fecedce210e2f15f: Status 404 returned error can't find the container with id e47c388d778b4b087acf8ae692184776b130e9b4340db531fecedce210e2f15f
Sep 11 17:26:24.187232 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:26:24.187198    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7bcfd8f9cd-tfwmr" event={"ID":"a24537f0-4662-40d2-8175-fdaf4d750d9f","Type":"ContainerStarted","Data":"e47c388d778b4b087acf8ae692184776b130e9b4340db531fecedce210e2f15f"}
Sep 11 17:26:26.403174 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:26:26.403136    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-tfwmr"]
Sep 11 17:26:27.205947 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:26:27.205908    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-84994f7ffc-nkmkm" event={"ID":"a0d4403a-011b-4bf2-ae88-a27c8ee0fa86","Type":"ContainerStarted","Data":"98e7e81eca89c9c2974aee138afa4ae013e6303a780957c1711161fca4aef38c"}
Sep 11 17:26:27.206162 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:26:27.205988    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-84994f7ffc-nkmkm"
Sep 11 17:26:27.207435 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:26:27.207414    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7bcfd8f9cd-tfwmr" event={"ID":"a24537f0-4662-40d2-8175-fdaf4d750d9f","Type":"ContainerStarted","Data":"778fb3b11cf420c17b07e4874dae0f1f7b2395db3f6cba69dd5bf577988ee89f"}
Sep 11 17:26:27.207538 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:26:27.207478    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-7bcfd8f9cd-tfwmr" podUID="a24537f0-4662-40d2-8175-fdaf4d750d9f" containerName="authorino" containerID="cri-o://778fb3b11cf420c17b07e4874dae0f1f7b2395db3f6cba69dd5bf577988ee89f" gracePeriod=30
Sep 11 17:26:27.229206 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:26:27.229157    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-84994f7ffc-nkmkm" podStartSLOduration=1.693589148 podStartE2EDuration="5.2291412s" totalImagesPullingTime="3.535552052s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 17:26:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 17:26:27.228819008 +0000 UTC m=+464.802039122" watchObservedRunningTime="2026-09-11 17:26:27.2291412 +0000 UTC m=+464.802361311"
Sep 11 17:26:27.251833 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:26:27.251781    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-7bcfd8f9cd-tfwmr" podStartSLOduration=2.105159677 podStartE2EDuration="5.251764207s" totalImagesPullingTime="3.14660453s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 17:26:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 17:26:27.250175912 +0000 UTC m=+464.823396026" watchObservedRunningTime="2026-09-11 17:26:27.251764207 +0000 UTC m=+464.824984319"
Sep 11 17:26:27.448050 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:26:27.448026    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7bcfd8f9cd-tfwmr"
Sep 11 17:26:27.560066 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:26:27.560026    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8pcr2\" (UniqueName: \"kubernetes.io/projected/a24537f0-4662-40d2-8175-fdaf4d750d9f-kube-api-access-8pcr2\") pod \"a24537f0-4662-40d2-8175-fdaf4d750d9f\" (UID: \"a24537f0-4662-40d2-8175-fdaf4d750d9f\") "
Sep 11 17:26:27.562388 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:26:27.562360    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a24537f0-4662-40d2-8175-fdaf4d750d9f-kube-api-access-8pcr2" (OuterVolumeSpecName: "kube-api-access-8pcr2") pod "a24537f0-4662-40d2-8175-fdaf4d750d9f" (UID: "a24537f0-4662-40d2-8175-fdaf4d750d9f"). InnerVolumeSpecName "kube-api-access-8pcr2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 17:26:27.661064 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:26:27.661024    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8pcr2\" (UniqueName: \"kubernetes.io/projected/a24537f0-4662-40d2-8175-fdaf4d750d9f-kube-api-access-8pcr2\") on node \"ip-10-0-141-139.ec2.internal\" DevicePath \"\""
Sep 11 17:26:28.213788 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:26:28.213747    2577 generic.go:356] "Generic (PLEG): container finished" podID="a24537f0-4662-40d2-8175-fdaf4d750d9f" containerID="778fb3b11cf420c17b07e4874dae0f1f7b2395db3f6cba69dd5bf577988ee89f" exitCode=0
Sep 11 17:26:28.213961 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:26:28.213812    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7bcfd8f9cd-tfwmr"
Sep 11 17:26:28.213961 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:26:28.213837    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7bcfd8f9cd-tfwmr" event={"ID":"a24537f0-4662-40d2-8175-fdaf4d750d9f","Type":"ContainerDied","Data":"778fb3b11cf420c17b07e4874dae0f1f7b2395db3f6cba69dd5bf577988ee89f"}
Sep 11 17:26:28.213961 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:26:28.213882    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7bcfd8f9cd-tfwmr" event={"ID":"a24537f0-4662-40d2-8175-fdaf4d750d9f","Type":"ContainerDied","Data":"e47c388d778b4b087acf8ae692184776b130e9b4340db531fecedce210e2f15f"}
Sep 11 17:26:28.213961 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:26:28.213900    2577 scope.go:117] "RemoveContainer" containerID="778fb3b11cf420c17b07e4874dae0f1f7b2395db3f6cba69dd5bf577988ee89f"
Sep 11 17:26:28.222864 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:26:28.222841    2577 scope.go:117] "RemoveContainer" containerID="778fb3b11cf420c17b07e4874dae0f1f7b2395db3f6cba69dd5bf577988ee89f"
Sep 11 17:26:28.223145 ip-10-0-141-139 kubenswrapper[2577]: E0911 17:26:28.223125    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"778fb3b11cf420c17b07e4874dae0f1f7b2395db3f6cba69dd5bf577988ee89f\": container with ID starting with 778fb3b11cf420c17b07e4874dae0f1f7b2395db3f6cba69dd5bf577988ee89f not found: ID does not exist" containerID="778fb3b11cf420c17b07e4874dae0f1f7b2395db3f6cba69dd5bf577988ee89f"
Sep 11 17:26:28.223194 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:26:28.223153    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"778fb3b11cf420c17b07e4874dae0f1f7b2395db3f6cba69dd5bf577988ee89f"} err="failed to get container status \"778fb3b11cf420c17b07e4874dae0f1f7b2395db3f6cba69dd5bf577988ee89f\": rpc error: code = NotFound desc = could not find container \"778fb3b11cf420c17b07e4874dae0f1f7b2395db3f6cba69dd5bf577988ee89f\": container with ID starting with 778fb3b11cf420c17b07e4874dae0f1f7b2395db3f6cba69dd5bf577988ee89f not found: ID does not exist"
Sep 11 17:26:28.234841 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:26:28.234817    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-tfwmr"]
Sep 11 17:26:28.241379 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:26:28.241356    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-tfwmr"]
Sep 11 17:26:28.926710 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:26:28.926675    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a24537f0-4662-40d2-8175-fdaf4d750d9f" path="/var/lib/kubelet/pods/a24537f0-4662-40d2-8175-fdaf4d750d9f/volumes"
Sep 11 17:26:38.215614 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:26:38.215581    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-84994f7ffc-nkmkm"
Sep 11 17:26:56.556882 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:26:56.556845    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-557f876f69-6qpwj"]
Sep 11 17:26:56.557476 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:26:56.557433    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a24537f0-4662-40d2-8175-fdaf4d750d9f" containerName="authorino"
Sep 11 17:26:56.557476 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:26:56.557451    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="a24537f0-4662-40d2-8175-fdaf4d750d9f" containerName="authorino"
Sep 11 17:26:56.557613 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:26:56.557553    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="a24537f0-4662-40d2-8175-fdaf4d750d9f" containerName="authorino"
Sep 11 17:26:56.560305 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:26:56.560275    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-557f876f69-6qpwj"
Sep 11 17:26:56.562517 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:26:56.562475    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-nx4ln\""
Sep 11 17:26:56.567737 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:26:56.567446    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-557f876f69-6qpwj"]
Sep 11 17:26:56.727176 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:26:56.727137    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-thfjv\" (UniqueName: \"kubernetes.io/projected/57195130-33a8-4a04-be39-db490ef63f4f-kube-api-access-thfjv\") pod \"authorino-557f876f69-6qpwj\" (UID: \"57195130-33a8-4a04-be39-db490ef63f4f\") " pod="kuadrant-system/authorino-557f876f69-6qpwj"
Sep 11 17:26:56.792936 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:26:56.792896    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-557f876f69-6qpwj"]
Sep 11 17:26:56.793313 ip-10-0-141-139 kubenswrapper[2577]: E0911 17:26:56.793288    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kube-api-access-thfjv], unattached volumes=[], failed to process volumes=[]: context canceled" pod="kuadrant-system/authorino-557f876f69-6qpwj" podUID="57195130-33a8-4a04-be39-db490ef63f4f"
Sep 11 17:26:56.819037 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:26:56.818958    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-895cf4fb6-nppzp"]
Sep 11 17:26:56.821236 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:26:56.821216    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-895cf4fb6-nppzp"
Sep 11 17:26:56.826791 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:26:56.826764    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-895cf4fb6-nppzp"]
Sep 11 17:26:56.828127 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:26:56.828102    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-thfjv\" (UniqueName: \"kubernetes.io/projected/57195130-33a8-4a04-be39-db490ef63f4f-kube-api-access-thfjv\") pod \"authorino-557f876f69-6qpwj\" (UID: \"57195130-33a8-4a04-be39-db490ef63f4f\") " pod="kuadrant-system/authorino-557f876f69-6qpwj"
Sep 11 17:26:56.842313 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:26:56.842287    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-thfjv\" (UniqueName: \"kubernetes.io/projected/57195130-33a8-4a04-be39-db490ef63f4f-kube-api-access-thfjv\") pod \"authorino-557f876f69-6qpwj\" (UID: \"57195130-33a8-4a04-be39-db490ef63f4f\") " pod="kuadrant-system/authorino-557f876f69-6qpwj"
Sep 11 17:26:56.929042 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:26:56.928996    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fv846\" (UniqueName: \"kubernetes.io/projected/9d5f5313-6a90-4253-8b9f-8850b7404dd8-kube-api-access-fv846\") pod \"authorino-895cf4fb6-nppzp\" (UID: \"9d5f5313-6a90-4253-8b9f-8850b7404dd8\") " pod="kuadrant-system/authorino-895cf4fb6-nppzp"
Sep 11 17:26:57.030566 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:26:57.030492    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fv846\" (UniqueName: \"kubernetes.io/projected/9d5f5313-6a90-4253-8b9f-8850b7404dd8-kube-api-access-fv846\") pod \"authorino-895cf4fb6-nppzp\" (UID: \"9d5f5313-6a90-4253-8b9f-8850b7404dd8\") " pod="kuadrant-system/authorino-895cf4fb6-nppzp"
Sep 11 17:26:57.044362 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:26:57.044323    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fv846\" (UniqueName: \"kubernetes.io/projected/9d5f5313-6a90-4253-8b9f-8850b7404dd8-kube-api-access-fv846\") pod \"authorino-895cf4fb6-nppzp\" (UID: \"9d5f5313-6a90-4253-8b9f-8850b7404dd8\") " pod="kuadrant-system/authorino-895cf4fb6-nppzp"
Sep 11 17:26:57.131104 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:26:57.131001    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-895cf4fb6-nppzp"
Sep 11 17:26:57.347301 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:26:57.347259    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-557f876f69-6qpwj"
Sep 11 17:26:57.354138 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:26:57.354114    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-557f876f69-6qpwj"
Sep 11 17:26:57.400360 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:26:57.400331    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-895cf4fb6-nppzp"]
Sep 11 17:26:57.402917 ip-10-0-141-139 kubenswrapper[2577]: W0911 17:26:57.402885    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod9d5f5313_6a90_4253_8b9f_8850b7404dd8.slice/crio-d5f17f9040615586677b14e45a964f583bd439ae0e592a1f55a3bb70ec9d3173 WatchSource:0}: Error finding container d5f17f9040615586677b14e45a964f583bd439ae0e592a1f55a3bb70ec9d3173: Status 404 returned error can't find the container with id d5f17f9040615586677b14e45a964f583bd439ae0e592a1f55a3bb70ec9d3173
Sep 11 17:26:57.433870 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:26:57.433835    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-thfjv\" (UniqueName: \"kubernetes.io/projected/57195130-33a8-4a04-be39-db490ef63f4f-kube-api-access-thfjv\") pod \"57195130-33a8-4a04-be39-db490ef63f4f\" (UID: \"57195130-33a8-4a04-be39-db490ef63f4f\") "
Sep 11 17:26:57.436127 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:26:57.436096    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/57195130-33a8-4a04-be39-db490ef63f4f-kube-api-access-thfjv" (OuterVolumeSpecName: "kube-api-access-thfjv") pod "57195130-33a8-4a04-be39-db490ef63f4f" (UID: "57195130-33a8-4a04-be39-db490ef63f4f"). InnerVolumeSpecName "kube-api-access-thfjv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 17:26:57.535354 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:26:57.535315    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-thfjv\" (UniqueName: \"kubernetes.io/projected/57195130-33a8-4a04-be39-db490ef63f4f-kube-api-access-thfjv\") on node \"ip-10-0-141-139.ec2.internal\" DevicePath \"\""
Sep 11 17:26:58.354409 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:26:58.354376    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-557f876f69-6qpwj"
Sep 11 17:26:58.354409 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:26:58.354405    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-895cf4fb6-nppzp" event={"ID":"9d5f5313-6a90-4253-8b9f-8850b7404dd8","Type":"ContainerStarted","Data":"2357c104ce77c90cb226028741a429eae64588360e3d9a39de4c5a23d1f13b54"}
Sep 11 17:26:58.354910 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:26:58.354444    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-895cf4fb6-nppzp" event={"ID":"9d5f5313-6a90-4253-8b9f-8850b7404dd8","Type":"ContainerStarted","Data":"d5f17f9040615586677b14e45a964f583bd439ae0e592a1f55a3bb70ec9d3173"}
Sep 11 17:26:58.369795 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:26:58.369740    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-895cf4fb6-nppzp" podStartSLOduration=1.5985681299999999 podStartE2EDuration="2.369722194s" totalImagesPullingTime="771.154064ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 17:26:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 17:26:58.369517224 +0000 UTC m=+495.942737334" watchObservedRunningTime="2026-09-11 17:26:58.369722194 +0000 UTC m=+495.942942317"
Sep 11 17:26:58.401165 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:26:58.401134    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-557f876f69-6qpwj"]
Sep 11 17:26:58.409767 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:26:58.409737    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-557f876f69-6qpwj"]
Sep 11 17:26:58.926876 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:26:58.926841    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="57195130-33a8-4a04-be39-db490ef63f4f" path="/var/lib/kubelet/pods/57195130-33a8-4a04-be39-db490ef63f4f/volumes"
Sep 11 17:27:00.603789 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:27:00.603750    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-895cf4fb6-nppzp"]
Sep 11 17:27:00.604301 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:27:00.603943    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-895cf4fb6-nppzp" podUID="9d5f5313-6a90-4253-8b9f-8850b7404dd8" containerName="authorino" containerID="cri-o://2357c104ce77c90cb226028741a429eae64588360e3d9a39de4c5a23d1f13b54" gracePeriod=30
Sep 11 17:27:00.840294 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:27:00.840263    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-895cf4fb6-nppzp"
Sep 11 17:27:00.965313 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:27:00.965289    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fv846\" (UniqueName: \"kubernetes.io/projected/9d5f5313-6a90-4253-8b9f-8850b7404dd8-kube-api-access-fv846\") pod \"9d5f5313-6a90-4253-8b9f-8850b7404dd8\" (UID: \"9d5f5313-6a90-4253-8b9f-8850b7404dd8\") "
Sep 11 17:27:00.967452 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:27:00.967414    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9d5f5313-6a90-4253-8b9f-8850b7404dd8-kube-api-access-fv846" (OuterVolumeSpecName: "kube-api-access-fv846") pod "9d5f5313-6a90-4253-8b9f-8850b7404dd8" (UID: "9d5f5313-6a90-4253-8b9f-8850b7404dd8"). InnerVolumeSpecName "kube-api-access-fv846". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 17:27:01.066088 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:27:01.066054    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fv846\" (UniqueName: \"kubernetes.io/projected/9d5f5313-6a90-4253-8b9f-8850b7404dd8-kube-api-access-fv846\") on node \"ip-10-0-141-139.ec2.internal\" DevicePath \"\""
Sep 11 17:27:01.372114 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:27:01.372027    2577 generic.go:356] "Generic (PLEG): container finished" podID="9d5f5313-6a90-4253-8b9f-8850b7404dd8" containerID="2357c104ce77c90cb226028741a429eae64588360e3d9a39de4c5a23d1f13b54" exitCode=0
Sep 11 17:27:01.372114 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:27:01.372070    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-895cf4fb6-nppzp"
Sep 11 17:27:01.372318 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:27:01.372108    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-895cf4fb6-nppzp" event={"ID":"9d5f5313-6a90-4253-8b9f-8850b7404dd8","Type":"ContainerDied","Data":"2357c104ce77c90cb226028741a429eae64588360e3d9a39de4c5a23d1f13b54"}
Sep 11 17:27:01.372318 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:27:01.372158    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-895cf4fb6-nppzp" event={"ID":"9d5f5313-6a90-4253-8b9f-8850b7404dd8","Type":"ContainerDied","Data":"d5f17f9040615586677b14e45a964f583bd439ae0e592a1f55a3bb70ec9d3173"}
Sep 11 17:27:01.372318 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:27:01.372176    2577 scope.go:117] "RemoveContainer" containerID="2357c104ce77c90cb226028741a429eae64588360e3d9a39de4c5a23d1f13b54"
Sep 11 17:27:01.381634 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:27:01.381618    2577 scope.go:117] "RemoveContainer" containerID="2357c104ce77c90cb226028741a429eae64588360e3d9a39de4c5a23d1f13b54"
Sep 11 17:27:01.381894 ip-10-0-141-139 kubenswrapper[2577]: E0911 17:27:01.381876    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2357c104ce77c90cb226028741a429eae64588360e3d9a39de4c5a23d1f13b54\": container with ID starting with 2357c104ce77c90cb226028741a429eae64588360e3d9a39de4c5a23d1f13b54 not found: ID does not exist" containerID="2357c104ce77c90cb226028741a429eae64588360e3d9a39de4c5a23d1f13b54"
Sep 11 17:27:01.381963 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:27:01.381907    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2357c104ce77c90cb226028741a429eae64588360e3d9a39de4c5a23d1f13b54"} err="failed to get container status \"2357c104ce77c90cb226028741a429eae64588360e3d9a39de4c5a23d1f13b54\": rpc error: code = NotFound desc = could not find container \"2357c104ce77c90cb226028741a429eae64588360e3d9a39de4c5a23d1f13b54\": container with ID starting with 2357c104ce77c90cb226028741a429eae64588360e3d9a39de4c5a23d1f13b54 not found: ID does not exist"
Sep 11 17:27:01.395978 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:27:01.395957    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-895cf4fb6-nppzp"]
Sep 11 17:27:01.403877 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:27:01.403857    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-895cf4fb6-nppzp"]
Sep 11 17:27:02.926834 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:27:02.926799    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9d5f5313-6a90-4253-8b9f-8850b7404dd8" path="/var/lib/kubelet/pods/9d5f5313-6a90-4253-8b9f-8850b7404dd8/volumes"
Sep 11 17:27:40.639048 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:27:40.638974    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-59fbc47bf8-r7l5l"]
Sep 11 17:27:40.639524 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:27:40.639351    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9d5f5313-6a90-4253-8b9f-8850b7404dd8" containerName="authorino"
Sep 11 17:27:40.639524 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:27:40.639362    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="9d5f5313-6a90-4253-8b9f-8850b7404dd8" containerName="authorino"
Sep 11 17:27:40.639524 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:27:40.639437    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="9d5f5313-6a90-4253-8b9f-8850b7404dd8" containerName="authorino"
Sep 11 17:27:40.641911 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:27:40.641896    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-59fbc47bf8-r7l5l"
Sep 11 17:27:40.644932 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:27:40.644910    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins\""
Sep 11 17:27:40.653950 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:27:40.653926    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-59fbc47bf8-r7l5l"]
Sep 11 17:27:40.699385 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:27:40.699350    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c779770c-0017-4b67-bf14-415f9ec89cc2-plugins-config-volume\") pod \"payload-pre-processing-59fbc47bf8-r7l5l\" (UID: \"c779770c-0017-4b67-bf14-415f9ec89cc2\") " pod="openshift-ingress/payload-pre-processing-59fbc47bf8-r7l5l"
Sep 11 17:27:40.699385 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:27:40.699391    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cmstt\" (UniqueName: \"kubernetes.io/projected/c779770c-0017-4b67-bf14-415f9ec89cc2-kube-api-access-cmstt\") pod \"payload-pre-processing-59fbc47bf8-r7l5l\" (UID: \"c779770c-0017-4b67-bf14-415f9ec89cc2\") " pod="openshift-ingress/payload-pre-processing-59fbc47bf8-r7l5l"
Sep 11 17:27:40.753421 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:27:40.753384    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-nkmkm"]
Sep 11 17:27:40.799963 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:27:40.799931    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c779770c-0017-4b67-bf14-415f9ec89cc2-plugins-config-volume\") pod \"payload-pre-processing-59fbc47bf8-r7l5l\" (UID: \"c779770c-0017-4b67-bf14-415f9ec89cc2\") " pod="openshift-ingress/payload-pre-processing-59fbc47bf8-r7l5l"
Sep 11 17:27:40.800151 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:27:40.799983    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cmstt\" (UniqueName: \"kubernetes.io/projected/c779770c-0017-4b67-bf14-415f9ec89cc2-kube-api-access-cmstt\") pod \"payload-pre-processing-59fbc47bf8-r7l5l\" (UID: \"c779770c-0017-4b67-bf14-415f9ec89cc2\") " pod="openshift-ingress/payload-pre-processing-59fbc47bf8-r7l5l"
Sep 11 17:27:40.800737 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:27:40.800718    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c779770c-0017-4b67-bf14-415f9ec89cc2-plugins-config-volume\") pod \"payload-pre-processing-59fbc47bf8-r7l5l\" (UID: \"c779770c-0017-4b67-bf14-415f9ec89cc2\") " pod="openshift-ingress/payload-pre-processing-59fbc47bf8-r7l5l"
Sep 11 17:27:40.809633 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:27:40.809607    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cmstt\" (UniqueName: \"kubernetes.io/projected/c779770c-0017-4b67-bf14-415f9ec89cc2-kube-api-access-cmstt\") pod \"payload-pre-processing-59fbc47bf8-r7l5l\" (UID: \"c779770c-0017-4b67-bf14-415f9ec89cc2\") " pod="openshift-ingress/payload-pre-processing-59fbc47bf8-r7l5l"
Sep 11 17:27:40.953001 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:27:40.952912    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-59fbc47bf8-r7l5l"
Sep 11 17:27:41.234809 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:27:41.234754    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-59fbc47bf8-r7l5l"]
Sep 11 17:27:41.237378 ip-10-0-141-139 kubenswrapper[2577]: W0911 17:27:41.237352    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc779770c_0017_4b67_bf14_415f9ec89cc2.slice/crio-77063ae3c7568a828f8010931bd29f243db1e7b45cef146cb222a98101270cc4 WatchSource:0}: Error finding container 77063ae3c7568a828f8010931bd29f243db1e7b45cef146cb222a98101270cc4: Status 404 returned error can't find the container with id 77063ae3c7568a828f8010931bd29f243db1e7b45cef146cb222a98101270cc4
Sep 11 17:27:41.558234 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:27:41.558152    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-59fbc47bf8-r7l5l" event={"ID":"c779770c-0017-4b67-bf14-415f9ec89cc2","Type":"ContainerStarted","Data":"77063ae3c7568a828f8010931bd29f243db1e7b45cef146cb222a98101270cc4"}
Sep 11 17:27:43.569759 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:27:43.569678    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-59fbc47bf8-r7l5l" event={"ID":"c779770c-0017-4b67-bf14-415f9ec89cc2","Type":"ContainerStarted","Data":"7f9a3d873d69e62c1a278bd17848bbcec50db5583eb57f48e09b3ba53faca665"}
Sep 11 17:27:43.570131 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:27:43.569800    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-59fbc47bf8-r7l5l"
Sep 11 17:27:43.587353 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:27:43.587304    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-59fbc47bf8-r7l5l" podStartSLOduration=1.524813227 podStartE2EDuration="3.587288298s" totalImagesPullingTime="2.062475071s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 17:27:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 17:27:43.587051694 +0000 UTC m=+541.160271807" watchObservedRunningTime="2026-09-11 17:27:43.587288298 +0000 UTC m=+541.160508486"
Sep 11 17:27:54.577305 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:27:54.577274    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-59fbc47bf8-r7l5l"
Sep 11 17:27:57.209868 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:27:57.209805    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-jrgvr"]
Sep 11 17:27:57.214193 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:27:57.214166    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-jrgvr"
Sep 11 17:27:57.217420 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:27:57.217391    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\""
Sep 11 17:27:57.217809 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:27:57.217789    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-2-simulated-kserve-self-signed-certs\""
Sep 11 17:27:57.217809 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:27:57.217806    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-d66tb\""
Sep 11 17:27:57.217990 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:27:57.217843    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\""
Sep 11 17:27:57.228651 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:27:57.228441    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-jrgvr"]
Sep 11 17:27:57.344276 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:27:57.344229    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/53d4883a-1f41-4db0-a78d-fc4d2fcf1857-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-jrgvr\" (UID: \"53d4883a-1f41-4db0-a78d-fc4d2fcf1857\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-jrgvr"
Sep 11 17:27:57.344276 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:27:57.344273    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mvv7t\" (UniqueName: \"kubernetes.io/projected/53d4883a-1f41-4db0-a78d-fc4d2fcf1857-kube-api-access-mvv7t\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-jrgvr\" (UID: \"53d4883a-1f41-4db0-a78d-fc4d2fcf1857\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-jrgvr"
Sep 11 17:27:57.344544 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:27:57.344318    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/53d4883a-1f41-4db0-a78d-fc4d2fcf1857-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-jrgvr\" (UID: \"53d4883a-1f41-4db0-a78d-fc4d2fcf1857\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-jrgvr"
Sep 11 17:27:57.344544 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:27:57.344428    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/53d4883a-1f41-4db0-a78d-fc4d2fcf1857-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-jrgvr\" (UID: \"53d4883a-1f41-4db0-a78d-fc4d2fcf1857\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-jrgvr"
Sep 11 17:27:57.344544 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:27:57.344491    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/53d4883a-1f41-4db0-a78d-fc4d2fcf1857-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-jrgvr\" (UID: \"53d4883a-1f41-4db0-a78d-fc4d2fcf1857\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-jrgvr"
Sep 11 17:27:57.344667 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:27:57.344583    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/53d4883a-1f41-4db0-a78d-fc4d2fcf1857-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-jrgvr\" (UID: \"53d4883a-1f41-4db0-a78d-fc4d2fcf1857\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-jrgvr"
Sep 11 17:27:57.445935 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:27:57.445895    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/53d4883a-1f41-4db0-a78d-fc4d2fcf1857-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-jrgvr\" (UID: \"53d4883a-1f41-4db0-a78d-fc4d2fcf1857\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-jrgvr"
Sep 11 17:27:57.446124 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:27:57.445945    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/53d4883a-1f41-4db0-a78d-fc4d2fcf1857-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-jrgvr\" (UID: \"53d4883a-1f41-4db0-a78d-fc4d2fcf1857\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-jrgvr"
Sep 11 17:27:57.446124 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:27:57.445977    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/53d4883a-1f41-4db0-a78d-fc4d2fcf1857-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-jrgvr\" (UID: \"53d4883a-1f41-4db0-a78d-fc4d2fcf1857\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-jrgvr"
Sep 11 17:27:57.446124 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:27:57.446008    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/53d4883a-1f41-4db0-a78d-fc4d2fcf1857-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-jrgvr\" (UID: \"53d4883a-1f41-4db0-a78d-fc4d2fcf1857\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-jrgvr"
Sep 11 17:27:57.446124 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:27:57.446025    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mvv7t\" (UniqueName: \"kubernetes.io/projected/53d4883a-1f41-4db0-a78d-fc4d2fcf1857-kube-api-access-mvv7t\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-jrgvr\" (UID: \"53d4883a-1f41-4db0-a78d-fc4d2fcf1857\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-jrgvr"
Sep 11 17:27:57.446124 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:27:57.446049    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/53d4883a-1f41-4db0-a78d-fc4d2fcf1857-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-jrgvr\" (UID: \"53d4883a-1f41-4db0-a78d-fc4d2fcf1857\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-jrgvr"
Sep 11 17:27:57.446414 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:27:57.446392    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/53d4883a-1f41-4db0-a78d-fc4d2fcf1857-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-jrgvr\" (UID: \"53d4883a-1f41-4db0-a78d-fc4d2fcf1857\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-jrgvr"
Sep 11 17:27:57.446464 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:27:57.446412    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/53d4883a-1f41-4db0-a78d-fc4d2fcf1857-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-jrgvr\" (UID: \"53d4883a-1f41-4db0-a78d-fc4d2fcf1857\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-jrgvr"
Sep 11 17:27:57.446464 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:27:57.446452    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/53d4883a-1f41-4db0-a78d-fc4d2fcf1857-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-jrgvr\" (UID: \"53d4883a-1f41-4db0-a78d-fc4d2fcf1857\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-jrgvr"
Sep 11 17:27:57.448537 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:27:57.448494    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/53d4883a-1f41-4db0-a78d-fc4d2fcf1857-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-jrgvr\" (UID: \"53d4883a-1f41-4db0-a78d-fc4d2fcf1857\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-jrgvr"
Sep 11 17:27:57.448788 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:27:57.448771    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/53d4883a-1f41-4db0-a78d-fc4d2fcf1857-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-jrgvr\" (UID: \"53d4883a-1f41-4db0-a78d-fc4d2fcf1857\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-jrgvr"
Sep 11 17:27:57.454916 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:27:57.454894    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mvv7t\" (UniqueName: \"kubernetes.io/projected/53d4883a-1f41-4db0-a78d-fc4d2fcf1857-kube-api-access-mvv7t\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-jrgvr\" (UID: \"53d4883a-1f41-4db0-a78d-fc4d2fcf1857\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-jrgvr"
Sep 11 17:27:57.525042 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:27:57.524988    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-jrgvr"
Sep 11 17:27:57.734142 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:27:57.734109    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-nkmkm"]
Sep 11 17:27:57.799212 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:27:57.799139    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-jrgvr"]
Sep 11 17:27:57.802776 ip-10-0-141-139 kubenswrapper[2577]: W0911 17:27:57.802742    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod53d4883a_1f41_4db0_a78d_fc4d2fcf1857.slice/crio-27dcdd119b6c01196ba94e95cd9c2ca64e7a2908f324adb2ff6f94fcb5780676 WatchSource:0}: Error finding container 27dcdd119b6c01196ba94e95cd9c2ca64e7a2908f324adb2ff6f94fcb5780676: Status 404 returned error can't find the container with id 27dcdd119b6c01196ba94e95cd9c2ca64e7a2908f324adb2ff6f94fcb5780676
Sep 11 17:27:58.643409 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:27:58.643373    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-jrgvr" event={"ID":"53d4883a-1f41-4db0-a78d-fc4d2fcf1857","Type":"ContainerStarted","Data":"27dcdd119b6c01196ba94e95cd9c2ca64e7a2908f324adb2ff6f94fcb5780676"}
Sep 11 17:27:59.650491 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:27:59.650454    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-jrgvr" event={"ID":"53d4883a-1f41-4db0-a78d-fc4d2fcf1857","Type":"ContainerStarted","Data":"eeba01db434f0db47667a0c2b6f2c53935beb1fdb2812f4e0f0bef5e601dbbbe"}
Sep 11 17:27:59.668518 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:27:59.668444    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-jrgvr" podStartSLOduration=1.593096267 podStartE2EDuration="2.66842844s" totalImagesPullingTime="1.075332173s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 17:27:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 17:27:59.666407981 +0000 UTC m=+557.239628093" watchObservedRunningTime="2026-09-11 17:27:59.66842844 +0000 UTC m=+557.241648552"
Sep 11 17:28:07.525953 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:28:07.525912    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-jrgvr"
Sep 11 17:28:07.526382 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:28:07.526011    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-jrgvr"
Sep 11 17:28:07.538472 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:28:07.538439    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-jrgvr"
Sep 11 17:28:07.699110 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:28:07.699076    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-jrgvr"
Sep 11 17:28:17.393893 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:28:17.393848    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-nkmkm"]
Sep 11 17:28:23.016442 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:28:23.016406    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-nkmkm"]
Sep 11 17:28:34.720823 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:28:34.720786    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-598466548d-6vpzl"]
Sep 11 17:28:34.724292 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:28:34.724273    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-6vpzl"
Sep 11 17:28:34.726820 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:28:34.726799    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-trlp-test-simulated-kserve-self-signed-certs\""
Sep 11 17:28:34.736316 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:28:34.736282    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-598466548d-6vpzl"]
Sep 11 17:28:34.797070 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:28:34.796705    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/334bafcb-51cd-4cc6-b9ea-da39d516395b-dshm\") pod \"e2e-trlp-test-simulated-kserve-598466548d-6vpzl\" (UID: \"334bafcb-51cd-4cc6-b9ea-da39d516395b\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-6vpzl"
Sep 11 17:28:34.797070 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:28:34.796783    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/334bafcb-51cd-4cc6-b9ea-da39d516395b-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-598466548d-6vpzl\" (UID: \"334bafcb-51cd-4cc6-b9ea-da39d516395b\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-6vpzl"
Sep 11 17:28:34.797070 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:28:34.796824    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/334bafcb-51cd-4cc6-b9ea-da39d516395b-home\") pod \"e2e-trlp-test-simulated-kserve-598466548d-6vpzl\" (UID: \"334bafcb-51cd-4cc6-b9ea-da39d516395b\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-6vpzl"
Sep 11 17:28:34.797070 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:28:34.796858    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7wxmh\" (UniqueName: \"kubernetes.io/projected/334bafcb-51cd-4cc6-b9ea-da39d516395b-kube-api-access-7wxmh\") pod \"e2e-trlp-test-simulated-kserve-598466548d-6vpzl\" (UID: \"334bafcb-51cd-4cc6-b9ea-da39d516395b\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-6vpzl"
Sep 11 17:28:34.797070 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:28:34.796914    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/334bafcb-51cd-4cc6-b9ea-da39d516395b-model-cache\") pod \"e2e-trlp-test-simulated-kserve-598466548d-6vpzl\" (UID: \"334bafcb-51cd-4cc6-b9ea-da39d516395b\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-6vpzl"
Sep 11 17:28:34.797070 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:28:34.796938    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/334bafcb-51cd-4cc6-b9ea-da39d516395b-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-598466548d-6vpzl\" (UID: \"334bafcb-51cd-4cc6-b9ea-da39d516395b\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-6vpzl"
Sep 11 17:28:34.898235 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:28:34.898199    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/334bafcb-51cd-4cc6-b9ea-da39d516395b-dshm\") pod \"e2e-trlp-test-simulated-kserve-598466548d-6vpzl\" (UID: \"334bafcb-51cd-4cc6-b9ea-da39d516395b\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-6vpzl"
Sep 11 17:28:34.898235 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:28:34.898236    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/334bafcb-51cd-4cc6-b9ea-da39d516395b-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-598466548d-6vpzl\" (UID: \"334bafcb-51cd-4cc6-b9ea-da39d516395b\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-6vpzl"
Sep 11 17:28:34.898529 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:28:34.898262    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/334bafcb-51cd-4cc6-b9ea-da39d516395b-home\") pod \"e2e-trlp-test-simulated-kserve-598466548d-6vpzl\" (UID: \"334bafcb-51cd-4cc6-b9ea-da39d516395b\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-6vpzl"
Sep 11 17:28:34.898529 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:28:34.898281    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7wxmh\" (UniqueName: \"kubernetes.io/projected/334bafcb-51cd-4cc6-b9ea-da39d516395b-kube-api-access-7wxmh\") pod \"e2e-trlp-test-simulated-kserve-598466548d-6vpzl\" (UID: \"334bafcb-51cd-4cc6-b9ea-da39d516395b\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-6vpzl"
Sep 11 17:28:34.898529 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:28:34.898324    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/334bafcb-51cd-4cc6-b9ea-da39d516395b-model-cache\") pod \"e2e-trlp-test-simulated-kserve-598466548d-6vpzl\" (UID: \"334bafcb-51cd-4cc6-b9ea-da39d516395b\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-6vpzl"
Sep 11 17:28:34.898529 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:28:34.898340    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/334bafcb-51cd-4cc6-b9ea-da39d516395b-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-598466548d-6vpzl\" (UID: \"334bafcb-51cd-4cc6-b9ea-da39d516395b\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-6vpzl"
Sep 11 17:28:34.898784 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:28:34.898764    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/334bafcb-51cd-4cc6-b9ea-da39d516395b-home\") pod \"e2e-trlp-test-simulated-kserve-598466548d-6vpzl\" (UID: \"334bafcb-51cd-4cc6-b9ea-da39d516395b\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-6vpzl"
Sep 11 17:28:34.898837 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:28:34.898797    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/334bafcb-51cd-4cc6-b9ea-da39d516395b-model-cache\") pod \"e2e-trlp-test-simulated-kserve-598466548d-6vpzl\" (UID: \"334bafcb-51cd-4cc6-b9ea-da39d516395b\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-6vpzl"
Sep 11 17:28:34.898837 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:28:34.898816    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/334bafcb-51cd-4cc6-b9ea-da39d516395b-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-598466548d-6vpzl\" (UID: \"334bafcb-51cd-4cc6-b9ea-da39d516395b\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-6vpzl"
Sep 11 17:28:34.900729 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:28:34.900706    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/334bafcb-51cd-4cc6-b9ea-da39d516395b-dshm\") pod \"e2e-trlp-test-simulated-kserve-598466548d-6vpzl\" (UID: \"334bafcb-51cd-4cc6-b9ea-da39d516395b\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-6vpzl"
Sep 11 17:28:34.900863 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:28:34.900847    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/334bafcb-51cd-4cc6-b9ea-da39d516395b-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-598466548d-6vpzl\" (UID: \"334bafcb-51cd-4cc6-b9ea-da39d516395b\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-6vpzl"
Sep 11 17:28:34.906156 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:28:34.906134    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7wxmh\" (UniqueName: \"kubernetes.io/projected/334bafcb-51cd-4cc6-b9ea-da39d516395b-kube-api-access-7wxmh\") pod \"e2e-trlp-test-simulated-kserve-598466548d-6vpzl\" (UID: \"334bafcb-51cd-4cc6-b9ea-da39d516395b\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-6vpzl"
Sep 11 17:28:35.034218 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:28:35.034122    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-6vpzl"
Sep 11 17:28:35.307362 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:28:35.307337    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-598466548d-6vpzl"]
Sep 11 17:28:35.309866 ip-10-0-141-139 kubenswrapper[2577]: W0911 17:28:35.309836    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod334bafcb_51cd_4cc6_b9ea_da39d516395b.slice/crio-a23e136685fba31cdd5726cd82884d3324489faf6b7656f4273f38e5a24b9e0c WatchSource:0}: Error finding container a23e136685fba31cdd5726cd82884d3324489faf6b7656f4273f38e5a24b9e0c: Status 404 returned error can't find the container with id a23e136685fba31cdd5726cd82884d3324489faf6b7656f4273f38e5a24b9e0c
Sep 11 17:28:35.822142 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:28:35.822104    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-6vpzl" event={"ID":"334bafcb-51cd-4cc6-b9ea-da39d516395b","Type":"ContainerStarted","Data":"932e87789f8185905542bac48e33899f7a32f1f44b3d851df447d03de135e67b"}
Sep 11 17:28:35.822645 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:28:35.822158    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-6vpzl" event={"ID":"334bafcb-51cd-4cc6-b9ea-da39d516395b","Type":"ContainerStarted","Data":"a23e136685fba31cdd5726cd82884d3324489faf6b7656f4273f38e5a24b9e0c"}
Sep 11 17:28:35.839979 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:28:35.839930    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-6vpzl" podStartSLOduration=1.666996239 podStartE2EDuration="1.839909399s" totalImagesPullingTime="172.91316ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 17:28:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 17:28:35.839106893 +0000 UTC m=+593.412327041" watchObservedRunningTime="2026-09-11 17:28:35.839909399 +0000 UTC m=+593.413129512"
Sep 11 17:28:36.102420 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:28:36.102321    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-nkmkm"]
Sep 11 17:28:45.034695 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:28:45.034654    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-6vpzl"
Sep 11 17:28:45.035267 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:28:45.034742    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-6vpzl"
Sep 11 17:28:45.047175 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:28:45.047144    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-6vpzl"
Sep 11 17:28:45.482881 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:28:45.482853    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-nkmkm"]
Sep 11 17:28:45.885244 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:28:45.885152    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-6vpzl"
Sep 11 17:28:45.894338 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:28:45.894309    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-xjqr7"]
Sep 11 17:28:45.896960 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:28:45.896938    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-xjqr7"
Sep 11 17:28:45.898618 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:28:45.898591    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cf363870-c91b-4c09-8a71-c1daa17229c8-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-xjqr7\" (UID: \"cf363870-c91b-4c09-8a71-c1daa17229c8\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-xjqr7"
Sep 11 17:28:45.898748 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:28:45.898652    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cf363870-c91b-4c09-8a71-c1daa17229c8-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-xjqr7\" (UID: \"cf363870-c91b-4c09-8a71-c1daa17229c8\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-xjqr7"
Sep 11 17:28:45.898748 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:28:45.898710    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cf363870-c91b-4c09-8a71-c1daa17229c8-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-xjqr7\" (UID: \"cf363870-c91b-4c09-8a71-c1daa17229c8\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-xjqr7"
Sep 11 17:28:45.898874 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:28:45.898757    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cf363870-c91b-4c09-8a71-c1daa17229c8-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-xjqr7\" (UID: \"cf363870-c91b-4c09-8a71-c1daa17229c8\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-xjqr7"
Sep 11 17:28:45.898874 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:28:45.898804    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cf363870-c91b-4c09-8a71-c1daa17229c8-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-xjqr7\" (UID: \"cf363870-c91b-4c09-8a71-c1daa17229c8\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-xjqr7"
Sep 11 17:28:45.898874 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:28:45.898831    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7nxnc\" (UniqueName: \"kubernetes.io/projected/cf363870-c91b-4c09-8a71-c1daa17229c8-kube-api-access-7nxnc\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-xjqr7\" (UID: \"cf363870-c91b-4c09-8a71-c1daa17229c8\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-xjqr7"
Sep 11 17:28:45.899569 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:28:45.899548    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"facebook-opt-125m-simulated-kserve-self-signed-certs\""
Sep 11 17:28:45.908729 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:28:45.908704    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-xjqr7"]
Sep 11 17:28:45.999497 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:28:45.999462    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cf363870-c91b-4c09-8a71-c1daa17229c8-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-xjqr7\" (UID: \"cf363870-c91b-4c09-8a71-c1daa17229c8\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-xjqr7"
Sep 11 17:28:45.999497 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:28:45.999524    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7nxnc\" (UniqueName: \"kubernetes.io/projected/cf363870-c91b-4c09-8a71-c1daa17229c8-kube-api-access-7nxnc\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-xjqr7\" (UID: \"cf363870-c91b-4c09-8a71-c1daa17229c8\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-xjqr7"
Sep 11 17:28:45.999789 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:28:45.999689    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cf363870-c91b-4c09-8a71-c1daa17229c8-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-xjqr7\" (UID: \"cf363870-c91b-4c09-8a71-c1daa17229c8\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-xjqr7"
Sep 11 17:28:45.999789 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:28:45.999763    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cf363870-c91b-4c09-8a71-c1daa17229c8-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-xjqr7\" (UID: \"cf363870-c91b-4c09-8a71-c1daa17229c8\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-xjqr7"
Sep 11 17:28:45.999906 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:28:45.999796    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cf363870-c91b-4c09-8a71-c1daa17229c8-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-xjqr7\" (UID: \"cf363870-c91b-4c09-8a71-c1daa17229c8\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-xjqr7"
Sep 11 17:28:45.999906 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:28:45.999875    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cf363870-c91b-4c09-8a71-c1daa17229c8-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-xjqr7\" (UID: \"cf363870-c91b-4c09-8a71-c1daa17229c8\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-xjqr7"
Sep 11 17:28:46.000022 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:28:45.999988    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cf363870-c91b-4c09-8a71-c1daa17229c8-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-xjqr7\" (UID: \"cf363870-c91b-4c09-8a71-c1daa17229c8\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-xjqr7"
Sep 11 17:28:46.000175 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:28:46.000150    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cf363870-c91b-4c09-8a71-c1daa17229c8-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-xjqr7\" (UID: \"cf363870-c91b-4c09-8a71-c1daa17229c8\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-xjqr7"
Sep 11 17:28:46.000246 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:28:46.000235    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cf363870-c91b-4c09-8a71-c1daa17229c8-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-xjqr7\" (UID: \"cf363870-c91b-4c09-8a71-c1daa17229c8\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-xjqr7"
Sep 11 17:28:46.002192 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:28:46.002170    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cf363870-c91b-4c09-8a71-c1daa17229c8-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-xjqr7\" (UID: \"cf363870-c91b-4c09-8a71-c1daa17229c8\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-xjqr7"
Sep 11 17:28:46.002340 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:28:46.002321    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cf363870-c91b-4c09-8a71-c1daa17229c8-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-xjqr7\" (UID: \"cf363870-c91b-4c09-8a71-c1daa17229c8\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-xjqr7"
Sep 11 17:28:46.007752 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:28:46.007732    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7nxnc\" (UniqueName: \"kubernetes.io/projected/cf363870-c91b-4c09-8a71-c1daa17229c8-kube-api-access-7nxnc\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-xjqr7\" (UID: \"cf363870-c91b-4c09-8a71-c1daa17229c8\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-xjqr7"
Sep 11 17:28:46.209206 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:28:46.209123    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-xjqr7"
Sep 11 17:28:46.481340 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:28:46.481314    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-xjqr7"]
Sep 11 17:28:46.483982 ip-10-0-141-139 kubenswrapper[2577]: W0911 17:28:46.483938    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcf363870_c91b_4c09_8a71_c1daa17229c8.slice/crio-edec047ea3c52648dafbfefaad80738e347d43214409c2ceb2b2b6708c0c4e15 WatchSource:0}: Error finding container edec047ea3c52648dafbfefaad80738e347d43214409c2ceb2b2b6708c0c4e15: Status 404 returned error can't find the container with id edec047ea3c52648dafbfefaad80738e347d43214409c2ceb2b2b6708c0c4e15
Sep 11 17:28:46.878614 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:28:46.878487    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-xjqr7" event={"ID":"cf363870-c91b-4c09-8a71-c1daa17229c8","Type":"ContainerStarted","Data":"713cc4bcab9fb8f416fe2b8cd7519b9b18b3873fcc21573fc48e653b4fb5878e"}
Sep 11 17:28:46.878614 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:28:46.878572    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-xjqr7" event={"ID":"cf363870-c91b-4c09-8a71-c1daa17229c8","Type":"ContainerStarted","Data":"edec047ea3c52648dafbfefaad80738e347d43214409c2ceb2b2b6708c0c4e15"}
Sep 11 17:28:46.899997 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:28:46.899933    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-xjqr7" podStartSLOduration=1.710756181 podStartE2EDuration="1.899915668s" totalImagesPullingTime="189.159487ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 17:28:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 17:28:46.897771822 +0000 UTC m=+604.470991958" watchObservedRunningTime="2026-09-11 17:28:46.899915668 +0000 UTC m=+604.473135816"
Sep 11 17:28:51.394599 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:28:51.394561    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-nkmkm"]
Sep 11 17:28:52.305895 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:28:52.305859    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-57fc66f697-lrzlk"]
Sep 11 17:28:52.308564 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:28:52.308544    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-lrzlk"
Sep 11 17:28:52.311519 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:28:52.311483    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"premium-simulated-simulated-premium-kserve-self-signed-certs\""
Sep 11 17:28:52.321230 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:28:52.321205    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-57fc66f697-lrzlk"]
Sep 11 17:28:52.359721 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:28:52.359690    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/647175ee-c483-455d-8bd0-987ca40fb2bd-home\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-lrzlk\" (UID: \"647175ee-c483-455d-8bd0-987ca40fb2bd\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-lrzlk"
Sep 11 17:28:52.359921 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:28:52.359778    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/647175ee-c483-455d-8bd0-987ca40fb2bd-model-cache\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-lrzlk\" (UID: \"647175ee-c483-455d-8bd0-987ca40fb2bd\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-lrzlk"
Sep 11 17:28:52.359921 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:28:52.359828    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/647175ee-c483-455d-8bd0-987ca40fb2bd-dshm\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-lrzlk\" (UID: \"647175ee-c483-455d-8bd0-987ca40fb2bd\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-lrzlk"
Sep 11 17:28:52.360038 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:28:52.359919    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kdkkk\" (UniqueName: \"kubernetes.io/projected/647175ee-c483-455d-8bd0-987ca40fb2bd-kube-api-access-kdkkk\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-lrzlk\" (UID: \"647175ee-c483-455d-8bd0-987ca40fb2bd\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-lrzlk"
Sep 11 17:28:52.360038 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:28:52.359968    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/647175ee-c483-455d-8bd0-987ca40fb2bd-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-lrzlk\" (UID: \"647175ee-c483-455d-8bd0-987ca40fb2bd\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-lrzlk"
Sep 11 17:28:52.360038 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:28:52.359987    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/647175ee-c483-455d-8bd0-987ca40fb2bd-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-lrzlk\" (UID: \"647175ee-c483-455d-8bd0-987ca40fb2bd\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-lrzlk"
Sep 11 17:28:52.461095 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:28:52.461054    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/647175ee-c483-455d-8bd0-987ca40fb2bd-model-cache\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-lrzlk\" (UID: \"647175ee-c483-455d-8bd0-987ca40fb2bd\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-lrzlk"
Sep 11 17:28:52.461590 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:28:52.461110    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/647175ee-c483-455d-8bd0-987ca40fb2bd-dshm\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-lrzlk\" (UID: \"647175ee-c483-455d-8bd0-987ca40fb2bd\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-lrzlk"
Sep 11 17:28:52.461590 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:28:52.461131    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kdkkk\" (UniqueName: \"kubernetes.io/projected/647175ee-c483-455d-8bd0-987ca40fb2bd-kube-api-access-kdkkk\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-lrzlk\" (UID: \"647175ee-c483-455d-8bd0-987ca40fb2bd\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-lrzlk"
Sep 11 17:28:52.461590 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:28:52.461161    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/647175ee-c483-455d-8bd0-987ca40fb2bd-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-lrzlk\" (UID: \"647175ee-c483-455d-8bd0-987ca40fb2bd\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-lrzlk"
Sep 11 17:28:52.461590 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:28:52.461177    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/647175ee-c483-455d-8bd0-987ca40fb2bd-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-lrzlk\" (UID: \"647175ee-c483-455d-8bd0-987ca40fb2bd\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-lrzlk"
Sep 11 17:28:52.461590 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:28:52.461198    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/647175ee-c483-455d-8bd0-987ca40fb2bd-home\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-lrzlk\" (UID: \"647175ee-c483-455d-8bd0-987ca40fb2bd\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-lrzlk"
Sep 11 17:28:52.461829 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:28:52.461586    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/647175ee-c483-455d-8bd0-987ca40fb2bd-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-lrzlk\" (UID: \"647175ee-c483-455d-8bd0-987ca40fb2bd\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-lrzlk"
Sep 11 17:28:52.461829 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:28:52.461596    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/647175ee-c483-455d-8bd0-987ca40fb2bd-model-cache\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-lrzlk\" (UID: \"647175ee-c483-455d-8bd0-987ca40fb2bd\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-lrzlk"
Sep 11 17:28:52.461829 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:28:52.461639    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/647175ee-c483-455d-8bd0-987ca40fb2bd-home\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-lrzlk\" (UID: \"647175ee-c483-455d-8bd0-987ca40fb2bd\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-lrzlk"
Sep 11 17:28:52.463667 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:28:52.463640    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/647175ee-c483-455d-8bd0-987ca40fb2bd-dshm\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-lrzlk\" (UID: \"647175ee-c483-455d-8bd0-987ca40fb2bd\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-lrzlk"
Sep 11 17:28:52.463916 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:28:52.463898    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/647175ee-c483-455d-8bd0-987ca40fb2bd-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-lrzlk\" (UID: \"647175ee-c483-455d-8bd0-987ca40fb2bd\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-lrzlk"
Sep 11 17:28:52.481144 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:28:52.481116    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kdkkk\" (UniqueName: \"kubernetes.io/projected/647175ee-c483-455d-8bd0-987ca40fb2bd-kube-api-access-kdkkk\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-lrzlk\" (UID: \"647175ee-c483-455d-8bd0-987ca40fb2bd\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-lrzlk"
Sep 11 17:28:52.618571 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:28:52.618479    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-lrzlk"
Sep 11 17:28:52.889994 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:28:52.889971    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-57fc66f697-lrzlk"]
Sep 11 17:28:52.892152 ip-10-0-141-139 kubenswrapper[2577]: W0911 17:28:52.892111    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod647175ee_c483_455d_8bd0_987ca40fb2bd.slice/crio-3e2e9e3ea17f4f28988d7c471b8d8b421c4689ea2c2dc1165066ef7d21bcd1bd WatchSource:0}: Error finding container 3e2e9e3ea17f4f28988d7c471b8d8b421c4689ea2c2dc1165066ef7d21bcd1bd: Status 404 returned error can't find the container with id 3e2e9e3ea17f4f28988d7c471b8d8b421c4689ea2c2dc1165066ef7d21bcd1bd
Sep 11 17:28:52.908826 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:28:52.908792    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-lrzlk" event={"ID":"647175ee-c483-455d-8bd0-987ca40fb2bd","Type":"ContainerStarted","Data":"3e2e9e3ea17f4f28988d7c471b8d8b421c4689ea2c2dc1165066ef7d21bcd1bd"}
Sep 11 17:28:53.914680 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:28:53.914644    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-lrzlk" event={"ID":"647175ee-c483-455d-8bd0-987ca40fb2bd","Type":"ContainerStarted","Data":"bf889951cfbccf172ff32cafee438cf65b8144490fd6326a5d5633b00d1b566c"}
Sep 11 17:28:53.933755 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:28:53.933701    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-lrzlk" podStartSLOduration=1.746375849 podStartE2EDuration="1.933686311s" totalImagesPullingTime="187.310462ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 17:28:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 17:28:53.932148308 +0000 UTC m=+611.505368413" watchObservedRunningTime="2026-09-11 17:28:53.933686311 +0000 UTC m=+611.506906422"
Sep 11 17:28:56.210195 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:28:56.210162    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-xjqr7"
Sep 11 17:28:56.210614 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:28:56.210261    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-xjqr7"
Sep 11 17:28:56.222853 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:28:56.222826    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-xjqr7"
Sep 11 17:28:56.939666 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:28:56.939635    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-xjqr7"
Sep 11 17:28:58.585944 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:28:58.585902    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-nkmkm"]
Sep 11 17:29:02.619426 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:29:02.619393    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-lrzlk"
Sep 11 17:29:02.619857 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:29:02.619546    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-lrzlk"
Sep 11 17:29:02.632111 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:29:02.632084    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-lrzlk"
Sep 11 17:29:02.968856 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:29:02.968832    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-lrzlk"
Sep 11 17:31:11.297580 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:31:11.297546    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-nkmkm"]
Sep 11 17:31:14.783711 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:31:14.783674    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-144bfa-7664866778-kh2p9"]
Sep 11 17:31:14.786613 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:31:14.786589    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-144bfa-7664866778-kh2p9"
Sep 11 17:31:14.788978 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:31:14.788951    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w4-144bfa-dockercfg-sz45c\""
Sep 11 17:31:14.788978 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:31:14.788952    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w4-144bfa\""
Sep 11 17:31:14.797710 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:31:14.797687    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-144bfa-7664866778-kh2p9"]
Sep 11 17:31:14.815922 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:31:14.815890    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-144bfa-56f9948455-8n2pv"]
Sep 11 17:31:14.818817 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:31:14.818802    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w4-144bfa-56f9948455-8n2pv"
Sep 11 17:31:14.832725 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:31:14.832696    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-144bfa-56f9948455-8n2pv"]
Sep 11 17:31:14.911641 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:31:14.911604    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h25l5\" (UniqueName: \"kubernetes.io/projected/638cfdad-0c58-4a69-8cee-cc50f7ca2655-kube-api-access-h25l5\") pod \"payload-pre-processing-e2e-worker-w4-144bfa-7664866778-kh2p9\" (UID: \"638cfdad-0c58-4a69-8cee-cc50f7ca2655\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-144bfa-7664866778-kh2p9"
Sep 11 17:31:14.911811 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:31:14.911647    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/401d5058-6997-4c1c-97b7-3dce976fbd62-plugins-config-volume\") pod \"payload-processing-e2e-worker-w4-144bfa-56f9948455-8n2pv\" (UID: \"401d5058-6997-4c1c-97b7-3dce976fbd62\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-144bfa-56f9948455-8n2pv"
Sep 11 17:31:14.911811 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:31:14.911728    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/638cfdad-0c58-4a69-8cee-cc50f7ca2655-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w4-144bfa-7664866778-kh2p9\" (UID: \"638cfdad-0c58-4a69-8cee-cc50f7ca2655\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-144bfa-7664866778-kh2p9"
Sep 11 17:31:14.911811 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:31:14.911781    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wv6sj\" (UniqueName: \"kubernetes.io/projected/401d5058-6997-4c1c-97b7-3dce976fbd62-kube-api-access-wv6sj\") pod \"payload-processing-e2e-worker-w4-144bfa-56f9948455-8n2pv\" (UID: \"401d5058-6997-4c1c-97b7-3dce976fbd62\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-144bfa-56f9948455-8n2pv"
Sep 11 17:31:15.013400 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:31:15.013361    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h25l5\" (UniqueName: \"kubernetes.io/projected/638cfdad-0c58-4a69-8cee-cc50f7ca2655-kube-api-access-h25l5\") pod \"payload-pre-processing-e2e-worker-w4-144bfa-7664866778-kh2p9\" (UID: \"638cfdad-0c58-4a69-8cee-cc50f7ca2655\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-144bfa-7664866778-kh2p9"
Sep 11 17:31:15.013400 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:31:15.013405    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/401d5058-6997-4c1c-97b7-3dce976fbd62-plugins-config-volume\") pod \"payload-processing-e2e-worker-w4-144bfa-56f9948455-8n2pv\" (UID: \"401d5058-6997-4c1c-97b7-3dce976fbd62\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-144bfa-56f9948455-8n2pv"
Sep 11 17:31:15.013691 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:31:15.013472    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/638cfdad-0c58-4a69-8cee-cc50f7ca2655-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w4-144bfa-7664866778-kh2p9\" (UID: \"638cfdad-0c58-4a69-8cee-cc50f7ca2655\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-144bfa-7664866778-kh2p9"
Sep 11 17:31:15.013691 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:31:15.013574    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wv6sj\" (UniqueName: \"kubernetes.io/projected/401d5058-6997-4c1c-97b7-3dce976fbd62-kube-api-access-wv6sj\") pod \"payload-processing-e2e-worker-w4-144bfa-56f9948455-8n2pv\" (UID: \"401d5058-6997-4c1c-97b7-3dce976fbd62\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-144bfa-56f9948455-8n2pv"
Sep 11 17:31:15.014211 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:31:15.014189    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/401d5058-6997-4c1c-97b7-3dce976fbd62-plugins-config-volume\") pod \"payload-processing-e2e-worker-w4-144bfa-56f9948455-8n2pv\" (UID: \"401d5058-6997-4c1c-97b7-3dce976fbd62\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-144bfa-56f9948455-8n2pv"
Sep 11 17:31:15.014265 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:31:15.014249    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/638cfdad-0c58-4a69-8cee-cc50f7ca2655-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w4-144bfa-7664866778-kh2p9\" (UID: \"638cfdad-0c58-4a69-8cee-cc50f7ca2655\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-144bfa-7664866778-kh2p9"
Sep 11 17:31:15.022052 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:31:15.022021    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-h25l5\" (UniqueName: \"kubernetes.io/projected/638cfdad-0c58-4a69-8cee-cc50f7ca2655-kube-api-access-h25l5\") pod \"payload-pre-processing-e2e-worker-w4-144bfa-7664866778-kh2p9\" (UID: \"638cfdad-0c58-4a69-8cee-cc50f7ca2655\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-144bfa-7664866778-kh2p9"
Sep 11 17:31:15.022176 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:31:15.022122    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wv6sj\" (UniqueName: \"kubernetes.io/projected/401d5058-6997-4c1c-97b7-3dce976fbd62-kube-api-access-wv6sj\") pod \"payload-processing-e2e-worker-w4-144bfa-56f9948455-8n2pv\" (UID: \"401d5058-6997-4c1c-97b7-3dce976fbd62\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-144bfa-56f9948455-8n2pv"
Sep 11 17:31:15.097889 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:31:15.097797    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-144bfa-7664866778-kh2p9"
Sep 11 17:31:15.128749 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:31:15.128718    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w4-144bfa-56f9948455-8n2pv"
Sep 11 17:31:15.387734 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:31:15.387712    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-144bfa-7664866778-kh2p9"]
Sep 11 17:31:15.389865 ip-10-0-141-139 kubenswrapper[2577]: W0911 17:31:15.389837    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod638cfdad_0c58_4a69_8cee_cc50f7ca2655.slice/crio-74d82bb671a5a99860f8cfe56f2152125f2956a50503a2261f6c4e0121996598 WatchSource:0}: Error finding container 74d82bb671a5a99860f8cfe56f2152125f2956a50503a2261f6c4e0121996598: Status 404 returned error can't find the container with id 74d82bb671a5a99860f8cfe56f2152125f2956a50503a2261f6c4e0121996598
Sep 11 17:31:15.432033 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:31:15.432008    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-144bfa-56f9948455-8n2pv"]
Sep 11 17:31:15.433902 ip-10-0-141-139 kubenswrapper[2577]: W0911 17:31:15.433880    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod401d5058_6997_4c1c_97b7_3dce976fbd62.slice/crio-0e99839d9216f3b884534a025515e47ad45e2cfe953fdae4079f01bc6e6e73e1 WatchSource:0}: Error finding container 0e99839d9216f3b884534a025515e47ad45e2cfe953fdae4079f01bc6e6e73e1: Status 404 returned error can't find the container with id 0e99839d9216f3b884534a025515e47ad45e2cfe953fdae4079f01bc6e6e73e1
Sep 11 17:31:15.609587 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:31:15.609542    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w4-144bfa-56f9948455-8n2pv" event={"ID":"401d5058-6997-4c1c-97b7-3dce976fbd62","Type":"ContainerStarted","Data":"9592b808c1e0905d5e86370bfb2a3d30f41acbb6da93199b48c0031a67ddb26d"}
Sep 11 17:31:15.609771 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:31:15.609620    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w4-144bfa-56f9948455-8n2pv"
Sep 11 17:31:15.609771 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:31:15.609645    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w4-144bfa-56f9948455-8n2pv" event={"ID":"401d5058-6997-4c1c-97b7-3dce976fbd62","Type":"ContainerStarted","Data":"0e99839d9216f3b884534a025515e47ad45e2cfe953fdae4079f01bc6e6e73e1"}
Sep 11 17:31:15.611452 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:31:15.611426    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-144bfa-7664866778-kh2p9" event={"ID":"638cfdad-0c58-4a69-8cee-cc50f7ca2655","Type":"ContainerStarted","Data":"63d3bd8cc15b82ed68fcab6c03e7a4651bd8da07e8530aa5b0417ce80f0c1b6b"}
Sep 11 17:31:15.611595 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:31:15.611478    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-144bfa-7664866778-kh2p9"
Sep 11 17:31:15.611595 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:31:15.611530    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-144bfa-7664866778-kh2p9" event={"ID":"638cfdad-0c58-4a69-8cee-cc50f7ca2655","Type":"ContainerStarted","Data":"74d82bb671a5a99860f8cfe56f2152125f2956a50503a2261f6c4e0121996598"}
Sep 11 17:31:15.628401 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:31:15.628346    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w4-144bfa-56f9948455-8n2pv" podStartSLOduration=1.6283312159999999 podStartE2EDuration="1.628331216s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 17:31:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 17:31:15.626873024 +0000 UTC m=+753.200093137" watchObservedRunningTime="2026-09-11 17:31:15.628331216 +0000 UTC m=+753.201551319"
Sep 11 17:31:15.660562 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:31:15.660462    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-144bfa-7664866778-kh2p9" podStartSLOduration=1.660446432 podStartE2EDuration="1.660446432s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 17:31:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 17:31:15.658780296 +0000 UTC m=+753.232000432" watchObservedRunningTime="2026-09-11 17:31:15.660446432 +0000 UTC m=+753.233666544"
Sep 11 17:31:16.094732 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:31:16.094700    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-08c6b2-7984f858bc-nmlzk"]
Sep 11 17:31:16.097650 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:31:16.097631    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-08c6b2-7984f858bc-nmlzk"
Sep 11 17:31:16.101292 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:31:16.101264    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-08c6b2\""
Sep 11 17:31:16.101427 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:31:16.101409    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-08c6b2-dockercfg-2dnf7\""
Sep 11 17:31:16.110965 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:31:16.110937    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-08c6b2-7984f858bc-nmlzk"]
Sep 11 17:31:16.226137 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:31:16.226103    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e4dc0839-a155-4a0c-b4f3-e8a1462e7390-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-08c6b2-7984f858bc-nmlzk\" (UID: \"e4dc0839-a155-4a0c-b4f3-e8a1462e7390\") " pod="openshift-ingress/payload-processing-e2e-inf-a-08c6b2-7984f858bc-nmlzk"
Sep 11 17:31:16.226318 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:31:16.226179    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8p4n6\" (UniqueName: \"kubernetes.io/projected/e4dc0839-a155-4a0c-b4f3-e8a1462e7390-kube-api-access-8p4n6\") pod \"payload-processing-e2e-inf-a-08c6b2-7984f858bc-nmlzk\" (UID: \"e4dc0839-a155-4a0c-b4f3-e8a1462e7390\") " pod="openshift-ingress/payload-processing-e2e-inf-a-08c6b2-7984f858bc-nmlzk"
Sep 11 17:31:16.327112 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:31:16.327072    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8p4n6\" (UniqueName: \"kubernetes.io/projected/e4dc0839-a155-4a0c-b4f3-e8a1462e7390-kube-api-access-8p4n6\") pod \"payload-processing-e2e-inf-a-08c6b2-7984f858bc-nmlzk\" (UID: \"e4dc0839-a155-4a0c-b4f3-e8a1462e7390\") " pod="openshift-ingress/payload-processing-e2e-inf-a-08c6b2-7984f858bc-nmlzk"
Sep 11 17:31:16.327303 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:31:16.327170    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e4dc0839-a155-4a0c-b4f3-e8a1462e7390-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-08c6b2-7984f858bc-nmlzk\" (UID: \"e4dc0839-a155-4a0c-b4f3-e8a1462e7390\") " pod="openshift-ingress/payload-processing-e2e-inf-a-08c6b2-7984f858bc-nmlzk"
Sep 11 17:31:16.327856 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:31:16.327839    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e4dc0839-a155-4a0c-b4f3-e8a1462e7390-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-08c6b2-7984f858bc-nmlzk\" (UID: \"e4dc0839-a155-4a0c-b4f3-e8a1462e7390\") " pod="openshift-ingress/payload-processing-e2e-inf-a-08c6b2-7984f858bc-nmlzk"
Sep 11 17:31:16.341801 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:31:16.341770    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8p4n6\" (UniqueName: \"kubernetes.io/projected/e4dc0839-a155-4a0c-b4f3-e8a1462e7390-kube-api-access-8p4n6\") pod \"payload-processing-e2e-inf-a-08c6b2-7984f858bc-nmlzk\" (UID: \"e4dc0839-a155-4a0c-b4f3-e8a1462e7390\") " pod="openshift-ingress/payload-processing-e2e-inf-a-08c6b2-7984f858bc-nmlzk"
Sep 11 17:31:16.409644 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:31:16.409550    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-08c6b2-7984f858bc-nmlzk"
Sep 11 17:31:16.692998 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:31:16.692971    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-08c6b2-7984f858bc-nmlzk"]
Sep 11 17:31:16.695917 ip-10-0-141-139 kubenswrapper[2577]: W0911 17:31:16.695890    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode4dc0839_a155_4a0c_b4f3_e8a1462e7390.slice/crio-45ad6f725d2fe5423b5b7bbdfc70d04b671942bad1376cede6ca816a5eaa56f7 WatchSource:0}: Error finding container 45ad6f725d2fe5423b5b7bbdfc70d04b671942bad1376cede6ca816a5eaa56f7: Status 404 returned error can't find the container with id 45ad6f725d2fe5423b5b7bbdfc70d04b671942bad1376cede6ca816a5eaa56f7
Sep 11 17:31:17.354311 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:31:17.354279    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-c7d192-67844f9ccf-5kttv"]
Sep 11 17:31:17.357480 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:31:17.357455    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-c7d192-67844f9ccf-5kttv"
Sep 11 17:31:17.358667 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:31:17.358645    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-c7d192-fb57bbbcd-nkspc"]
Sep 11 17:31:17.359701 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:31:17.359681    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w0-c7d192\""
Sep 11 17:31:17.361518 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:31:17.361488    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-c7d192-fb57bbbcd-nkspc"
Sep 11 17:31:17.364820 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:31:17.364799    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w0-c7d192-dockercfg-286rb\""
Sep 11 17:31:17.377197 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:31:17.377146    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-c7d192-67844f9ccf-5kttv"]
Sep 11 17:31:17.380527 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:31:17.380479    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-c7d192-fb57bbbcd-nkspc"]
Sep 11 17:31:17.540405 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:31:17.540375    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cb6ad0f4-dfc7-4a72-91f1-4612e23f1bbe-plugins-config-volume\") pod \"payload-processing-e2e-worker-w0-c7d192-67844f9ccf-5kttv\" (UID: \"cb6ad0f4-dfc7-4a72-91f1-4612e23f1bbe\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-c7d192-67844f9ccf-5kttv"
Sep 11 17:31:17.540405 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:31:17.540411    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wg876\" (UniqueName: \"kubernetes.io/projected/cb6ad0f4-dfc7-4a72-91f1-4612e23f1bbe-kube-api-access-wg876\") pod \"payload-processing-e2e-worker-w0-c7d192-67844f9ccf-5kttv\" (UID: \"cb6ad0f4-dfc7-4a72-91f1-4612e23f1bbe\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-c7d192-67844f9ccf-5kttv"
Sep 11 17:31:17.540699 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:31:17.540477    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d8aafb09-db77-4dce-9138-fb92ed438794-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w0-c7d192-fb57bbbcd-nkspc\" (UID: \"d8aafb09-db77-4dce-9138-fb92ed438794\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-c7d192-fb57bbbcd-nkspc"
Sep 11 17:31:17.540699 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:31:17.540569    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k452z\" (UniqueName: \"kubernetes.io/projected/d8aafb09-db77-4dce-9138-fb92ed438794-kube-api-access-k452z\") pod \"payload-pre-processing-e2e-worker-w0-c7d192-fb57bbbcd-nkspc\" (UID: \"d8aafb09-db77-4dce-9138-fb92ed438794\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-c7d192-fb57bbbcd-nkspc"
Sep 11 17:31:17.626519 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:31:17.626390    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-08c6b2-7984f858bc-nmlzk" event={"ID":"e4dc0839-a155-4a0c-b4f3-e8a1462e7390","Type":"ContainerStarted","Data":"ac93a5162d972950c1b4fefdef917aa826775dc83a63be708edff15f38250ca0"}
Sep 11 17:31:17.626519 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:31:17.626456    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-08c6b2-7984f858bc-nmlzk" event={"ID":"e4dc0839-a155-4a0c-b4f3-e8a1462e7390","Type":"ContainerStarted","Data":"45ad6f725d2fe5423b5b7bbdfc70d04b671942bad1376cede6ca816a5eaa56f7"}
Sep 11 17:31:17.626519 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:31:17.626486    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-a-08c6b2-7984f858bc-nmlzk"
Sep 11 17:31:17.641846 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:31:17.641798    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cb6ad0f4-dfc7-4a72-91f1-4612e23f1bbe-plugins-config-volume\") pod \"payload-processing-e2e-worker-w0-c7d192-67844f9ccf-5kttv\" (UID: \"cb6ad0f4-dfc7-4a72-91f1-4612e23f1bbe\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-c7d192-67844f9ccf-5kttv"
Sep 11 17:31:17.642022 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:31:17.641854    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wg876\" (UniqueName: \"kubernetes.io/projected/cb6ad0f4-dfc7-4a72-91f1-4612e23f1bbe-kube-api-access-wg876\") pod \"payload-processing-e2e-worker-w0-c7d192-67844f9ccf-5kttv\" (UID: \"cb6ad0f4-dfc7-4a72-91f1-4612e23f1bbe\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-c7d192-67844f9ccf-5kttv"
Sep 11 17:31:17.642022 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:31:17.641908    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d8aafb09-db77-4dce-9138-fb92ed438794-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w0-c7d192-fb57bbbcd-nkspc\" (UID: \"d8aafb09-db77-4dce-9138-fb92ed438794\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-c7d192-fb57bbbcd-nkspc"
Sep 11 17:31:17.642022 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:31:17.641993    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k452z\" (UniqueName: \"kubernetes.io/projected/d8aafb09-db77-4dce-9138-fb92ed438794-kube-api-access-k452z\") pod \"payload-pre-processing-e2e-worker-w0-c7d192-fb57bbbcd-nkspc\" (UID: \"d8aafb09-db77-4dce-9138-fb92ed438794\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-c7d192-fb57bbbcd-nkspc"
Sep 11 17:31:17.643095 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:31:17.643059    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d8aafb09-db77-4dce-9138-fb92ed438794-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w0-c7d192-fb57bbbcd-nkspc\" (UID: \"d8aafb09-db77-4dce-9138-fb92ed438794\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-c7d192-fb57bbbcd-nkspc"
Sep 11 17:31:17.643239 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:31:17.643146    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cb6ad0f4-dfc7-4a72-91f1-4612e23f1bbe-plugins-config-volume\") pod \"payload-processing-e2e-worker-w0-c7d192-67844f9ccf-5kttv\" (UID: \"cb6ad0f4-dfc7-4a72-91f1-4612e23f1bbe\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-c7d192-67844f9ccf-5kttv"
Sep 11 17:31:17.644732 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:31:17.644688    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-a-08c6b2-7984f858bc-nmlzk" podStartSLOduration=1.644669495 podStartE2EDuration="1.644669495s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 17:31:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 17:31:17.642046102 +0000 UTC m=+755.215266215" watchObservedRunningTime="2026-09-11 17:31:17.644669495 +0000 UTC m=+755.217889614"
Sep 11 17:31:17.653155 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:31:17.653128    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wg876\" (UniqueName: \"kubernetes.io/projected/cb6ad0f4-dfc7-4a72-91f1-4612e23f1bbe-kube-api-access-wg876\") pod \"payload-processing-e2e-worker-w0-c7d192-67844f9ccf-5kttv\" (UID: \"cb6ad0f4-dfc7-4a72-91f1-4612e23f1bbe\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-c7d192-67844f9ccf-5kttv"
Sep 11 17:31:17.653565 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:31:17.653547    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-k452z\" (UniqueName: \"kubernetes.io/projected/d8aafb09-db77-4dce-9138-fb92ed438794-kube-api-access-k452z\") pod \"payload-pre-processing-e2e-worker-w0-c7d192-fb57bbbcd-nkspc\" (UID: \"d8aafb09-db77-4dce-9138-fb92ed438794\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-c7d192-fb57bbbcd-nkspc"
Sep 11 17:31:17.672541 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:31:17.672488    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-c7d192-67844f9ccf-5kttv"
Sep 11 17:31:17.680146 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:31:17.680118    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-c7d192-fb57bbbcd-nkspc"
Sep 11 17:31:17.960038 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:31:17.960010    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-c7d192-67844f9ccf-5kttv"]
Sep 11 17:31:17.962806 ip-10-0-141-139 kubenswrapper[2577]: W0911 17:31:17.962775    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcb6ad0f4_dfc7_4a72_91f1_4612e23f1bbe.slice/crio-10c385af04a76f51c161878f82b8fbe0546cea20a3c27d27235b4febe23ce3b6 WatchSource:0}: Error finding container 10c385af04a76f51c161878f82b8fbe0546cea20a3c27d27235b4febe23ce3b6: Status 404 returned error can't find the container with id 10c385af04a76f51c161878f82b8fbe0546cea20a3c27d27235b4febe23ce3b6
Sep 11 17:31:17.988115 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:31:17.988085    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-c7d192-fb57bbbcd-nkspc"]
Sep 11 17:31:17.993970 ip-10-0-141-139 kubenswrapper[2577]: W0911 17:31:17.993939    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd8aafb09_db77_4dce_9138_fb92ed438794.slice/crio-2bd7bb9259da560b031f75c0c9e236b61d784c64e8e0df8f4aaeac5c6436b9d0 WatchSource:0}: Error finding container 2bd7bb9259da560b031f75c0c9e236b61d784c64e8e0df8f4aaeac5c6436b9d0: Status 404 returned error can't find the container with id 2bd7bb9259da560b031f75c0c9e236b61d784c64e8e0df8f4aaeac5c6436b9d0
Sep 11 17:31:18.633118 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:31:18.633083    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-c7d192-fb57bbbcd-nkspc" event={"ID":"d8aafb09-db77-4dce-9138-fb92ed438794","Type":"ContainerStarted","Data":"e8a1d60d0cdccd32768b1f8cf3a352466610194311e32547ca40236c5850f67d"}
Sep 11 17:31:18.633576 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:31:18.633133    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-c7d192-fb57bbbcd-nkspc" event={"ID":"d8aafb09-db77-4dce-9138-fb92ed438794","Type":"ContainerStarted","Data":"2bd7bb9259da560b031f75c0c9e236b61d784c64e8e0df8f4aaeac5c6436b9d0"}
Sep 11 17:31:18.633576 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:31:18.633175    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-c7d192-fb57bbbcd-nkspc"
Sep 11 17:31:18.634785 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:31:18.634762    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-c7d192-67844f9ccf-5kttv" event={"ID":"cb6ad0f4-dfc7-4a72-91f1-4612e23f1bbe","Type":"ContainerStarted","Data":"d13559ac08f3a8f852fa18ed2bde7109545647482e12198a7848c632a9715b4b"}
Sep 11 17:31:18.634846 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:31:18.634808    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-c7d192-67844f9ccf-5kttv" event={"ID":"cb6ad0f4-dfc7-4a72-91f1-4612e23f1bbe","Type":"ContainerStarted","Data":"10c385af04a76f51c161878f82b8fbe0546cea20a3c27d27235b4febe23ce3b6"}
Sep 11 17:31:18.634917 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:31:18.634902    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w0-c7d192-67844f9ccf-5kttv"
Sep 11 17:31:18.651820 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:31:18.651769    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-c7d192-fb57bbbcd-nkspc" podStartSLOduration=1.651752611 podStartE2EDuration="1.651752611s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 17:31:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 17:31:18.651288427 +0000 UTC m=+756.224508545" watchObservedRunningTime="2026-09-11 17:31:18.651752611 +0000 UTC m=+756.224972747"
Sep 11 17:31:18.678095 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:31:18.678045    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w0-c7d192-67844f9ccf-5kttv" podStartSLOduration=1.6780302790000001 podStartE2EDuration="1.678030279s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 17:31:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 17:31:18.675664613 +0000 UTC m=+756.248884727" watchObservedRunningTime="2026-09-11 17:31:18.678030279 +0000 UTC m=+756.251250388"
Sep 11 17:31:20.708134 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:31:20.708097    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-nkmkm"]
Sep 11 17:31:24.823796 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:31:24.823758    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-23a9cc4a-7475fb94f9-nfsm6"]
Sep 11 17:31:24.827350 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:31:24.827331    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-23a9cc4a-7475fb94f9-nfsm6"
Sep 11 17:31:24.830462 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:31:24.830437    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-23a9cc4a\""
Sep 11 17:31:24.830906 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:31:24.830887    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-23a9cc4a-dockercfg-4gsxn\""
Sep 11 17:31:24.842681 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:31:24.842651    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-23a9cc4a-7475fb94f9-nfsm6"]
Sep 11 17:31:24.922668 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:31:24.922637    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tj6mv\" (UniqueName: \"kubernetes.io/projected/58eab99b-e56c-4ef4-838a-5448cc9758b6-kube-api-access-tj6mv\") pod \"payload-processing-e2e-ait-23a9cc4a-7475fb94f9-nfsm6\" (UID: \"58eab99b-e56c-4ef4-838a-5448cc9758b6\") " pod="openshift-ingress/payload-processing-e2e-ait-23a9cc4a-7475fb94f9-nfsm6"
Sep 11 17:31:24.922849 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:31:24.922729    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/58eab99b-e56c-4ef4-838a-5448cc9758b6-plugins-config-volume\") pod \"payload-processing-e2e-ait-23a9cc4a-7475fb94f9-nfsm6\" (UID: \"58eab99b-e56c-4ef4-838a-5448cc9758b6\") " pod="openshift-ingress/payload-processing-e2e-ait-23a9cc4a-7475fb94f9-nfsm6"
Sep 11 17:31:25.023555 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:31:25.023474    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/58eab99b-e56c-4ef4-838a-5448cc9758b6-plugins-config-volume\") pod \"payload-processing-e2e-ait-23a9cc4a-7475fb94f9-nfsm6\" (UID: \"58eab99b-e56c-4ef4-838a-5448cc9758b6\") " pod="openshift-ingress/payload-processing-e2e-ait-23a9cc4a-7475fb94f9-nfsm6"
Sep 11 17:31:25.023759 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:31:25.023676    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tj6mv\" (UniqueName: \"kubernetes.io/projected/58eab99b-e56c-4ef4-838a-5448cc9758b6-kube-api-access-tj6mv\") pod \"payload-processing-e2e-ait-23a9cc4a-7475fb94f9-nfsm6\" (UID: \"58eab99b-e56c-4ef4-838a-5448cc9758b6\") " pod="openshift-ingress/payload-processing-e2e-ait-23a9cc4a-7475fb94f9-nfsm6"
Sep 11 17:31:25.024302 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:31:25.024267    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/58eab99b-e56c-4ef4-838a-5448cc9758b6-plugins-config-volume\") pod \"payload-processing-e2e-ait-23a9cc4a-7475fb94f9-nfsm6\" (UID: \"58eab99b-e56c-4ef4-838a-5448cc9758b6\") " pod="openshift-ingress/payload-processing-e2e-ait-23a9cc4a-7475fb94f9-nfsm6"
Sep 11 17:31:25.053672 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:31:25.053641    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tj6mv\" (UniqueName: \"kubernetes.io/projected/58eab99b-e56c-4ef4-838a-5448cc9758b6-kube-api-access-tj6mv\") pod \"payload-processing-e2e-ait-23a9cc4a-7475fb94f9-nfsm6\" (UID: \"58eab99b-e56c-4ef4-838a-5448cc9758b6\") " pod="openshift-ingress/payload-processing-e2e-ait-23a9cc4a-7475fb94f9-nfsm6"
Sep 11 17:31:25.138064 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:31:25.137976    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-23a9cc4a-7475fb94f9-nfsm6"
Sep 11 17:31:25.415667 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:31:25.415640    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-23a9cc4a-7475fb94f9-nfsm6"]
Sep 11 17:31:25.417480 ip-10-0-141-139 kubenswrapper[2577]: W0911 17:31:25.417447    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod58eab99b_e56c_4ef4_838a_5448cc9758b6.slice/crio-33fae3778b8dbbc49c505783cbadf8dd9ce14f32d059ea644510ac2ba45c2548 WatchSource:0}: Error finding container 33fae3778b8dbbc49c505783cbadf8dd9ce14f32d059ea644510ac2ba45c2548: Status 404 returned error can't find the container with id 33fae3778b8dbbc49c505783cbadf8dd9ce14f32d059ea644510ac2ba45c2548
Sep 11 17:31:25.674197 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:31:25.674096    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-23a9cc4a-7475fb94f9-nfsm6" event={"ID":"58eab99b-e56c-4ef4-838a-5448cc9758b6","Type":"ContainerStarted","Data":"6c30cc191eb0bcff5a5407a83260c97dee9c9481228e539adcf7898cfe883758"}
Sep 11 17:31:25.674197 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:31:25.674150    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-23a9cc4a-7475fb94f9-nfsm6" event={"ID":"58eab99b-e56c-4ef4-838a-5448cc9758b6","Type":"ContainerStarted","Data":"33fae3778b8dbbc49c505783cbadf8dd9ce14f32d059ea644510ac2ba45c2548"}
Sep 11 17:31:25.674197 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:31:25.674172    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-23a9cc4a-7475fb94f9-nfsm6"
Sep 11 17:31:25.698159 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:31:25.696418    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-23a9cc4a-7475fb94f9-nfsm6" podStartSLOduration=1.69639461 podStartE2EDuration="1.69639461s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 17:31:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 17:31:25.691301991 +0000 UTC m=+763.264522103" watchObservedRunningTime="2026-09-11 17:31:25.69639461 +0000 UTC m=+763.269614724"
Sep 11 17:31:26.620716 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:31:26.620684    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-144bfa-7664866778-kh2p9"
Sep 11 17:31:26.621116 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:31:26.620772    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w4-144bfa-56f9948455-8n2pv"
Sep 11 17:31:27.231175 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:31:27.231136    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-49e398-8fccbc594-v2cgl"]
Sep 11 17:31:27.234180 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:31:27.234159    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-autores-49e398-8fccbc594-v2cgl"
Sep 11 17:31:27.236618 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:31:27.236592    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-autores-49e398\""
Sep 11 17:31:27.244466 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:31:27.244439    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-49e398-8fccbc594-v2cgl"]
Sep 11 17:31:27.344952 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:31:27.344905    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p58xm\" (UniqueName: \"kubernetes.io/projected/65b33dd3-cc20-465a-9da7-9c8c31dffa9d-kube-api-access-p58xm\") pod \"payload-pre-processing-e2e-autores-49e398-8fccbc594-v2cgl\" (UID: \"65b33dd3-cc20-465a-9da7-9c8c31dffa9d\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-49e398-8fccbc594-v2cgl"
Sep 11 17:31:27.345134 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:31:27.344967    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/65b33dd3-cc20-465a-9da7-9c8c31dffa9d-plugins-config-volume\") pod \"payload-pre-processing-e2e-autores-49e398-8fccbc594-v2cgl\" (UID: \"65b33dd3-cc20-465a-9da7-9c8c31dffa9d\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-49e398-8fccbc594-v2cgl"
Sep 11 17:31:27.446006 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:31:27.445966    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p58xm\" (UniqueName: \"kubernetes.io/projected/65b33dd3-cc20-465a-9da7-9c8c31dffa9d-kube-api-access-p58xm\") pod \"payload-pre-processing-e2e-autores-49e398-8fccbc594-v2cgl\" (UID: \"65b33dd3-cc20-465a-9da7-9c8c31dffa9d\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-49e398-8fccbc594-v2cgl"
Sep 11 17:31:27.446192 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:31:27.446016    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/65b33dd3-cc20-465a-9da7-9c8c31dffa9d-plugins-config-volume\") pod \"payload-pre-processing-e2e-autores-49e398-8fccbc594-v2cgl\" (UID: \"65b33dd3-cc20-465a-9da7-9c8c31dffa9d\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-49e398-8fccbc594-v2cgl"
Sep 11 17:31:27.446759 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:31:27.446741    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/65b33dd3-cc20-465a-9da7-9c8c31dffa9d-plugins-config-volume\") pod \"payload-pre-processing-e2e-autores-49e398-8fccbc594-v2cgl\" (UID: \"65b33dd3-cc20-465a-9da7-9c8c31dffa9d\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-49e398-8fccbc594-v2cgl"
Sep 11 17:31:27.454379 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:31:27.454348    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-p58xm\" (UniqueName: \"kubernetes.io/projected/65b33dd3-cc20-465a-9da7-9c8c31dffa9d-kube-api-access-p58xm\") pod \"payload-pre-processing-e2e-autores-49e398-8fccbc594-v2cgl\" (UID: \"65b33dd3-cc20-465a-9da7-9c8c31dffa9d\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-49e398-8fccbc594-v2cgl"
Sep 11 17:31:27.545245 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:31:27.545161    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-autores-49e398-8fccbc594-v2cgl"
Sep 11 17:31:27.817523 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:31:27.817483    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-49e398-8fccbc594-v2cgl"]
Sep 11 17:31:27.819774 ip-10-0-141-139 kubenswrapper[2577]: W0911 17:31:27.819746    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod65b33dd3_cc20_465a_9da7_9c8c31dffa9d.slice/crio-f6044ae6c69ec8809c2d03f2ed13b6340a68792200c0cace5a214f7591603587 WatchSource:0}: Error finding container f6044ae6c69ec8809c2d03f2ed13b6340a68792200c0cace5a214f7591603587: Status 404 returned error can't find the container with id f6044ae6c69ec8809c2d03f2ed13b6340a68792200c0cace5a214f7591603587
Sep 11 17:31:28.637057 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:31:28.637027    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-a-08c6b2-7984f858bc-nmlzk"
Sep 11 17:31:28.694098 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:31:28.694062    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-autores-49e398-8fccbc594-v2cgl" event={"ID":"65b33dd3-cc20-465a-9da7-9c8c31dffa9d","Type":"ContainerStarted","Data":"db784bbc86c7b0bc76f569a3bb9d20bd42b5d6fc05c28dfa7974c63c426187e0"}
Sep 11 17:31:28.694295 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:31:28.694116    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-autores-49e398-8fccbc594-v2cgl" event={"ID":"65b33dd3-cc20-465a-9da7-9c8c31dffa9d","Type":"ContainerStarted","Data":"f6044ae6c69ec8809c2d03f2ed13b6340a68792200c0cace5a214f7591603587"}
Sep 11 17:31:28.694295 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:31:28.694201    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-autores-49e398-8fccbc594-v2cgl"
Sep 11 17:31:28.712291 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:31:28.712231    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-autores-49e398-8fccbc594-v2cgl" podStartSLOduration=1.712214725 podStartE2EDuration="1.712214725s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 17:31:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 17:31:28.710624739 +0000 UTC m=+766.283844854" watchObservedRunningTime="2026-09-11 17:31:28.712214725 +0000 UTC m=+766.285434837"
Sep 11 17:31:29.642320 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:31:29.642292    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w0-c7d192-67844f9ccf-5kttv"
Sep 11 17:31:29.642724 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:31:29.642354    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-c7d192-fb57bbbcd-nkspc"
Sep 11 17:31:36.683079 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:31:36.683045    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-23a9cc4a-7475fb94f9-nfsm6"
Sep 11 17:31:36.699967 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:31:36.699929    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23a9cc4a-gx6kd"]
Sep 11 17:31:36.703479 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:31:36.703450    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23a9cc4a-gx6kd"
Sep 11 17:31:36.705610 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:31:36.705587    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-9n8gf\""
Sep 11 17:31:36.705739 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:31:36.705658    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 11 17:31:36.714180 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:31:36.714151    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23a9cc4a-gx6kd"]
Sep 11 17:31:36.839048 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:31:36.839009    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7acb539f-d736-49c4-9b72-8f9f2fee6c71-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-23a9cc4a-gx6kd\" (UID: \"7acb539f-d736-49c4-9b72-8f9f2fee6c71\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23a9cc4a-gx6kd"
Sep 11 17:31:36.940890 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:31:36.940801    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7acb539f-d736-49c4-9b72-8f9f2fee6c71-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-23a9cc4a-gx6kd\" (UID: \"7acb539f-d736-49c4-9b72-8f9f2fee6c71\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23a9cc4a-gx6kd"
Sep 11 17:31:36.941434 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:31:36.941417    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7acb539f-d736-49c4-9b72-8f9f2fee6c71-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-23a9cc4a-gx6kd\" (UID: \"7acb539f-d736-49c4-9b72-8f9f2fee6c71\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23a9cc4a-gx6kd"
Sep 11 17:31:37.016805 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:31:37.016765    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23a9cc4a-gx6kd"
Sep 11 17:31:37.293543 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:31:37.293497    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23a9cc4a-gx6kd"]
Sep 11 17:31:37.295534 ip-10-0-141-139 kubenswrapper[2577]: W0911 17:31:37.295479    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7acb539f_d736_49c4_9b72_8f9f2fee6c71.slice/crio-b8e220bca102c3d550b93e9919afa09bffe3ffa07c55e42511f7baf442bc355a WatchSource:0}: Error finding container b8e220bca102c3d550b93e9919afa09bffe3ffa07c55e42511f7baf442bc355a: Status 404 returned error can't find the container with id b8e220bca102c3d550b93e9919afa09bffe3ffa07c55e42511f7baf442bc355a
Sep 11 17:31:37.297690 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:31:37.297670    2577 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 11 17:31:37.746755 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:31:37.746724    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23a9cc4a-gx6kd" event={"ID":"7acb539f-d736-49c4-9b72-8f9f2fee6c71","Type":"ContainerStarted","Data":"b8e220bca102c3d550b93e9919afa09bffe3ffa07c55e42511f7baf442bc355a"}
Sep 11 17:31:39.703012 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:31:39.702980    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-autores-49e398-8fccbc594-v2cgl"
Sep 11 17:31:40.765872 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:31:40.765838    2577 generic.go:356] "Generic (PLEG): container finished" podID="7acb539f-d736-49c4-9b72-8f9f2fee6c71" containerID="f60c471ce3383d36ef28c63fcf45d8bd0f617e464d4c387038d8657d0b903308" exitCode=0
Sep 11 17:31:40.766342 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:31:40.765927    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23a9cc4a-gx6kd" event={"ID":"7acb539f-d736-49c4-9b72-8f9f2fee6c71","Type":"ContainerDied","Data":"f60c471ce3383d36ef28c63fcf45d8bd0f617e464d4c387038d8657d0b903308"}
Sep 11 17:31:41.911049 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:31:41.911018    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23a9cc4a-gx6kd"
Sep 11 17:31:41.997268 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:31:41.997229    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7acb539f-d736-49c4-9b72-8f9f2fee6c71-maas-api-service-ca\") pod \"7acb539f-d736-49c4-9b72-8f9f2fee6c71\" (UID: \"7acb539f-d736-49c4-9b72-8f9f2fee6c71\") "
Sep 11 17:31:41.997640 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:31:41.997615    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7acb539f-d736-49c4-9b72-8f9f2fee6c71-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "7acb539f-d736-49c4-9b72-8f9f2fee6c71" (UID: "7acb539f-d736-49c4-9b72-8f9f2fee6c71"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 17:31:42.098484 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:31:42.098392    2577 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7acb539f-d736-49c4-9b72-8f9f2fee6c71-maas-api-service-ca\") on node \"ip-10-0-141-139.ec2.internal\" DevicePath \"\""
Sep 11 17:31:42.778020 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:31:42.777989    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23a9cc4a-gx6kd" event={"ID":"7acb539f-d736-49c4-9b72-8f9f2fee6c71","Type":"ContainerDied","Data":"b8e220bca102c3d550b93e9919afa09bffe3ffa07c55e42511f7baf442bc355a"}
Sep 11 17:31:42.778020 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:31:42.778022    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23a9cc4a-gx6kd"
Sep 11 17:31:42.778311 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:31:42.778043    2577 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b8e220bca102c3d550b93e9919afa09bffe3ffa07c55e42511f7baf442bc355a"
Sep 11 17:31:45.887427 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:31:45.887394    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-cbbb09-544b668766-zmvwq"]
Sep 11 17:31:45.887880 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:31:45.887865    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7acb539f-d736-49c4-9b72-8f9f2fee6c71" containerName="revoke-keys"
Sep 11 17:31:45.887925 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:31:45.887881    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="7acb539f-d736-49c4-9b72-8f9f2fee6c71" containerName="revoke-keys"
Sep 11 17:31:45.887993 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:31:45.887984    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="7acb539f-d736-49c4-9b72-8f9f2fee6c71" containerName="revoke-keys"
Sep 11 17:31:45.890249 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:31:45.890233    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-cbbb09-544b668766-zmvwq"
Sep 11 17:31:45.893043 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:31:45.893020    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-b-cbbb09-dockercfg-gznj9\""
Sep 11 17:31:45.893157 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:31:45.893085    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-cbbb09\""
Sep 11 17:31:45.902543 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:31:45.902519    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-cbbb09-544b668766-zmvwq"]
Sep 11 17:31:45.909584 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:31:45.909544    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-cbbb09-85f75d469f-fmfwt"]
Sep 11 17:31:45.913295 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:31:45.913275    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-cbbb09-85f75d469f-fmfwt"
Sep 11 17:31:45.927170 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:31:45.927147    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-cbbb09-85f75d469f-fmfwt"]
Sep 11 17:31:46.039256 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:31:46.039223    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pdtf2\" (UniqueName: \"kubernetes.io/projected/8e1a7c02-0df2-4958-bd88-2e212f3d89e4-kube-api-access-pdtf2\") pod \"payload-processing-e2e-inf-b-cbbb09-85f75d469f-fmfwt\" (UID: \"8e1a7c02-0df2-4958-bd88-2e212f3d89e4\") " pod="openshift-ingress/payload-processing-e2e-inf-b-cbbb09-85f75d469f-fmfwt"
Sep 11 17:31:46.039470 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:31:46.039285    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c3346092-b638-42cc-b901-aa4b12a0e13e-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-cbbb09-544b668766-zmvwq\" (UID: \"c3346092-b638-42cc-b901-aa4b12a0e13e\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-cbbb09-544b668766-zmvwq"
Sep 11 17:31:46.039470 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:31:46.039362    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8e1a7c02-0df2-4958-bd88-2e212f3d89e4-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-cbbb09-85f75d469f-fmfwt\" (UID: \"8e1a7c02-0df2-4958-bd88-2e212f3d89e4\") " pod="openshift-ingress/payload-processing-e2e-inf-b-cbbb09-85f75d469f-fmfwt"
Sep 11 17:31:46.039609 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:31:46.039484    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b4ms2\" (UniqueName: \"kubernetes.io/projected/c3346092-b638-42cc-b901-aa4b12a0e13e-kube-api-access-b4ms2\") pod \"payload-pre-processing-e2e-inf-b-cbbb09-544b668766-zmvwq\" (UID: \"c3346092-b638-42cc-b901-aa4b12a0e13e\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-cbbb09-544b668766-zmvwq"
Sep 11 17:31:46.140837 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:31:46.140742    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pdtf2\" (UniqueName: \"kubernetes.io/projected/8e1a7c02-0df2-4958-bd88-2e212f3d89e4-kube-api-access-pdtf2\") pod \"payload-processing-e2e-inf-b-cbbb09-85f75d469f-fmfwt\" (UID: \"8e1a7c02-0df2-4958-bd88-2e212f3d89e4\") " pod="openshift-ingress/payload-processing-e2e-inf-b-cbbb09-85f75d469f-fmfwt"
Sep 11 17:31:46.140837 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:31:46.140830    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c3346092-b638-42cc-b901-aa4b12a0e13e-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-cbbb09-544b668766-zmvwq\" (UID: \"c3346092-b638-42cc-b901-aa4b12a0e13e\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-cbbb09-544b668766-zmvwq"
Sep 11 17:31:46.141069 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:31:46.140866    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8e1a7c02-0df2-4958-bd88-2e212f3d89e4-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-cbbb09-85f75d469f-fmfwt\" (UID: \"8e1a7c02-0df2-4958-bd88-2e212f3d89e4\") " pod="openshift-ingress/payload-processing-e2e-inf-b-cbbb09-85f75d469f-fmfwt"
Sep 11 17:31:46.141069 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:31:46.140944    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b4ms2\" (UniqueName: \"kubernetes.io/projected/c3346092-b638-42cc-b901-aa4b12a0e13e-kube-api-access-b4ms2\") pod \"payload-pre-processing-e2e-inf-b-cbbb09-544b668766-zmvwq\" (UID: \"c3346092-b638-42cc-b901-aa4b12a0e13e\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-cbbb09-544b668766-zmvwq"
Sep 11 17:31:46.141603 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:31:46.141581    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c3346092-b638-42cc-b901-aa4b12a0e13e-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-cbbb09-544b668766-zmvwq\" (UID: \"c3346092-b638-42cc-b901-aa4b12a0e13e\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-cbbb09-544b668766-zmvwq"
Sep 11 17:31:46.141667 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:31:46.141613    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8e1a7c02-0df2-4958-bd88-2e212f3d89e4-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-cbbb09-85f75d469f-fmfwt\" (UID: \"8e1a7c02-0df2-4958-bd88-2e212f3d89e4\") " pod="openshift-ingress/payload-processing-e2e-inf-b-cbbb09-85f75d469f-fmfwt"
Sep 11 17:31:46.149035 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:31:46.149016    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pdtf2\" (UniqueName: \"kubernetes.io/projected/8e1a7c02-0df2-4958-bd88-2e212f3d89e4-kube-api-access-pdtf2\") pod \"payload-processing-e2e-inf-b-cbbb09-85f75d469f-fmfwt\" (UID: \"8e1a7c02-0df2-4958-bd88-2e212f3d89e4\") " pod="openshift-ingress/payload-processing-e2e-inf-b-cbbb09-85f75d469f-fmfwt"
Sep 11 17:31:46.149102 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:31:46.149073    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-b4ms2\" (UniqueName: \"kubernetes.io/projected/c3346092-b638-42cc-b901-aa4b12a0e13e-kube-api-access-b4ms2\") pod \"payload-pre-processing-e2e-inf-b-cbbb09-544b668766-zmvwq\" (UID: \"c3346092-b638-42cc-b901-aa4b12a0e13e\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-cbbb09-544b668766-zmvwq"
Sep 11 17:31:46.201354 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:31:46.201327    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-cbbb09-544b668766-zmvwq"
Sep 11 17:31:46.224130 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:31:46.224106    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-cbbb09-85f75d469f-fmfwt"
Sep 11 17:31:46.476370 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:31:46.476337    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-cbbb09-544b668766-zmvwq"]
Sep 11 17:31:46.479479 ip-10-0-141-139 kubenswrapper[2577]: W0911 17:31:46.479451    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc3346092_b638_42cc_b901_aa4b12a0e13e.slice/crio-cfc8dc24f60f6f911be35f6b64271d82798529efcb0c016c8870485d8b759f0a WatchSource:0}: Error finding container cfc8dc24f60f6f911be35f6b64271d82798529efcb0c016c8870485d8b759f0a: Status 404 returned error can't find the container with id cfc8dc24f60f6f911be35f6b64271d82798529efcb0c016c8870485d8b759f0a
Sep 11 17:31:46.511424 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:31:46.511400    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-cbbb09-85f75d469f-fmfwt"]
Sep 11 17:31:46.516609 ip-10-0-141-139 kubenswrapper[2577]: W0911 17:31:46.516581    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8e1a7c02_0df2_4958_bd88_2e212f3d89e4.slice/crio-646517111edce307e265218752c82cdb96744ca93113e4d897997b95525708b8 WatchSource:0}: Error finding container 646517111edce307e265218752c82cdb96744ca93113e4d897997b95525708b8: Status 404 returned error can't find the container with id 646517111edce307e265218752c82cdb96744ca93113e4d897997b95525708b8
Sep 11 17:31:46.801533 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:31:46.801418    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-cbbb09-544b668766-zmvwq" event={"ID":"c3346092-b638-42cc-b901-aa4b12a0e13e","Type":"ContainerStarted","Data":"101cabb20d179f58646161b79f96e1827a3e69c5df9bf8eb8d8da3129dc1bcb4"}
Sep 11 17:31:46.801533 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:31:46.801479    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-cbbb09-544b668766-zmvwq" event={"ID":"c3346092-b638-42cc-b901-aa4b12a0e13e","Type":"ContainerStarted","Data":"cfc8dc24f60f6f911be35f6b64271d82798529efcb0c016c8870485d8b759f0a"}
Sep 11 17:31:46.801533 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:31:46.801524    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-cbbb09-544b668766-zmvwq"
Sep 11 17:31:46.803080 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:31:46.803053    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-cbbb09-85f75d469f-fmfwt" event={"ID":"8e1a7c02-0df2-4958-bd88-2e212f3d89e4","Type":"ContainerStarted","Data":"09a3d4081916f90191d4e96de393c264ae180c8eec9d3ac76f317e32ed62d5dd"}
Sep 11 17:31:46.803203 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:31:46.803103    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-cbbb09-85f75d469f-fmfwt" event={"ID":"8e1a7c02-0df2-4958-bd88-2e212f3d89e4","Type":"ContainerStarted","Data":"646517111edce307e265218752c82cdb96744ca93113e4d897997b95525708b8"}
Sep 11 17:31:46.803203 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:31:46.803125    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-b-cbbb09-85f75d469f-fmfwt"
Sep 11 17:31:46.818267 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:31:46.818227    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-cbbb09-544b668766-zmvwq" podStartSLOduration=1.818212914 podStartE2EDuration="1.818212914s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 17:31:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 17:31:46.817027578 +0000 UTC m=+784.390247691" watchObservedRunningTime="2026-09-11 17:31:46.818212914 +0000 UTC m=+784.391433026"
Sep 11 17:31:46.833365 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:31:46.833304    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-b-cbbb09-85f75d469f-fmfwt" podStartSLOduration=1.833280894 podStartE2EDuration="1.833280894s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 17:31:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 17:31:46.832279883 +0000 UTC m=+784.405500019" watchObservedRunningTime="2026-09-11 17:31:46.833280894 +0000 UTC m=+784.406501011"
Sep 11 17:31:48.306817 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:31:48.306770    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-23a9cc4a-7475fb94f9-nfsm6"]
Sep 11 17:31:48.307523 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:31:48.307263    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-23a9cc4a-7475fb94f9-nfsm6" podUID="58eab99b-e56c-4ef4-838a-5448cc9758b6" containerName="payload-processing" containerID="cri-o://6c30cc191eb0bcff5a5407a83260c97dee9c9481228e539adcf7898cfe883758" gracePeriod=30
Sep 11 17:31:48.576173 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:31:48.576145    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-23a9cc4a-7475fb94f9-nfsm6"
Sep 11 17:31:48.580766 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:31:48.580737    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23a9cc4a-gx6kd"]
Sep 11 17:31:48.583063 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:31:48.583043    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23a9cc4a-gx6kd"]
Sep 11 17:31:48.667755 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:31:48.667710    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/58eab99b-e56c-4ef4-838a-5448cc9758b6-plugins-config-volume\") pod \"58eab99b-e56c-4ef4-838a-5448cc9758b6\" (UID: \"58eab99b-e56c-4ef4-838a-5448cc9758b6\") "
Sep 11 17:31:48.667955 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:31:48.667823    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tj6mv\" (UniqueName: \"kubernetes.io/projected/58eab99b-e56c-4ef4-838a-5448cc9758b6-kube-api-access-tj6mv\") pod \"58eab99b-e56c-4ef4-838a-5448cc9758b6\" (UID: \"58eab99b-e56c-4ef4-838a-5448cc9758b6\") "
Sep 11 17:31:48.668123 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:31:48.668096    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/58eab99b-e56c-4ef4-838a-5448cc9758b6-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "58eab99b-e56c-4ef4-838a-5448cc9758b6" (UID: "58eab99b-e56c-4ef4-838a-5448cc9758b6"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 17:31:48.670522 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:31:48.670478    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/58eab99b-e56c-4ef4-838a-5448cc9758b6-kube-api-access-tj6mv" (OuterVolumeSpecName: "kube-api-access-tj6mv") pod "58eab99b-e56c-4ef4-838a-5448cc9758b6" (UID: "58eab99b-e56c-4ef4-838a-5448cc9758b6"). InnerVolumeSpecName "kube-api-access-tj6mv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 17:31:48.768627 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:31:48.768579    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tj6mv\" (UniqueName: \"kubernetes.io/projected/58eab99b-e56c-4ef4-838a-5448cc9758b6-kube-api-access-tj6mv\") on node \"ip-10-0-141-139.ec2.internal\" DevicePath \"\""
Sep 11 17:31:48.768627 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:31:48.768617    2577 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/58eab99b-e56c-4ef4-838a-5448cc9758b6-plugins-config-volume\") on node \"ip-10-0-141-139.ec2.internal\" DevicePath \"\""
Sep 11 17:31:48.815392 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:31:48.815290    2577 generic.go:356] "Generic (PLEG): container finished" podID="58eab99b-e56c-4ef4-838a-5448cc9758b6" containerID="6c30cc191eb0bcff5a5407a83260c97dee9c9481228e539adcf7898cfe883758" exitCode=0
Sep 11 17:31:48.815392 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:31:48.815353    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-23a9cc4a-7475fb94f9-nfsm6" event={"ID":"58eab99b-e56c-4ef4-838a-5448cc9758b6","Type":"ContainerDied","Data":"6c30cc191eb0bcff5a5407a83260c97dee9c9481228e539adcf7898cfe883758"}
Sep 11 17:31:48.815392 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:31:48.815367    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-23a9cc4a-7475fb94f9-nfsm6"
Sep 11 17:31:48.815669 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:31:48.815420    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-23a9cc4a-7475fb94f9-nfsm6" event={"ID":"58eab99b-e56c-4ef4-838a-5448cc9758b6","Type":"ContainerDied","Data":"33fae3778b8dbbc49c505783cbadf8dd9ce14f32d059ea644510ac2ba45c2548"}
Sep 11 17:31:48.815669 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:31:48.815447    2577 scope.go:117] "RemoveContainer" containerID="6c30cc191eb0bcff5a5407a83260c97dee9c9481228e539adcf7898cfe883758"
Sep 11 17:31:48.826456 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:31:48.826410    2577 scope.go:117] "RemoveContainer" containerID="6c30cc191eb0bcff5a5407a83260c97dee9c9481228e539adcf7898cfe883758"
Sep 11 17:31:48.826829 ip-10-0-141-139 kubenswrapper[2577]: E0911 17:31:48.826806    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6c30cc191eb0bcff5a5407a83260c97dee9c9481228e539adcf7898cfe883758\": container with ID starting with 6c30cc191eb0bcff5a5407a83260c97dee9c9481228e539adcf7898cfe883758 not found: ID does not exist" containerID="6c30cc191eb0bcff5a5407a83260c97dee9c9481228e539adcf7898cfe883758"
Sep 11 17:31:48.826921 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:31:48.826842    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6c30cc191eb0bcff5a5407a83260c97dee9c9481228e539adcf7898cfe883758"} err="failed to get container status \"6c30cc191eb0bcff5a5407a83260c97dee9c9481228e539adcf7898cfe883758\": rpc error: code = NotFound desc = could not find container \"6c30cc191eb0bcff5a5407a83260c97dee9c9481228e539adcf7898cfe883758\": container with ID starting with 6c30cc191eb0bcff5a5407a83260c97dee9c9481228e539adcf7898cfe883758 not found: ID does not exist"
Sep 11 17:31:48.852718 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:31:48.852675    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-23a9cc4a-7475fb94f9-nfsm6"]
Sep 11 17:31:48.855920 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:31:48.855893    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-23a9cc4a-7475fb94f9-nfsm6"]
Sep 11 17:31:48.927489 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:31:48.927449    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="58eab99b-e56c-4ef4-838a-5448cc9758b6" path="/var/lib/kubelet/pods/58eab99b-e56c-4ef4-838a-5448cc9758b6/volumes"
Sep 11 17:31:48.927834 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:31:48.927819    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7acb539f-d736-49c4-9b72-8f9f2fee6c71" path="/var/lib/kubelet/pods/7acb539f-d736-49c4-9b72-8f9f2fee6c71/volumes"
Sep 11 17:31:52.605284 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:31:52.605244    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-nkmkm"]
Sep 11 17:31:55.474152 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:31:55.474113    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-d19de295-9c99867d-6jq7l"]
Sep 11 17:31:55.474613 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:31:55.474599    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="58eab99b-e56c-4ef4-838a-5448cc9758b6" containerName="payload-processing"
Sep 11 17:31:55.474671 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:31:55.474615    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="58eab99b-e56c-4ef4-838a-5448cc9758b6" containerName="payload-processing"
Sep 11 17:31:55.474708 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:31:55.474699    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="58eab99b-e56c-4ef4-838a-5448cc9758b6" containerName="payload-processing"
Sep 11 17:31:55.477789 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:31:55.477773    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-d19de295-9c99867d-6jq7l"
Sep 11 17:31:55.480267 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:31:55.480247    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-migrate-d19de295-dockercfg-rd722\""
Sep 11 17:31:55.480400 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:31:55.480287    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-migrate-d19de295\""
Sep 11 17:31:55.488824 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:31:55.488787    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-d19de295-9c99867d-6jq7l"]
Sep 11 17:31:55.540567 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:31:55.540524    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/19da6207-f8bd-4314-a058-1e332f16c3e8-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-d19de295-9c99867d-6jq7l\" (UID: \"19da6207-f8bd-4314-a058-1e332f16c3e8\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-d19de295-9c99867d-6jq7l"
Sep 11 17:31:55.540785 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:31:55.540660    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zztnq\" (UniqueName: \"kubernetes.io/projected/19da6207-f8bd-4314-a058-1e332f16c3e8-kube-api-access-zztnq\") pod \"payload-pre-processing-e2e-migrate-d19de295-9c99867d-6jq7l\" (UID: \"19da6207-f8bd-4314-a058-1e332f16c3e8\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-d19de295-9c99867d-6jq7l"
Sep 11 17:31:55.641910 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:31:55.641863    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zztnq\" (UniqueName: \"kubernetes.io/projected/19da6207-f8bd-4314-a058-1e332f16c3e8-kube-api-access-zztnq\") pod \"payload-pre-processing-e2e-migrate-d19de295-9c99867d-6jq7l\" (UID: \"19da6207-f8bd-4314-a058-1e332f16c3e8\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-d19de295-9c99867d-6jq7l"
Sep 11 17:31:55.642125 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:31:55.641962    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/19da6207-f8bd-4314-a058-1e332f16c3e8-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-d19de295-9c99867d-6jq7l\" (UID: \"19da6207-f8bd-4314-a058-1e332f16c3e8\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-d19de295-9c99867d-6jq7l"
Sep 11 17:31:55.642717 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:31:55.642698    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/19da6207-f8bd-4314-a058-1e332f16c3e8-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-d19de295-9c99867d-6jq7l\" (UID: \"19da6207-f8bd-4314-a058-1e332f16c3e8\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-d19de295-9c99867d-6jq7l"
Sep 11 17:31:55.651892 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:31:55.651858    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zztnq\" (UniqueName: \"kubernetes.io/projected/19da6207-f8bd-4314-a058-1e332f16c3e8-kube-api-access-zztnq\") pod \"payload-pre-processing-e2e-migrate-d19de295-9c99867d-6jq7l\" (UID: \"19da6207-f8bd-4314-a058-1e332f16c3e8\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-d19de295-9c99867d-6jq7l"
Sep 11 17:31:55.790626 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:31:55.790525    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-d19de295-9c99867d-6jq7l"
Sep 11 17:31:56.067527 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:31:56.067487    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-d19de295-9c99867d-6jq7l"]
Sep 11 17:31:56.069331 ip-10-0-141-139 kubenswrapper[2577]: W0911 17:31:56.069301    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod19da6207_f8bd_4314_a058_1e332f16c3e8.slice/crio-07a333e35b59c60eaa98610ea11d3c0ba3018d96f4f8dd690ac2ad006d97f8d4 WatchSource:0}: Error finding container 07a333e35b59c60eaa98610ea11d3c0ba3018d96f4f8dd690ac2ad006d97f8d4: Status 404 returned error can't find the container with id 07a333e35b59c60eaa98610ea11d3c0ba3018d96f4f8dd690ac2ad006d97f8d4
Sep 11 17:31:56.596400 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:31:56.596354    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-144bfa/simulator-w4-144bfa-kserve-6ddd858bc-4vs4s"]
Sep 11 17:31:56.599353 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:31:56.599326    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-144bfa/simulator-w4-144bfa-kserve-6ddd858bc-4vs4s"
Sep 11 17:31:56.602945 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:31:56.602907    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-144bfa\"/\"default-dockercfg-chr7w\""
Sep 11 17:31:56.602945 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:31:56.602923    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-144bfa\"/\"openshift-service-ca.crt\""
Sep 11 17:31:56.603143 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:31:56.602950    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-144bfa\"/\"simulator-w4-144bfa-kserve-self-signed-certs\""
Sep 11 17:31:56.603143 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:31:56.602985    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-144bfa\"/\"kube-root-ca.crt\""
Sep 11 17:31:56.608896 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:31:56.608874    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-144bfa/simulator-w4-144bfa-kserve-6ddd858bc-4vs4s"]
Sep 11 17:31:56.652394 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:31:56.652357    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1a60ac0d-485c-48b2-8ea4-6633cdbcd138-model-cache\") pod \"simulator-w4-144bfa-kserve-6ddd858bc-4vs4s\" (UID: \"1a60ac0d-485c-48b2-8ea4-6633cdbcd138\") " pod="e2e-models-e2e-worker-w4-144bfa/simulator-w4-144bfa-kserve-6ddd858bc-4vs4s"
Sep 11 17:31:56.652394 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:31:56.652399    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1a60ac0d-485c-48b2-8ea4-6633cdbcd138-tmp-dir\") pod \"simulator-w4-144bfa-kserve-6ddd858bc-4vs4s\" (UID: \"1a60ac0d-485c-48b2-8ea4-6633cdbcd138\") " pod="e2e-models-e2e-worker-w4-144bfa/simulator-w4-144bfa-kserve-6ddd858bc-4vs4s"
Sep 11 17:31:56.652635 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:31:56.652420    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1a60ac0d-485c-48b2-8ea4-6633cdbcd138-tls-certs\") pod \"simulator-w4-144bfa-kserve-6ddd858bc-4vs4s\" (UID: \"1a60ac0d-485c-48b2-8ea4-6633cdbcd138\") " pod="e2e-models-e2e-worker-w4-144bfa/simulator-w4-144bfa-kserve-6ddd858bc-4vs4s"
Sep 11 17:31:56.652635 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:31:56.652580    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1a60ac0d-485c-48b2-8ea4-6633cdbcd138-home\") pod \"simulator-w4-144bfa-kserve-6ddd858bc-4vs4s\" (UID: \"1a60ac0d-485c-48b2-8ea4-6633cdbcd138\") " pod="e2e-models-e2e-worker-w4-144bfa/simulator-w4-144bfa-kserve-6ddd858bc-4vs4s"
Sep 11 17:31:56.652702 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:31:56.652639    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rsw29\" (UniqueName: \"kubernetes.io/projected/1a60ac0d-485c-48b2-8ea4-6633cdbcd138-kube-api-access-rsw29\") pod \"simulator-w4-144bfa-kserve-6ddd858bc-4vs4s\" (UID: \"1a60ac0d-485c-48b2-8ea4-6633cdbcd138\") " pod="e2e-models-e2e-worker-w4-144bfa/simulator-w4-144bfa-kserve-6ddd858bc-4vs4s"
Sep 11 17:31:56.652762 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:31:56.652744    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1a60ac0d-485c-48b2-8ea4-6633cdbcd138-dshm\") pod \"simulator-w4-144bfa-kserve-6ddd858bc-4vs4s\" (UID: \"1a60ac0d-485c-48b2-8ea4-6633cdbcd138\") " pod="e2e-models-e2e-worker-w4-144bfa/simulator-w4-144bfa-kserve-6ddd858bc-4vs4s"
Sep 11 17:31:56.753739 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:31:56.753703    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1a60ac0d-485c-48b2-8ea4-6633cdbcd138-model-cache\") pod \"simulator-w4-144bfa-kserve-6ddd858bc-4vs4s\" (UID: \"1a60ac0d-485c-48b2-8ea4-6633cdbcd138\") " pod="e2e-models-e2e-worker-w4-144bfa/simulator-w4-144bfa-kserve-6ddd858bc-4vs4s"
Sep 11 17:31:56.753739 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:31:56.753743    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1a60ac0d-485c-48b2-8ea4-6633cdbcd138-tmp-dir\") pod \"simulator-w4-144bfa-kserve-6ddd858bc-4vs4s\" (UID: \"1a60ac0d-485c-48b2-8ea4-6633cdbcd138\") " pod="e2e-models-e2e-worker-w4-144bfa/simulator-w4-144bfa-kserve-6ddd858bc-4vs4s"
Sep 11 17:31:56.753984 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:31:56.753759    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1a60ac0d-485c-48b2-8ea4-6633cdbcd138-tls-certs\") pod \"simulator-w4-144bfa-kserve-6ddd858bc-4vs4s\" (UID: \"1a60ac0d-485c-48b2-8ea4-6633cdbcd138\") " pod="e2e-models-e2e-worker-w4-144bfa/simulator-w4-144bfa-kserve-6ddd858bc-4vs4s"
Sep 11 17:31:56.753984 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:31:56.753795    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1a60ac0d-485c-48b2-8ea4-6633cdbcd138-home\") pod \"simulator-w4-144bfa-kserve-6ddd858bc-4vs4s\" (UID: \"1a60ac0d-485c-48b2-8ea4-6633cdbcd138\") " pod="e2e-models-e2e-worker-w4-144bfa/simulator-w4-144bfa-kserve-6ddd858bc-4vs4s"
Sep 11 17:31:56.753984 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:31:56.753835    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rsw29\" (UniqueName: \"kubernetes.io/projected/1a60ac0d-485c-48b2-8ea4-6633cdbcd138-kube-api-access-rsw29\") pod \"simulator-w4-144bfa-kserve-6ddd858bc-4vs4s\" (UID: \"1a60ac0d-485c-48b2-8ea4-6633cdbcd138\") " pod="e2e-models-e2e-worker-w4-144bfa/simulator-w4-144bfa-kserve-6ddd858bc-4vs4s"
Sep 11 17:31:56.753984 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:31:56.753904    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1a60ac0d-485c-48b2-8ea4-6633cdbcd138-dshm\") pod \"simulator-w4-144bfa-kserve-6ddd858bc-4vs4s\" (UID: \"1a60ac0d-485c-48b2-8ea4-6633cdbcd138\") " pod="e2e-models-e2e-worker-w4-144bfa/simulator-w4-144bfa-kserve-6ddd858bc-4vs4s"
Sep 11 17:31:56.754215 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:31:56.754115    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1a60ac0d-485c-48b2-8ea4-6633cdbcd138-model-cache\") pod \"simulator-w4-144bfa-kserve-6ddd858bc-4vs4s\" (UID: \"1a60ac0d-485c-48b2-8ea4-6633cdbcd138\") " pod="e2e-models-e2e-worker-w4-144bfa/simulator-w4-144bfa-kserve-6ddd858bc-4vs4s"
Sep 11 17:31:56.754215 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:31:56.754195    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1a60ac0d-485c-48b2-8ea4-6633cdbcd138-tmp-dir\") pod \"simulator-w4-144bfa-kserve-6ddd858bc-4vs4s\" (UID: \"1a60ac0d-485c-48b2-8ea4-6633cdbcd138\") " pod="e2e-models-e2e-worker-w4-144bfa/simulator-w4-144bfa-kserve-6ddd858bc-4vs4s"
Sep 11 17:31:56.754341 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:31:56.754314    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1a60ac0d-485c-48b2-8ea4-6633cdbcd138-home\") pod \"simulator-w4-144bfa-kserve-6ddd858bc-4vs4s\" (UID: \"1a60ac0d-485c-48b2-8ea4-6633cdbcd138\") " pod="e2e-models-e2e-worker-w4-144bfa/simulator-w4-144bfa-kserve-6ddd858bc-4vs4s"
Sep 11 17:31:56.756258 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:31:56.756232    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1a60ac0d-485c-48b2-8ea4-6633cdbcd138-dshm\") pod \"simulator-w4-144bfa-kserve-6ddd858bc-4vs4s\" (UID: \"1a60ac0d-485c-48b2-8ea4-6633cdbcd138\") " pod="e2e-models-e2e-worker-w4-144bfa/simulator-w4-144bfa-kserve-6ddd858bc-4vs4s"
Sep 11 17:31:56.756406 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:31:56.756389    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1a60ac0d-485c-48b2-8ea4-6633cdbcd138-tls-certs\") pod \"simulator-w4-144bfa-kserve-6ddd858bc-4vs4s\" (UID: \"1a60ac0d-485c-48b2-8ea4-6633cdbcd138\") " pod="e2e-models-e2e-worker-w4-144bfa/simulator-w4-144bfa-kserve-6ddd858bc-4vs4s"
Sep 11 17:31:56.761752 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:31:56.761725    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rsw29\" (UniqueName: \"kubernetes.io/projected/1a60ac0d-485c-48b2-8ea4-6633cdbcd138-kube-api-access-rsw29\") pod \"simulator-w4-144bfa-kserve-6ddd858bc-4vs4s\" (UID: \"1a60ac0d-485c-48b2-8ea4-6633cdbcd138\") " pod="e2e-models-e2e-worker-w4-144bfa/simulator-w4-144bfa-kserve-6ddd858bc-4vs4s"
Sep 11 17:31:56.865272 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:31:56.865172    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-d19de295-9c99867d-6jq7l" event={"ID":"19da6207-f8bd-4314-a058-1e332f16c3e8","Type":"ContainerStarted","Data":"7a3c471ce44fe8cbb8920b7f5be53a0295747bf81c27c1ec67264654e34942c5"}
Sep 11 17:31:56.865272 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:31:56.865231    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-d19de295-9c99867d-6jq7l" event={"ID":"19da6207-f8bd-4314-a058-1e332f16c3e8","Type":"ContainerStarted","Data":"07a333e35b59c60eaa98610ea11d3c0ba3018d96f4f8dd690ac2ad006d97f8d4"}
Sep 11 17:31:56.865475 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:31:56.865377    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-migrate-d19de295-9c99867d-6jq7l"
Sep 11 17:31:56.881557 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:31:56.881471    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-migrate-d19de295-9c99867d-6jq7l" podStartSLOduration=1.8814524000000001 podStartE2EDuration="1.8814524s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 17:31:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 17:31:56.880318157 +0000 UTC m=+794.453538271" watchObservedRunningTime="2026-09-11 17:31:56.8814524 +0000 UTC m=+794.454672515"
Sep 11 17:31:56.910587 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:31:56.910551    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-144bfa/simulator-w4-144bfa-kserve-6ddd858bc-4vs4s"
Sep 11 17:31:57.188750 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:31:57.188718    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-144bfa/simulator-w4-144bfa-kserve-6ddd858bc-4vs4s"]
Sep 11 17:31:57.190891 ip-10-0-141-139 kubenswrapper[2577]: W0911 17:31:57.190865    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod1a60ac0d_485c_48b2_8ea4_6633cdbcd138.slice/crio-8993064e6203d5e7848ede64b47cd1a9aa528e303c5453d2c7334ee9633d3649 WatchSource:0}: Error finding container 8993064e6203d5e7848ede64b47cd1a9aa528e303c5453d2c7334ee9633d3649: Status 404 returned error can't find the container with id 8993064e6203d5e7848ede64b47cd1a9aa528e303c5453d2c7334ee9633d3649
Sep 11 17:31:57.811112 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:31:57.811078    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-cbbb09-544b668766-zmvwq"
Sep 11 17:31:57.811529 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:31:57.811262    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-b-cbbb09-85f75d469f-fmfwt"
Sep 11 17:31:57.874971 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:31:57.874935    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-144bfa/simulator-w4-144bfa-kserve-6ddd858bc-4vs4s" event={"ID":"1a60ac0d-485c-48b2-8ea4-6633cdbcd138","Type":"ContainerStarted","Data":"1b1f22351c03b1e69349f6e8b5008c6a239520ea1e97a05676c19abf0fe06c85"}
Sep 11 17:31:57.874971 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:31:57.874982    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-144bfa/simulator-w4-144bfa-kserve-6ddd858bc-4vs4s" event={"ID":"1a60ac0d-485c-48b2-8ea4-6633cdbcd138","Type":"ContainerStarted","Data":"8993064e6203d5e7848ede64b47cd1a9aa528e303c5453d2c7334ee9633d3649"}
Sep 11 17:31:57.893986 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:31:57.893927    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-144bfa/simulator-w4-144bfa-kserve-6ddd858bc-4vs4s" podStartSLOduration=1.893908712 podStartE2EDuration="1.893908712s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 17:31:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 17:31:57.891478837 +0000 UTC m=+795.464698950" watchObservedRunningTime="2026-09-11 17:31:57.893908712 +0000 UTC m=+795.467128823"
Sep 11 17:32:06.911171 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:32:06.911069    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-144bfa/simulator-w4-144bfa-kserve-6ddd858bc-4vs4s"
Sep 11 17:32:06.911171 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:32:06.911126    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-144bfa/simulator-w4-144bfa-kserve-6ddd858bc-4vs4s"
Sep 11 17:32:06.926614 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:32:06.926581    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-144bfa/simulator-w4-144bfa-kserve-6ddd858bc-4vs4s"
Sep 11 17:32:06.938398 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:32:06.938368    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-144bfa/simulator-w4-144bfa-kserve-6ddd858bc-4vs4s"
Sep 11 17:32:07.877048 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:32:07.877021    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-migrate-d19de295-9c99867d-6jq7l"
Sep 11 17:32:17.754455 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:32:17.754419    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-d19de295-thh6c"]
Sep 11 17:32:17.759370 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:32:17.759351    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-d19de295-thh6c"
Sep 11 17:32:17.761892 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:32:17.761857    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 11 17:32:17.762058 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:32:17.761924    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-9n8gf\""
Sep 11 17:32:17.765946 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:32:17.765917    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-d19de295-thh6c"]
Sep 11 17:32:17.880378 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:32:17.880340    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/430e4710-8b46-4d9f-a094-8f0e536c17f4-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-d19de295-thh6c\" (UID: \"430e4710-8b46-4d9f-a094-8f0e536c17f4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-d19de295-thh6c"
Sep 11 17:32:17.981346 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:32:17.981310    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/430e4710-8b46-4d9f-a094-8f0e536c17f4-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-d19de295-thh6c\" (UID: \"430e4710-8b46-4d9f-a094-8f0e536c17f4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-d19de295-thh6c"
Sep 11 17:32:17.982027 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:32:17.982005    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/430e4710-8b46-4d9f-a094-8f0e536c17f4-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-d19de295-thh6c\" (UID: \"430e4710-8b46-4d9f-a094-8f0e536c17f4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-d19de295-thh6c"
Sep 11 17:32:18.072876 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:32:18.072790    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-d19de295-thh6c"
Sep 11 17:32:18.296981 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:32:18.296943    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-nkmkm"]
Sep 11 17:32:18.345667 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:32:18.345640    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-d19de295-thh6c"]
Sep 11 17:32:18.347131 ip-10-0-141-139 kubenswrapper[2577]: W0911 17:32:18.347105    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod430e4710_8b46_4d9f_a094_8f0e536c17f4.slice/crio-d44e2f921b9e42474ae1cfdb681ce6b6e33abe1778379a4bc869f6c5282d28bf WatchSource:0}: Error finding container d44e2f921b9e42474ae1cfdb681ce6b6e33abe1778379a4bc869f6c5282d28bf: Status 404 returned error can't find the container with id d44e2f921b9e42474ae1cfdb681ce6b6e33abe1778379a4bc869f6c5282d28bf
Sep 11 17:32:18.991749 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:32:18.991706    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-d19de295-thh6c" event={"ID":"430e4710-8b46-4d9f-a094-8f0e536c17f4","Type":"ContainerStarted","Data":"afc2706c0b7cc82ab9b88b12696e98cc53957038c0e7edde31698486d11b6d69"}
Sep 11 17:32:18.992143 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:32:18.991773    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-d19de295-thh6c" event={"ID":"430e4710-8b46-4d9f-a094-8f0e536c17f4","Type":"ContainerStarted","Data":"d44e2f921b9e42474ae1cfdb681ce6b6e33abe1778379a4bc869f6c5282d28bf"}
Sep 11 17:32:19.008003 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:32:19.007950    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-d19de295-thh6c" podStartSLOduration=2.007935251 podStartE2EDuration="2.007935251s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 17:32:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 17:32:19.00533546 +0000 UTC m=+816.578555588" watchObservedRunningTime="2026-09-11 17:32:19.007935251 +0000 UTC m=+816.581155380"
Sep 11 17:32:19.999168 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:32:19.999128    2577 generic.go:356] "Generic (PLEG): container finished" podID="430e4710-8b46-4d9f-a094-8f0e536c17f4" containerID="afc2706c0b7cc82ab9b88b12696e98cc53957038c0e7edde31698486d11b6d69" exitCode=0
Sep 11 17:32:19.999584 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:32:19.999213    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-d19de295-thh6c" event={"ID":"430e4710-8b46-4d9f-a094-8f0e536c17f4","Type":"ContainerDied","Data":"afc2706c0b7cc82ab9b88b12696e98cc53957038c0e7edde31698486d11b6d69"}
Sep 11 17:32:21.147851 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:32:21.147824    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-d19de295-thh6c"
Sep 11 17:32:21.313697 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:32:21.313602    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/430e4710-8b46-4d9f-a094-8f0e536c17f4-maas-api-service-ca\") pod \"430e4710-8b46-4d9f-a094-8f0e536c17f4\" (UID: \"430e4710-8b46-4d9f-a094-8f0e536c17f4\") "
Sep 11 17:32:21.313913 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:32:21.313890    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/430e4710-8b46-4d9f-a094-8f0e536c17f4-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "430e4710-8b46-4d9f-a094-8f0e536c17f4" (UID: "430e4710-8b46-4d9f-a094-8f0e536c17f4"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 17:32:21.314016 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:32:21.313998    2577 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/430e4710-8b46-4d9f-a094-8f0e536c17f4-maas-api-service-ca\") on node \"ip-10-0-141-139.ec2.internal\" DevicePath \"\""
Sep 11 17:32:22.012329 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:32:22.012297    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-d19de295-thh6c"
Sep 11 17:32:22.012519 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:32:22.012326    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-d19de295-thh6c" event={"ID":"430e4710-8b46-4d9f-a094-8f0e536c17f4","Type":"ContainerDied","Data":"d44e2f921b9e42474ae1cfdb681ce6b6e33abe1778379a4bc869f6c5282d28bf"}
Sep 11 17:32:22.012519 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:32:22.012372    2577 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d44e2f921b9e42474ae1cfdb681ce6b6e33abe1778379a4bc869f6c5282d28bf"
Sep 11 17:32:24.261425 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:32:24.261384    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-d19de295-9c99867d-6jq7l"]
Sep 11 17:32:24.261880 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:32:24.261673    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-migrate-d19de295-9c99867d-6jq7l" podUID="19da6207-f8bd-4314-a058-1e332f16c3e8" containerName="payload-pre-processing" containerID="cri-o://7a3c471ce44fe8cbb8920b7f5be53a0295747bf81c27c1ec67264654e34942c5" gracePeriod=30
Sep 11 17:32:24.426621 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:32:24.426588    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-d19de295-thh6c"]
Sep 11 17:32:24.429283 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:32:24.429257    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-d19de295-thh6c"]
Sep 11 17:32:24.512575 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:32:24.511129    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-d19de295-9c99867d-6jq7l"
Sep 11 17:32:24.646707 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:32:24.646674    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/19da6207-f8bd-4314-a058-1e332f16c3e8-plugins-config-volume\") pod \"19da6207-f8bd-4314-a058-1e332f16c3e8\" (UID: \"19da6207-f8bd-4314-a058-1e332f16c3e8\") "
Sep 11 17:32:24.646881 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:32:24.646724    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zztnq\" (UniqueName: \"kubernetes.io/projected/19da6207-f8bd-4314-a058-1e332f16c3e8-kube-api-access-zztnq\") pod \"19da6207-f8bd-4314-a058-1e332f16c3e8\" (UID: \"19da6207-f8bd-4314-a058-1e332f16c3e8\") "
Sep 11 17:32:24.647074 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:32:24.647052    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/19da6207-f8bd-4314-a058-1e332f16c3e8-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "19da6207-f8bd-4314-a058-1e332f16c3e8" (UID: "19da6207-f8bd-4314-a058-1e332f16c3e8"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 17:32:24.649053 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:32:24.649037    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/19da6207-f8bd-4314-a058-1e332f16c3e8-kube-api-access-zztnq" (OuterVolumeSpecName: "kube-api-access-zztnq") pod "19da6207-f8bd-4314-a058-1e332f16c3e8" (UID: "19da6207-f8bd-4314-a058-1e332f16c3e8"). InnerVolumeSpecName "kube-api-access-zztnq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 17:32:24.747965 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:32:24.747928    2577 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/19da6207-f8bd-4314-a058-1e332f16c3e8-plugins-config-volume\") on node \"ip-10-0-141-139.ec2.internal\" DevicePath \"\""
Sep 11 17:32:24.747965 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:32:24.747961    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zztnq\" (UniqueName: \"kubernetes.io/projected/19da6207-f8bd-4314-a058-1e332f16c3e8-kube-api-access-zztnq\") on node \"ip-10-0-141-139.ec2.internal\" DevicePath \"\""
Sep 11 17:32:24.927655 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:32:24.927574    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="430e4710-8b46-4d9f-a094-8f0e536c17f4" path="/var/lib/kubelet/pods/430e4710-8b46-4d9f-a094-8f0e536c17f4/volumes"
Sep 11 17:32:25.030285 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:32:25.030247    2577 generic.go:356] "Generic (PLEG): container finished" podID="19da6207-f8bd-4314-a058-1e332f16c3e8" containerID="7a3c471ce44fe8cbb8920b7f5be53a0295747bf81c27c1ec67264654e34942c5" exitCode=0
Sep 11 17:32:25.030463 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:32:25.030302    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-d19de295-9c99867d-6jq7l"
Sep 11 17:32:25.030463 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:32:25.030342    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-d19de295-9c99867d-6jq7l" event={"ID":"19da6207-f8bd-4314-a058-1e332f16c3e8","Type":"ContainerDied","Data":"7a3c471ce44fe8cbb8920b7f5be53a0295747bf81c27c1ec67264654e34942c5"}
Sep 11 17:32:25.030463 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:32:25.030399    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-d19de295-9c99867d-6jq7l" event={"ID":"19da6207-f8bd-4314-a058-1e332f16c3e8","Type":"ContainerDied","Data":"07a333e35b59c60eaa98610ea11d3c0ba3018d96f4f8dd690ac2ad006d97f8d4"}
Sep 11 17:32:25.030463 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:32:25.030428    2577 scope.go:117] "RemoveContainer" containerID="7a3c471ce44fe8cbb8920b7f5be53a0295747bf81c27c1ec67264654e34942c5"
Sep 11 17:32:25.039979 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:32:25.039956    2577 scope.go:117] "RemoveContainer" containerID="7a3c471ce44fe8cbb8920b7f5be53a0295747bf81c27c1ec67264654e34942c5"
Sep 11 17:32:25.040332 ip-10-0-141-139 kubenswrapper[2577]: E0911 17:32:25.040305    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7a3c471ce44fe8cbb8920b7f5be53a0295747bf81c27c1ec67264654e34942c5\": container with ID starting with 7a3c471ce44fe8cbb8920b7f5be53a0295747bf81c27c1ec67264654e34942c5 not found: ID does not exist" containerID="7a3c471ce44fe8cbb8920b7f5be53a0295747bf81c27c1ec67264654e34942c5"
Sep 11 17:32:25.040385 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:32:25.040346    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7a3c471ce44fe8cbb8920b7f5be53a0295747bf81c27c1ec67264654e34942c5"} err="failed to get container status \"7a3c471ce44fe8cbb8920b7f5be53a0295747bf81c27c1ec67264654e34942c5\": rpc error: code = NotFound desc = could not find container \"7a3c471ce44fe8cbb8920b7f5be53a0295747bf81c27c1ec67264654e34942c5\": container with ID starting with 7a3c471ce44fe8cbb8920b7f5be53a0295747bf81c27c1ec67264654e34942c5 not found: ID does not exist"
Sep 11 17:32:25.046781 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:32:25.046753    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-d19de295-9c99867d-6jq7l"]
Sep 11 17:32:25.048756 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:32:25.048734    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-d19de295-9c99867d-6jq7l"]
Sep 11 17:32:26.929113 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:32:26.929080    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="19da6207-f8bd-4314-a058-1e332f16c3e8" path="/var/lib/kubelet/pods/19da6207-f8bd-4314-a058-1e332f16c3e8/volumes"
Sep 11 17:32:30.891162 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:32:30.891125    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-nkmkm"]
Sep 11 17:32:31.979955 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:32:31.979913    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-c7d192/simulator-w0-c7d192-kserve-6bfc765cf5-klnxw"]
Sep 11 17:32:31.980630 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:32:31.980609    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="19da6207-f8bd-4314-a058-1e332f16c3e8" containerName="payload-pre-processing"
Sep 11 17:32:31.980722 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:32:31.980631    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="19da6207-f8bd-4314-a058-1e332f16c3e8" containerName="payload-pre-processing"
Sep 11 17:32:31.980722 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:32:31.980653    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="430e4710-8b46-4d9f-a094-8f0e536c17f4" containerName="revoke-keys"
Sep 11 17:32:31.980722 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:32:31.980662    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="430e4710-8b46-4d9f-a094-8f0e536c17f4" containerName="revoke-keys"
Sep 11 17:32:31.980889 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:32:31.980785    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="19da6207-f8bd-4314-a058-1e332f16c3e8" containerName="payload-pre-processing"
Sep 11 17:32:31.980889 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:32:31.980797    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="430e4710-8b46-4d9f-a094-8f0e536c17f4" containerName="revoke-keys"
Sep 11 17:32:31.984283 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:32:31.984253    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-c7d192/simulator-w0-c7d192-kserve-6bfc765cf5-klnxw"
Sep 11 17:32:31.986896 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:32:31.986876    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-c7d192\"/\"kube-root-ca.crt\""
Sep 11 17:32:31.987011 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:32:31.986914    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-c7d192\"/\"openshift-service-ca.crt\""
Sep 11 17:32:31.988368 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:32:31.988343    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-c7d192\"/\"default-dockercfg-jxfg5\""
Sep 11 17:32:31.988475 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:32:31.988376    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-c7d192\"/\"simulator-w0-c7d192-kserve-self-signed-certs\""
Sep 11 17:32:31.993473 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:32:31.993451    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-c7d192/simulator-w0-c7d192-kserve-6bfc765cf5-klnxw"]
Sep 11 17:32:32.016229 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:32:32.016196    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c30aa08c-b2d4-4edc-8561-e78ad422fd1f-dshm\") pod \"simulator-w0-c7d192-kserve-6bfc765cf5-klnxw\" (UID: \"c30aa08c-b2d4-4edc-8561-e78ad422fd1f\") " pod="e2e-models-e2e-worker-w0-c7d192/simulator-w0-c7d192-kserve-6bfc765cf5-klnxw"
Sep 11 17:32:32.016421 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:32:32.016297    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c30aa08c-b2d4-4edc-8561-e78ad422fd1f-home\") pod \"simulator-w0-c7d192-kserve-6bfc765cf5-klnxw\" (UID: \"c30aa08c-b2d4-4edc-8561-e78ad422fd1f\") " pod="e2e-models-e2e-worker-w0-c7d192/simulator-w0-c7d192-kserve-6bfc765cf5-klnxw"
Sep 11 17:32:32.016421 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:32:32.016349    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c30aa08c-b2d4-4edc-8561-e78ad422fd1f-tmp-dir\") pod \"simulator-w0-c7d192-kserve-6bfc765cf5-klnxw\" (UID: \"c30aa08c-b2d4-4edc-8561-e78ad422fd1f\") " pod="e2e-models-e2e-worker-w0-c7d192/simulator-w0-c7d192-kserve-6bfc765cf5-klnxw"
Sep 11 17:32:32.016421 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:32:32.016402    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c30aa08c-b2d4-4edc-8561-e78ad422fd1f-model-cache\") pod \"simulator-w0-c7d192-kserve-6bfc765cf5-klnxw\" (UID: \"c30aa08c-b2d4-4edc-8561-e78ad422fd1f\") " pod="e2e-models-e2e-worker-w0-c7d192/simulator-w0-c7d192-kserve-6bfc765cf5-klnxw"
Sep 11 17:32:32.016564 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:32:32.016541    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m4drh\" (UniqueName: \"kubernetes.io/projected/c30aa08c-b2d4-4edc-8561-e78ad422fd1f-kube-api-access-m4drh\") pod \"simulator-w0-c7d192-kserve-6bfc765cf5-klnxw\" (UID: \"c30aa08c-b2d4-4edc-8561-e78ad422fd1f\") " pod="e2e-models-e2e-worker-w0-c7d192/simulator-w0-c7d192-kserve-6bfc765cf5-klnxw"
Sep 11 17:32:32.016616 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:32:32.016600    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c30aa08c-b2d4-4edc-8561-e78ad422fd1f-tls-certs\") pod \"simulator-w0-c7d192-kserve-6bfc765cf5-klnxw\" (UID: \"c30aa08c-b2d4-4edc-8561-e78ad422fd1f\") " pod="e2e-models-e2e-worker-w0-c7d192/simulator-w0-c7d192-kserve-6bfc765cf5-klnxw"
Sep 11 17:32:32.117204 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:32:32.117170    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c30aa08c-b2d4-4edc-8561-e78ad422fd1f-model-cache\") pod \"simulator-w0-c7d192-kserve-6bfc765cf5-klnxw\" (UID: \"c30aa08c-b2d4-4edc-8561-e78ad422fd1f\") " pod="e2e-models-e2e-worker-w0-c7d192/simulator-w0-c7d192-kserve-6bfc765cf5-klnxw"
Sep 11 17:32:32.117399 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:32:32.117229    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m4drh\" (UniqueName: \"kubernetes.io/projected/c30aa08c-b2d4-4edc-8561-e78ad422fd1f-kube-api-access-m4drh\") pod \"simulator-w0-c7d192-kserve-6bfc765cf5-klnxw\" (UID: \"c30aa08c-b2d4-4edc-8561-e78ad422fd1f\") " pod="e2e-models-e2e-worker-w0-c7d192/simulator-w0-c7d192-kserve-6bfc765cf5-klnxw"
Sep 11 17:32:32.117399 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:32:32.117261    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c30aa08c-b2d4-4edc-8561-e78ad422fd1f-tls-certs\") pod \"simulator-w0-c7d192-kserve-6bfc765cf5-klnxw\" (UID: \"c30aa08c-b2d4-4edc-8561-e78ad422fd1f\") " pod="e2e-models-e2e-worker-w0-c7d192/simulator-w0-c7d192-kserve-6bfc765cf5-klnxw"
Sep 11 17:32:32.117399 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:32:32.117299    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c30aa08c-b2d4-4edc-8561-e78ad422fd1f-dshm\") pod \"simulator-w0-c7d192-kserve-6bfc765cf5-klnxw\" (UID: \"c30aa08c-b2d4-4edc-8561-e78ad422fd1f\") " pod="e2e-models-e2e-worker-w0-c7d192/simulator-w0-c7d192-kserve-6bfc765cf5-klnxw"
Sep 11 17:32:32.117599 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:32:32.117452    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c30aa08c-b2d4-4edc-8561-e78ad422fd1f-home\") pod \"simulator-w0-c7d192-kserve-6bfc765cf5-klnxw\" (UID: \"c30aa08c-b2d4-4edc-8561-e78ad422fd1f\") " pod="e2e-models-e2e-worker-w0-c7d192/simulator-w0-c7d192-kserve-6bfc765cf5-klnxw"
Sep 11 17:32:32.117599 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:32:32.117546    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c30aa08c-b2d4-4edc-8561-e78ad422fd1f-tmp-dir\") pod \"simulator-w0-c7d192-kserve-6bfc765cf5-klnxw\" (UID: \"c30aa08c-b2d4-4edc-8561-e78ad422fd1f\") " pod="e2e-models-e2e-worker-w0-c7d192/simulator-w0-c7d192-kserve-6bfc765cf5-klnxw"
Sep 11 17:32:32.117715 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:32:32.117638    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c30aa08c-b2d4-4edc-8561-e78ad422fd1f-model-cache\") pod \"simulator-w0-c7d192-kserve-6bfc765cf5-klnxw\" (UID: \"c30aa08c-b2d4-4edc-8561-e78ad422fd1f\") " pod="e2e-models-e2e-worker-w0-c7d192/simulator-w0-c7d192-kserve-6bfc765cf5-klnxw"
Sep 11 17:32:32.117789 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:32:32.117770    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c30aa08c-b2d4-4edc-8561-e78ad422fd1f-home\") pod \"simulator-w0-c7d192-kserve-6bfc765cf5-klnxw\" (UID: \"c30aa08c-b2d4-4edc-8561-e78ad422fd1f\") " pod="e2e-models-e2e-worker-w0-c7d192/simulator-w0-c7d192-kserve-6bfc765cf5-klnxw"
Sep 11 17:32:32.117891 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:32:32.117871    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c30aa08c-b2d4-4edc-8561-e78ad422fd1f-tmp-dir\") pod \"simulator-w0-c7d192-kserve-6bfc765cf5-klnxw\" (UID: \"c30aa08c-b2d4-4edc-8561-e78ad422fd1f\") " pod="e2e-models-e2e-worker-w0-c7d192/simulator-w0-c7d192-kserve-6bfc765cf5-klnxw"
Sep 11 17:32:32.119832 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:32:32.119812    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c30aa08c-b2d4-4edc-8561-e78ad422fd1f-dshm\") pod \"simulator-w0-c7d192-kserve-6bfc765cf5-klnxw\" (UID: \"c30aa08c-b2d4-4edc-8561-e78ad422fd1f\") " pod="e2e-models-e2e-worker-w0-c7d192/simulator-w0-c7d192-kserve-6bfc765cf5-klnxw"
Sep 11 17:32:32.120081 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:32:32.120065    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c30aa08c-b2d4-4edc-8561-e78ad422fd1f-tls-certs\") pod \"simulator-w0-c7d192-kserve-6bfc765cf5-klnxw\" (UID: \"c30aa08c-b2d4-4edc-8561-e78ad422fd1f\") " pod="e2e-models-e2e-worker-w0-c7d192/simulator-w0-c7d192-kserve-6bfc765cf5-klnxw"
Sep 11 17:32:32.125580 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:32:32.125558    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-m4drh\" (UniqueName: \"kubernetes.io/projected/c30aa08c-b2d4-4edc-8561-e78ad422fd1f-kube-api-access-m4drh\") pod \"simulator-w0-c7d192-kserve-6bfc765cf5-klnxw\" (UID: \"c30aa08c-b2d4-4edc-8561-e78ad422fd1f\") " pod="e2e-models-e2e-worker-w0-c7d192/simulator-w0-c7d192-kserve-6bfc765cf5-klnxw"
Sep 11 17:32:32.294784 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:32:32.294691    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-c7d192/simulator-w0-c7d192-kserve-6bfc765cf5-klnxw"
Sep 11 17:32:32.569393 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:32:32.569368    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-c7d192/simulator-w0-c7d192-kserve-6bfc765cf5-klnxw"]
Sep 11 17:32:32.571886 ip-10-0-141-139 kubenswrapper[2577]: W0911 17:32:32.571860    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc30aa08c_b2d4_4edc_8561_e78ad422fd1f.slice/crio-1c9da7dd3c09e732d511a6da43370ce0d9a8827d33963f68c9f25d1b4629be42 WatchSource:0}: Error finding container 1c9da7dd3c09e732d511a6da43370ce0d9a8827d33963f68c9f25d1b4629be42: Status 404 returned error can't find the container with id 1c9da7dd3c09e732d511a6da43370ce0d9a8827d33963f68c9f25d1b4629be42
Sep 11 17:32:33.078561 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:32:33.078525    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-c7d192/simulator-w0-c7d192-kserve-6bfc765cf5-klnxw" event={"ID":"c30aa08c-b2d4-4edc-8561-e78ad422fd1f","Type":"ContainerStarted","Data":"4967c83f7832a12d0f739396188f205f644df6605ff208a482fbc3255a33323a"}
Sep 11 17:32:33.079047 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:32:33.078574    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-c7d192/simulator-w0-c7d192-kserve-6bfc765cf5-klnxw" event={"ID":"c30aa08c-b2d4-4edc-8561-e78ad422fd1f","Type":"ContainerStarted","Data":"1c9da7dd3c09e732d511a6da43370ce0d9a8827d33963f68c9f25d1b4629be42"}
Sep 11 17:32:33.094429 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:32:33.094377    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-c7d192/simulator-w0-c7d192-kserve-6bfc765cf5-klnxw" podStartSLOduration=2.094357187 podStartE2EDuration="2.094357187s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 17:32:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 17:32:33.09395187 +0000 UTC m=+830.667171977" watchObservedRunningTime="2026-09-11 17:32:33.094357187 +0000 UTC m=+830.667577299"
Sep 11 17:32:42.295720 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:32:42.295680    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-c7d192/simulator-w0-c7d192-kserve-6bfc765cf5-klnxw"
Sep 11 17:32:42.296133 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:32:42.295956    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-c7d192/simulator-w0-c7d192-kserve-6bfc765cf5-klnxw"
Sep 11 17:32:42.308397 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:32:42.308372    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-c7d192/simulator-w0-c7d192-kserve-6bfc765cf5-klnxw"
Sep 11 17:32:43.147005 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:32:43.146969    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-c7d192/simulator-w0-c7d192-kserve-6bfc765cf5-klnxw"
Sep 11 17:32:48.993157 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:32:48.993115    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-nkmkm"]
Sep 11 17:32:53.704141 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:32:53.704101    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0ad997be-k6fw6"]
Sep 11 17:32:53.710263 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:32:53.710200    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0ad997be-k6fw6"
Sep 11 17:32:53.713040 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:32:53.713006    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-9n8gf\""
Sep 11 17:32:53.713227 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:32:53.713206    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 11 17:32:53.714014 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:32:53.713984    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0ad997be-k6fw6"]
Sep 11 17:32:53.845322 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:32:53.845277    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e54d3854-c9ee-43a3-b847-9e805e038de2-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-0ad997be-k6fw6\" (UID: \"e54d3854-c9ee-43a3-b847-9e805e038de2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0ad997be-k6fw6"
Sep 11 17:32:53.947049 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:32:53.947009    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e54d3854-c9ee-43a3-b847-9e805e038de2-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-0ad997be-k6fw6\" (UID: \"e54d3854-c9ee-43a3-b847-9e805e038de2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0ad997be-k6fw6"
Sep 11 17:32:53.947668 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:32:53.947648    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e54d3854-c9ee-43a3-b847-9e805e038de2-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-0ad997be-k6fw6\" (UID: \"e54d3854-c9ee-43a3-b847-9e805e038de2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0ad997be-k6fw6"
Sep 11 17:32:54.023613 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:32:54.023577    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0ad997be-k6fw6"
Sep 11 17:32:54.294267 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:32:54.294240    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0ad997be-k6fw6"]
Sep 11 17:32:54.296367 ip-10-0-141-139 kubenswrapper[2577]: W0911 17:32:54.296333    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode54d3854_c9ee_43a3_b847_9e805e038de2.slice/crio-dca674f7e762e11979a6bb6c569413fdde46ab63fd1829b07cab5acb6c7581af WatchSource:0}: Error finding container dca674f7e762e11979a6bb6c569413fdde46ab63fd1829b07cab5acb6c7581af: Status 404 returned error can't find the container with id dca674f7e762e11979a6bb6c569413fdde46ab63fd1829b07cab5acb6c7581af
Sep 11 17:32:55.210466 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:32:55.210366    2577 generic.go:356] "Generic (PLEG): container finished" podID="e54d3854-c9ee-43a3-b847-9e805e038de2" containerID="e793dcd6dfb34202cbf43a6c08a1cda3a19d55336ba4e0e1259f0b600b653508" exitCode=0
Sep 11 17:32:55.210466 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:32:55.210430    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0ad997be-k6fw6" event={"ID":"e54d3854-c9ee-43a3-b847-9e805e038de2","Type":"ContainerDied","Data":"e793dcd6dfb34202cbf43a6c08a1cda3a19d55336ba4e0e1259f0b600b653508"}
Sep 11 17:32:55.210915 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:32:55.210494    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0ad997be-k6fw6" event={"ID":"e54d3854-c9ee-43a3-b847-9e805e038de2","Type":"ContainerStarted","Data":"dca674f7e762e11979a6bb6c569413fdde46ab63fd1829b07cab5acb6c7581af"}
Sep 11 17:32:56.351368 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:32:56.351346    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0ad997be-k6fw6"
Sep 11 17:32:56.475607 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:32:56.475496    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e54d3854-c9ee-43a3-b847-9e805e038de2-maas-api-service-ca\") pod \"e54d3854-c9ee-43a3-b847-9e805e038de2\" (UID: \"e54d3854-c9ee-43a3-b847-9e805e038de2\") "
Sep 11 17:32:56.475850 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:32:56.475826    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e54d3854-c9ee-43a3-b847-9e805e038de2-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "e54d3854-c9ee-43a3-b847-9e805e038de2" (UID: "e54d3854-c9ee-43a3-b847-9e805e038de2"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 17:32:56.576938 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:32:56.576906    2577 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e54d3854-c9ee-43a3-b847-9e805e038de2-maas-api-service-ca\") on node \"ip-10-0-141-139.ec2.internal\" DevicePath \"\""
Sep 11 17:32:57.222117 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:32:57.222088    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0ad997be-k6fw6"
Sep 11 17:32:57.222117 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:32:57.222110    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0ad997be-k6fw6" event={"ID":"e54d3854-c9ee-43a3-b847-9e805e038de2","Type":"ContainerDied","Data":"dca674f7e762e11979a6bb6c569413fdde46ab63fd1829b07cab5acb6c7581af"}
Sep 11 17:32:57.222321 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:32:57.222154    2577 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="dca674f7e762e11979a6bb6c569413fdde46ab63fd1829b07cab5acb6c7581af"
Sep 11 17:33:00.770155 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:33:00.770113    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0ad997be-k6fw6"]
Sep 11 17:33:00.773515 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:33:00.773480    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0ad997be-k6fw6"]
Sep 11 17:33:00.927090 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:33:00.927057    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e54d3854-c9ee-43a3-b847-9e805e038de2" path="/var/lib/kubelet/pods/e54d3854-c9ee-43a3-b847-9e805e038de2/volumes"
Sep 11 17:33:07.795487 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:33:07.795448    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-nkmkm"]
Sep 11 17:33:28.032727 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:33:28.032692    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-nkmkm"]
Sep 11 17:33:35.210168 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:33:35.210082    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-357aed65-6cs2d"]
Sep 11 17:33:35.210659 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:33:35.210573    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e54d3854-c9ee-43a3-b847-9e805e038de2" containerName="revoke-keys"
Sep 11 17:33:35.210659 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:33:35.210587    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="e54d3854-c9ee-43a3-b847-9e805e038de2" containerName="revoke-keys"
Sep 11 17:33:35.210760 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:33:35.210678    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="e54d3854-c9ee-43a3-b847-9e805e038de2" containerName="revoke-keys"
Sep 11 17:33:35.212753 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:33:35.212737    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-357aed65-6cs2d"
Sep 11 17:33:35.215122 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:33:35.215080    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 11 17:33:35.215241 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:33:35.215169    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-9n8gf\""
Sep 11 17:33:35.220350 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:33:35.220317    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-357aed65-6cs2d"]
Sep 11 17:33:35.277682 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:33:35.277641    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/106a5585-d585-4c71-98f9-af278547caf3-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-357aed65-6cs2d\" (UID: \"106a5585-d585-4c71-98f9-af278547caf3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-357aed65-6cs2d"
Sep 11 17:33:35.378433 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:33:35.378388    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/106a5585-d585-4c71-98f9-af278547caf3-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-357aed65-6cs2d\" (UID: \"106a5585-d585-4c71-98f9-af278547caf3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-357aed65-6cs2d"
Sep 11 17:33:35.379048 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:33:35.379023    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/106a5585-d585-4c71-98f9-af278547caf3-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-357aed65-6cs2d\" (UID: \"106a5585-d585-4c71-98f9-af278547caf3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-357aed65-6cs2d"
Sep 11 17:33:35.546113 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:33:35.546073    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-357aed65-6cs2d"
Sep 11 17:33:35.824902 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:33:35.824868    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-357aed65-6cs2d"]
Sep 11 17:33:35.826704 ip-10-0-141-139 kubenswrapper[2577]: W0911 17:33:35.826679    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod106a5585_d585_4c71_98f9_af278547caf3.slice/crio-ed4805a92a5cd7ab47e0456cd48c868af448024ff207d2cd1b7d657976ee1208 WatchSource:0}: Error finding container ed4805a92a5cd7ab47e0456cd48c868af448024ff207d2cd1b7d657976ee1208: Status 404 returned error can't find the container with id ed4805a92a5cd7ab47e0456cd48c868af448024ff207d2cd1b7d657976ee1208
Sep 11 17:33:36.444465 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:33:36.444356    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-357aed65-6cs2d" event={"ID":"106a5585-d585-4c71-98f9-af278547caf3","Type":"ContainerStarted","Data":"e4983553c2196f922eb1738b7936880709e79f2f72d3eca749f8644615287aca"}
Sep 11 17:33:36.444465 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:33:36.444456    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-357aed65-6cs2d" event={"ID":"106a5585-d585-4c71-98f9-af278547caf3","Type":"ContainerStarted","Data":"ed4805a92a5cd7ab47e0456cd48c868af448024ff207d2cd1b7d657976ee1208"}
Sep 11 17:33:36.463442 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:33:36.463387    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-357aed65-6cs2d" podStartSLOduration=1.46336577 podStartE2EDuration="1.46336577s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 17:33:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 17:33:36.462801988 +0000 UTC m=+894.036022113" watchObservedRunningTime="2026-09-11 17:33:36.46336577 +0000 UTC m=+894.036585885"
Sep 11 17:33:37.451793 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:33:37.451753    2577 generic.go:356] "Generic (PLEG): container finished" podID="106a5585-d585-4c71-98f9-af278547caf3" containerID="e4983553c2196f922eb1738b7936880709e79f2f72d3eca749f8644615287aca" exitCode=0
Sep 11 17:33:37.452192 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:33:37.451812    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-357aed65-6cs2d" event={"ID":"106a5585-d585-4c71-98f9-af278547caf3","Type":"ContainerDied","Data":"e4983553c2196f922eb1738b7936880709e79f2f72d3eca749f8644615287aca"}
Sep 11 17:33:38.589479 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:33:38.589451    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-357aed65-6cs2d"
Sep 11 17:33:38.612419 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:33:38.612391    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/106a5585-d585-4c71-98f9-af278547caf3-maas-api-service-ca\") pod \"106a5585-d585-4c71-98f9-af278547caf3\" (UID: \"106a5585-d585-4c71-98f9-af278547caf3\") "
Sep 11 17:33:38.612754 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:33:38.612731    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/106a5585-d585-4c71-98f9-af278547caf3-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "106a5585-d585-4c71-98f9-af278547caf3" (UID: "106a5585-d585-4c71-98f9-af278547caf3"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 17:33:38.714086 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:33:38.713996    2577 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/106a5585-d585-4c71-98f9-af278547caf3-maas-api-service-ca\") on node \"ip-10-0-141-139.ec2.internal\" DevicePath \"\""
Sep 11 17:33:39.463961 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:33:39.463931    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-357aed65-6cs2d"
Sep 11 17:33:39.464141 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:33:39.463978    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-357aed65-6cs2d" event={"ID":"106a5585-d585-4c71-98f9-af278547caf3","Type":"ContainerDied","Data":"ed4805a92a5cd7ab47e0456cd48c868af448024ff207d2cd1b7d657976ee1208"}
Sep 11 17:33:39.464141 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:33:39.464027    2577 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ed4805a92a5cd7ab47e0456cd48c868af448024ff207d2cd1b7d657976ee1208"
Sep 11 17:33:47.479278 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:33:47.479238    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-357aed65-6cs2d"]
Sep 11 17:33:47.481570 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:33:47.481547    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-357aed65-6cs2d"]
Sep 11 17:33:48.927556 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:33:48.927493    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="106a5585-d585-4c71-98f9-af278547caf3" path="/var/lib/kubelet/pods/106a5585-d585-4c71-98f9-af278547caf3/volumes"
Sep 11 17:33:50.099358 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:33:50.099316    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-nkmkm"]
Sep 11 17:34:05.010712 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:34:05.010682    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-nkmkm"]
Sep 11 17:34:19.312641 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:34:19.312606    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-49e398-t2bcf"]
Sep 11 17:34:19.313095 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:34:19.313078    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="106a5585-d585-4c71-98f9-af278547caf3" containerName="revoke-keys"
Sep 11 17:34:19.313095 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:34:19.313089    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="106a5585-d585-4c71-98f9-af278547caf3" containerName="revoke-keys"
Sep 11 17:34:19.313187 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:34:19.313177    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="106a5585-d585-4c71-98f9-af278547caf3" containerName="revoke-keys"
Sep 11 17:34:19.315532 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:34:19.315484    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-49e398-t2bcf"
Sep 11 17:34:19.317758 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:34:19.317733    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 11 17:34:19.317879 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:34:19.317801    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-9n8gf\""
Sep 11 17:34:19.322800 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:34:19.322777    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-49e398-t2bcf"]
Sep 11 17:34:19.433723 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:34:19.433683    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/52e51d5c-d496-41ee-bf44-f4c52a877a71-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-autores-49e398-t2bcf\" (UID: \"52e51d5c-d496-41ee-bf44-f4c52a877a71\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-49e398-t2bcf"
Sep 11 17:34:19.534569 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:34:19.534528    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/52e51d5c-d496-41ee-bf44-f4c52a877a71-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-autores-49e398-t2bcf\" (UID: \"52e51d5c-d496-41ee-bf44-f4c52a877a71\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-49e398-t2bcf"
Sep 11 17:34:19.535165 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:34:19.535144    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/52e51d5c-d496-41ee-bf44-f4c52a877a71-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-autores-49e398-t2bcf\" (UID: \"52e51d5c-d496-41ee-bf44-f4c52a877a71\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-49e398-t2bcf"
Sep 11 17:34:19.627663 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:34:19.627571    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-49e398-t2bcf"
Sep 11 17:34:19.903204 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:34:19.903174    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-49e398-t2bcf"]
Sep 11 17:34:19.904768 ip-10-0-141-139 kubenswrapper[2577]: W0911 17:34:19.904743    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod52e51d5c_d496_41ee_bf44_f4c52a877a71.slice/crio-210ecec726e71b7fb5eb05f93b3d0e31507c4891959ecffb44efa8528d44a773 WatchSource:0}: Error finding container 210ecec726e71b7fb5eb05f93b3d0e31507c4891959ecffb44efa8528d44a773: Status 404 returned error can't find the container with id 210ecec726e71b7fb5eb05f93b3d0e31507c4891959ecffb44efa8528d44a773
Sep 11 17:34:20.694142 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:34:20.694104    2577 generic.go:356] "Generic (PLEG): container finished" podID="52e51d5c-d496-41ee-bf44-f4c52a877a71" containerID="5c5a1481a0517a06b2eed7d2e6fcaca271c9463fcd19cdd044e92a37b0410012" exitCode=0
Sep 11 17:34:20.694546 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:34:20.694197    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-49e398-t2bcf" event={"ID":"52e51d5c-d496-41ee-bf44-f4c52a877a71","Type":"ContainerDied","Data":"5c5a1481a0517a06b2eed7d2e6fcaca271c9463fcd19cdd044e92a37b0410012"}
Sep 11 17:34:20.694546 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:34:20.694286    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-49e398-t2bcf" event={"ID":"52e51d5c-d496-41ee-bf44-f4c52a877a71","Type":"ContainerStarted","Data":"210ecec726e71b7fb5eb05f93b3d0e31507c4891959ecffb44efa8528d44a773"}
Sep 11 17:34:21.838838 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:34:21.838814    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-49e398-t2bcf"
Sep 11 17:34:21.961777 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:34:21.961687    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/52e51d5c-d496-41ee-bf44-f4c52a877a71-maas-api-service-ca\") pod \"52e51d5c-d496-41ee-bf44-f4c52a877a71\" (UID: \"52e51d5c-d496-41ee-bf44-f4c52a877a71\") "
Sep 11 17:34:21.962116 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:34:21.962092    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/52e51d5c-d496-41ee-bf44-f4c52a877a71-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "52e51d5c-d496-41ee-bf44-f4c52a877a71" (UID: "52e51d5c-d496-41ee-bf44-f4c52a877a71"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 17:34:22.062925 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:34:22.062896    2577 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/52e51d5c-d496-41ee-bf44-f4c52a877a71-maas-api-service-ca\") on node \"ip-10-0-141-139.ec2.internal\" DevicePath \"\""
Sep 11 17:34:22.705778 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:34:22.705744    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-49e398-t2bcf"
Sep 11 17:34:22.705947 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:34:22.705740    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-49e398-t2bcf" event={"ID":"52e51d5c-d496-41ee-bf44-f4c52a877a71","Type":"ContainerDied","Data":"210ecec726e71b7fb5eb05f93b3d0e31507c4891959ecffb44efa8528d44a773"}
Sep 11 17:34:22.705947 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:34:22.705871    2577 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="210ecec726e71b7fb5eb05f93b3d0e31507c4891959ecffb44efa8528d44a773"
Sep 11 17:34:30.015266 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:34:30.015223    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-49e398-8fccbc594-v2cgl"]
Sep 11 17:34:30.015782 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:34:30.015490    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-autores-49e398-8fccbc594-v2cgl" podUID="65b33dd3-cc20-465a-9da7-9c8c31dffa9d" containerName="payload-pre-processing" containerID="cri-o://db784bbc86c7b0bc76f569a3bb9d20bd42b5d6fc05c28dfa7974c63c426187e0" gracePeriod=30
Sep 11 17:34:30.207865 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:34:30.207813    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-49e398-t2bcf"]
Sep 11 17:34:30.210087 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:34:30.210057    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-49e398-t2bcf"]
Sep 11 17:34:30.273456 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:34:30.273385    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-autores-49e398-8fccbc594-v2cgl"
Sep 11 17:34:30.456408 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:34:30.456375    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-p58xm\" (UniqueName: \"kubernetes.io/projected/65b33dd3-cc20-465a-9da7-9c8c31dffa9d-kube-api-access-p58xm\") pod \"65b33dd3-cc20-465a-9da7-9c8c31dffa9d\" (UID: \"65b33dd3-cc20-465a-9da7-9c8c31dffa9d\") "
Sep 11 17:34:30.456630 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:34:30.456456    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/65b33dd3-cc20-465a-9da7-9c8c31dffa9d-plugins-config-volume\") pod \"65b33dd3-cc20-465a-9da7-9c8c31dffa9d\" (UID: \"65b33dd3-cc20-465a-9da7-9c8c31dffa9d\") "
Sep 11 17:34:30.456823 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:34:30.456799    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/65b33dd3-cc20-465a-9da7-9c8c31dffa9d-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "65b33dd3-cc20-465a-9da7-9c8c31dffa9d" (UID: "65b33dd3-cc20-465a-9da7-9c8c31dffa9d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 17:34:30.458987 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:34:30.458962    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/65b33dd3-cc20-465a-9da7-9c8c31dffa9d-kube-api-access-p58xm" (OuterVolumeSpecName: "kube-api-access-p58xm") pod "65b33dd3-cc20-465a-9da7-9c8c31dffa9d" (UID: "65b33dd3-cc20-465a-9da7-9c8c31dffa9d"). InnerVolumeSpecName "kube-api-access-p58xm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 17:34:30.557684 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:34:30.557592    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p58xm\" (UniqueName: \"kubernetes.io/projected/65b33dd3-cc20-465a-9da7-9c8c31dffa9d-kube-api-access-p58xm\") on node \"ip-10-0-141-139.ec2.internal\" DevicePath \"\""
Sep 11 17:34:30.557684 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:34:30.557623    2577 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/65b33dd3-cc20-465a-9da7-9c8c31dffa9d-plugins-config-volume\") on node \"ip-10-0-141-139.ec2.internal\" DevicePath \"\""
Sep 11 17:34:30.753100 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:34:30.753063    2577 generic.go:356] "Generic (PLEG): container finished" podID="65b33dd3-cc20-465a-9da7-9c8c31dffa9d" containerID="db784bbc86c7b0bc76f569a3bb9d20bd42b5d6fc05c28dfa7974c63c426187e0" exitCode=0
Sep 11 17:34:30.753308 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:34:30.753115    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-autores-49e398-8fccbc594-v2cgl"
Sep 11 17:34:30.753308 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:34:30.753151    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-autores-49e398-8fccbc594-v2cgl" event={"ID":"65b33dd3-cc20-465a-9da7-9c8c31dffa9d","Type":"ContainerDied","Data":"db784bbc86c7b0bc76f569a3bb9d20bd42b5d6fc05c28dfa7974c63c426187e0"}
Sep 11 17:34:30.753308 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:34:30.753238    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-autores-49e398-8fccbc594-v2cgl" event={"ID":"65b33dd3-cc20-465a-9da7-9c8c31dffa9d","Type":"ContainerDied","Data":"f6044ae6c69ec8809c2d03f2ed13b6340a68792200c0cace5a214f7591603587"}
Sep 11 17:34:30.753308 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:34:30.753264    2577 scope.go:117] "RemoveContainer" containerID="db784bbc86c7b0bc76f569a3bb9d20bd42b5d6fc05c28dfa7974c63c426187e0"
Sep 11 17:34:30.763029 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:34:30.763011    2577 scope.go:117] "RemoveContainer" containerID="db784bbc86c7b0bc76f569a3bb9d20bd42b5d6fc05c28dfa7974c63c426187e0"
Sep 11 17:34:30.763364 ip-10-0-141-139 kubenswrapper[2577]: E0911 17:34:30.763345    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"db784bbc86c7b0bc76f569a3bb9d20bd42b5d6fc05c28dfa7974c63c426187e0\": container with ID starting with db784bbc86c7b0bc76f569a3bb9d20bd42b5d6fc05c28dfa7974c63c426187e0 not found: ID does not exist" containerID="db784bbc86c7b0bc76f569a3bb9d20bd42b5d6fc05c28dfa7974c63c426187e0"
Sep 11 17:34:30.763414 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:34:30.763374    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"db784bbc86c7b0bc76f569a3bb9d20bd42b5d6fc05c28dfa7974c63c426187e0"} err="failed to get container status \"db784bbc86c7b0bc76f569a3bb9d20bd42b5d6fc05c28dfa7974c63c426187e0\": rpc error: code = NotFound desc = could not find container \"db784bbc86c7b0bc76f569a3bb9d20bd42b5d6fc05c28dfa7974c63c426187e0\": container with ID starting with db784bbc86c7b0bc76f569a3bb9d20bd42b5d6fc05c28dfa7974c63c426187e0 not found: ID does not exist"
Sep 11 17:34:30.780114 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:34:30.780059    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-49e398-8fccbc594-v2cgl"]
Sep 11 17:34:30.782691 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:34:30.782666    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-49e398-8fccbc594-v2cgl"]
Sep 11 17:34:30.927136 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:34:30.927060    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="52e51d5c-d496-41ee-bf44-f4c52a877a71" path="/var/lib/kubelet/pods/52e51d5c-d496-41ee-bf44-f4c52a877a71/volumes"
Sep 11 17:34:30.927432 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:34:30.927418    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="65b33dd3-cc20-465a-9da7-9c8c31dffa9d" path="/var/lib/kubelet/pods/65b33dd3-cc20-465a-9da7-9c8c31dffa9d/volumes"
Sep 11 17:34:34.994023 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:34:34.993987    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-nkmkm"]
Sep 11 17:35:08.512471 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:35:08.512434    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-nkmkm"]
Sep 11 17:35:27.684663 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:35:27.684628    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-c7d192/premium-w0-c7d192-kserve-6f977d4fcd-n9nhd"]
Sep 11 17:35:27.685271 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:35:27.685250    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="65b33dd3-cc20-465a-9da7-9c8c31dffa9d" containerName="payload-pre-processing"
Sep 11 17:35:27.685363 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:35:27.685274    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="65b33dd3-cc20-465a-9da7-9c8c31dffa9d" containerName="payload-pre-processing"
Sep 11 17:35:27.685363 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:35:27.685322    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="52e51d5c-d496-41ee-bf44-f4c52a877a71" containerName="revoke-keys"
Sep 11 17:35:27.685363 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:35:27.685331    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="52e51d5c-d496-41ee-bf44-f4c52a877a71" containerName="revoke-keys"
Sep 11 17:35:27.685570 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:35:27.685435    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="65b33dd3-cc20-465a-9da7-9c8c31dffa9d" containerName="payload-pre-processing"
Sep 11 17:35:27.685570 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:35:27.685449    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="52e51d5c-d496-41ee-bf44-f4c52a877a71" containerName="revoke-keys"
Sep 11 17:35:27.688838 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:35:27.688816    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-c7d192/premium-w0-c7d192-kserve-6f977d4fcd-n9nhd"
Sep 11 17:35:27.691228 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:35:27.691209    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-c7d192\"/\"premium-w0-c7d192-kserve-self-signed-certs\""
Sep 11 17:35:27.698691 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:35:27.698660    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-c7d192/premium-w0-c7d192-kserve-6f977d4fcd-n9nhd"]
Sep 11 17:35:27.859353 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:35:27.859307    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/55806c17-765a-43a1-babc-46540c0423bd-home\") pod \"premium-w0-c7d192-kserve-6f977d4fcd-n9nhd\" (UID: \"55806c17-765a-43a1-babc-46540c0423bd\") " pod="e2e-models-e2e-worker-w0-c7d192/premium-w0-c7d192-kserve-6f977d4fcd-n9nhd"
Sep 11 17:35:27.859570 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:35:27.859370    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/55806c17-765a-43a1-babc-46540c0423bd-model-cache\") pod \"premium-w0-c7d192-kserve-6f977d4fcd-n9nhd\" (UID: \"55806c17-765a-43a1-babc-46540c0423bd\") " pod="e2e-models-e2e-worker-w0-c7d192/premium-w0-c7d192-kserve-6f977d4fcd-n9nhd"
Sep 11 17:35:27.859570 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:35:27.859429    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/55806c17-765a-43a1-babc-46540c0423bd-tmp-dir\") pod \"premium-w0-c7d192-kserve-6f977d4fcd-n9nhd\" (UID: \"55806c17-765a-43a1-babc-46540c0423bd\") " pod="e2e-models-e2e-worker-w0-c7d192/premium-w0-c7d192-kserve-6f977d4fcd-n9nhd"
Sep 11 17:35:27.859570 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:35:27.859488    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fjnp9\" (UniqueName: \"kubernetes.io/projected/55806c17-765a-43a1-babc-46540c0423bd-kube-api-access-fjnp9\") pod \"premium-w0-c7d192-kserve-6f977d4fcd-n9nhd\" (UID: \"55806c17-765a-43a1-babc-46540c0423bd\") " pod="e2e-models-e2e-worker-w0-c7d192/premium-w0-c7d192-kserve-6f977d4fcd-n9nhd"
Sep 11 17:35:27.859706 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:35:27.859590    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/55806c17-765a-43a1-babc-46540c0423bd-tls-certs\") pod \"premium-w0-c7d192-kserve-6f977d4fcd-n9nhd\" (UID: \"55806c17-765a-43a1-babc-46540c0423bd\") " pod="e2e-models-e2e-worker-w0-c7d192/premium-w0-c7d192-kserve-6f977d4fcd-n9nhd"
Sep 11 17:35:27.859706 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:35:27.859666    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/55806c17-765a-43a1-babc-46540c0423bd-dshm\") pod \"premium-w0-c7d192-kserve-6f977d4fcd-n9nhd\" (UID: \"55806c17-765a-43a1-babc-46540c0423bd\") " pod="e2e-models-e2e-worker-w0-c7d192/premium-w0-c7d192-kserve-6f977d4fcd-n9nhd"
Sep 11 17:35:27.961228 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:35:27.961135    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fjnp9\" (UniqueName: \"kubernetes.io/projected/55806c17-765a-43a1-babc-46540c0423bd-kube-api-access-fjnp9\") pod \"premium-w0-c7d192-kserve-6f977d4fcd-n9nhd\" (UID: \"55806c17-765a-43a1-babc-46540c0423bd\") " pod="e2e-models-e2e-worker-w0-c7d192/premium-w0-c7d192-kserve-6f977d4fcd-n9nhd"
Sep 11 17:35:27.961228 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:35:27.961193    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/55806c17-765a-43a1-babc-46540c0423bd-tls-certs\") pod \"premium-w0-c7d192-kserve-6f977d4fcd-n9nhd\" (UID: \"55806c17-765a-43a1-babc-46540c0423bd\") " pod="e2e-models-e2e-worker-w0-c7d192/premium-w0-c7d192-kserve-6f977d4fcd-n9nhd"
Sep 11 17:35:27.961228 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:35:27.961228    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/55806c17-765a-43a1-babc-46540c0423bd-dshm\") pod \"premium-w0-c7d192-kserve-6f977d4fcd-n9nhd\" (UID: \"55806c17-765a-43a1-babc-46540c0423bd\") " pod="e2e-models-e2e-worker-w0-c7d192/premium-w0-c7d192-kserve-6f977d4fcd-n9nhd"
Sep 11 17:35:27.961547 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:35:27.961248    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/55806c17-765a-43a1-babc-46540c0423bd-home\") pod \"premium-w0-c7d192-kserve-6f977d4fcd-n9nhd\" (UID: \"55806c17-765a-43a1-babc-46540c0423bd\") " pod="e2e-models-e2e-worker-w0-c7d192/premium-w0-c7d192-kserve-6f977d4fcd-n9nhd"
Sep 11 17:35:27.961547 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:35:27.961387    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/55806c17-765a-43a1-babc-46540c0423bd-model-cache\") pod \"premium-w0-c7d192-kserve-6f977d4fcd-n9nhd\" (UID: \"55806c17-765a-43a1-babc-46540c0423bd\") " pod="e2e-models-e2e-worker-w0-c7d192/premium-w0-c7d192-kserve-6f977d4fcd-n9nhd"
Sep 11 17:35:27.961547 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:35:27.961443    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/55806c17-765a-43a1-babc-46540c0423bd-tmp-dir\") pod \"premium-w0-c7d192-kserve-6f977d4fcd-n9nhd\" (UID: \"55806c17-765a-43a1-babc-46540c0423bd\") " pod="e2e-models-e2e-worker-w0-c7d192/premium-w0-c7d192-kserve-6f977d4fcd-n9nhd"
Sep 11 17:35:27.961825 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:35:27.961803    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/55806c17-765a-43a1-babc-46540c0423bd-model-cache\") pod \"premium-w0-c7d192-kserve-6f977d4fcd-n9nhd\" (UID: \"55806c17-765a-43a1-babc-46540c0423bd\") " pod="e2e-models-e2e-worker-w0-c7d192/premium-w0-c7d192-kserve-6f977d4fcd-n9nhd"
Sep 11 17:35:27.961893 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:35:27.961857    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/55806c17-765a-43a1-babc-46540c0423bd-tmp-dir\") pod \"premium-w0-c7d192-kserve-6f977d4fcd-n9nhd\" (UID: \"55806c17-765a-43a1-babc-46540c0423bd\") " pod="e2e-models-e2e-worker-w0-c7d192/premium-w0-c7d192-kserve-6f977d4fcd-n9nhd"
Sep 11 17:35:27.961933 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:35:27.961860    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/55806c17-765a-43a1-babc-46540c0423bd-home\") pod \"premium-w0-c7d192-kserve-6f977d4fcd-n9nhd\" (UID: \"55806c17-765a-43a1-babc-46540c0423bd\") " pod="e2e-models-e2e-worker-w0-c7d192/premium-w0-c7d192-kserve-6f977d4fcd-n9nhd"
Sep 11 17:35:27.963704 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:35:27.963687    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/55806c17-765a-43a1-babc-46540c0423bd-dshm\") pod \"premium-w0-c7d192-kserve-6f977d4fcd-n9nhd\" (UID: \"55806c17-765a-43a1-babc-46540c0423bd\") " pod="e2e-models-e2e-worker-w0-c7d192/premium-w0-c7d192-kserve-6f977d4fcd-n9nhd"
Sep 11 17:35:27.963870 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:35:27.963853    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/55806c17-765a-43a1-babc-46540c0423bd-tls-certs\") pod \"premium-w0-c7d192-kserve-6f977d4fcd-n9nhd\" (UID: \"55806c17-765a-43a1-babc-46540c0423bd\") " pod="e2e-models-e2e-worker-w0-c7d192/premium-w0-c7d192-kserve-6f977d4fcd-n9nhd"
Sep 11 17:35:27.969446 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:35:27.969421    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fjnp9\" (UniqueName: \"kubernetes.io/projected/55806c17-765a-43a1-babc-46540c0423bd-kube-api-access-fjnp9\") pod \"premium-w0-c7d192-kserve-6f977d4fcd-n9nhd\" (UID: \"55806c17-765a-43a1-babc-46540c0423bd\") " pod="e2e-models-e2e-worker-w0-c7d192/premium-w0-c7d192-kserve-6f977d4fcd-n9nhd"
Sep 11 17:35:28.001544 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:35:28.001481    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-c7d192/premium-w0-c7d192-kserve-6f977d4fcd-n9nhd"
Sep 11 17:35:28.275278 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:35:28.275238    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-c7d192/premium-w0-c7d192-kserve-6f977d4fcd-n9nhd"]
Sep 11 17:35:28.278422 ip-10-0-141-139 kubenswrapper[2577]: W0911 17:35:28.278395    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod55806c17_765a_43a1_babc_46540c0423bd.slice/crio-f3eef0bef2b51f86e102732c1c4a39b09e989edbd4ceb160aecf57ba795512b5 WatchSource:0}: Error finding container f3eef0bef2b51f86e102732c1c4a39b09e989edbd4ceb160aecf57ba795512b5: Status 404 returned error can't find the container with id f3eef0bef2b51f86e102732c1c4a39b09e989edbd4ceb160aecf57ba795512b5
Sep 11 17:35:29.082582 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:35:29.082535    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-c7d192/premium-w0-c7d192-kserve-6f977d4fcd-n9nhd" event={"ID":"55806c17-765a-43a1-babc-46540c0423bd","Type":"ContainerStarted","Data":"d07916d97da58fa3f509688c4b9efe0ccdf7310d10eb8e8f733ef1bc63967c60"}
Sep 11 17:35:29.082582 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:35:29.082586    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-c7d192/premium-w0-c7d192-kserve-6f977d4fcd-n9nhd" event={"ID":"55806c17-765a-43a1-babc-46540c0423bd","Type":"ContainerStarted","Data":"f3eef0bef2b51f86e102732c1c4a39b09e989edbd4ceb160aecf57ba795512b5"}
Sep 11 17:35:29.099606 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:35:29.099552    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-c7d192/premium-w0-c7d192-kserve-6f977d4fcd-n9nhd" podStartSLOduration=2.099538086 podStartE2EDuration="2.099538086s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 17:35:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 17:35:29.099074254 +0000 UTC m=+1006.672294366" watchObservedRunningTime="2026-09-11 17:35:29.099538086 +0000 UTC m=+1006.672758195"
Sep 11 17:35:29.800591 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:35:29.800558    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-nkmkm"]
Sep 11 17:35:34.883147 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:35:34.883109    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-65fdd4fd-7f8dc674cd-4zklz"]
Sep 11 17:35:34.887766 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:35:34.887744    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-65fdd4fd-7f8dc674cd-4zklz"
Sep 11 17:35:34.890999 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:35:34.890975    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-65fdd4fd\""
Sep 11 17:35:34.891116 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:35:34.890976    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-65fdd4fd-dockercfg-kmlrm\""
Sep 11 17:35:34.895545 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:35:34.895495    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-65fdd4fd-7f8dc674cd-4zklz"]
Sep 11 17:35:34.932819 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:35:34.932778    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dvrjk\" (UniqueName: \"kubernetes.io/projected/96bfb668-ad05-4e2a-8c7d-8ad3b62b3d2a-kube-api-access-dvrjk\") pod \"payload-pre-processing-e2e-mt-65fdd4fd-7f8dc674cd-4zklz\" (UID: \"96bfb668-ad05-4e2a-8c7d-8ad3b62b3d2a\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-65fdd4fd-7f8dc674cd-4zklz"
Sep 11 17:35:34.933016 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:35:34.932952    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/96bfb668-ad05-4e2a-8c7d-8ad3b62b3d2a-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-65fdd4fd-7f8dc674cd-4zklz\" (UID: \"96bfb668-ad05-4e2a-8c7d-8ad3b62b3d2a\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-65fdd4fd-7f8dc674cd-4zklz"
Sep 11 17:35:35.033542 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:35:35.033470    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dvrjk\" (UniqueName: \"kubernetes.io/projected/96bfb668-ad05-4e2a-8c7d-8ad3b62b3d2a-kube-api-access-dvrjk\") pod \"payload-pre-processing-e2e-mt-65fdd4fd-7f8dc674cd-4zklz\" (UID: \"96bfb668-ad05-4e2a-8c7d-8ad3b62b3d2a\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-65fdd4fd-7f8dc674cd-4zklz"
Sep 11 17:35:35.033730 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:35:35.033603    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/96bfb668-ad05-4e2a-8c7d-8ad3b62b3d2a-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-65fdd4fd-7f8dc674cd-4zklz\" (UID: \"96bfb668-ad05-4e2a-8c7d-8ad3b62b3d2a\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-65fdd4fd-7f8dc674cd-4zklz"
Sep 11 17:35:35.034275 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:35:35.034256    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/96bfb668-ad05-4e2a-8c7d-8ad3b62b3d2a-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-65fdd4fd-7f8dc674cd-4zklz\" (UID: \"96bfb668-ad05-4e2a-8c7d-8ad3b62b3d2a\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-65fdd4fd-7f8dc674cd-4zklz"
Sep 11 17:35:35.042335 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:35:35.042307    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dvrjk\" (UniqueName: \"kubernetes.io/projected/96bfb668-ad05-4e2a-8c7d-8ad3b62b3d2a-kube-api-access-dvrjk\") pod \"payload-pre-processing-e2e-mt-65fdd4fd-7f8dc674cd-4zklz\" (UID: \"96bfb668-ad05-4e2a-8c7d-8ad3b62b3d2a\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-65fdd4fd-7f8dc674cd-4zklz"
Sep 11 17:35:35.201205 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:35:35.201112    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-65fdd4fd-7f8dc674cd-4zklz"
Sep 11 17:35:35.478711 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:35:35.478683    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-65fdd4fd-7f8dc674cd-4zklz"]
Sep 11 17:35:35.480718 ip-10-0-141-139 kubenswrapper[2577]: W0911 17:35:35.480690    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod96bfb668_ad05_4e2a_8c7d_8ad3b62b3d2a.slice/crio-36da4c2d7694e63e701741cd6fbc0ab4daf8ab29b241438a10bd65d019ae7bcc WatchSource:0}: Error finding container 36da4c2d7694e63e701741cd6fbc0ab4daf8ab29b241438a10bd65d019ae7bcc: Status 404 returned error can't find the container with id 36da4c2d7694e63e701741cd6fbc0ab4daf8ab29b241438a10bd65d019ae7bcc
Sep 11 17:35:36.129403 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:35:36.129363    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-65fdd4fd-7f8dc674cd-4zklz" event={"ID":"96bfb668-ad05-4e2a-8c7d-8ad3b62b3d2a","Type":"ContainerStarted","Data":"08b2a840a862b7057e3bae9ca63dd7f30df1bce04bc275e4df8c8466dbc2ea68"}
Sep 11 17:35:36.129801 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:35:36.129458    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-65fdd4fd-7f8dc674cd-4zklz" event={"ID":"96bfb668-ad05-4e2a-8c7d-8ad3b62b3d2a","Type":"ContainerStarted","Data":"36da4c2d7694e63e701741cd6fbc0ab4daf8ab29b241438a10bd65d019ae7bcc"}
Sep 11 17:35:36.129801 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:35:36.129521    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-65fdd4fd-7f8dc674cd-4zklz"
Sep 11 17:35:36.153187 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:35:36.153123    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-mt-65fdd4fd-7f8dc674cd-4zklz" podStartSLOduration=2.15310399 podStartE2EDuration="2.15310399s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 17:35:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 17:35:36.149727206 +0000 UTC m=+1013.722947318" watchObservedRunningTime="2026-09-11 17:35:36.15310399 +0000 UTC m=+1013.726324102"
Sep 11 17:35:38.002020 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:35:38.001976    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-c7d192/premium-w0-c7d192-kserve-6f977d4fcd-n9nhd"
Sep 11 17:35:38.002404 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:35:38.002043    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-c7d192/premium-w0-c7d192-kserve-6f977d4fcd-n9nhd"
Sep 11 17:35:38.014765 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:35:38.014737    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-c7d192/premium-w0-c7d192-kserve-6f977d4fcd-n9nhd"
Sep 11 17:35:38.151791 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:35:38.151759    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-c7d192/premium-w0-c7d192-kserve-6f977d4fcd-n9nhd"
Sep 11 17:35:47.136661 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:35:47.136624    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-65fdd4fd-7f8dc674cd-4zklz"
Sep 11 17:36:03.995387 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:36:03.995347    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-nkmkm"]
Sep 11 17:36:07.197126 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:36:07.197082    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-65fdd4fd-cfww6"]
Sep 11 17:36:07.201339 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:36:07.201314    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-65fdd4fd-cfww6"
Sep 11 17:36:07.204076 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:36:07.204043    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 11 17:36:07.204223 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:36:07.204082    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-9n8gf\""
Sep 11 17:36:07.208600 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:36:07.208575    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-65fdd4fd-cfww6"]
Sep 11 17:36:07.351553 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:36:07.351523    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8f4775eb-df86-45cc-8e0c-ec25394169ae-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-65fdd4fd-cfww6\" (UID: \"8f4775eb-df86-45cc-8e0c-ec25394169ae\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-65fdd4fd-cfww6"
Sep 11 17:36:07.452543 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:36:07.452422    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8f4775eb-df86-45cc-8e0c-ec25394169ae-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-65fdd4fd-cfww6\" (UID: \"8f4775eb-df86-45cc-8e0c-ec25394169ae\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-65fdd4fd-cfww6"
Sep 11 17:36:07.453036 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:36:07.453015    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8f4775eb-df86-45cc-8e0c-ec25394169ae-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-65fdd4fd-cfww6\" (UID: \"8f4775eb-df86-45cc-8e0c-ec25394169ae\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-65fdd4fd-cfww6"
Sep 11 17:36:07.513419 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:36:07.513377    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-65fdd4fd-cfww6"
Sep 11 17:36:07.788206 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:36:07.788179    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-65fdd4fd-cfww6"]
Sep 11 17:36:07.790163 ip-10-0-141-139 kubenswrapper[2577]: W0911 17:36:07.790139    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8f4775eb_df86_45cc_8e0c_ec25394169ae.slice/crio-af0d2e3a92dcf8f29cecf6358cace4b0376b75fe37c9b779cbd717550db83101 WatchSource:0}: Error finding container af0d2e3a92dcf8f29cecf6358cace4b0376b75fe37c9b779cbd717550db83101: Status 404 returned error can't find the container with id af0d2e3a92dcf8f29cecf6358cace4b0376b75fe37c9b779cbd717550db83101
Sep 11 17:36:08.323170 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:36:08.323078    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-65fdd4fd-cfww6" event={"ID":"8f4775eb-df86-45cc-8e0c-ec25394169ae","Type":"ContainerStarted","Data":"12a9428bcdc5ff5ac0183f420e89d3d64ae44016d34bb488a5244b8823794657"}
Sep 11 17:36:08.323170 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:36:08.323128    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-65fdd4fd-cfww6" event={"ID":"8f4775eb-df86-45cc-8e0c-ec25394169ae","Type":"ContainerStarted","Data":"af0d2e3a92dcf8f29cecf6358cace4b0376b75fe37c9b779cbd717550db83101"}
Sep 11 17:36:09.330395 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:36:09.330359    2577 generic.go:356] "Generic (PLEG): container finished" podID="8f4775eb-df86-45cc-8e0c-ec25394169ae" containerID="12a9428bcdc5ff5ac0183f420e89d3d64ae44016d34bb488a5244b8823794657" exitCode=0
Sep 11 17:36:09.330813 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:36:09.330459    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-65fdd4fd-cfww6" event={"ID":"8f4775eb-df86-45cc-8e0c-ec25394169ae","Type":"ContainerDied","Data":"12a9428bcdc5ff5ac0183f420e89d3d64ae44016d34bb488a5244b8823794657"}
Sep 11 17:36:10.476672 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:36:10.476644    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-65fdd4fd-cfww6"
Sep 11 17:36:10.586156 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:36:10.586128    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8f4775eb-df86-45cc-8e0c-ec25394169ae-maas-api-service-ca\") pod \"8f4775eb-df86-45cc-8e0c-ec25394169ae\" (UID: \"8f4775eb-df86-45cc-8e0c-ec25394169ae\") "
Sep 11 17:36:10.586481 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:36:10.586457    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8f4775eb-df86-45cc-8e0c-ec25394169ae-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "8f4775eb-df86-45cc-8e0c-ec25394169ae" (UID: "8f4775eb-df86-45cc-8e0c-ec25394169ae"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 17:36:10.687732 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:36:10.687649    2577 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8f4775eb-df86-45cc-8e0c-ec25394169ae-maas-api-service-ca\") on node \"ip-10-0-141-139.ec2.internal\" DevicePath \"\""
Sep 11 17:36:11.347411 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:36:11.347384    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-65fdd4fd-cfww6"
Sep 11 17:36:11.347411 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:36:11.347392    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-65fdd4fd-cfww6" event={"ID":"8f4775eb-df86-45cc-8e0c-ec25394169ae","Type":"ContainerDied","Data":"af0d2e3a92dcf8f29cecf6358cace4b0376b75fe37c9b779cbd717550db83101"}
Sep 11 17:36:11.347643 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:36:11.347438    2577 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="af0d2e3a92dcf8f29cecf6358cace4b0376b75fe37c9b779cbd717550db83101"
Sep 11 17:36:17.964566 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:36:17.964529    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-65fdd4fd-7f8dc674cd-4zklz"]
Sep 11 17:36:17.964956 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:36:17.964735    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-mt-65fdd4fd-7f8dc674cd-4zklz" podUID="96bfb668-ad05-4e2a-8c7d-8ad3b62b3d2a" containerName="payload-pre-processing" containerID="cri-o://08b2a840a862b7057e3bae9ca63dd7f30df1bce04bc275e4df8c8466dbc2ea68" gracePeriod=30
Sep 11 17:36:18.222466 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:36:18.222441    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-65fdd4fd-7f8dc674cd-4zklz"
Sep 11 17:36:18.224252 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:36:18.224229    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-65fdd4fd-cfww6"]
Sep 11 17:36:18.231249 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:36:18.231227    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-65fdd4fd-cfww6"]
Sep 11 17:36:18.268920 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:36:18.268893    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dvrjk\" (UniqueName: \"kubernetes.io/projected/96bfb668-ad05-4e2a-8c7d-8ad3b62b3d2a-kube-api-access-dvrjk\") pod \"96bfb668-ad05-4e2a-8c7d-8ad3b62b3d2a\" (UID: \"96bfb668-ad05-4e2a-8c7d-8ad3b62b3d2a\") "
Sep 11 17:36:18.269103 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:36:18.268950    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/96bfb668-ad05-4e2a-8c7d-8ad3b62b3d2a-plugins-config-volume\") pod \"96bfb668-ad05-4e2a-8c7d-8ad3b62b3d2a\" (UID: \"96bfb668-ad05-4e2a-8c7d-8ad3b62b3d2a\") "
Sep 11 17:36:18.269449 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:36:18.269415    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/96bfb668-ad05-4e2a-8c7d-8ad3b62b3d2a-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "96bfb668-ad05-4e2a-8c7d-8ad3b62b3d2a" (UID: "96bfb668-ad05-4e2a-8c7d-8ad3b62b3d2a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 17:36:18.271272 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:36:18.271244    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/96bfb668-ad05-4e2a-8c7d-8ad3b62b3d2a-kube-api-access-dvrjk" (OuterVolumeSpecName: "kube-api-access-dvrjk") pod "96bfb668-ad05-4e2a-8c7d-8ad3b62b3d2a" (UID: "96bfb668-ad05-4e2a-8c7d-8ad3b62b3d2a"). InnerVolumeSpecName "kube-api-access-dvrjk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 17:36:18.369871 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:36:18.369838    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dvrjk\" (UniqueName: \"kubernetes.io/projected/96bfb668-ad05-4e2a-8c7d-8ad3b62b3d2a-kube-api-access-dvrjk\") on node \"ip-10-0-141-139.ec2.internal\" DevicePath \"\""
Sep 11 17:36:18.369871 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:36:18.369866    2577 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/96bfb668-ad05-4e2a-8c7d-8ad3b62b3d2a-plugins-config-volume\") on node \"ip-10-0-141-139.ec2.internal\" DevicePath \"\""
Sep 11 17:36:18.389900 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:36:18.389866    2577 generic.go:356] "Generic (PLEG): container finished" podID="96bfb668-ad05-4e2a-8c7d-8ad3b62b3d2a" containerID="08b2a840a862b7057e3bae9ca63dd7f30df1bce04bc275e4df8c8466dbc2ea68" exitCode=0
Sep 11 17:36:18.390071 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:36:18.389910    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-65fdd4fd-7f8dc674cd-4zklz"
Sep 11 17:36:18.390071 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:36:18.389956    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-65fdd4fd-7f8dc674cd-4zklz" event={"ID":"96bfb668-ad05-4e2a-8c7d-8ad3b62b3d2a","Type":"ContainerDied","Data":"08b2a840a862b7057e3bae9ca63dd7f30df1bce04bc275e4df8c8466dbc2ea68"}
Sep 11 17:36:18.390071 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:36:18.390004    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-65fdd4fd-7f8dc674cd-4zklz" event={"ID":"96bfb668-ad05-4e2a-8c7d-8ad3b62b3d2a","Type":"ContainerDied","Data":"36da4c2d7694e63e701741cd6fbc0ab4daf8ab29b241438a10bd65d019ae7bcc"}
Sep 11 17:36:18.390071 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:36:18.390029    2577 scope.go:117] "RemoveContainer" containerID="08b2a840a862b7057e3bae9ca63dd7f30df1bce04bc275e4df8c8466dbc2ea68"
Sep 11 17:36:18.400068 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:36:18.400053    2577 scope.go:117] "RemoveContainer" containerID="08b2a840a862b7057e3bae9ca63dd7f30df1bce04bc275e4df8c8466dbc2ea68"
Sep 11 17:36:18.400365 ip-10-0-141-139 kubenswrapper[2577]: E0911 17:36:18.400348    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"08b2a840a862b7057e3bae9ca63dd7f30df1bce04bc275e4df8c8466dbc2ea68\": container with ID starting with 08b2a840a862b7057e3bae9ca63dd7f30df1bce04bc275e4df8c8466dbc2ea68 not found: ID does not exist" containerID="08b2a840a862b7057e3bae9ca63dd7f30df1bce04bc275e4df8c8466dbc2ea68"
Sep 11 17:36:18.400426 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:36:18.400372    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"08b2a840a862b7057e3bae9ca63dd7f30df1bce04bc275e4df8c8466dbc2ea68"} err="failed to get container status \"08b2a840a862b7057e3bae9ca63dd7f30df1bce04bc275e4df8c8466dbc2ea68\": rpc error: code = NotFound desc = could not find container \"08b2a840a862b7057e3bae9ca63dd7f30df1bce04bc275e4df8c8466dbc2ea68\": container with ID starting with 08b2a840a862b7057e3bae9ca63dd7f30df1bce04bc275e4df8c8466dbc2ea68 not found: ID does not exist"
Sep 11 17:36:18.425478 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:36:18.425444    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-65fdd4fd-7f8dc674cd-4zklz"]
Sep 11 17:36:18.431748 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:36:18.431724    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-65fdd4fd-7f8dc674cd-4zklz"]
Sep 11 17:36:18.927099 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:36:18.927063    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8f4775eb-df86-45cc-8e0c-ec25394169ae" path="/var/lib/kubelet/pods/8f4775eb-df86-45cc-8e0c-ec25394169ae/volumes"
Sep 11 17:36:18.927450 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:36:18.927436    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="96bfb668-ad05-4e2a-8c7d-8ad3b62b3d2a" path="/var/lib/kubelet/pods/96bfb668-ad05-4e2a-8c7d-8ad3b62b3d2a/volumes"
Sep 11 17:36:32.900146 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:36:32.900099    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-nkmkm"]
Sep 11 17:37:04.428041 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:37:04.428006    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-explicit-ec3fed/explicit-model-ec3fed-kserve-9d5c6f865-b987k"]
Sep 11 17:37:04.428488 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:37:04.428457    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="96bfb668-ad05-4e2a-8c7d-8ad3b62b3d2a" containerName="payload-pre-processing"
Sep 11 17:37:04.428488 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:37:04.428470    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="96bfb668-ad05-4e2a-8c7d-8ad3b62b3d2a" containerName="payload-pre-processing"
Sep 11 17:37:04.428488 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:37:04.428481    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8f4775eb-df86-45cc-8e0c-ec25394169ae" containerName="revoke-keys"
Sep 11 17:37:04.428488 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:37:04.428487    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="8f4775eb-df86-45cc-8e0c-ec25394169ae" containerName="revoke-keys"
Sep 11 17:37:04.428670 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:37:04.428583    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="96bfb668-ad05-4e2a-8c7d-8ad3b62b3d2a" containerName="payload-pre-processing"
Sep 11 17:37:04.428670 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:37:04.428594    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="8f4775eb-df86-45cc-8e0c-ec25394169ae" containerName="revoke-keys"
Sep 11 17:37:04.433189 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:37:04.433172    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-explicit-ec3fed/explicit-model-ec3fed-kserve-9d5c6f865-b987k"
Sep 11 17:37:04.435735 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:37:04.435714    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-explicit-ec3fed\"/\"explicit-model-ec3fed-kserve-self-signed-certs\""
Sep 11 17:37:04.436514 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:37:04.436483    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-explicit-ec3fed\"/\"openshift-service-ca.crt\""
Sep 11 17:37:04.436641 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:37:04.436621    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-explicit-ec3fed\"/\"kube-root-ca.crt\""
Sep 11 17:37:04.436721 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:37:04.436642    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-explicit-ec3fed\"/\"default-dockercfg-9qtbj\""
Sep 11 17:37:04.444828 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:37:04.444802    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-explicit-ec3fed/explicit-model-ec3fed-kserve-9d5c6f865-b987k"]
Sep 11 17:37:04.508209 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:37:04.508174    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-nkmkm"]
Sep 11 17:37:04.551629 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:37:04.551596    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/562d9f98-29ec-4d37-95bb-6ab5a4c6c983-home\") pod \"explicit-model-ec3fed-kserve-9d5c6f865-b987k\" (UID: \"562d9f98-29ec-4d37-95bb-6ab5a4c6c983\") " pod="ai-tenant-e2e-explicit-ec3fed/explicit-model-ec3fed-kserve-9d5c6f865-b987k"
Sep 11 17:37:04.551840 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:37:04.551647    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/562d9f98-29ec-4d37-95bb-6ab5a4c6c983-tmp-dir\") pod \"explicit-model-ec3fed-kserve-9d5c6f865-b987k\" (UID: \"562d9f98-29ec-4d37-95bb-6ab5a4c6c983\") " pod="ai-tenant-e2e-explicit-ec3fed/explicit-model-ec3fed-kserve-9d5c6f865-b987k"
Sep 11 17:37:04.551840 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:37:04.551671    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/562d9f98-29ec-4d37-95bb-6ab5a4c6c983-model-cache\") pod \"explicit-model-ec3fed-kserve-9d5c6f865-b987k\" (UID: \"562d9f98-29ec-4d37-95bb-6ab5a4c6c983\") " pod="ai-tenant-e2e-explicit-ec3fed/explicit-model-ec3fed-kserve-9d5c6f865-b987k"
Sep 11 17:37:04.551840 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:37:04.551807    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/562d9f98-29ec-4d37-95bb-6ab5a4c6c983-dshm\") pod \"explicit-model-ec3fed-kserve-9d5c6f865-b987k\" (UID: \"562d9f98-29ec-4d37-95bb-6ab5a4c6c983\") " pod="ai-tenant-e2e-explicit-ec3fed/explicit-model-ec3fed-kserve-9d5c6f865-b987k"
Sep 11 17:37:04.552020 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:37:04.551854    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/562d9f98-29ec-4d37-95bb-6ab5a4c6c983-tls-certs\") pod \"explicit-model-ec3fed-kserve-9d5c6f865-b987k\" (UID: \"562d9f98-29ec-4d37-95bb-6ab5a4c6c983\") " pod="ai-tenant-e2e-explicit-ec3fed/explicit-model-ec3fed-kserve-9d5c6f865-b987k"
Sep 11 17:37:04.552020 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:37:04.551919    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ns42g\" (UniqueName: \"kubernetes.io/projected/562d9f98-29ec-4d37-95bb-6ab5a4c6c983-kube-api-access-ns42g\") pod \"explicit-model-ec3fed-kserve-9d5c6f865-b987k\" (UID: \"562d9f98-29ec-4d37-95bb-6ab5a4c6c983\") " pod="ai-tenant-e2e-explicit-ec3fed/explicit-model-ec3fed-kserve-9d5c6f865-b987k"
Sep 11 17:37:04.653057 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:37:04.653016    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/562d9f98-29ec-4d37-95bb-6ab5a4c6c983-dshm\") pod \"explicit-model-ec3fed-kserve-9d5c6f865-b987k\" (UID: \"562d9f98-29ec-4d37-95bb-6ab5a4c6c983\") " pod="ai-tenant-e2e-explicit-ec3fed/explicit-model-ec3fed-kserve-9d5c6f865-b987k"
Sep 11 17:37:04.653241 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:37:04.653156    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/562d9f98-29ec-4d37-95bb-6ab5a4c6c983-tls-certs\") pod \"explicit-model-ec3fed-kserve-9d5c6f865-b987k\" (UID: \"562d9f98-29ec-4d37-95bb-6ab5a4c6c983\") " pod="ai-tenant-e2e-explicit-ec3fed/explicit-model-ec3fed-kserve-9d5c6f865-b987k"
Sep 11 17:37:04.653241 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:37:04.653234    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ns42g\" (UniqueName: \"kubernetes.io/projected/562d9f98-29ec-4d37-95bb-6ab5a4c6c983-kube-api-access-ns42g\") pod \"explicit-model-ec3fed-kserve-9d5c6f865-b987k\" (UID: \"562d9f98-29ec-4d37-95bb-6ab5a4c6c983\") " pod="ai-tenant-e2e-explicit-ec3fed/explicit-model-ec3fed-kserve-9d5c6f865-b987k"
Sep 11 17:37:04.653367 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:37:04.653306    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/562d9f98-29ec-4d37-95bb-6ab5a4c6c983-home\") pod \"explicit-model-ec3fed-kserve-9d5c6f865-b987k\" (UID: \"562d9f98-29ec-4d37-95bb-6ab5a4c6c983\") " pod="ai-tenant-e2e-explicit-ec3fed/explicit-model-ec3fed-kserve-9d5c6f865-b987k"
Sep 11 17:37:04.653430 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:37:04.653377    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/562d9f98-29ec-4d37-95bb-6ab5a4c6c983-tmp-dir\") pod \"explicit-model-ec3fed-kserve-9d5c6f865-b987k\" (UID: \"562d9f98-29ec-4d37-95bb-6ab5a4c6c983\") " pod="ai-tenant-e2e-explicit-ec3fed/explicit-model-ec3fed-kserve-9d5c6f865-b987k"
Sep 11 17:37:04.653430 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:37:04.653410    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/562d9f98-29ec-4d37-95bb-6ab5a4c6c983-model-cache\") pod \"explicit-model-ec3fed-kserve-9d5c6f865-b987k\" (UID: \"562d9f98-29ec-4d37-95bb-6ab5a4c6c983\") " pod="ai-tenant-e2e-explicit-ec3fed/explicit-model-ec3fed-kserve-9d5c6f865-b987k"
Sep 11 17:37:04.653815 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:37:04.653792    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/562d9f98-29ec-4d37-95bb-6ab5a4c6c983-model-cache\") pod \"explicit-model-ec3fed-kserve-9d5c6f865-b987k\" (UID: \"562d9f98-29ec-4d37-95bb-6ab5a4c6c983\") " pod="ai-tenant-e2e-explicit-ec3fed/explicit-model-ec3fed-kserve-9d5c6f865-b987k"
Sep 11 17:37:04.654084 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:37:04.653940    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/562d9f98-29ec-4d37-95bb-6ab5a4c6c983-tmp-dir\") pod \"explicit-model-ec3fed-kserve-9d5c6f865-b987k\" (UID: \"562d9f98-29ec-4d37-95bb-6ab5a4c6c983\") " pod="ai-tenant-e2e-explicit-ec3fed/explicit-model-ec3fed-kserve-9d5c6f865-b987k"
Sep 11 17:37:04.654245 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:37:04.654216    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/562d9f98-29ec-4d37-95bb-6ab5a4c6c983-home\") pod \"explicit-model-ec3fed-kserve-9d5c6f865-b987k\" (UID: \"562d9f98-29ec-4d37-95bb-6ab5a4c6c983\") " pod="ai-tenant-e2e-explicit-ec3fed/explicit-model-ec3fed-kserve-9d5c6f865-b987k"
Sep 11 17:37:04.655794 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:37:04.655772    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/562d9f98-29ec-4d37-95bb-6ab5a4c6c983-dshm\") pod \"explicit-model-ec3fed-kserve-9d5c6f865-b987k\" (UID: \"562d9f98-29ec-4d37-95bb-6ab5a4c6c983\") " pod="ai-tenant-e2e-explicit-ec3fed/explicit-model-ec3fed-kserve-9d5c6f865-b987k"
Sep 11 17:37:04.655980 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:37:04.655965    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/562d9f98-29ec-4d37-95bb-6ab5a4c6c983-tls-certs\") pod \"explicit-model-ec3fed-kserve-9d5c6f865-b987k\" (UID: \"562d9f98-29ec-4d37-95bb-6ab5a4c6c983\") " pod="ai-tenant-e2e-explicit-ec3fed/explicit-model-ec3fed-kserve-9d5c6f865-b987k"
Sep 11 17:37:04.661842 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:37:04.661814    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ns42g\" (UniqueName: \"kubernetes.io/projected/562d9f98-29ec-4d37-95bb-6ab5a4c6c983-kube-api-access-ns42g\") pod \"explicit-model-ec3fed-kserve-9d5c6f865-b987k\" (UID: \"562d9f98-29ec-4d37-95bb-6ab5a4c6c983\") " pod="ai-tenant-e2e-explicit-ec3fed/explicit-model-ec3fed-kserve-9d5c6f865-b987k"
Sep 11 17:37:04.743301 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:37:04.743262    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-explicit-ec3fed/explicit-model-ec3fed-kserve-9d5c6f865-b987k"
Sep 11 17:37:05.026281 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:37:05.026253    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-explicit-ec3fed/explicit-model-ec3fed-kserve-9d5c6f865-b987k"]
Sep 11 17:37:05.027107 ip-10-0-141-139 kubenswrapper[2577]: W0911 17:37:05.027080    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod562d9f98_29ec_4d37_95bb_6ab5a4c6c983.slice/crio-0d46d3a3b786ec30909151d4ab860f6b66c47ddc67e49d64b98b37b47205d561 WatchSource:0}: Error finding container 0d46d3a3b786ec30909151d4ab860f6b66c47ddc67e49d64b98b37b47205d561: Status 404 returned error can't find the container with id 0d46d3a3b786ec30909151d4ab860f6b66c47ddc67e49d64b98b37b47205d561
Sep 11 17:37:05.666943 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:37:05.666907    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-explicit-ec3fed/explicit-model-ec3fed-kserve-9d5c6f865-b987k" event={"ID":"562d9f98-29ec-4d37-95bb-6ab5a4c6c983","Type":"ContainerStarted","Data":"f2409464c7843cfbacdfa3795ed206d0a6058ec650da5a3f5d1397da3c7db2b6"}
Sep 11 17:37:05.667315 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:37:05.666949    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-explicit-ec3fed/explicit-model-ec3fed-kserve-9d5c6f865-b987k" event={"ID":"562d9f98-29ec-4d37-95bb-6ab5a4c6c983","Type":"ContainerStarted","Data":"0d46d3a3b786ec30909151d4ab860f6b66c47ddc67e49d64b98b37b47205d561"}
Sep 11 17:37:05.688743 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:37:05.688693    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-explicit-ec3fed/explicit-model-ec3fed-kserve-9d5c6f865-b987k" podStartSLOduration=1.688680682 podStartE2EDuration="1.688680682s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 17:37:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 17:37:05.686196954 +0000 UTC m=+1103.259417089" watchObservedRunningTime="2026-09-11 17:37:05.688680682 +0000 UTC m=+1103.261900854"
Sep 11 17:37:14.743746 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:37:14.743711    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-explicit-ec3fed/explicit-model-ec3fed-kserve-9d5c6f865-b987k"
Sep 11 17:37:14.744243 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:37:14.743762    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-explicit-ec3fed/explicit-model-ec3fed-kserve-9d5c6f865-b987k"
Sep 11 17:37:14.757200 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:37:14.757176    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-explicit-ec3fed/explicit-model-ec3fed-kserve-9d5c6f865-b987k"
Sep 11 17:37:15.734083 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:37:15.734051    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-explicit-ec3fed/explicit-model-ec3fed-kserve-9d5c6f865-b987k"
Sep 11 17:37:40.231520 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:37:40.231471    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-ec3fed-wsdc7"]
Sep 11 17:37:40.235127 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:37:40.235109    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-ec3fed-wsdc7"
Sep 11 17:37:40.238185 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:37:40.238155    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-9n8gf\""
Sep 11 17:37:40.239162 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:37:40.239144    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 11 17:37:40.246119 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:37:40.246074    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-ec3fed-wsdc7"]
Sep 11 17:37:40.326337 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:37:40.326295    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d2697195-3574-448c-be29-1ef00a02e5cb-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-explicit-ec3fed-wsdc7\" (UID: \"d2697195-3574-448c-be29-1ef00a02e5cb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-ec3fed-wsdc7"
Sep 11 17:37:40.427422 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:37:40.427376    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d2697195-3574-448c-be29-1ef00a02e5cb-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-explicit-ec3fed-wsdc7\" (UID: \"d2697195-3574-448c-be29-1ef00a02e5cb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-ec3fed-wsdc7"
Sep 11 17:37:40.428054 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:37:40.428033    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d2697195-3574-448c-be29-1ef00a02e5cb-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-explicit-ec3fed-wsdc7\" (UID: \"d2697195-3574-448c-be29-1ef00a02e5cb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-ec3fed-wsdc7"
Sep 11 17:37:40.547328 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:37:40.547241    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-ec3fed-wsdc7"
Sep 11 17:37:40.824123 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:37:40.824095    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-ec3fed-wsdc7"]
Sep 11 17:37:40.826278 ip-10-0-141-139 kubenswrapper[2577]: W0911 17:37:40.826242    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd2697195_3574_448c_be29_1ef00a02e5cb.slice/crio-2608517a6994261bab6ddc5832f4df92a59807a66555bdbdb6e8b08a9dfba2c0 WatchSource:0}: Error finding container 2608517a6994261bab6ddc5832f4df92a59807a66555bdbdb6e8b08a9dfba2c0: Status 404 returned error can't find the container with id 2608517a6994261bab6ddc5832f4df92a59807a66555bdbdb6e8b08a9dfba2c0
Sep 11 17:37:40.863156 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:37:40.863056    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-ec3fed-wsdc7" event={"ID":"d2697195-3574-448c-be29-1ef00a02e5cb","Type":"ContainerStarted","Data":"2608517a6994261bab6ddc5832f4df92a59807a66555bdbdb6e8b08a9dfba2c0"}
Sep 11 17:37:41.870403 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:37:41.870315    2577 generic.go:356] "Generic (PLEG): container finished" podID="d2697195-3574-448c-be29-1ef00a02e5cb" containerID="ba8bcea1c2f00505544e3ae214586029eec2b7c1ea7cf384ad1a2c8eb1c7109c" exitCode=0
Sep 11 17:37:41.870804 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:37:41.870409    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-ec3fed-wsdc7" event={"ID":"d2697195-3574-448c-be29-1ef00a02e5cb","Type":"ContainerDied","Data":"ba8bcea1c2f00505544e3ae214586029eec2b7c1ea7cf384ad1a2c8eb1c7109c"}
Sep 11 17:37:42.705253 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:37:42.705217    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-144bfa/e2e-conflict-a-a3b46de5-kserve-6c7467f57f-xxm7v"]
Sep 11 17:37:42.708981 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:37:42.708957    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-144bfa/e2e-conflict-a-a3b46de5-kserve-6c7467f57f-xxm7v"
Sep 11 17:37:42.711372 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:37:42.711343    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-144bfa\"/\"e2e-conflict-a-a3b46de5-kserve-self-signed-certs\""
Sep 11 17:37:42.719029 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:37:42.718993    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-144bfa/e2e-conflict-a-a3b46de5-kserve-6c7467f57f-xxm7v"]
Sep 11 17:37:42.851662 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:37:42.851615    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sgjvr\" (UniqueName: \"kubernetes.io/projected/1c2898b6-c293-4ed2-b759-19b104df5f39-kube-api-access-sgjvr\") pod \"e2e-conflict-a-a3b46de5-kserve-6c7467f57f-xxm7v\" (UID: \"1c2898b6-c293-4ed2-b759-19b104df5f39\") " pod="e2e-models-e2e-worker-w4-144bfa/e2e-conflict-a-a3b46de5-kserve-6c7467f57f-xxm7v"
Sep 11 17:37:42.851852 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:37:42.851682    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1c2898b6-c293-4ed2-b759-19b104df5f39-tls-certs\") pod \"e2e-conflict-a-a3b46de5-kserve-6c7467f57f-xxm7v\" (UID: \"1c2898b6-c293-4ed2-b759-19b104df5f39\") " pod="e2e-models-e2e-worker-w4-144bfa/e2e-conflict-a-a3b46de5-kserve-6c7467f57f-xxm7v"
Sep 11 17:37:42.851852 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:37:42.851729    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1c2898b6-c293-4ed2-b759-19b104df5f39-model-cache\") pod \"e2e-conflict-a-a3b46de5-kserve-6c7467f57f-xxm7v\" (UID: \"1c2898b6-c293-4ed2-b759-19b104df5f39\") " pod="e2e-models-e2e-worker-w4-144bfa/e2e-conflict-a-a3b46de5-kserve-6c7467f57f-xxm7v"
Sep 11 17:37:42.851852 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:37:42.851751    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1c2898b6-c293-4ed2-b759-19b104df5f39-tmp-dir\") pod \"e2e-conflict-a-a3b46de5-kserve-6c7467f57f-xxm7v\" (UID: \"1c2898b6-c293-4ed2-b759-19b104df5f39\") " pod="e2e-models-e2e-worker-w4-144bfa/e2e-conflict-a-a3b46de5-kserve-6c7467f57f-xxm7v"
Sep 11 17:37:42.851852 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:37:42.851833    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1c2898b6-c293-4ed2-b759-19b104df5f39-home\") pod \"e2e-conflict-a-a3b46de5-kserve-6c7467f57f-xxm7v\" (UID: \"1c2898b6-c293-4ed2-b759-19b104df5f39\") " pod="e2e-models-e2e-worker-w4-144bfa/e2e-conflict-a-a3b46de5-kserve-6c7467f57f-xxm7v"
Sep 11 17:37:42.851998 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:37:42.851932    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1c2898b6-c293-4ed2-b759-19b104df5f39-dshm\") pod \"e2e-conflict-a-a3b46de5-kserve-6c7467f57f-xxm7v\" (UID: \"1c2898b6-c293-4ed2-b759-19b104df5f39\") " pod="e2e-models-e2e-worker-w4-144bfa/e2e-conflict-a-a3b46de5-kserve-6c7467f57f-xxm7v"
Sep 11 17:37:42.952628 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:37:42.952595    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sgjvr\" (UniqueName: \"kubernetes.io/projected/1c2898b6-c293-4ed2-b759-19b104df5f39-kube-api-access-sgjvr\") pod \"e2e-conflict-a-a3b46de5-kserve-6c7467f57f-xxm7v\" (UID: \"1c2898b6-c293-4ed2-b759-19b104df5f39\") " pod="e2e-models-e2e-worker-w4-144bfa/e2e-conflict-a-a3b46de5-kserve-6c7467f57f-xxm7v"
Sep 11 17:37:42.953038 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:37:42.952642    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1c2898b6-c293-4ed2-b759-19b104df5f39-tls-certs\") pod \"e2e-conflict-a-a3b46de5-kserve-6c7467f57f-xxm7v\" (UID: \"1c2898b6-c293-4ed2-b759-19b104df5f39\") " pod="e2e-models-e2e-worker-w4-144bfa/e2e-conflict-a-a3b46de5-kserve-6c7467f57f-xxm7v"
Sep 11 17:37:42.953038 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:37:42.952686    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1c2898b6-c293-4ed2-b759-19b104df5f39-model-cache\") pod \"e2e-conflict-a-a3b46de5-kserve-6c7467f57f-xxm7v\" (UID: \"1c2898b6-c293-4ed2-b759-19b104df5f39\") " pod="e2e-models-e2e-worker-w4-144bfa/e2e-conflict-a-a3b46de5-kserve-6c7467f57f-xxm7v"
Sep 11 17:37:42.953038 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:37:42.952704    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1c2898b6-c293-4ed2-b759-19b104df5f39-tmp-dir\") pod \"e2e-conflict-a-a3b46de5-kserve-6c7467f57f-xxm7v\" (UID: \"1c2898b6-c293-4ed2-b759-19b104df5f39\") " pod="e2e-models-e2e-worker-w4-144bfa/e2e-conflict-a-a3b46de5-kserve-6c7467f57f-xxm7v"
Sep 11 17:37:42.953038 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:37:42.952733    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1c2898b6-c293-4ed2-b759-19b104df5f39-home\") pod \"e2e-conflict-a-a3b46de5-kserve-6c7467f57f-xxm7v\" (UID: \"1c2898b6-c293-4ed2-b759-19b104df5f39\") " pod="e2e-models-e2e-worker-w4-144bfa/e2e-conflict-a-a3b46de5-kserve-6c7467f57f-xxm7v"
Sep 11 17:37:42.953038 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:37:42.952825    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1c2898b6-c293-4ed2-b759-19b104df5f39-dshm\") pod \"e2e-conflict-a-a3b46de5-kserve-6c7467f57f-xxm7v\" (UID: \"1c2898b6-c293-4ed2-b759-19b104df5f39\") " pod="e2e-models-e2e-worker-w4-144bfa/e2e-conflict-a-a3b46de5-kserve-6c7467f57f-xxm7v"
Sep 11 17:37:42.953333 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:37:42.953109    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1c2898b6-c293-4ed2-b759-19b104df5f39-model-cache\") pod \"e2e-conflict-a-a3b46de5-kserve-6c7467f57f-xxm7v\" (UID: \"1c2898b6-c293-4ed2-b759-19b104df5f39\") " pod="e2e-models-e2e-worker-w4-144bfa/e2e-conflict-a-a3b46de5-kserve-6c7467f57f-xxm7v"
Sep 11 17:37:42.953333 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:37:42.953167    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1c2898b6-c293-4ed2-b759-19b104df5f39-tmp-dir\") pod \"e2e-conflict-a-a3b46de5-kserve-6c7467f57f-xxm7v\" (UID: \"1c2898b6-c293-4ed2-b759-19b104df5f39\") " pod="e2e-models-e2e-worker-w4-144bfa/e2e-conflict-a-a3b46de5-kserve-6c7467f57f-xxm7v"
Sep 11 17:37:42.953333 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:37:42.953180    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1c2898b6-c293-4ed2-b759-19b104df5f39-home\") pod \"e2e-conflict-a-a3b46de5-kserve-6c7467f57f-xxm7v\" (UID: \"1c2898b6-c293-4ed2-b759-19b104df5f39\") " pod="e2e-models-e2e-worker-w4-144bfa/e2e-conflict-a-a3b46de5-kserve-6c7467f57f-xxm7v"
Sep 11 17:37:42.955303 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:37:42.955231    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1c2898b6-c293-4ed2-b759-19b104df5f39-dshm\") pod \"e2e-conflict-a-a3b46de5-kserve-6c7467f57f-xxm7v\" (UID: \"1c2898b6-c293-4ed2-b759-19b104df5f39\") " pod="e2e-models-e2e-worker-w4-144bfa/e2e-conflict-a-a3b46de5-kserve-6c7467f57f-xxm7v"
Sep 11 17:37:42.955303 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:37:42.955269    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-144bfa\"/\"e2e-conflict-a-a3b46de5-kserve-self-signed-certs\""
Sep 11 17:37:42.961878 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:37:42.961852    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sgjvr\" (UniqueName: \"kubernetes.io/projected/1c2898b6-c293-4ed2-b759-19b104df5f39-kube-api-access-sgjvr\") pod \"e2e-conflict-a-a3b46de5-kserve-6c7467f57f-xxm7v\" (UID: \"1c2898b6-c293-4ed2-b759-19b104df5f39\") " pod="e2e-models-e2e-worker-w4-144bfa/e2e-conflict-a-a3b46de5-kserve-6c7467f57f-xxm7v"
Sep 11 17:37:42.965804 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:37:42.965768    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1c2898b6-c293-4ed2-b759-19b104df5f39-tls-certs\") pod \"e2e-conflict-a-a3b46de5-kserve-6c7467f57f-xxm7v\" (UID: \"1c2898b6-c293-4ed2-b759-19b104df5f39\") " pod="e2e-models-e2e-worker-w4-144bfa/e2e-conflict-a-a3b46de5-kserve-6c7467f57f-xxm7v"
Sep 11 17:37:42.969134 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:37:42.969111    2577 scope.go:117] "RemoveContainer" containerID="f60c471ce3383d36ef28c63fcf45d8bd0f617e464d4c387038d8657d0b903308"
Sep 11 17:37:43.020206 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:37:43.020182    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-144bfa/e2e-conflict-a-a3b46de5-kserve-6c7467f57f-xxm7v"
Sep 11 17:37:43.025900 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:37:43.025881    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-ec3fed-wsdc7"
Sep 11 17:37:43.155650 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:37:43.155616    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d2697195-3574-448c-be29-1ef00a02e5cb-maas-api-service-ca\") pod \"d2697195-3574-448c-be29-1ef00a02e5cb\" (UID: \"d2697195-3574-448c-be29-1ef00a02e5cb\") "
Sep 11 17:37:43.156015 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:37:43.155990    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d2697195-3574-448c-be29-1ef00a02e5cb-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "d2697195-3574-448c-be29-1ef00a02e5cb" (UID: "d2697195-3574-448c-be29-1ef00a02e5cb"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 17:37:43.257026 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:37:43.256987    2577 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d2697195-3574-448c-be29-1ef00a02e5cb-maas-api-service-ca\") on node \"ip-10-0-141-139.ec2.internal\" DevicePath \"\""
Sep 11 17:37:43.299391 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:37:43.299366    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-144bfa/e2e-conflict-a-a3b46de5-kserve-6c7467f57f-xxm7v"]
Sep 11 17:37:43.301418 ip-10-0-141-139 kubenswrapper[2577]: W0911 17:37:43.301390    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod1c2898b6_c293_4ed2_b759_19b104df5f39.slice/crio-d3aed22a82bb01ef846116299539d76f3c9bfce2ce0e103210f347ddf01dc728 WatchSource:0}: Error finding container d3aed22a82bb01ef846116299539d76f3c9bfce2ce0e103210f347ddf01dc728: Status 404 returned error can't find the container with id d3aed22a82bb01ef846116299539d76f3c9bfce2ce0e103210f347ddf01dc728
Sep 11 17:37:43.884288 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:37:43.884255    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-ec3fed-wsdc7"
Sep 11 17:37:43.884288 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:37:43.884276    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-ec3fed-wsdc7" event={"ID":"d2697195-3574-448c-be29-1ef00a02e5cb","Type":"ContainerDied","Data":"2608517a6994261bab6ddc5832f4df92a59807a66555bdbdb6e8b08a9dfba2c0"}
Sep 11 17:37:43.884584 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:37:43.884339    2577 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2608517a6994261bab6ddc5832f4df92a59807a66555bdbdb6e8b08a9dfba2c0"
Sep 11 17:37:43.885990 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:37:43.885967    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-144bfa/e2e-conflict-a-a3b46de5-kserve-6c7467f57f-xxm7v" event={"ID":"1c2898b6-c293-4ed2-b759-19b104df5f39","Type":"ContainerStarted","Data":"8f7e4ec76358c3f77eda16db7ab5de402e785b935ba22f8a7e7a028073f29e65"}
Sep 11 17:37:43.886132 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:37:43.885996    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-144bfa/e2e-conflict-a-a3b46de5-kserve-6c7467f57f-xxm7v" event={"ID":"1c2898b6-c293-4ed2-b759-19b104df5f39","Type":"ContainerStarted","Data":"d3aed22a82bb01ef846116299539d76f3c9bfce2ce0e103210f347ddf01dc728"}
Sep 11 17:37:43.903200 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:37:43.903155    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-144bfa/e2e-conflict-a-a3b46de5-kserve-6c7467f57f-xxm7v" podStartSLOduration=1.903139044 podStartE2EDuration="1.903139044s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 17:37:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 17:37:43.902206784 +0000 UTC m=+1141.475426926" watchObservedRunningTime="2026-09-11 17:37:43.903139044 +0000 UTC m=+1141.476359197"
Sep 11 17:37:44.215891 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:37:44.215863    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-nkmkm"]
Sep 11 17:37:45.619470 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:37:45.619432    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-explicit-ec3fed/explicit-model-ec3fed-kserve-9d5c6f865-b987k"]
Sep 11 17:37:53.020328 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:37:53.020284    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-144bfa/e2e-conflict-a-a3b46de5-kserve-6c7467f57f-xxm7v"
Sep 11 17:37:53.020828 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:37:53.020562    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-144bfa/e2e-conflict-a-a3b46de5-kserve-6c7467f57f-xxm7v"
Sep 11 17:37:53.032987 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:37:53.032961    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-144bfa/e2e-conflict-a-a3b46de5-kserve-6c7467f57f-xxm7v"
Sep 11 17:37:53.956330 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:37:53.956290    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-144bfa/e2e-conflict-a-a3b46de5-kserve-6c7467f57f-xxm7v"
Sep 11 17:37:55.221648 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:37:55.221610    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-ec3fed-wsdc7"]
Sep 11 17:37:55.227884 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:37:55.227858    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-ec3fed-wsdc7"]
Sep 11 17:37:56.927442 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:37:56.927404    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d2697195-3574-448c-be29-1ef00a02e5cb" path="/var/lib/kubelet/pods/d2697195-3574-448c-be29-1ef00a02e5cb/volumes"
Sep 11 17:38:00.639446 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:38:00.639407    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-explicit-ec3fed/explicit-model-ec3fed-kserve-9d5c6f865-b987k" podUID="562d9f98-29ec-4d37-95bb-6ab5a4c6c983" containerName="main" containerID="cri-o://f2409464c7843cfbacdfa3795ed206d0a6058ec650da5a3f5d1397da3c7db2b6" gracePeriod=45
Sep 11 17:38:00.892374 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:38:00.892306    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-explicit-ec3fed/explicit-model-ec3fed-kserve-9d5c6f865-b987k"
Sep 11 17:38:00.948359 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:38:00.948324    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/562d9f98-29ec-4d37-95bb-6ab5a4c6c983-home\") pod \"562d9f98-29ec-4d37-95bb-6ab5a4c6c983\" (UID: \"562d9f98-29ec-4d37-95bb-6ab5a4c6c983\") "
Sep 11 17:38:00.948557 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:38:00.948466    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/562d9f98-29ec-4d37-95bb-6ab5a4c6c983-model-cache\") pod \"562d9f98-29ec-4d37-95bb-6ab5a4c6c983\" (UID: \"562d9f98-29ec-4d37-95bb-6ab5a4c6c983\") "
Sep 11 17:38:00.948557 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:38:00.948548    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/562d9f98-29ec-4d37-95bb-6ab5a4c6c983-dshm\") pod \"562d9f98-29ec-4d37-95bb-6ab5a4c6c983\" (UID: \"562d9f98-29ec-4d37-95bb-6ab5a4c6c983\") "
Sep 11 17:38:00.948691 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:38:00.948578    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/562d9f98-29ec-4d37-95bb-6ab5a4c6c983-tmp-dir\") pod \"562d9f98-29ec-4d37-95bb-6ab5a4c6c983\" (UID: \"562d9f98-29ec-4d37-95bb-6ab5a4c6c983\") "
Sep 11 17:38:00.948691 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:38:00.948601    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/562d9f98-29ec-4d37-95bb-6ab5a4c6c983-home" (OuterVolumeSpecName: "home") pod "562d9f98-29ec-4d37-95bb-6ab5a4c6c983" (UID: "562d9f98-29ec-4d37-95bb-6ab5a4c6c983"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 17:38:00.948691 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:38:00.948613    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ns42g\" (UniqueName: \"kubernetes.io/projected/562d9f98-29ec-4d37-95bb-6ab5a4c6c983-kube-api-access-ns42g\") pod \"562d9f98-29ec-4d37-95bb-6ab5a4c6c983\" (UID: \"562d9f98-29ec-4d37-95bb-6ab5a4c6c983\") "
Sep 11 17:38:00.948845 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:38:00.948726    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/562d9f98-29ec-4d37-95bb-6ab5a4c6c983-tls-certs\") pod \"562d9f98-29ec-4d37-95bb-6ab5a4c6c983\" (UID: \"562d9f98-29ec-4d37-95bb-6ab5a4c6c983\") "
Sep 11 17:38:00.948845 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:38:00.948725    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/562d9f98-29ec-4d37-95bb-6ab5a4c6c983-model-cache" (OuterVolumeSpecName: "model-cache") pod "562d9f98-29ec-4d37-95bb-6ab5a4c6c983" (UID: "562d9f98-29ec-4d37-95bb-6ab5a4c6c983"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 17:38:00.948951 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:38:00.948801    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/562d9f98-29ec-4d37-95bb-6ab5a4c6c983-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "562d9f98-29ec-4d37-95bb-6ab5a4c6c983" (UID: "562d9f98-29ec-4d37-95bb-6ab5a4c6c983"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 17:38:00.949181 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:38:00.949155    2577 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/562d9f98-29ec-4d37-95bb-6ab5a4c6c983-model-cache\") on node \"ip-10-0-141-139.ec2.internal\" DevicePath \"\""
Sep 11 17:38:00.949181 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:38:00.949183    2577 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/562d9f98-29ec-4d37-95bb-6ab5a4c6c983-tmp-dir\") on node \"ip-10-0-141-139.ec2.internal\" DevicePath \"\""
Sep 11 17:38:00.949551 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:38:00.949197    2577 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/562d9f98-29ec-4d37-95bb-6ab5a4c6c983-home\") on node \"ip-10-0-141-139.ec2.internal\" DevicePath \"\""
Sep 11 17:38:00.951045 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:38:00.951021    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/562d9f98-29ec-4d37-95bb-6ab5a4c6c983-dshm" (OuterVolumeSpecName: "dshm") pod "562d9f98-29ec-4d37-95bb-6ab5a4c6c983" (UID: "562d9f98-29ec-4d37-95bb-6ab5a4c6c983"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 17:38:00.951143 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:38:00.951093    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/562d9f98-29ec-4d37-95bb-6ab5a4c6c983-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "562d9f98-29ec-4d37-95bb-6ab5a4c6c983" (UID: "562d9f98-29ec-4d37-95bb-6ab5a4c6c983"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 17:38:00.951143 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:38:00.951082    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/562d9f98-29ec-4d37-95bb-6ab5a4c6c983-kube-api-access-ns42g" (OuterVolumeSpecName: "kube-api-access-ns42g") pod "562d9f98-29ec-4d37-95bb-6ab5a4c6c983" (UID: "562d9f98-29ec-4d37-95bb-6ab5a4c6c983"). InnerVolumeSpecName "kube-api-access-ns42g". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 17:38:00.986545 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:38:00.986495    2577 generic.go:356] "Generic (PLEG): container finished" podID="562d9f98-29ec-4d37-95bb-6ab5a4c6c983" containerID="f2409464c7843cfbacdfa3795ed206d0a6058ec650da5a3f5d1397da3c7db2b6" exitCode=0
Sep 11 17:38:00.986715 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:38:00.986572    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-explicit-ec3fed/explicit-model-ec3fed-kserve-9d5c6f865-b987k"
Sep 11 17:38:00.986715 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:38:00.986600    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-explicit-ec3fed/explicit-model-ec3fed-kserve-9d5c6f865-b987k" event={"ID":"562d9f98-29ec-4d37-95bb-6ab5a4c6c983","Type":"ContainerDied","Data":"f2409464c7843cfbacdfa3795ed206d0a6058ec650da5a3f5d1397da3c7db2b6"}
Sep 11 17:38:00.986715 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:38:00.986648    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-explicit-ec3fed/explicit-model-ec3fed-kserve-9d5c6f865-b987k" event={"ID":"562d9f98-29ec-4d37-95bb-6ab5a4c6c983","Type":"ContainerDied","Data":"0d46d3a3b786ec30909151d4ab860f6b66c47ddc67e49d64b98b37b47205d561"}
Sep 11 17:38:00.986715 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:38:00.986667    2577 scope.go:117] "RemoveContainer" containerID="f2409464c7843cfbacdfa3795ed206d0a6058ec650da5a3f5d1397da3c7db2b6"
Sep 11 17:38:00.996517 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:38:00.996487    2577 scope.go:117] "RemoveContainer" containerID="f2409464c7843cfbacdfa3795ed206d0a6058ec650da5a3f5d1397da3c7db2b6"
Sep 11 17:38:00.996795 ip-10-0-141-139 kubenswrapper[2577]: E0911 17:38:00.996774    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f2409464c7843cfbacdfa3795ed206d0a6058ec650da5a3f5d1397da3c7db2b6\": container with ID starting with f2409464c7843cfbacdfa3795ed206d0a6058ec650da5a3f5d1397da3c7db2b6 not found: ID does not exist" containerID="f2409464c7843cfbacdfa3795ed206d0a6058ec650da5a3f5d1397da3c7db2b6"
Sep 11 17:38:00.996869 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:38:00.996801    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f2409464c7843cfbacdfa3795ed206d0a6058ec650da5a3f5d1397da3c7db2b6"} err="failed to get container status \"f2409464c7843cfbacdfa3795ed206d0a6058ec650da5a3f5d1397da3c7db2b6\": rpc error: code = NotFound desc = could not find container \"f2409464c7843cfbacdfa3795ed206d0a6058ec650da5a3f5d1397da3c7db2b6\": container with ID starting with f2409464c7843cfbacdfa3795ed206d0a6058ec650da5a3f5d1397da3c7db2b6 not found: ID does not exist"
Sep 11 17:38:01.011162 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:38:01.011128    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-explicit-ec3fed/explicit-model-ec3fed-kserve-9d5c6f865-b987k"]
Sep 11 17:38:01.017050 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:38:01.017023    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-explicit-ec3fed/explicit-model-ec3fed-kserve-9d5c6f865-b987k"]
Sep 11 17:38:01.049825 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:38:01.049796    2577 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/562d9f98-29ec-4d37-95bb-6ab5a4c6c983-dshm\") on node \"ip-10-0-141-139.ec2.internal\" DevicePath \"\""
Sep 11 17:38:01.049825 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:38:01.049826    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ns42g\" (UniqueName: \"kubernetes.io/projected/562d9f98-29ec-4d37-95bb-6ab5a4c6c983-kube-api-access-ns42g\") on node \"ip-10-0-141-139.ec2.internal\" DevicePath \"\""
Sep 11 17:38:01.049994 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:38:01.049839    2577 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/562d9f98-29ec-4d37-95bb-6ab5a4c6c983-tls-certs\") on node \"ip-10-0-141-139.ec2.internal\" DevicePath \"\""
Sep 11 17:38:02.928970 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:38:02.928868    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="562d9f98-29ec-4d37-95bb-6ab5a4c6c983" path="/var/lib/kubelet/pods/562d9f98-29ec-4d37-95bb-6ab5a4c6c983/volumes"
Sep 11 17:38:03.662844 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:38:03.662766    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-6709f1a7-5dbf546cbf-jq9n7"]
Sep 11 17:38:03.663228 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:38:03.663216    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="562d9f98-29ec-4d37-95bb-6ab5a4c6c983" containerName="main"
Sep 11 17:38:03.663290 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:38:03.663230    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="562d9f98-29ec-4d37-95bb-6ab5a4c6c983" containerName="main"
Sep 11 17:38:03.663290 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:38:03.663251    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d2697195-3574-448c-be29-1ef00a02e5cb" containerName="revoke-keys"
Sep 11 17:38:03.663290 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:38:03.663258    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="d2697195-3574-448c-be29-1ef00a02e5cb" containerName="revoke-keys"
Sep 11 17:38:03.663402 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:38:03.663325    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="562d9f98-29ec-4d37-95bb-6ab5a4c6c983" containerName="main"
Sep 11 17:38:03.663402 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:38:03.663335    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="d2697195-3574-448c-be29-1ef00a02e5cb" containerName="revoke-keys"
Sep 11 17:38:03.667645 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:38:03.667626    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-6709f1a7-5dbf546cbf-jq9n7"
Sep 11 17:38:03.670076 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:38:03.670049    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-6709f1a7\""
Sep 11 17:38:03.670183 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:38:03.670099    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-6709f1a7-dockercfg-sc2gj\""
Sep 11 17:38:03.676855 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:38:03.676831    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-6709f1a7-5dbf546cbf-jq9n7"]
Sep 11 17:38:03.780809 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:38:03.780766    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4w79s\" (UniqueName: \"kubernetes.io/projected/8b097356-9bdb-4504-9c22-4982c9ecaca8-kube-api-access-4w79s\") pod \"payload-processing-e2e-mt-6709f1a7-5dbf546cbf-jq9n7\" (UID: \"8b097356-9bdb-4504-9c22-4982c9ecaca8\") " pod="openshift-ingress/payload-processing-e2e-mt-6709f1a7-5dbf546cbf-jq9n7"
Sep 11 17:38:03.780991 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:38:03.780834    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8b097356-9bdb-4504-9c22-4982c9ecaca8-plugins-config-volume\") pod \"payload-processing-e2e-mt-6709f1a7-5dbf546cbf-jq9n7\" (UID: \"8b097356-9bdb-4504-9c22-4982c9ecaca8\") " pod="openshift-ingress/payload-processing-e2e-mt-6709f1a7-5dbf546cbf-jq9n7"
Sep 11 17:38:03.882442 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:38:03.882409    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4w79s\" (UniqueName: \"kubernetes.io/projected/8b097356-9bdb-4504-9c22-4982c9ecaca8-kube-api-access-4w79s\") pod \"payload-processing-e2e-mt-6709f1a7-5dbf546cbf-jq9n7\" (UID: \"8b097356-9bdb-4504-9c22-4982c9ecaca8\") " pod="openshift-ingress/payload-processing-e2e-mt-6709f1a7-5dbf546cbf-jq9n7"
Sep 11 17:38:03.882658 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:38:03.882465    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8b097356-9bdb-4504-9c22-4982c9ecaca8-plugins-config-volume\") pod \"payload-processing-e2e-mt-6709f1a7-5dbf546cbf-jq9n7\" (UID: \"8b097356-9bdb-4504-9c22-4982c9ecaca8\") " pod="openshift-ingress/payload-processing-e2e-mt-6709f1a7-5dbf546cbf-jq9n7"
Sep 11 17:38:03.883234 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:38:03.883214    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8b097356-9bdb-4504-9c22-4982c9ecaca8-plugins-config-volume\") pod \"payload-processing-e2e-mt-6709f1a7-5dbf546cbf-jq9n7\" (UID: \"8b097356-9bdb-4504-9c22-4982c9ecaca8\") " pod="openshift-ingress/payload-processing-e2e-mt-6709f1a7-5dbf546cbf-jq9n7"
Sep 11 17:38:03.893011 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:38:03.892987    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4w79s\" (UniqueName: \"kubernetes.io/projected/8b097356-9bdb-4504-9c22-4982c9ecaca8-kube-api-access-4w79s\") pod \"payload-processing-e2e-mt-6709f1a7-5dbf546cbf-jq9n7\" (UID: \"8b097356-9bdb-4504-9c22-4982c9ecaca8\") " pod="openshift-ingress/payload-processing-e2e-mt-6709f1a7-5dbf546cbf-jq9n7"
Sep 11 17:38:03.979494 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:38:03.979462    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-6709f1a7-5dbf546cbf-jq9n7"
Sep 11 17:38:04.255733 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:38:04.255709    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-6709f1a7-5dbf546cbf-jq9n7"]
Sep 11 17:38:04.257752 ip-10-0-141-139 kubenswrapper[2577]: W0911 17:38:04.257724    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8b097356_9bdb_4504_9c22_4982c9ecaca8.slice/crio-30a7d1d60aae45c1416e66819c7c07e4f093484bc8f81df00805fe7226889c65 WatchSource:0}: Error finding container 30a7d1d60aae45c1416e66819c7c07e4f093484bc8f81df00805fe7226889c65: Status 404 returned error can't find the container with id 30a7d1d60aae45c1416e66819c7c07e4f093484bc8f81df00805fe7226889c65
Sep 11 17:38:05.016166 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:38:05.016123    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-6709f1a7-5dbf546cbf-jq9n7" event={"ID":"8b097356-9bdb-4504-9c22-4982c9ecaca8","Type":"ContainerStarted","Data":"8c8a5643c23b5bbb714ce973c849047c7774f497ee27e37ce7000c4f3c258851"}
Sep 11 17:38:05.016721 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:38:05.016182    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-6709f1a7-5dbf546cbf-jq9n7" event={"ID":"8b097356-9bdb-4504-9c22-4982c9ecaca8","Type":"ContainerStarted","Data":"30a7d1d60aae45c1416e66819c7c07e4f093484bc8f81df00805fe7226889c65"}
Sep 11 17:38:05.016721 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:38:05.016218    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-mt-6709f1a7-5dbf546cbf-jq9n7"
Sep 11 17:38:05.040705 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:38:05.040658    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-mt-6709f1a7-5dbf546cbf-jq9n7" podStartSLOduration=2.040643309 podStartE2EDuration="2.040643309s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 17:38:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 17:38:05.03779971 +0000 UTC m=+1162.611019824" watchObservedRunningTime="2026-09-11 17:38:05.040643309 +0000 UTC m=+1162.613863421"
Sep 11 17:38:15.135816 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:38:15.135779    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-nkmkm"]
Sep 11 17:38:16.024368 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:38:16.024333    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-mt-6709f1a7-5dbf546cbf-jq9n7"
Sep 11 17:38:20.295297 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:38:20.295265    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-c7d192/distinct-w0-c7d192-kserve-645f648f77-lxn5g"]
Sep 11 17:38:20.299351 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:38:20.299329    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-c7d192/distinct-w0-c7d192-kserve-645f648f77-lxn5g"
Sep 11 17:38:20.302810 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:38:20.302785    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-c7d192\"/\"distinct-w0-c7d192-kserve-self-signed-certs\""
Sep 11 17:38:20.317095 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:38:20.317063    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-c7d192/distinct-w0-c7d192-kserve-645f648f77-lxn5g"]
Sep 11 17:38:20.349108 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:38:20.349074    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/66e0a151-0ad9-46ce-9d4e-d0f2f8b03d4d-tls-certs\") pod \"distinct-w0-c7d192-kserve-645f648f77-lxn5g\" (UID: \"66e0a151-0ad9-46ce-9d4e-d0f2f8b03d4d\") " pod="e2e-models-e2e-worker-w0-c7d192/distinct-w0-c7d192-kserve-645f648f77-lxn5g"
Sep 11 17:38:20.349292 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:38:20.349174    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/66e0a151-0ad9-46ce-9d4e-d0f2f8b03d4d-home\") pod \"distinct-w0-c7d192-kserve-645f648f77-lxn5g\" (UID: \"66e0a151-0ad9-46ce-9d4e-d0f2f8b03d4d\") " pod="e2e-models-e2e-worker-w0-c7d192/distinct-w0-c7d192-kserve-645f648f77-lxn5g"
Sep 11 17:38:20.349292 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:38:20.349209    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/66e0a151-0ad9-46ce-9d4e-d0f2f8b03d4d-tmp-dir\") pod \"distinct-w0-c7d192-kserve-645f648f77-lxn5g\" (UID: \"66e0a151-0ad9-46ce-9d4e-d0f2f8b03d4d\") " pod="e2e-models-e2e-worker-w0-c7d192/distinct-w0-c7d192-kserve-645f648f77-lxn5g"
Sep 11 17:38:20.349422 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:38:20.349309    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7csn8\" (UniqueName: \"kubernetes.io/projected/66e0a151-0ad9-46ce-9d4e-d0f2f8b03d4d-kube-api-access-7csn8\") pod \"distinct-w0-c7d192-kserve-645f648f77-lxn5g\" (UID: \"66e0a151-0ad9-46ce-9d4e-d0f2f8b03d4d\") " pod="e2e-models-e2e-worker-w0-c7d192/distinct-w0-c7d192-kserve-645f648f77-lxn5g"
Sep 11 17:38:20.349422 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:38:20.349367    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/66e0a151-0ad9-46ce-9d4e-d0f2f8b03d4d-dshm\") pod \"distinct-w0-c7d192-kserve-645f648f77-lxn5g\" (UID: \"66e0a151-0ad9-46ce-9d4e-d0f2f8b03d4d\") " pod="e2e-models-e2e-worker-w0-c7d192/distinct-w0-c7d192-kserve-645f648f77-lxn5g"
Sep 11 17:38:20.349567 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:38:20.349419    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/66e0a151-0ad9-46ce-9d4e-d0f2f8b03d4d-model-cache\") pod \"distinct-w0-c7d192-kserve-645f648f77-lxn5g\" (UID: \"66e0a151-0ad9-46ce-9d4e-d0f2f8b03d4d\") " pod="e2e-models-e2e-worker-w0-c7d192/distinct-w0-c7d192-kserve-645f648f77-lxn5g"
Sep 11 17:38:20.450795 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:38:20.450754    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/66e0a151-0ad9-46ce-9d4e-d0f2f8b03d4d-home\") pod \"distinct-w0-c7d192-kserve-645f648f77-lxn5g\" (UID: \"66e0a151-0ad9-46ce-9d4e-d0f2f8b03d4d\") " pod="e2e-models-e2e-worker-w0-c7d192/distinct-w0-c7d192-kserve-645f648f77-lxn5g"
Sep 11 17:38:20.450795 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:38:20.450799    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/66e0a151-0ad9-46ce-9d4e-d0f2f8b03d4d-tmp-dir\") pod \"distinct-w0-c7d192-kserve-645f648f77-lxn5g\" (UID: \"66e0a151-0ad9-46ce-9d4e-d0f2f8b03d4d\") " pod="e2e-models-e2e-worker-w0-c7d192/distinct-w0-c7d192-kserve-645f648f77-lxn5g"
Sep 11 17:38:20.451054 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:38:20.450947    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7csn8\" (UniqueName: \"kubernetes.io/projected/66e0a151-0ad9-46ce-9d4e-d0f2f8b03d4d-kube-api-access-7csn8\") pod \"distinct-w0-c7d192-kserve-645f648f77-lxn5g\" (UID: \"66e0a151-0ad9-46ce-9d4e-d0f2f8b03d4d\") " pod="e2e-models-e2e-worker-w0-c7d192/distinct-w0-c7d192-kserve-645f648f77-lxn5g"
Sep 11 17:38:20.451054 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:38:20.451010    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/66e0a151-0ad9-46ce-9d4e-d0f2f8b03d4d-dshm\") pod \"distinct-w0-c7d192-kserve-645f648f77-lxn5g\" (UID: \"66e0a151-0ad9-46ce-9d4e-d0f2f8b03d4d\") " pod="e2e-models-e2e-worker-w0-c7d192/distinct-w0-c7d192-kserve-645f648f77-lxn5g"
Sep 11 17:38:20.451183 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:38:20.451067    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/66e0a151-0ad9-46ce-9d4e-d0f2f8b03d4d-model-cache\") pod \"distinct-w0-c7d192-kserve-645f648f77-lxn5g\" (UID: \"66e0a151-0ad9-46ce-9d4e-d0f2f8b03d4d\") " pod="e2e-models-e2e-worker-w0-c7d192/distinct-w0-c7d192-kserve-645f648f77-lxn5g"
Sep 11 17:38:20.451183 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:38:20.451104    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/66e0a151-0ad9-46ce-9d4e-d0f2f8b03d4d-tls-certs\") pod \"distinct-w0-c7d192-kserve-645f648f77-lxn5g\" (UID: \"66e0a151-0ad9-46ce-9d4e-d0f2f8b03d4d\") " pod="e2e-models-e2e-worker-w0-c7d192/distinct-w0-c7d192-kserve-645f648f77-lxn5g"
Sep 11 17:38:20.451183 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:38:20.451143    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/66e0a151-0ad9-46ce-9d4e-d0f2f8b03d4d-tmp-dir\") pod \"distinct-w0-c7d192-kserve-645f648f77-lxn5g\" (UID: \"66e0a151-0ad9-46ce-9d4e-d0f2f8b03d4d\") " pod="e2e-models-e2e-worker-w0-c7d192/distinct-w0-c7d192-kserve-645f648f77-lxn5g"
Sep 11 17:38:20.451317 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:38:20.451208    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/66e0a151-0ad9-46ce-9d4e-d0f2f8b03d4d-home\") pod \"distinct-w0-c7d192-kserve-645f648f77-lxn5g\" (UID: \"66e0a151-0ad9-46ce-9d4e-d0f2f8b03d4d\") " pod="e2e-models-e2e-worker-w0-c7d192/distinct-w0-c7d192-kserve-645f648f77-lxn5g"
Sep 11 17:38:20.451381 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:38:20.451364    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/66e0a151-0ad9-46ce-9d4e-d0f2f8b03d4d-model-cache\") pod \"distinct-w0-c7d192-kserve-645f648f77-lxn5g\" (UID: \"66e0a151-0ad9-46ce-9d4e-d0f2f8b03d4d\") " pod="e2e-models-e2e-worker-w0-c7d192/distinct-w0-c7d192-kserve-645f648f77-lxn5g"
Sep 11 17:38:20.453420 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:38:20.453393    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/66e0a151-0ad9-46ce-9d4e-d0f2f8b03d4d-dshm\") pod \"distinct-w0-c7d192-kserve-645f648f77-lxn5g\" (UID: \"66e0a151-0ad9-46ce-9d4e-d0f2f8b03d4d\") " pod="e2e-models-e2e-worker-w0-c7d192/distinct-w0-c7d192-kserve-645f648f77-lxn5g"
Sep 11 17:38:20.453571 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:38:20.453555    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/66e0a151-0ad9-46ce-9d4e-d0f2f8b03d4d-tls-certs\") pod \"distinct-w0-c7d192-kserve-645f648f77-lxn5g\" (UID: \"66e0a151-0ad9-46ce-9d4e-d0f2f8b03d4d\") " pod="e2e-models-e2e-worker-w0-c7d192/distinct-w0-c7d192-kserve-645f648f77-lxn5g"
Sep 11 17:38:20.512332 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:38:20.512280    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7csn8\" (UniqueName: \"kubernetes.io/projected/66e0a151-0ad9-46ce-9d4e-d0f2f8b03d4d-kube-api-access-7csn8\") pod \"distinct-w0-c7d192-kserve-645f648f77-lxn5g\" (UID: \"66e0a151-0ad9-46ce-9d4e-d0f2f8b03d4d\") " pod="e2e-models-e2e-worker-w0-c7d192/distinct-w0-c7d192-kserve-645f648f77-lxn5g"
Sep 11 17:38:20.610089 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:38:20.609998    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-c7d192/distinct-w0-c7d192-kserve-645f648f77-lxn5g"
Sep 11 17:38:20.910944 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:38:20.910919    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-c7d192/distinct-w0-c7d192-kserve-645f648f77-lxn5g"]
Sep 11 17:38:20.913263 ip-10-0-141-139 kubenswrapper[2577]: W0911 17:38:20.913239    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod66e0a151_0ad9_46ce_9d4e_d0f2f8b03d4d.slice/crio-767ed3d9c3fa1fec9a82eb8d4c695336055a352ce9981d6bbd05fff25ed8850a WatchSource:0}: Error finding container 767ed3d9c3fa1fec9a82eb8d4c695336055a352ce9981d6bbd05fff25ed8850a: Status 404 returned error can't find the container with id 767ed3d9c3fa1fec9a82eb8d4c695336055a352ce9981d6bbd05fff25ed8850a
Sep 11 17:38:21.117608 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:38:21.117572    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-c7d192/distinct-w0-c7d192-kserve-645f648f77-lxn5g" event={"ID":"66e0a151-0ad9-46ce-9d4e-d0f2f8b03d4d","Type":"ContainerStarted","Data":"8fb5e9789ab3a1fbf975b73256bd830adb7948376aadb7d182f466e765ec6ba7"}
Sep 11 17:38:21.117764 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:38:21.117615    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-c7d192/distinct-w0-c7d192-kserve-645f648f77-lxn5g" event={"ID":"66e0a151-0ad9-46ce-9d4e-d0f2f8b03d4d","Type":"ContainerStarted","Data":"767ed3d9c3fa1fec9a82eb8d4c695336055a352ce9981d6bbd05fff25ed8850a"}
Sep 11 17:38:30.610608 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:38:30.610558    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-c7d192/distinct-w0-c7d192-kserve-645f648f77-lxn5g"
Sep 11 17:38:30.611039 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:38:30.610641    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-c7d192/distinct-w0-c7d192-kserve-645f648f77-lxn5g"
Sep 11 17:38:30.623133 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:38:30.623100    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-c7d192/distinct-w0-c7d192-kserve-645f648f77-lxn5g"
Sep 11 17:38:30.643694 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:38:30.643638    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-c7d192/distinct-w0-c7d192-kserve-645f648f77-lxn5g" podStartSLOduration=10.643616104 podStartE2EDuration="10.643616104s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 17:38:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 17:38:21.168218211 +0000 UTC m=+1178.741438322" watchObservedRunningTime="2026-09-11 17:38:30.643616104 +0000 UTC m=+1188.216836217"
Sep 11 17:38:31.189057 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:38:31.189021    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-c7d192/distinct-w0-c7d192-kserve-645f648f77-lxn5g"
Sep 11 17:38:43.003006 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:38:43.002976    2577 scope.go:117] "RemoveContainer" containerID="afc2706c0b7cc82ab9b88b12696e98cc53957038c0e7edde31698486d11b6d69"
Sep 11 17:38:45.602343 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:38:45.602303    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-nkmkm"]
Sep 11 17:39:07.748457 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:39:07.748415    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-08c6b2-s86kz"]
Sep 11 17:39:07.752127 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:39:07.752111    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-08c6b2-s86kz"
Sep 11 17:39:07.754667 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:39:07.754648    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 11 17:39:07.754784 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:39:07.754651    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-9n8gf\""
Sep 11 17:39:07.759738 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:39:07.759713    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-08c6b2-s86kz"]
Sep 11 17:39:07.867642 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:39:07.867611    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/32ae1663-f75e-4b65-8d48-d6eb375c8648-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-08c6b2-s86kz\" (UID: \"32ae1663-f75e-4b65-8d48-d6eb375c8648\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-08c6b2-s86kz"
Sep 11 17:39:07.968553 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:39:07.968487    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/32ae1663-f75e-4b65-8d48-d6eb375c8648-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-08c6b2-s86kz\" (UID: \"32ae1663-f75e-4b65-8d48-d6eb375c8648\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-08c6b2-s86kz"
Sep 11 17:39:07.969170 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:39:07.969144    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/32ae1663-f75e-4b65-8d48-d6eb375c8648-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-08c6b2-s86kz\" (UID: \"32ae1663-f75e-4b65-8d48-d6eb375c8648\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-08c6b2-s86kz"
Sep 11 17:39:08.064596 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:39:08.064515    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-08c6b2-s86kz"
Sep 11 17:39:08.339807 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:39:08.339776    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-08c6b2-s86kz"]
Sep 11 17:39:08.341241 ip-10-0-141-139 kubenswrapper[2577]: W0911 17:39:08.341215    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod32ae1663_f75e_4b65_8d48_d6eb375c8648.slice/crio-b83af7a88c9971d1bd31848333ea257ed60434533425653f6e14b5497c177b93 WatchSource:0}: Error finding container b83af7a88c9971d1bd31848333ea257ed60434533425653f6e14b5497c177b93: Status 404 returned error can't find the container with id b83af7a88c9971d1bd31848333ea257ed60434533425653f6e14b5497c177b93
Sep 11 17:39:08.402554 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:39:08.402522    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-08c6b2-s86kz" event={"ID":"32ae1663-f75e-4b65-8d48-d6eb375c8648","Type":"ContainerStarted","Data":"b83af7a88c9971d1bd31848333ea257ed60434533425653f6e14b5497c177b93"}
Sep 11 17:39:09.410420 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:39:09.410330    2577 generic.go:356] "Generic (PLEG): container finished" podID="32ae1663-f75e-4b65-8d48-d6eb375c8648" containerID="ff5545207aeab7b03618f24f8ea7f22a66bf85861bf396df4e7e4e99d8c3c126" exitCode=0
Sep 11 17:39:09.410794 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:39:09.410414    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-08c6b2-s86kz" event={"ID":"32ae1663-f75e-4b65-8d48-d6eb375c8648","Type":"ContainerDied","Data":"ff5545207aeab7b03618f24f8ea7f22a66bf85861bf396df4e7e4e99d8c3c126"}
Sep 11 17:39:10.558767 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:39:10.558743    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-08c6b2-s86kz"
Sep 11 17:39:10.698061 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:39:10.697971    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/32ae1663-f75e-4b65-8d48-d6eb375c8648-maas-api-service-ca\") pod \"32ae1663-f75e-4b65-8d48-d6eb375c8648\" (UID: \"32ae1663-f75e-4b65-8d48-d6eb375c8648\") "
Sep 11 17:39:10.698353 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:39:10.698329    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/32ae1663-f75e-4b65-8d48-d6eb375c8648-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "32ae1663-f75e-4b65-8d48-d6eb375c8648" (UID: "32ae1663-f75e-4b65-8d48-d6eb375c8648"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 17:39:10.799662 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:39:10.799628    2577 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/32ae1663-f75e-4b65-8d48-d6eb375c8648-maas-api-service-ca\") on node \"ip-10-0-141-139.ec2.internal\" DevicePath \"\""
Sep 11 17:39:11.423024 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:39:11.422995    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-08c6b2-s86kz"
Sep 11 17:39:11.423206 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:39:11.422993    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-08c6b2-s86kz" event={"ID":"32ae1663-f75e-4b65-8d48-d6eb375c8648","Type":"ContainerDied","Data":"b83af7a88c9971d1bd31848333ea257ed60434533425653f6e14b5497c177b93"}
Sep 11 17:39:11.423206 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:39:11.423118    2577 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b83af7a88c9971d1bd31848333ea257ed60434533425653f6e14b5497c177b93"
Sep 11 17:39:17.111483 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:39:17.111448    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-nkmkm"]
Sep 11 17:39:18.421933 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:39:18.421900    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-08c6b2-7984f858bc-nmlzk"]
Sep 11 17:39:18.422429 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:39:18.422123    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-a-08c6b2-7984f858bc-nmlzk" podUID="e4dc0839-a155-4a0c-b4f3-e8a1462e7390" containerName="payload-processing" containerID="cri-o://ac93a5162d972950c1b4fefdef917aa826775dc83a63be708edff15f38250ca0" gracePeriod=30
Sep 11 17:39:18.605584 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:39:18.605549    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-08c6b2-s86kz"]
Sep 11 17:39:18.609162 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:39:18.609137    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-08c6b2-s86kz"]
Sep 11 17:39:18.689192 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:39:18.689122    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-08c6b2-7984f858bc-nmlzk"
Sep 11 17:39:18.779873 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:39:18.779836    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e4dc0839-a155-4a0c-b4f3-e8a1462e7390-plugins-config-volume\") pod \"e4dc0839-a155-4a0c-b4f3-e8a1462e7390\" (UID: \"e4dc0839-a155-4a0c-b4f3-e8a1462e7390\") "
Sep 11 17:39:18.780040 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:39:18.779913    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8p4n6\" (UniqueName: \"kubernetes.io/projected/e4dc0839-a155-4a0c-b4f3-e8a1462e7390-kube-api-access-8p4n6\") pod \"e4dc0839-a155-4a0c-b4f3-e8a1462e7390\" (UID: \"e4dc0839-a155-4a0c-b4f3-e8a1462e7390\") "
Sep 11 17:39:18.780216 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:39:18.780194    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e4dc0839-a155-4a0c-b4f3-e8a1462e7390-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "e4dc0839-a155-4a0c-b4f3-e8a1462e7390" (UID: "e4dc0839-a155-4a0c-b4f3-e8a1462e7390"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 17:39:18.782231 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:39:18.782209    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e4dc0839-a155-4a0c-b4f3-e8a1462e7390-kube-api-access-8p4n6" (OuterVolumeSpecName: "kube-api-access-8p4n6") pod "e4dc0839-a155-4a0c-b4f3-e8a1462e7390" (UID: "e4dc0839-a155-4a0c-b4f3-e8a1462e7390"). InnerVolumeSpecName "kube-api-access-8p4n6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 17:39:18.881497 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:39:18.881462    2577 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e4dc0839-a155-4a0c-b4f3-e8a1462e7390-plugins-config-volume\") on node \"ip-10-0-141-139.ec2.internal\" DevicePath \"\""
Sep 11 17:39:18.881497 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:39:18.881488    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8p4n6\" (UniqueName: \"kubernetes.io/projected/e4dc0839-a155-4a0c-b4f3-e8a1462e7390-kube-api-access-8p4n6\") on node \"ip-10-0-141-139.ec2.internal\" DevicePath \"\""
Sep 11 17:39:18.927052 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:39:18.927017    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="32ae1663-f75e-4b65-8d48-d6eb375c8648" path="/var/lib/kubelet/pods/32ae1663-f75e-4b65-8d48-d6eb375c8648/volumes"
Sep 11 17:39:19.473051 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:39:19.473018    2577 generic.go:356] "Generic (PLEG): container finished" podID="e4dc0839-a155-4a0c-b4f3-e8a1462e7390" containerID="ac93a5162d972950c1b4fefdef917aa826775dc83a63be708edff15f38250ca0" exitCode=0
Sep 11 17:39:19.473488 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:39:19.473072    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-08c6b2-7984f858bc-nmlzk"
Sep 11 17:39:19.473488 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:39:19.473110    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-08c6b2-7984f858bc-nmlzk" event={"ID":"e4dc0839-a155-4a0c-b4f3-e8a1462e7390","Type":"ContainerDied","Data":"ac93a5162d972950c1b4fefdef917aa826775dc83a63be708edff15f38250ca0"}
Sep 11 17:39:19.473488 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:39:19.473170    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-08c6b2-7984f858bc-nmlzk" event={"ID":"e4dc0839-a155-4a0c-b4f3-e8a1462e7390","Type":"ContainerDied","Data":"45ad6f725d2fe5423b5b7bbdfc70d04b671942bad1376cede6ca816a5eaa56f7"}
Sep 11 17:39:19.473488 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:39:19.473201    2577 scope.go:117] "RemoveContainer" containerID="ac93a5162d972950c1b4fefdef917aa826775dc83a63be708edff15f38250ca0"
Sep 11 17:39:19.485164 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:39:19.485089    2577 scope.go:117] "RemoveContainer" containerID="ac93a5162d972950c1b4fefdef917aa826775dc83a63be708edff15f38250ca0"
Sep 11 17:39:19.485695 ip-10-0-141-139 kubenswrapper[2577]: E0911 17:39:19.485667    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ac93a5162d972950c1b4fefdef917aa826775dc83a63be708edff15f38250ca0\": container with ID starting with ac93a5162d972950c1b4fefdef917aa826775dc83a63be708edff15f38250ca0 not found: ID does not exist" containerID="ac93a5162d972950c1b4fefdef917aa826775dc83a63be708edff15f38250ca0"
Sep 11 17:39:19.485794 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:39:19.485700    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ac93a5162d972950c1b4fefdef917aa826775dc83a63be708edff15f38250ca0"} err="failed to get container status \"ac93a5162d972950c1b4fefdef917aa826775dc83a63be708edff15f38250ca0\": rpc error: code = NotFound desc = could not find container \"ac93a5162d972950c1b4fefdef917aa826775dc83a63be708edff15f38250ca0\": container with ID starting with ac93a5162d972950c1b4fefdef917aa826775dc83a63be708edff15f38250ca0 not found: ID does not exist"
Sep 11 17:39:19.499748 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:39:19.499709    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-08c6b2-7984f858bc-nmlzk"]
Sep 11 17:39:19.506366 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:39:19.506342    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-08c6b2-7984f858bc-nmlzk"]
Sep 11 17:39:19.636067 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:39:19.636018    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-a-08c6b2-7984f858bc-nmlzk" podUID="e4dc0839-a155-4a0c-b4f3-e8a1462e7390" containerName="payload-processing" probeResult="failure" output="dial tcp 10.134.0.40:9004: i/o timeout"
Sep 11 17:39:20.926996 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:39:20.926962    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e4dc0839-a155-4a0c-b4f3-e8a1462e7390" path="/var/lib/kubelet/pods/e4dc0839-a155-4a0c-b4f3-e8a1462e7390/volumes"
Sep 11 17:39:25.676257 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:39:25.676226    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-cbbb09-6b4kj"]
Sep 11 17:39:25.676716 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:39:25.676702    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="32ae1663-f75e-4b65-8d48-d6eb375c8648" containerName="revoke-keys"
Sep 11 17:39:25.676775 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:39:25.676717    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="32ae1663-f75e-4b65-8d48-d6eb375c8648" containerName="revoke-keys"
Sep 11 17:39:25.676775 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:39:25.676733    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e4dc0839-a155-4a0c-b4f3-e8a1462e7390" containerName="payload-processing"
Sep 11 17:39:25.676775 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:39:25.676740    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="e4dc0839-a155-4a0c-b4f3-e8a1462e7390" containerName="payload-processing"
Sep 11 17:39:25.676868 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:39:25.676823    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="32ae1663-f75e-4b65-8d48-d6eb375c8648" containerName="revoke-keys"
Sep 11 17:39:25.676868 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:39:25.676831    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="e4dc0839-a155-4a0c-b4f3-e8a1462e7390" containerName="payload-processing"
Sep 11 17:39:25.680204 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:39:25.680190    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-cbbb09-6b4kj"
Sep 11 17:39:25.682724 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:39:25.682703    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 11 17:39:25.682854 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:39:25.682707    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-9n8gf\""
Sep 11 17:39:25.688945 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:39:25.688916    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-cbbb09-6b4kj"]
Sep 11 17:39:25.752053 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:39:25.752016    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ba11ee6d-7493-4368-a532-4390e647b0ac-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-cbbb09-6b4kj\" (UID: \"ba11ee6d-7493-4368-a532-4390e647b0ac\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-cbbb09-6b4kj"
Sep 11 17:39:25.853061 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:39:25.853026    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ba11ee6d-7493-4368-a532-4390e647b0ac-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-cbbb09-6b4kj\" (UID: \"ba11ee6d-7493-4368-a532-4390e647b0ac\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-cbbb09-6b4kj"
Sep 11 17:39:25.853693 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:39:25.853675    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ba11ee6d-7493-4368-a532-4390e647b0ac-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-cbbb09-6b4kj\" (UID: \"ba11ee6d-7493-4368-a532-4390e647b0ac\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-cbbb09-6b4kj"
Sep 11 17:39:25.992802 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:39:25.992767    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-cbbb09-6b4kj"
Sep 11 17:39:26.269035 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:39:26.269011    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-cbbb09-6b4kj"]
Sep 11 17:39:26.270848 ip-10-0-141-139 kubenswrapper[2577]: W0911 17:39:26.270823    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podba11ee6d_7493_4368_a532_4390e647b0ac.slice/crio-99b35fc3d2c4566424934b01980c5a1645eef8fdeb65369f04e5242610f9b07a WatchSource:0}: Error finding container 99b35fc3d2c4566424934b01980c5a1645eef8fdeb65369f04e5242610f9b07a: Status 404 returned error can't find the container with id 99b35fc3d2c4566424934b01980c5a1645eef8fdeb65369f04e5242610f9b07a
Sep 11 17:39:26.517900 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:39:26.517861    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-cbbb09-6b4kj" event={"ID":"ba11ee6d-7493-4368-a532-4390e647b0ac","Type":"ContainerStarted","Data":"99b35fc3d2c4566424934b01980c5a1645eef8fdeb65369f04e5242610f9b07a"}
Sep 11 17:39:27.526110 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:39:27.526077    2577 generic.go:356] "Generic (PLEG): container finished" podID="ba11ee6d-7493-4368-a532-4390e647b0ac" containerID="504109a314b0a7117a009b758797a7a7f77383cb9fbbf58029946c7b10ec74a7" exitCode=0
Sep 11 17:39:27.526600 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:39:27.526168    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-cbbb09-6b4kj" event={"ID":"ba11ee6d-7493-4368-a532-4390e647b0ac","Type":"ContainerDied","Data":"504109a314b0a7117a009b758797a7a7f77383cb9fbbf58029946c7b10ec74a7"}
Sep 11 17:39:28.670424 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:39:28.670399    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-cbbb09-6b4kj"
Sep 11 17:39:28.784669 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:39:28.784558    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ba11ee6d-7493-4368-a532-4390e647b0ac-maas-api-service-ca\") pod \"ba11ee6d-7493-4368-a532-4390e647b0ac\" (UID: \"ba11ee6d-7493-4368-a532-4390e647b0ac\") "
Sep 11 17:39:28.784949 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:39:28.784924    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ba11ee6d-7493-4368-a532-4390e647b0ac-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "ba11ee6d-7493-4368-a532-4390e647b0ac" (UID: "ba11ee6d-7493-4368-a532-4390e647b0ac"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 17:39:28.886180 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:39:28.886143    2577 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ba11ee6d-7493-4368-a532-4390e647b0ac-maas-api-service-ca\") on node \"ip-10-0-141-139.ec2.internal\" DevicePath \"\""
Sep 11 17:39:29.539710 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:39:29.539684    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-cbbb09-6b4kj"
Sep 11 17:39:29.539710 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:39:29.539683    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-cbbb09-6b4kj" event={"ID":"ba11ee6d-7493-4368-a532-4390e647b0ac","Type":"ContainerDied","Data":"99b35fc3d2c4566424934b01980c5a1645eef8fdeb65369f04e5242610f9b07a"}
Sep 11 17:39:29.539930 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:39:29.539745    2577 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="99b35fc3d2c4566424934b01980c5a1645eef8fdeb65369f04e5242610f9b07a"
Sep 11 17:39:36.226852 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:39:36.226777    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-cbbb09-85f75d469f-fmfwt"]
Sep 11 17:39:36.227231 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:39:36.227013    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-b-cbbb09-85f75d469f-fmfwt" podUID="8e1a7c02-0df2-4958-bd88-2e212f3d89e4" containerName="payload-processing" containerID="cri-o://09a3d4081916f90191d4e96de393c264ae180c8eec9d3ac76f317e32ed62d5dd" gracePeriod=30
Sep 11 17:39:36.236989 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:39:36.236761    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-cbbb09-544b668766-zmvwq"]
Sep 11 17:39:36.237069 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:39:36.236994    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-cbbb09-544b668766-zmvwq" podUID="c3346092-b638-42cc-b901-aa4b12a0e13e" containerName="payload-pre-processing" containerID="cri-o://101cabb20d179f58646161b79f96e1827a3e69c5df9bf8eb8d8da3129dc1bcb4" gracePeriod=30
Sep 11 17:39:36.410041 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:39:36.409987    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-cbbb09-6b4kj"]
Sep 11 17:39:36.412961 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:39:36.412885    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-cbbb09-6b4kj"]
Sep 11 17:39:36.528709 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:39:36.528685    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-cbbb09-85f75d469f-fmfwt"
Sep 11 17:39:36.531730 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:39:36.531708    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-cbbb09-544b668766-zmvwq"
Sep 11 17:39:36.589235 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:39:36.589196    2577 generic.go:356] "Generic (PLEG): container finished" podID="c3346092-b638-42cc-b901-aa4b12a0e13e" containerID="101cabb20d179f58646161b79f96e1827a3e69c5df9bf8eb8d8da3129dc1bcb4" exitCode=0
Sep 11 17:39:36.589405 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:39:36.589261    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-cbbb09-544b668766-zmvwq"
Sep 11 17:39:36.589405 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:39:36.589284    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-cbbb09-544b668766-zmvwq" event={"ID":"c3346092-b638-42cc-b901-aa4b12a0e13e","Type":"ContainerDied","Data":"101cabb20d179f58646161b79f96e1827a3e69c5df9bf8eb8d8da3129dc1bcb4"}
Sep 11 17:39:36.589405 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:39:36.589370    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-cbbb09-544b668766-zmvwq" event={"ID":"c3346092-b638-42cc-b901-aa4b12a0e13e","Type":"ContainerDied","Data":"cfc8dc24f60f6f911be35f6b64271d82798529efcb0c016c8870485d8b759f0a"}
Sep 11 17:39:36.589405 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:39:36.589400    2577 scope.go:117] "RemoveContainer" containerID="101cabb20d179f58646161b79f96e1827a3e69c5df9bf8eb8d8da3129dc1bcb4"
Sep 11 17:39:36.590802 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:39:36.590780    2577 generic.go:356] "Generic (PLEG): container finished" podID="8e1a7c02-0df2-4958-bd88-2e212f3d89e4" containerID="09a3d4081916f90191d4e96de393c264ae180c8eec9d3ac76f317e32ed62d5dd" exitCode=0
Sep 11 17:39:36.590921 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:39:36.590825    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-cbbb09-85f75d469f-fmfwt"
Sep 11 17:39:36.590921 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:39:36.590863    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-cbbb09-85f75d469f-fmfwt" event={"ID":"8e1a7c02-0df2-4958-bd88-2e212f3d89e4","Type":"ContainerDied","Data":"09a3d4081916f90191d4e96de393c264ae180c8eec9d3ac76f317e32ed62d5dd"}
Sep 11 17:39:36.590921 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:39:36.590909    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-cbbb09-85f75d469f-fmfwt" event={"ID":"8e1a7c02-0df2-4958-bd88-2e212f3d89e4","Type":"ContainerDied","Data":"646517111edce307e265218752c82cdb96744ca93113e4d897997b95525708b8"}
Sep 11 17:39:36.598754 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:39:36.598734    2577 scope.go:117] "RemoveContainer" containerID="101cabb20d179f58646161b79f96e1827a3e69c5df9bf8eb8d8da3129dc1bcb4"
Sep 11 17:39:36.599063 ip-10-0-141-139 kubenswrapper[2577]: E0911 17:39:36.599047    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"101cabb20d179f58646161b79f96e1827a3e69c5df9bf8eb8d8da3129dc1bcb4\": container with ID starting with 101cabb20d179f58646161b79f96e1827a3e69c5df9bf8eb8d8da3129dc1bcb4 not found: ID does not exist" containerID="101cabb20d179f58646161b79f96e1827a3e69c5df9bf8eb8d8da3129dc1bcb4"
Sep 11 17:39:36.599106 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:39:36.599072    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"101cabb20d179f58646161b79f96e1827a3e69c5df9bf8eb8d8da3129dc1bcb4"} err="failed to get container status \"101cabb20d179f58646161b79f96e1827a3e69c5df9bf8eb8d8da3129dc1bcb4\": rpc error: code = NotFound desc = could not find container \"101cabb20d179f58646161b79f96e1827a3e69c5df9bf8eb8d8da3129dc1bcb4\": container with ID starting with 101cabb20d179f58646161b79f96e1827a3e69c5df9bf8eb8d8da3129dc1bcb4 not found: ID does not exist"
Sep 11 17:39:36.599106 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:39:36.599088    2577 scope.go:117] "RemoveContainer" containerID="09a3d4081916f90191d4e96de393c264ae180c8eec9d3ac76f317e32ed62d5dd"
Sep 11 17:39:36.608111 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:39:36.608087    2577 scope.go:117] "RemoveContainer" containerID="09a3d4081916f90191d4e96de393c264ae180c8eec9d3ac76f317e32ed62d5dd"
Sep 11 17:39:36.608383 ip-10-0-141-139 kubenswrapper[2577]: E0911 17:39:36.608368    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"09a3d4081916f90191d4e96de393c264ae180c8eec9d3ac76f317e32ed62d5dd\": container with ID starting with 09a3d4081916f90191d4e96de393c264ae180c8eec9d3ac76f317e32ed62d5dd not found: ID does not exist" containerID="09a3d4081916f90191d4e96de393c264ae180c8eec9d3ac76f317e32ed62d5dd"
Sep 11 17:39:36.608456 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:39:36.608390    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"09a3d4081916f90191d4e96de393c264ae180c8eec9d3ac76f317e32ed62d5dd"} err="failed to get container status \"09a3d4081916f90191d4e96de393c264ae180c8eec9d3ac76f317e32ed62d5dd\": rpc error: code = NotFound desc = could not find container \"09a3d4081916f90191d4e96de393c264ae180c8eec9d3ac76f317e32ed62d5dd\": container with ID starting with 09a3d4081916f90191d4e96de393c264ae180c8eec9d3ac76f317e32ed62d5dd not found: ID does not exist"
Sep 11 17:39:36.668391 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:39:36.668344    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-b4ms2\" (UniqueName: \"kubernetes.io/projected/c3346092-b638-42cc-b901-aa4b12a0e13e-kube-api-access-b4ms2\") pod \"c3346092-b638-42cc-b901-aa4b12a0e13e\" (UID: \"c3346092-b638-42cc-b901-aa4b12a0e13e\") "
Sep 11 17:39:36.668391 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:39:36.668415    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c3346092-b638-42cc-b901-aa4b12a0e13e-plugins-config-volume\") pod \"c3346092-b638-42cc-b901-aa4b12a0e13e\" (UID: \"c3346092-b638-42cc-b901-aa4b12a0e13e\") "
Sep 11 17:39:36.668653 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:39:36.668538    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pdtf2\" (UniqueName: \"kubernetes.io/projected/8e1a7c02-0df2-4958-bd88-2e212f3d89e4-kube-api-access-pdtf2\") pod \"8e1a7c02-0df2-4958-bd88-2e212f3d89e4\" (UID: \"8e1a7c02-0df2-4958-bd88-2e212f3d89e4\") "
Sep 11 17:39:36.668653 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:39:36.668587    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8e1a7c02-0df2-4958-bd88-2e212f3d89e4-plugins-config-volume\") pod \"8e1a7c02-0df2-4958-bd88-2e212f3d89e4\" (UID: \"8e1a7c02-0df2-4958-bd88-2e212f3d89e4\") "
Sep 11 17:39:36.668867 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:39:36.668843    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c3346092-b638-42cc-b901-aa4b12a0e13e-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "c3346092-b638-42cc-b901-aa4b12a0e13e" (UID: "c3346092-b638-42cc-b901-aa4b12a0e13e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 17:39:36.668970 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:39:36.668954    2577 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c3346092-b638-42cc-b901-aa4b12a0e13e-plugins-config-volume\") on node \"ip-10-0-141-139.ec2.internal\" DevicePath \"\""
Sep 11 17:39:36.669029 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:39:36.668977    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8e1a7c02-0df2-4958-bd88-2e212f3d89e4-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "8e1a7c02-0df2-4958-bd88-2e212f3d89e4" (UID: "8e1a7c02-0df2-4958-bd88-2e212f3d89e4"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 17:39:36.670808 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:39:36.670784    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c3346092-b638-42cc-b901-aa4b12a0e13e-kube-api-access-b4ms2" (OuterVolumeSpecName: "kube-api-access-b4ms2") pod "c3346092-b638-42cc-b901-aa4b12a0e13e" (UID: "c3346092-b638-42cc-b901-aa4b12a0e13e"). InnerVolumeSpecName "kube-api-access-b4ms2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 17:39:36.670914 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:39:36.670827    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8e1a7c02-0df2-4958-bd88-2e212f3d89e4-kube-api-access-pdtf2" (OuterVolumeSpecName: "kube-api-access-pdtf2") pod "8e1a7c02-0df2-4958-bd88-2e212f3d89e4" (UID: "8e1a7c02-0df2-4958-bd88-2e212f3d89e4"). InnerVolumeSpecName "kube-api-access-pdtf2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 17:39:36.770361 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:39:36.770240    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b4ms2\" (UniqueName: \"kubernetes.io/projected/c3346092-b638-42cc-b901-aa4b12a0e13e-kube-api-access-b4ms2\") on node \"ip-10-0-141-139.ec2.internal\" DevicePath \"\""
Sep 11 17:39:36.770361 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:39:36.770279    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pdtf2\" (UniqueName: \"kubernetes.io/projected/8e1a7c02-0df2-4958-bd88-2e212f3d89e4-kube-api-access-pdtf2\") on node \"ip-10-0-141-139.ec2.internal\" DevicePath \"\""
Sep 11 17:39:36.770361 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:39:36.770301    2577 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8e1a7c02-0df2-4958-bd88-2e212f3d89e4-plugins-config-volume\") on node \"ip-10-0-141-139.ec2.internal\" DevicePath \"\""
Sep 11 17:39:36.911296 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:39:36.911267    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-cbbb09-544b668766-zmvwq"]
Sep 11 17:39:36.917240 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:39:36.917214    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-cbbb09-544b668766-zmvwq"]
Sep 11 17:39:36.927438 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:39:36.927405    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ba11ee6d-7493-4368-a532-4390e647b0ac" path="/var/lib/kubelet/pods/ba11ee6d-7493-4368-a532-4390e647b0ac/volumes"
Sep 11 17:39:36.927880 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:39:36.927865    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c3346092-b638-42cc-b901-aa4b12a0e13e" path="/var/lib/kubelet/pods/c3346092-b638-42cc-b901-aa4b12a0e13e/volumes"
Sep 11 17:39:36.928132 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:39:36.928120    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-cbbb09-85f75d469f-fmfwt"]
Sep 11 17:39:36.932320 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:39:36.932281    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-cbbb09-85f75d469f-fmfwt"]
Sep 11 17:39:38.927095 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:39:38.927061    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8e1a7c02-0df2-4958-bd88-2e212f3d89e4" path="/var/lib/kubelet/pods/8e1a7c02-0df2-4958-bd88-2e212f3d89e4/volumes"
Sep 11 17:39:43.031450 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:39:43.031419    2577 scope.go:117] "RemoveContainer" containerID="e4983553c2196f922eb1738b7936880709e79f2f72d3eca749f8644615287aca"
Sep 11 17:39:43.040371 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:39:43.040347    2577 scope.go:117] "RemoveContainer" containerID="e793dcd6dfb34202cbf43a6c08a1cda3a19d55336ba4e0e1259f0b600b653508"
Sep 11 17:39:47.397795 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:39:47.397762    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-nkmkm"]
Sep 11 17:39:53.311073 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:39:53.311034    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-2fac54-5796b4b6c7-c29rj"]
Sep 11 17:39:53.311492 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:39:53.311476    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c3346092-b638-42cc-b901-aa4b12a0e13e" containerName="payload-pre-processing"
Sep 11 17:39:53.311592 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:39:53.311493    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="c3346092-b638-42cc-b901-aa4b12a0e13e" containerName="payload-pre-processing"
Sep 11 17:39:53.311592 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:39:53.311518    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ba11ee6d-7493-4368-a532-4390e647b0ac" containerName="revoke-keys"
Sep 11 17:39:53.311592 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:39:53.311523    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="ba11ee6d-7493-4368-a532-4390e647b0ac" containerName="revoke-keys"
Sep 11 17:39:53.311592 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:39:53.311562    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8e1a7c02-0df2-4958-bd88-2e212f3d89e4" containerName="payload-processing"
Sep 11 17:39:53.311592 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:39:53.311568    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="8e1a7c02-0df2-4958-bd88-2e212f3d89e4" containerName="payload-processing"
Sep 11 17:39:53.311763 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:39:53.311654    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="8e1a7c02-0df2-4958-bd88-2e212f3d89e4" containerName="payload-processing"
Sep 11 17:39:53.311763 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:39:53.311668    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="ba11ee6d-7493-4368-a532-4390e647b0ac" containerName="revoke-keys"
Sep 11 17:39:53.311763 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:39:53.311678    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="c3346092-b638-42cc-b901-aa4b12a0e13e" containerName="payload-pre-processing"
Sep 11 17:39:53.316559 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:39:53.316542    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-2fac54-5796b4b6c7-c29rj"
Sep 11 17:39:53.319557 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:39:53.319534    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w3-2fac54\""
Sep 11 17:39:53.319702 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:39:53.319656    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w3-2fac54-dockercfg-hc2pw\""
Sep 11 17:39:53.325959 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:39:53.325931    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-2fac54-5796b4b6c7-c29rj"]
Sep 11 17:39:53.340853 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:39:53.340817    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c5842af6-6103-4759-951c-0fb4717aff70-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w3-2fac54-5796b4b6c7-c29rj\" (UID: \"c5842af6-6103-4759-951c-0fb4717aff70\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-2fac54-5796b4b6c7-c29rj"
Sep 11 17:39:53.341050 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:39:53.340866    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hq2wf\" (UniqueName: \"kubernetes.io/projected/c5842af6-6103-4759-951c-0fb4717aff70-kube-api-access-hq2wf\") pod \"payload-processing-e2e-shared-a-w3-2fac54-5796b4b6c7-c29rj\" (UID: \"c5842af6-6103-4759-951c-0fb4717aff70\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-2fac54-5796b4b6c7-c29rj"
Sep 11 17:39:53.441995 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:39:53.441954    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c5842af6-6103-4759-951c-0fb4717aff70-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w3-2fac54-5796b4b6c7-c29rj\" (UID: \"c5842af6-6103-4759-951c-0fb4717aff70\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-2fac54-5796b4b6c7-c29rj"
Sep 11 17:39:53.441995 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:39:53.442001    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hq2wf\" (UniqueName: \"kubernetes.io/projected/c5842af6-6103-4759-951c-0fb4717aff70-kube-api-access-hq2wf\") pod \"payload-processing-e2e-shared-a-w3-2fac54-5796b4b6c7-c29rj\" (UID: \"c5842af6-6103-4759-951c-0fb4717aff70\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-2fac54-5796b4b6c7-c29rj"
Sep 11 17:39:53.442774 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:39:53.442749    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c5842af6-6103-4759-951c-0fb4717aff70-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w3-2fac54-5796b4b6c7-c29rj\" (UID: \"c5842af6-6103-4759-951c-0fb4717aff70\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-2fac54-5796b4b6c7-c29rj"
Sep 11 17:39:53.452173 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:39:53.452149    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hq2wf\" (UniqueName: \"kubernetes.io/projected/c5842af6-6103-4759-951c-0fb4717aff70-kube-api-access-hq2wf\") pod \"payload-processing-e2e-shared-a-w3-2fac54-5796b4b6c7-c29rj\" (UID: \"c5842af6-6103-4759-951c-0fb4717aff70\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-2fac54-5796b4b6c7-c29rj"
Sep 11 17:39:53.628726 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:39:53.628635    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-2fac54-5796b4b6c7-c29rj"
Sep 11 17:39:53.913489 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:39:53.913462    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-2fac54-5796b4b6c7-c29rj"]
Sep 11 17:39:53.915664 ip-10-0-141-139 kubenswrapper[2577]: W0911 17:39:53.915634    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc5842af6_6103_4759_951c_0fb4717aff70.slice/crio-2afefeb2ed0de0a1a24ebcf3082d143f7010860610f7271b5124329d4a11208d WatchSource:0}: Error finding container 2afefeb2ed0de0a1a24ebcf3082d143f7010860610f7271b5124329d4a11208d: Status 404 returned error can't find the container with id 2afefeb2ed0de0a1a24ebcf3082d143f7010860610f7271b5124329d4a11208d
Sep 11 17:39:54.732523 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:39:54.732468    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-2fac54-5796b4b6c7-c29rj" event={"ID":"c5842af6-6103-4759-951c-0fb4717aff70","Type":"ContainerStarted","Data":"e93da037698548f38d28624bad4e6de8af9dda61926acf27becf0099acf043af"}
Sep 11 17:39:54.732997 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:39:54.732540    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-2fac54-5796b4b6c7-c29rj"
Sep 11 17:39:54.732997 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:39:54.732557    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-2fac54-5796b4b6c7-c29rj" event={"ID":"c5842af6-6103-4759-951c-0fb4717aff70","Type":"ContainerStarted","Data":"2afefeb2ed0de0a1a24ebcf3082d143f7010860610f7271b5124329d4a11208d"}
Sep 11 17:39:54.752149 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:39:54.752082    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-2fac54-5796b4b6c7-c29rj" podStartSLOduration=1.7520623560000002 podStartE2EDuration="1.752062356s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 17:39:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 17:39:54.748996964 +0000 UTC m=+1272.322217089" watchObservedRunningTime="2026-09-11 17:39:54.752062356 +0000 UTC m=+1272.325282470"
Sep 11 17:40:05.741480 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:40:05.741447    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-2fac54-5796b4b6c7-c29rj"
Sep 11 17:40:20.809560 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:40:20.804027    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-nkmkm"]
Sep 11 17:40:27.365194 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:40:27.365158    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-6709f1a7-k2c8z"]
Sep 11 17:40:27.369023 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:40:27.369004    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-6709f1a7-k2c8z"
Sep 11 17:40:27.371590 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:40:27.371463    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 11 17:40:27.371590 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:40:27.371555    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-9n8gf\""
Sep 11 17:40:27.378205 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:40:27.378180    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-6709f1a7-k2c8z"]
Sep 11 17:40:27.390029 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:40:27.390002    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4ac1435b-32f6-48f4-939d-f6d8071a7635-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-6709f1a7-k2c8z\" (UID: \"4ac1435b-32f6-48f4-939d-f6d8071a7635\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-6709f1a7-k2c8z"
Sep 11 17:40:27.490631 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:40:27.490593    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4ac1435b-32f6-48f4-939d-f6d8071a7635-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-6709f1a7-k2c8z\" (UID: \"4ac1435b-32f6-48f4-939d-f6d8071a7635\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-6709f1a7-k2c8z"
Sep 11 17:40:27.491271 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:40:27.491248    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4ac1435b-32f6-48f4-939d-f6d8071a7635-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-6709f1a7-k2c8z\" (UID: \"4ac1435b-32f6-48f4-939d-f6d8071a7635\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-6709f1a7-k2c8z"
Sep 11 17:40:27.681692 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:40:27.681595    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-6709f1a7-k2c8z"
Sep 11 17:40:27.955517 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:40:27.955474    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-6709f1a7-k2c8z"]
Sep 11 17:40:27.957645 ip-10-0-141-139 kubenswrapper[2577]: W0911 17:40:27.957618    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4ac1435b_32f6_48f4_939d_f6d8071a7635.slice/crio-953e3a9913c18956039d1c3c809d5546172be3b41926fa5e50e93d1b4aa96091 WatchSource:0}: Error finding container 953e3a9913c18956039d1c3c809d5546172be3b41926fa5e50e93d1b4aa96091: Status 404 returned error can't find the container with id 953e3a9913c18956039d1c3c809d5546172be3b41926fa5e50e93d1b4aa96091
Sep 11 17:40:28.031850 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:40:28.031809    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-6fc9e6-6447f977cd-rrrsv"]
Sep 11 17:40:28.035690 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:40:28.035672    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-6fc9e6-6447f977cd-rrrsv"
Sep 11 17:40:28.038246 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:40:28.038224    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w3-6fc9e6-dockercfg-9pxzh\""
Sep 11 17:40:28.038246 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:40:28.038242    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w3-6fc9e6\""
Sep 11 17:40:28.049372 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:40:28.049344    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-6fc9e6-6447f977cd-rrrsv"]
Sep 11 17:40:28.095950 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:40:28.095915    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/931af6e1-ab7b-46e8-83ab-ae9329e32a8f-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w3-6fc9e6-6447f977cd-rrrsv\" (UID: \"931af6e1-ab7b-46e8-83ab-ae9329e32a8f\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-6fc9e6-6447f977cd-rrrsv"
Sep 11 17:40:28.096138 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:40:28.096033    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nd5ql\" (UniqueName: \"kubernetes.io/projected/931af6e1-ab7b-46e8-83ab-ae9329e32a8f-kube-api-access-nd5ql\") pod \"payload-pre-processing-e2e-shared-b-w3-6fc9e6-6447f977cd-rrrsv\" (UID: \"931af6e1-ab7b-46e8-83ab-ae9329e32a8f\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-6fc9e6-6447f977cd-rrrsv"
Sep 11 17:40:28.197049 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:40:28.197006    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nd5ql\" (UniqueName: \"kubernetes.io/projected/931af6e1-ab7b-46e8-83ab-ae9329e32a8f-kube-api-access-nd5ql\") pod \"payload-pre-processing-e2e-shared-b-w3-6fc9e6-6447f977cd-rrrsv\" (UID: \"931af6e1-ab7b-46e8-83ab-ae9329e32a8f\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-6fc9e6-6447f977cd-rrrsv"
Sep 11 17:40:28.197249 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:40:28.197120    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/931af6e1-ab7b-46e8-83ab-ae9329e32a8f-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w3-6fc9e6-6447f977cd-rrrsv\" (UID: \"931af6e1-ab7b-46e8-83ab-ae9329e32a8f\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-6fc9e6-6447f977cd-rrrsv"
Sep 11 17:40:28.197878 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:40:28.197857    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/931af6e1-ab7b-46e8-83ab-ae9329e32a8f-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w3-6fc9e6-6447f977cd-rrrsv\" (UID: \"931af6e1-ab7b-46e8-83ab-ae9329e32a8f\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-6fc9e6-6447f977cd-rrrsv"
Sep 11 17:40:28.205668 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:40:28.205605    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nd5ql\" (UniqueName: \"kubernetes.io/projected/931af6e1-ab7b-46e8-83ab-ae9329e32a8f-kube-api-access-nd5ql\") pod \"payload-pre-processing-e2e-shared-b-w3-6fc9e6-6447f977cd-rrrsv\" (UID: \"931af6e1-ab7b-46e8-83ab-ae9329e32a8f\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-6fc9e6-6447f977cd-rrrsv"
Sep 11 17:40:28.347818 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:40:28.347783    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-6fc9e6-6447f977cd-rrrsv"
Sep 11 17:40:28.634517 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:40:28.634478    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-6fc9e6-6447f977cd-rrrsv"]
Sep 11 17:40:28.636292 ip-10-0-141-139 kubenswrapper[2577]: W0911 17:40:28.636258    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod931af6e1_ab7b_46e8_83ab_ae9329e32a8f.slice/crio-45e126b5f30ffaea090a1a3bcecd04173f3417356b79fd5f4d838b9ac5e5f60e WatchSource:0}: Error finding container 45e126b5f30ffaea090a1a3bcecd04173f3417356b79fd5f4d838b9ac5e5f60e: Status 404 returned error can't find the container with id 45e126b5f30ffaea090a1a3bcecd04173f3417356b79fd5f4d838b9ac5e5f60e
Sep 11 17:40:28.926714 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:40:28.926625    2577 generic.go:356] "Generic (PLEG): container finished" podID="4ac1435b-32f6-48f4-939d-f6d8071a7635" containerID="8f4ddcd41fe11ee377d709f3667e2d49a4697d87da721384c9f559ce5b2f7228" exitCode=0
Sep 11 17:40:28.928465 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:40:28.928435    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-6fc9e6-6447f977cd-rrrsv" event={"ID":"931af6e1-ab7b-46e8-83ab-ae9329e32a8f","Type":"ContainerStarted","Data":"df965416545f4f31401bed82fc54e6ca8b2c5d095b5382ee2f1e6fc7f5f6c9a0"}
Sep 11 17:40:28.928613 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:40:28.928485    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-6fc9e6-6447f977cd-rrrsv"
Sep 11 17:40:28.928613 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:40:28.928515    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-6fc9e6-6447f977cd-rrrsv" event={"ID":"931af6e1-ab7b-46e8-83ab-ae9329e32a8f","Type":"ContainerStarted","Data":"45e126b5f30ffaea090a1a3bcecd04173f3417356b79fd5f4d838b9ac5e5f60e"}
Sep 11 17:40:28.928613 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:40:28.928536    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-6709f1a7-k2c8z" event={"ID":"4ac1435b-32f6-48f4-939d-f6d8071a7635","Type":"ContainerDied","Data":"8f4ddcd41fe11ee377d709f3667e2d49a4697d87da721384c9f559ce5b2f7228"}
Sep 11 17:40:28.928613 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:40:28.928558    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-6709f1a7-k2c8z" event={"ID":"4ac1435b-32f6-48f4-939d-f6d8071a7635","Type":"ContainerStarted","Data":"953e3a9913c18956039d1c3c809d5546172be3b41926fa5e50e93d1b4aa96091"}
Sep 11 17:40:28.945143 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:40:28.945093    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-6fc9e6-6447f977cd-rrrsv" podStartSLOduration=0.945076674 podStartE2EDuration="945.076674ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 17:40:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 17:40:28.94260365 +0000 UTC m=+1306.515823764" watchObservedRunningTime="2026-09-11 17:40:28.945076674 +0000 UTC m=+1306.518296810"
Sep 11 17:40:30.068518 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:40:30.068471    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-6709f1a7-k2c8z"
Sep 11 17:40:30.116554 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:40:30.116494    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4ac1435b-32f6-48f4-939d-f6d8071a7635-maas-api-service-ca\") pod \"4ac1435b-32f6-48f4-939d-f6d8071a7635\" (UID: \"4ac1435b-32f6-48f4-939d-f6d8071a7635\") "
Sep 11 17:40:30.116860 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:40:30.116836    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4ac1435b-32f6-48f4-939d-f6d8071a7635-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "4ac1435b-32f6-48f4-939d-f6d8071a7635" (UID: "4ac1435b-32f6-48f4-939d-f6d8071a7635"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 17:40:30.218009 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:40:30.217979    2577 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4ac1435b-32f6-48f4-939d-f6d8071a7635-maas-api-service-ca\") on node \"ip-10-0-141-139.ec2.internal\" DevicePath \"\""
Sep 11 17:40:30.940171 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:40:30.940147    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-6709f1a7-k2c8z"
Sep 11 17:40:30.940337 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:40:30.940180    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-6709f1a7-k2c8z" event={"ID":"4ac1435b-32f6-48f4-939d-f6d8071a7635","Type":"ContainerDied","Data":"953e3a9913c18956039d1c3c809d5546172be3b41926fa5e50e93d1b4aa96091"}
Sep 11 17:40:30.940337 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:40:30.940227    2577 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="953e3a9913c18956039d1c3c809d5546172be3b41926fa5e50e93d1b4aa96091"
Sep 11 17:40:34.410758 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:40:34.410718    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-144bfa/e2e-conflict-a-a3b46de5-kserve-6c7467f57f-xxm7v"]
Sep 11 17:40:40.942356 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:40:40.942322    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-6fc9e6-6447f977cd-rrrsv"
Sep 11 17:40:43.079684 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:40:43.079645    2577 scope.go:117] "RemoveContainer" containerID="5c5a1481a0517a06b2eed7d2e6fcaca271c9463fcd19cdd044e92a37b0410012"
Sep 11 17:40:43.295080 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:40:43.295045    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-6709f1a7-5dbf546cbf-jq9n7"]
Sep 11 17:40:43.295439 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:40:43.295373    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-mt-6709f1a7-5dbf546cbf-jq9n7" podUID="8b097356-9bdb-4504-9c22-4982c9ecaca8" containerName="payload-processing" containerID="cri-o://8c8a5643c23b5bbb714ce973c849047c7774f497ee27e37ce7000c4f3c258851" gracePeriod=30
Sep 11 17:40:43.515295 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:40:43.515264    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-6709f1a7-k2c8z"]
Sep 11 17:40:43.519012 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:40:43.518980    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-6709f1a7-k2c8z"]
Sep 11 17:40:43.552427 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:40:43.552402    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-6709f1a7-5dbf546cbf-jq9n7"
Sep 11 17:40:43.656464 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:40:43.656416    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4w79s\" (UniqueName: \"kubernetes.io/projected/8b097356-9bdb-4504-9c22-4982c9ecaca8-kube-api-access-4w79s\") pod \"8b097356-9bdb-4504-9c22-4982c9ecaca8\" (UID: \"8b097356-9bdb-4504-9c22-4982c9ecaca8\") "
Sep 11 17:40:43.656670 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:40:43.656618    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8b097356-9bdb-4504-9c22-4982c9ecaca8-plugins-config-volume\") pod \"8b097356-9bdb-4504-9c22-4982c9ecaca8\" (UID: \"8b097356-9bdb-4504-9c22-4982c9ecaca8\") "
Sep 11 17:40:43.656999 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:40:43.656969    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8b097356-9bdb-4504-9c22-4982c9ecaca8-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "8b097356-9bdb-4504-9c22-4982c9ecaca8" (UID: "8b097356-9bdb-4504-9c22-4982c9ecaca8"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 17:40:43.658720 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:40:43.658693    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8b097356-9bdb-4504-9c22-4982c9ecaca8-kube-api-access-4w79s" (OuterVolumeSpecName: "kube-api-access-4w79s") pod "8b097356-9bdb-4504-9c22-4982c9ecaca8" (UID: "8b097356-9bdb-4504-9c22-4982c9ecaca8"). InnerVolumeSpecName "kube-api-access-4w79s". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 17:40:43.757928 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:40:43.757876    2577 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8b097356-9bdb-4504-9c22-4982c9ecaca8-plugins-config-volume\") on node \"ip-10-0-141-139.ec2.internal\" DevicePath \"\""
Sep 11 17:40:43.757928 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:40:43.757920    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4w79s\" (UniqueName: \"kubernetes.io/projected/8b097356-9bdb-4504-9c22-4982c9ecaca8-kube-api-access-4w79s\") on node \"ip-10-0-141-139.ec2.internal\" DevicePath \"\""
Sep 11 17:40:44.018833 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:40:44.018802    2577 generic.go:356] "Generic (PLEG): container finished" podID="8b097356-9bdb-4504-9c22-4982c9ecaca8" containerID="8c8a5643c23b5bbb714ce973c849047c7774f497ee27e37ce7000c4f3c258851" exitCode=0
Sep 11 17:40:44.019019 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:40:44.018846    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-6709f1a7-5dbf546cbf-jq9n7"
Sep 11 17:40:44.019019 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:40:44.018849    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-6709f1a7-5dbf546cbf-jq9n7" event={"ID":"8b097356-9bdb-4504-9c22-4982c9ecaca8","Type":"ContainerDied","Data":"8c8a5643c23b5bbb714ce973c849047c7774f497ee27e37ce7000c4f3c258851"}
Sep 11 17:40:44.019019 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:40:44.018965    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-6709f1a7-5dbf546cbf-jq9n7" event={"ID":"8b097356-9bdb-4504-9c22-4982c9ecaca8","Type":"ContainerDied","Data":"30a7d1d60aae45c1416e66819c7c07e4f093484bc8f81df00805fe7226889c65"}
Sep 11 17:40:44.019019 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:40:44.018987    2577 scope.go:117] "RemoveContainer" containerID="8c8a5643c23b5bbb714ce973c849047c7774f497ee27e37ce7000c4f3c258851"
Sep 11 17:40:44.031890 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:40:44.031815    2577 scope.go:117] "RemoveContainer" containerID="8c8a5643c23b5bbb714ce973c849047c7774f497ee27e37ce7000c4f3c258851"
Sep 11 17:40:44.032511 ip-10-0-141-139 kubenswrapper[2577]: E0911 17:40:44.032481    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8c8a5643c23b5bbb714ce973c849047c7774f497ee27e37ce7000c4f3c258851\": container with ID starting with 8c8a5643c23b5bbb714ce973c849047c7774f497ee27e37ce7000c4f3c258851 not found: ID does not exist" containerID="8c8a5643c23b5bbb714ce973c849047c7774f497ee27e37ce7000c4f3c258851"
Sep 11 17:40:44.032640 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:40:44.032539    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8c8a5643c23b5bbb714ce973c849047c7774f497ee27e37ce7000c4f3c258851"} err="failed to get container status \"8c8a5643c23b5bbb714ce973c849047c7774f497ee27e37ce7000c4f3c258851\": rpc error: code = NotFound desc = could not find container \"8c8a5643c23b5bbb714ce973c849047c7774f497ee27e37ce7000c4f3c258851\": container with ID starting with 8c8a5643c23b5bbb714ce973c849047c7774f497ee27e37ce7000c4f3c258851 not found: ID does not exist"
Sep 11 17:40:44.042482 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:40:44.042454    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-6709f1a7-5dbf546cbf-jq9n7"]
Sep 11 17:40:44.051921 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:40:44.051896    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-6709f1a7-5dbf546cbf-jq9n7"]
Sep 11 17:40:44.936295 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:40:44.936249    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4ac1435b-32f6-48f4-939d-f6d8071a7635" path="/var/lib/kubelet/pods/4ac1435b-32f6-48f4-939d-f6d8071a7635/volumes"
Sep 11 17:40:44.936789 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:40:44.936755    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8b097356-9bdb-4504-9c22-4982c9ecaca8" path="/var/lib/kubelet/pods/8b097356-9bdb-4504-9c22-4982c9ecaca8/volumes"
Sep 11 17:40:49.433536 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:40:49.433425    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-144bfa/e2e-conflict-a-a3b46de5-kserve-6c7467f57f-xxm7v" podUID="1c2898b6-c293-4ed2-b759-19b104df5f39" containerName="main" containerID="cri-o://8f7e4ec76358c3f77eda16db7ab5de402e785b935ba22f8a7e7a028073f29e65" gracePeriod=45
Sep 11 17:40:49.694315 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:40:49.694228    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-144bfa/distinct-w4-144bfa-kserve-648d4c7f9c-m2562"]
Sep 11 17:40:49.695012 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:40:49.694985    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8b097356-9bdb-4504-9c22-4982c9ecaca8" containerName="payload-processing"
Sep 11 17:40:49.695012 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:40:49.695014    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="8b097356-9bdb-4504-9c22-4982c9ecaca8" containerName="payload-processing"
Sep 11 17:40:49.695201 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:40:49.695035    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4ac1435b-32f6-48f4-939d-f6d8071a7635" containerName="revoke-keys"
Sep 11 17:40:49.695201 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:40:49.695044    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="4ac1435b-32f6-48f4-939d-f6d8071a7635" containerName="revoke-keys"
Sep 11 17:40:49.695201 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:40:49.695187    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="4ac1435b-32f6-48f4-939d-f6d8071a7635" containerName="revoke-keys"
Sep 11 17:40:49.695370 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:40:49.695204    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="8b097356-9bdb-4504-9c22-4982c9ecaca8" containerName="payload-processing"
Sep 11 17:40:49.701296 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:40:49.698858    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-144bfa/distinct-w4-144bfa-kserve-648d4c7f9c-m2562"
Sep 11 17:40:49.701667 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:40:49.701647    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-144bfa\"/\"distinct-w4-144bfa-kserve-self-signed-certs\""
Sep 11 17:40:49.711988 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:40:49.710338    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-144bfa/distinct-w4-144bfa-kserve-648d4c7f9c-m2562"]
Sep 11 17:40:49.766851 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:40:49.766823    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-144bfa/e2e-conflict-a-a3b46de5-kserve-6c7467f57f-xxm7v"
Sep 11 17:40:49.832714 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:40:49.832663    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/44bbc11d-62da-41bb-90e3-37c721b01e2f-model-cache\") pod \"distinct-w4-144bfa-kserve-648d4c7f9c-m2562\" (UID: \"44bbc11d-62da-41bb-90e3-37c721b01e2f\") " pod="e2e-models-e2e-worker-w4-144bfa/distinct-w4-144bfa-kserve-648d4c7f9c-m2562"
Sep 11 17:40:49.832714 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:40:49.832718    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/44bbc11d-62da-41bb-90e3-37c721b01e2f-home\") pod \"distinct-w4-144bfa-kserve-648d4c7f9c-m2562\" (UID: \"44bbc11d-62da-41bb-90e3-37c721b01e2f\") " pod="e2e-models-e2e-worker-w4-144bfa/distinct-w4-144bfa-kserve-648d4c7f9c-m2562"
Sep 11 17:40:49.832929 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:40:49.832769    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/44bbc11d-62da-41bb-90e3-37c721b01e2f-dshm\") pod \"distinct-w4-144bfa-kserve-648d4c7f9c-m2562\" (UID: \"44bbc11d-62da-41bb-90e3-37c721b01e2f\") " pod="e2e-models-e2e-worker-w4-144bfa/distinct-w4-144bfa-kserve-648d4c7f9c-m2562"
Sep 11 17:40:49.832929 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:40:49.832798    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/44bbc11d-62da-41bb-90e3-37c721b01e2f-tmp-dir\") pod \"distinct-w4-144bfa-kserve-648d4c7f9c-m2562\" (UID: \"44bbc11d-62da-41bb-90e3-37c721b01e2f\") " pod="e2e-models-e2e-worker-w4-144bfa/distinct-w4-144bfa-kserve-648d4c7f9c-m2562"
Sep 11 17:40:49.832929 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:40:49.832837    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/44bbc11d-62da-41bb-90e3-37c721b01e2f-tls-certs\") pod \"distinct-w4-144bfa-kserve-648d4c7f9c-m2562\" (UID: \"44bbc11d-62da-41bb-90e3-37c721b01e2f\") " pod="e2e-models-e2e-worker-w4-144bfa/distinct-w4-144bfa-kserve-648d4c7f9c-m2562"
Sep 11 17:40:49.832929 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:40:49.832872    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r6vbp\" (UniqueName: \"kubernetes.io/projected/44bbc11d-62da-41bb-90e3-37c721b01e2f-kube-api-access-r6vbp\") pod \"distinct-w4-144bfa-kserve-648d4c7f9c-m2562\" (UID: \"44bbc11d-62da-41bb-90e3-37c721b01e2f\") " pod="e2e-models-e2e-worker-w4-144bfa/distinct-w4-144bfa-kserve-648d4c7f9c-m2562"
Sep 11 17:40:49.934455 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:40:49.934424    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1c2898b6-c293-4ed2-b759-19b104df5f39-tls-certs\") pod \"1c2898b6-c293-4ed2-b759-19b104df5f39\" (UID: \"1c2898b6-c293-4ed2-b759-19b104df5f39\") "
Sep 11 17:40:49.934633 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:40:49.934467    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1c2898b6-c293-4ed2-b759-19b104df5f39-tmp-dir\") pod \"1c2898b6-c293-4ed2-b759-19b104df5f39\" (UID: \"1c2898b6-c293-4ed2-b759-19b104df5f39\") "
Sep 11 17:40:49.934633 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:40:49.934521    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1c2898b6-c293-4ed2-b759-19b104df5f39-model-cache\") pod \"1c2898b6-c293-4ed2-b759-19b104df5f39\" (UID: \"1c2898b6-c293-4ed2-b759-19b104df5f39\") "
Sep 11 17:40:49.934633 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:40:49.934550    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-sgjvr\" (UniqueName: \"kubernetes.io/projected/1c2898b6-c293-4ed2-b759-19b104df5f39-kube-api-access-sgjvr\") pod \"1c2898b6-c293-4ed2-b759-19b104df5f39\" (UID: \"1c2898b6-c293-4ed2-b759-19b104df5f39\") "
Sep 11 17:40:49.934633 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:40:49.934583    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1c2898b6-c293-4ed2-b759-19b104df5f39-dshm\") pod \"1c2898b6-c293-4ed2-b759-19b104df5f39\" (UID: \"1c2898b6-c293-4ed2-b759-19b104df5f39\") "
Sep 11 17:40:49.934633 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:40:49.934626    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1c2898b6-c293-4ed2-b759-19b104df5f39-home\") pod \"1c2898b6-c293-4ed2-b759-19b104df5f39\" (UID: \"1c2898b6-c293-4ed2-b759-19b104df5f39\") "
Sep 11 17:40:49.934934 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:40:49.934717    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1c2898b6-c293-4ed2-b759-19b104df5f39-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "1c2898b6-c293-4ed2-b759-19b104df5f39" (UID: "1c2898b6-c293-4ed2-b759-19b104df5f39"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 17:40:49.934934 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:40:49.934785    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1c2898b6-c293-4ed2-b759-19b104df5f39-model-cache" (OuterVolumeSpecName: "model-cache") pod "1c2898b6-c293-4ed2-b759-19b104df5f39" (UID: "1c2898b6-c293-4ed2-b759-19b104df5f39"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 17:40:49.934934 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:40:49.934891    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/44bbc11d-62da-41bb-90e3-37c721b01e2f-dshm\") pod \"distinct-w4-144bfa-kserve-648d4c7f9c-m2562\" (UID: \"44bbc11d-62da-41bb-90e3-37c721b01e2f\") " pod="e2e-models-e2e-worker-w4-144bfa/distinct-w4-144bfa-kserve-648d4c7f9c-m2562"
Sep 11 17:40:49.935102 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:40:49.934935    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/44bbc11d-62da-41bb-90e3-37c721b01e2f-tmp-dir\") pod \"distinct-w4-144bfa-kserve-648d4c7f9c-m2562\" (UID: \"44bbc11d-62da-41bb-90e3-37c721b01e2f\") " pod="e2e-models-e2e-worker-w4-144bfa/distinct-w4-144bfa-kserve-648d4c7f9c-m2562"
Sep 11 17:40:49.935102 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:40:49.934957    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1c2898b6-c293-4ed2-b759-19b104df5f39-home" (OuterVolumeSpecName: "home") pod "1c2898b6-c293-4ed2-b759-19b104df5f39" (UID: "1c2898b6-c293-4ed2-b759-19b104df5f39"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 17:40:49.935102 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:40:49.934976    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/44bbc11d-62da-41bb-90e3-37c721b01e2f-tls-certs\") pod \"distinct-w4-144bfa-kserve-648d4c7f9c-m2562\" (UID: \"44bbc11d-62da-41bb-90e3-37c721b01e2f\") " pod="e2e-models-e2e-worker-w4-144bfa/distinct-w4-144bfa-kserve-648d4c7f9c-m2562"
Sep 11 17:40:49.935102 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:40:49.935018    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r6vbp\" (UniqueName: \"kubernetes.io/projected/44bbc11d-62da-41bb-90e3-37c721b01e2f-kube-api-access-r6vbp\") pod \"distinct-w4-144bfa-kserve-648d4c7f9c-m2562\" (UID: \"44bbc11d-62da-41bb-90e3-37c721b01e2f\") " pod="e2e-models-e2e-worker-w4-144bfa/distinct-w4-144bfa-kserve-648d4c7f9c-m2562"
Sep 11 17:40:49.935357 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:40:49.935331    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/44bbc11d-62da-41bb-90e3-37c721b01e2f-tmp-dir\") pod \"distinct-w4-144bfa-kserve-648d4c7f9c-m2562\" (UID: \"44bbc11d-62da-41bb-90e3-37c721b01e2f\") " pod="e2e-models-e2e-worker-w4-144bfa/distinct-w4-144bfa-kserve-648d4c7f9c-m2562"
Sep 11 17:40:49.935435 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:40:49.935380    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/44bbc11d-62da-41bb-90e3-37c721b01e2f-model-cache\") pod \"distinct-w4-144bfa-kserve-648d4c7f9c-m2562\" (UID: \"44bbc11d-62da-41bb-90e3-37c721b01e2f\") " pod="e2e-models-e2e-worker-w4-144bfa/distinct-w4-144bfa-kserve-648d4c7f9c-m2562"
Sep 11 17:40:49.935591 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:40:49.935567    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/44bbc11d-62da-41bb-90e3-37c721b01e2f-home\") pod \"distinct-w4-144bfa-kserve-648d4c7f9c-m2562\" (UID: \"44bbc11d-62da-41bb-90e3-37c721b01e2f\") " pod="e2e-models-e2e-worker-w4-144bfa/distinct-w4-144bfa-kserve-648d4c7f9c-m2562"
Sep 11 17:40:49.935707 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:40:49.935691    2577 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1c2898b6-c293-4ed2-b759-19b104df5f39-home\") on node \"ip-10-0-141-139.ec2.internal\" DevicePath \"\""
Sep 11 17:40:49.935779 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:40:49.935714    2577 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1c2898b6-c293-4ed2-b759-19b104df5f39-tmp-dir\") on node \"ip-10-0-141-139.ec2.internal\" DevicePath \"\""
Sep 11 17:40:49.935779 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:40:49.935729    2577 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1c2898b6-c293-4ed2-b759-19b104df5f39-model-cache\") on node \"ip-10-0-141-139.ec2.internal\" DevicePath \"\""
Sep 11 17:40:49.935884 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:40:49.935834    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/44bbc11d-62da-41bb-90e3-37c721b01e2f-model-cache\") pod \"distinct-w4-144bfa-kserve-648d4c7f9c-m2562\" (UID: \"44bbc11d-62da-41bb-90e3-37c721b01e2f\") " pod="e2e-models-e2e-worker-w4-144bfa/distinct-w4-144bfa-kserve-648d4c7f9c-m2562"
Sep 11 17:40:49.936019 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:40:49.935994    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/44bbc11d-62da-41bb-90e3-37c721b01e2f-home\") pod \"distinct-w4-144bfa-kserve-648d4c7f9c-m2562\" (UID: \"44bbc11d-62da-41bb-90e3-37c721b01e2f\") " pod="e2e-models-e2e-worker-w4-144bfa/distinct-w4-144bfa-kserve-648d4c7f9c-m2562"
Sep 11 17:40:49.938400 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:40:49.938370    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1c2898b6-c293-4ed2-b759-19b104df5f39-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "1c2898b6-c293-4ed2-b759-19b104df5f39" (UID: "1c2898b6-c293-4ed2-b759-19b104df5f39"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 17:40:49.938400 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:40:49.938389    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1c2898b6-c293-4ed2-b759-19b104df5f39-kube-api-access-sgjvr" (OuterVolumeSpecName: "kube-api-access-sgjvr") pod "1c2898b6-c293-4ed2-b759-19b104df5f39" (UID: "1c2898b6-c293-4ed2-b759-19b104df5f39"). InnerVolumeSpecName "kube-api-access-sgjvr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 17:40:49.938583 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:40:49.938455    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/44bbc11d-62da-41bb-90e3-37c721b01e2f-dshm\") pod \"distinct-w4-144bfa-kserve-648d4c7f9c-m2562\" (UID: \"44bbc11d-62da-41bb-90e3-37c721b01e2f\") " pod="e2e-models-e2e-worker-w4-144bfa/distinct-w4-144bfa-kserve-648d4c7f9c-m2562"
Sep 11 17:40:49.938583 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:40:49.938482    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1c2898b6-c293-4ed2-b759-19b104df5f39-dshm" (OuterVolumeSpecName: "dshm") pod "1c2898b6-c293-4ed2-b759-19b104df5f39" (UID: "1c2898b6-c293-4ed2-b759-19b104df5f39"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 17:40:49.938720 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:40:49.938688    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/44bbc11d-62da-41bb-90e3-37c721b01e2f-tls-certs\") pod \"distinct-w4-144bfa-kserve-648d4c7f9c-m2562\" (UID: \"44bbc11d-62da-41bb-90e3-37c721b01e2f\") " pod="e2e-models-e2e-worker-w4-144bfa/distinct-w4-144bfa-kserve-648d4c7f9c-m2562"
Sep 11 17:40:49.954465 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:40:49.951967    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-r6vbp\" (UniqueName: \"kubernetes.io/projected/44bbc11d-62da-41bb-90e3-37c721b01e2f-kube-api-access-r6vbp\") pod \"distinct-w4-144bfa-kserve-648d4c7f9c-m2562\" (UID: \"44bbc11d-62da-41bb-90e3-37c721b01e2f\") " pod="e2e-models-e2e-worker-w4-144bfa/distinct-w4-144bfa-kserve-648d4c7f9c-m2562"
Sep 11 17:40:50.017583 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:40:50.017453    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-144bfa/distinct-w4-144bfa-kserve-648d4c7f9c-m2562"
Sep 11 17:40:50.037430 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:40:50.037405    2577 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1c2898b6-c293-4ed2-b759-19b104df5f39-dshm\") on node \"ip-10-0-141-139.ec2.internal\" DevicePath \"\""
Sep 11 17:40:50.037549 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:40:50.037434    2577 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1c2898b6-c293-4ed2-b759-19b104df5f39-tls-certs\") on node \"ip-10-0-141-139.ec2.internal\" DevicePath \"\""
Sep 11 17:40:50.037549 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:40:50.037445    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sgjvr\" (UniqueName: \"kubernetes.io/projected/1c2898b6-c293-4ed2-b759-19b104df5f39-kube-api-access-sgjvr\") on node \"ip-10-0-141-139.ec2.internal\" DevicePath \"\""
Sep 11 17:40:50.067156 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:40:50.067051    2577 generic.go:356] "Generic (PLEG): container finished" podID="1c2898b6-c293-4ed2-b759-19b104df5f39" containerID="8f7e4ec76358c3f77eda16db7ab5de402e785b935ba22f8a7e7a028073f29e65" exitCode=0
Sep 11 17:40:50.067364 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:40:50.067254    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-144bfa/e2e-conflict-a-a3b46de5-kserve-6c7467f57f-xxm7v" event={"ID":"1c2898b6-c293-4ed2-b759-19b104df5f39","Type":"ContainerDied","Data":"8f7e4ec76358c3f77eda16db7ab5de402e785b935ba22f8a7e7a028073f29e65"}
Sep 11 17:40:50.067364 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:40:50.067292    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-144bfa/e2e-conflict-a-a3b46de5-kserve-6c7467f57f-xxm7v" event={"ID":"1c2898b6-c293-4ed2-b759-19b104df5f39","Type":"ContainerDied","Data":"d3aed22a82bb01ef846116299539d76f3c9bfce2ce0e103210f347ddf01dc728"}
Sep 11 17:40:50.067364 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:40:50.067313    2577 scope.go:117] "RemoveContainer" containerID="8f7e4ec76358c3f77eda16db7ab5de402e785b935ba22f8a7e7a028073f29e65"
Sep 11 17:40:50.067565 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:40:50.067485    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-144bfa/e2e-conflict-a-a3b46de5-kserve-6c7467f57f-xxm7v"
Sep 11 17:40:50.101051 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:40:50.094753    2577 scope.go:117] "RemoveContainer" containerID="8f7e4ec76358c3f77eda16db7ab5de402e785b935ba22f8a7e7a028073f29e65"
Sep 11 17:40:50.101051 ip-10-0-141-139 kubenswrapper[2577]: E0911 17:40:50.095184    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8f7e4ec76358c3f77eda16db7ab5de402e785b935ba22f8a7e7a028073f29e65\": container with ID starting with 8f7e4ec76358c3f77eda16db7ab5de402e785b935ba22f8a7e7a028073f29e65 not found: ID does not exist" containerID="8f7e4ec76358c3f77eda16db7ab5de402e785b935ba22f8a7e7a028073f29e65"
Sep 11 17:40:50.101051 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:40:50.095223    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8f7e4ec76358c3f77eda16db7ab5de402e785b935ba22f8a7e7a028073f29e65"} err="failed to get container status \"8f7e4ec76358c3f77eda16db7ab5de402e785b935ba22f8a7e7a028073f29e65\": rpc error: code = NotFound desc = could not find container \"8f7e4ec76358c3f77eda16db7ab5de402e785b935ba22f8a7e7a028073f29e65\": container with ID starting with 8f7e4ec76358c3f77eda16db7ab5de402e785b935ba22f8a7e7a028073f29e65 not found: ID does not exist"
Sep 11 17:40:50.118760 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:40:50.118722    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-144bfa/e2e-conflict-a-a3b46de5-kserve-6c7467f57f-xxm7v"]
Sep 11 17:40:50.126451 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:40:50.121717    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-144bfa/e2e-conflict-a-a3b46de5-kserve-6c7467f57f-xxm7v"]
Sep 11 17:40:50.329805 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:40:50.329779    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-144bfa/distinct-w4-144bfa-kserve-648d4c7f9c-m2562"]
Sep 11 17:40:50.333143 ip-10-0-141-139 kubenswrapper[2577]: W0911 17:40:50.333112    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod44bbc11d_62da_41bb_90e3_37c721b01e2f.slice/crio-4c25a1faa0ed80229963cbffaef70e2382b6c62a9e538e7b20df64e14431cf67 WatchSource:0}: Error finding container 4c25a1faa0ed80229963cbffaef70e2382b6c62a9e538e7b20df64e14431cf67: Status 404 returned error can't find the container with id 4c25a1faa0ed80229963cbffaef70e2382b6c62a9e538e7b20df64e14431cf67
Sep 11 17:40:50.930721 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:40:50.930678    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1c2898b6-c293-4ed2-b759-19b104df5f39" path="/var/lib/kubelet/pods/1c2898b6-c293-4ed2-b759-19b104df5f39/volumes"
Sep 11 17:40:51.081689 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:40:51.081643    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-144bfa/distinct-w4-144bfa-kserve-648d4c7f9c-m2562" event={"ID":"44bbc11d-62da-41bb-90e3-37c721b01e2f","Type":"ContainerStarted","Data":"fe6a7652f020a1ab2148e020c86e0f136ac1d4b4b815d4770a965c718c3d7ae0"}
Sep 11 17:40:51.081904 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:40:51.081705    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-144bfa/distinct-w4-144bfa-kserve-648d4c7f9c-m2562" event={"ID":"44bbc11d-62da-41bb-90e3-37c721b01e2f","Type":"ContainerStarted","Data":"4c25a1faa0ed80229963cbffaef70e2382b6c62a9e538e7b20df64e14431cf67"}
Sep 11 17:40:51.101658 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:40:51.101599    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-144bfa/distinct-w4-144bfa-kserve-648d4c7f9c-m2562" podStartSLOduration=2.101579237 podStartE2EDuration="2.101579237s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 17:40:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 17:40:51.10064559 +0000 UTC m=+1328.673865714" watchObservedRunningTime="2026-09-11 17:40:51.101579237 +0000 UTC m=+1328.674799350"
Sep 11 17:40:59.002405 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:40:59.002364    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-nkmkm"]
Sep 11 17:41:00.018103 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:41:00.018062    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-144bfa/distinct-w4-144bfa-kserve-648d4c7f9c-m2562"
Sep 11 17:41:00.018103 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:41:00.018114    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-144bfa/distinct-w4-144bfa-kserve-648d4c7f9c-m2562"
Sep 11 17:41:00.030797 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:41:00.030777    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-144bfa/distinct-w4-144bfa-kserve-648d4c7f9c-m2562"
Sep 11 17:41:00.152255 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:41:00.152225    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-144bfa/distinct-w4-144bfa-kserve-648d4c7f9c-m2562"
Sep 11 17:41:38.513338 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:41:38.513300    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-nkmkm"]
Sep 11 17:42:14.516534 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:42:14.516484    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-nkmkm"]
Sep 11 17:42:40.521289 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:42:40.521202    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-15f6f2-5db9d8d8b7-pm48s"]
Sep 11 17:42:40.522210 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:42:40.521729    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1c2898b6-c293-4ed2-b759-19b104df5f39" containerName="main"
Sep 11 17:42:40.522210 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:42:40.521744    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="1c2898b6-c293-4ed2-b759-19b104df5f39" containerName="main"
Sep 11 17:42:40.522210 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:42:40.521836    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="1c2898b6-c293-4ed2-b759-19b104df5f39" containerName="main"
Sep 11 17:42:40.524990 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:42:40.524970    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-15f6f2-5db9d8d8b7-pm48s"
Sep 11 17:42:40.527601 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:42:40.527578    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-a-15f6f2-dockercfg-5mbtm\""
Sep 11 17:42:40.527726 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:42:40.527656    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-a-15f6f2\""
Sep 11 17:42:40.539239 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:42:40.539205    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-15f6f2-5db9d8d8b7-pm48s"]
Sep 11 17:42:40.562064 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:42:40.562031    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-15f6f2-77875d44c4-zf4pp"]
Sep 11 17:42:40.565914 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:42:40.565896    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-15f6f2-77875d44c4-zf4pp"
Sep 11 17:42:40.575252 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:42:40.575229    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-15f6f2-77875d44c4-zf4pp"]
Sep 11 17:42:40.605631 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:42:40.605596    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hc7tk\" (UniqueName: \"kubernetes.io/projected/dfb94eb0-7d89-4c81-80a2-2a11116fdd1b-kube-api-access-hc7tk\") pod \"payload-processing-e2e-api-a-15f6f2-5db9d8d8b7-pm48s\" (UID: \"dfb94eb0-7d89-4c81-80a2-2a11116fdd1b\") " pod="openshift-ingress/payload-processing-e2e-api-a-15f6f2-5db9d8d8b7-pm48s"
Sep 11 17:42:40.605831 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:42:40.605666    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6a29c245-ea49-4787-9405-d58d6587f401-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-15f6f2-77875d44c4-zf4pp\" (UID: \"6a29c245-ea49-4787-9405-d58d6587f401\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-15f6f2-77875d44c4-zf4pp"
Sep 11 17:42:40.605831 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:42:40.605698    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dfb94eb0-7d89-4c81-80a2-2a11116fdd1b-plugins-config-volume\") pod \"payload-processing-e2e-api-a-15f6f2-5db9d8d8b7-pm48s\" (UID: \"dfb94eb0-7d89-4c81-80a2-2a11116fdd1b\") " pod="openshift-ingress/payload-processing-e2e-api-a-15f6f2-5db9d8d8b7-pm48s"
Sep 11 17:42:40.605831 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:42:40.605746    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rhq2x\" (UniqueName: \"kubernetes.io/projected/6a29c245-ea49-4787-9405-d58d6587f401-kube-api-access-rhq2x\") pod \"payload-pre-processing-e2e-api-a-15f6f2-77875d44c4-zf4pp\" (UID: \"6a29c245-ea49-4787-9405-d58d6587f401\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-15f6f2-77875d44c4-zf4pp"
Sep 11 17:42:40.707098 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:42:40.707057    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rhq2x\" (UniqueName: \"kubernetes.io/projected/6a29c245-ea49-4787-9405-d58d6587f401-kube-api-access-rhq2x\") pod \"payload-pre-processing-e2e-api-a-15f6f2-77875d44c4-zf4pp\" (UID: \"6a29c245-ea49-4787-9405-d58d6587f401\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-15f6f2-77875d44c4-zf4pp"
Sep 11 17:42:40.707325 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:42:40.707179    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hc7tk\" (UniqueName: \"kubernetes.io/projected/dfb94eb0-7d89-4c81-80a2-2a11116fdd1b-kube-api-access-hc7tk\") pod \"payload-processing-e2e-api-a-15f6f2-5db9d8d8b7-pm48s\" (UID: \"dfb94eb0-7d89-4c81-80a2-2a11116fdd1b\") " pod="openshift-ingress/payload-processing-e2e-api-a-15f6f2-5db9d8d8b7-pm48s"
Sep 11 17:42:40.707325 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:42:40.707263    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6a29c245-ea49-4787-9405-d58d6587f401-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-15f6f2-77875d44c4-zf4pp\" (UID: \"6a29c245-ea49-4787-9405-d58d6587f401\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-15f6f2-77875d44c4-zf4pp"
Sep 11 17:42:40.707467 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:42:40.707430    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dfb94eb0-7d89-4c81-80a2-2a11116fdd1b-plugins-config-volume\") pod \"payload-processing-e2e-api-a-15f6f2-5db9d8d8b7-pm48s\" (UID: \"dfb94eb0-7d89-4c81-80a2-2a11116fdd1b\") " pod="openshift-ingress/payload-processing-e2e-api-a-15f6f2-5db9d8d8b7-pm48s"
Sep 11 17:42:40.708124 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:42:40.708104    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dfb94eb0-7d89-4c81-80a2-2a11116fdd1b-plugins-config-volume\") pod \"payload-processing-e2e-api-a-15f6f2-5db9d8d8b7-pm48s\" (UID: \"dfb94eb0-7d89-4c81-80a2-2a11116fdd1b\") " pod="openshift-ingress/payload-processing-e2e-api-a-15f6f2-5db9d8d8b7-pm48s"
Sep 11 17:42:40.708214 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:42:40.708181    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6a29c245-ea49-4787-9405-d58d6587f401-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-15f6f2-77875d44c4-zf4pp\" (UID: \"6a29c245-ea49-4787-9405-d58d6587f401\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-15f6f2-77875d44c4-zf4pp"
Sep 11 17:42:40.716478 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:42:40.716445    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rhq2x\" (UniqueName: \"kubernetes.io/projected/6a29c245-ea49-4787-9405-d58d6587f401-kube-api-access-rhq2x\") pod \"payload-pre-processing-e2e-api-a-15f6f2-77875d44c4-zf4pp\" (UID: \"6a29c245-ea49-4787-9405-d58d6587f401\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-15f6f2-77875d44c4-zf4pp"
Sep 11 17:42:40.717980 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:42:40.717959    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hc7tk\" (UniqueName: \"kubernetes.io/projected/dfb94eb0-7d89-4c81-80a2-2a11116fdd1b-kube-api-access-hc7tk\") pod \"payload-processing-e2e-api-a-15f6f2-5db9d8d8b7-pm48s\" (UID: \"dfb94eb0-7d89-4c81-80a2-2a11116fdd1b\") " pod="openshift-ingress/payload-processing-e2e-api-a-15f6f2-5db9d8d8b7-pm48s"
Sep 11 17:42:40.840000 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:42:40.839911    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-15f6f2-5db9d8d8b7-pm48s"
Sep 11 17:42:40.877479 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:42:40.877445    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-15f6f2-77875d44c4-zf4pp"
Sep 11 17:42:41.119876 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:42:41.119845    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-15f6f2-5db9d8d8b7-pm48s"]
Sep 11 17:42:41.121423 ip-10-0-141-139 kubenswrapper[2577]: W0911 17:42:41.121396    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddfb94eb0_7d89_4c81_80a2_2a11116fdd1b.slice/crio-9ba618c0c6f65130210fa01b8234e4fbe9b974f5b58462a1659f2d094785e19f WatchSource:0}: Error finding container 9ba618c0c6f65130210fa01b8234e4fbe9b974f5b58462a1659f2d094785e19f: Status 404 returned error can't find the container with id 9ba618c0c6f65130210fa01b8234e4fbe9b974f5b58462a1659f2d094785e19f
Sep 11 17:42:41.168485 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:42:41.168457    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-15f6f2-77875d44c4-zf4pp"]
Sep 11 17:42:41.182879 ip-10-0-141-139 kubenswrapper[2577]: W0911 17:42:41.182853    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6a29c245_ea49_4787_9405_d58d6587f401.slice/crio-1c72ac5813b212a7c024f298aeae1e0ecabb3c814c58ad6d2fb96fe380fab8ab WatchSource:0}: Error finding container 1c72ac5813b212a7c024f298aeae1e0ecabb3c814c58ad6d2fb96fe380fab8ab: Status 404 returned error can't find the container with id 1c72ac5813b212a7c024f298aeae1e0ecabb3c814c58ad6d2fb96fe380fab8ab
Sep 11 17:42:41.720521 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:42:41.720469    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-15f6f2-77875d44c4-zf4pp" event={"ID":"6a29c245-ea49-4787-9405-d58d6587f401","Type":"ContainerStarted","Data":"afe9f4d9c44dd95fda722bdfe27232b93f5e25d657cfc3d79dfdecdced2a17be"}
Sep 11 17:42:41.720980 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:42:41.720560    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-15f6f2-77875d44c4-zf4pp"
Sep 11 17:42:41.720980 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:42:41.720586    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-15f6f2-77875d44c4-zf4pp" event={"ID":"6a29c245-ea49-4787-9405-d58d6587f401","Type":"ContainerStarted","Data":"1c72ac5813b212a7c024f298aeae1e0ecabb3c814c58ad6d2fb96fe380fab8ab"}
Sep 11 17:42:41.722210 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:42:41.722184    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-15f6f2-5db9d8d8b7-pm48s" event={"ID":"dfb94eb0-7d89-4c81-80a2-2a11116fdd1b","Type":"ContainerStarted","Data":"a4334ff2442c5626fada39cc54b16cb59886a18a6168bfc64936db12b80b664b"}
Sep 11 17:42:41.722323 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:42:41.722231    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-15f6f2-5db9d8d8b7-pm48s" event={"ID":"dfb94eb0-7d89-4c81-80a2-2a11116fdd1b","Type":"ContainerStarted","Data":"9ba618c0c6f65130210fa01b8234e4fbe9b974f5b58462a1659f2d094785e19f"}
Sep 11 17:42:41.722323 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:42:41.722284    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-a-15f6f2-5db9d8d8b7-pm48s"
Sep 11 17:42:41.739257 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:42:41.739203    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-a-15f6f2-77875d44c4-zf4pp" podStartSLOduration=1.739188075 podStartE2EDuration="1.739188075s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 17:42:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 17:42:41.737296352 +0000 UTC m=+1439.310516465" watchObservedRunningTime="2026-09-11 17:42:41.739188075 +0000 UTC m=+1439.312408224"
Sep 11 17:42:41.756256 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:42:41.756207    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-a-15f6f2-5db9d8d8b7-pm48s" podStartSLOduration=1.756189451 podStartE2EDuration="1.756189451s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 17:42:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 17:42:41.753177087 +0000 UTC m=+1439.326397199" watchObservedRunningTime="2026-09-11 17:42:41.756189451 +0000 UTC m=+1439.329409563"
Sep 11 17:42:43.121072 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:42:43.121037    2577 scope.go:117] "RemoveContainer" containerID="12a9428bcdc5ff5ac0183f420e89d3d64ae44016d34bb488a5244b8823794657"
Sep 11 17:42:47.700439 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:42:47.700405    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-nkmkm"]
Sep 11 17:42:52.731714 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:42:52.731673    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-15f6f2-77875d44c4-zf4pp"
Sep 11 17:42:52.732159 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:42:52.731920    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-a-15f6f2-5db9d8d8b7-pm48s"
Sep 11 17:43:04.906274 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:43:04.906243    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-c7d192/embedding-w0-c7d192-kserve-5d5bfbb578-22zmh"]
Sep 11 17:43:04.911159 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:43:04.911129    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-c7d192/embedding-w0-c7d192-kserve-5d5bfbb578-22zmh"
Sep 11 17:43:04.914032 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:43:04.914010    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-c7d192\"/\"embedding-w0-c7d192-kserve-self-signed-certs\""
Sep 11 17:43:04.944136 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:43:04.944107    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-c7d192/embedding-w0-c7d192-kserve-5d5bfbb578-22zmh"]
Sep 11 17:43:04.962823 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:43:04.962790    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5aa9157d-fd25-4e77-bb11-b0fd137bf739-model-cache\") pod \"embedding-w0-c7d192-kserve-5d5bfbb578-22zmh\" (UID: \"5aa9157d-fd25-4e77-bb11-b0fd137bf739\") " pod="e2e-models-e2e-worker-w0-c7d192/embedding-w0-c7d192-kserve-5d5bfbb578-22zmh"
Sep 11 17:43:04.962961 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:43:04.962866    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5aa9157d-fd25-4e77-bb11-b0fd137bf739-tls-certs\") pod \"embedding-w0-c7d192-kserve-5d5bfbb578-22zmh\" (UID: \"5aa9157d-fd25-4e77-bb11-b0fd137bf739\") " pod="e2e-models-e2e-worker-w0-c7d192/embedding-w0-c7d192-kserve-5d5bfbb578-22zmh"
Sep 11 17:43:04.962961 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:43:04.962919    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5aa9157d-fd25-4e77-bb11-b0fd137bf739-dshm\") pod \"embedding-w0-c7d192-kserve-5d5bfbb578-22zmh\" (UID: \"5aa9157d-fd25-4e77-bb11-b0fd137bf739\") " pod="e2e-models-e2e-worker-w0-c7d192/embedding-w0-c7d192-kserve-5d5bfbb578-22zmh"
Sep 11 17:43:04.963142 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:43:04.962980    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5aa9157d-fd25-4e77-bb11-b0fd137bf739-tmp-dir\") pod \"embedding-w0-c7d192-kserve-5d5bfbb578-22zmh\" (UID: \"5aa9157d-fd25-4e77-bb11-b0fd137bf739\") " pod="e2e-models-e2e-worker-w0-c7d192/embedding-w0-c7d192-kserve-5d5bfbb578-22zmh"
Sep 11 17:43:04.963142 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:43:04.963060    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-65gcl\" (UniqueName: \"kubernetes.io/projected/5aa9157d-fd25-4e77-bb11-b0fd137bf739-kube-api-access-65gcl\") pod \"embedding-w0-c7d192-kserve-5d5bfbb578-22zmh\" (UID: \"5aa9157d-fd25-4e77-bb11-b0fd137bf739\") " pod="e2e-models-e2e-worker-w0-c7d192/embedding-w0-c7d192-kserve-5d5bfbb578-22zmh"
Sep 11 17:43:04.963142 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:43:04.963090    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5aa9157d-fd25-4e77-bb11-b0fd137bf739-home\") pod \"embedding-w0-c7d192-kserve-5d5bfbb578-22zmh\" (UID: \"5aa9157d-fd25-4e77-bb11-b0fd137bf739\") " pod="e2e-models-e2e-worker-w0-c7d192/embedding-w0-c7d192-kserve-5d5bfbb578-22zmh"
Sep 11 17:43:05.064021 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:43:05.063988    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5aa9157d-fd25-4e77-bb11-b0fd137bf739-dshm\") pod \"embedding-w0-c7d192-kserve-5d5bfbb578-22zmh\" (UID: \"5aa9157d-fd25-4e77-bb11-b0fd137bf739\") " pod="e2e-models-e2e-worker-w0-c7d192/embedding-w0-c7d192-kserve-5d5bfbb578-22zmh"
Sep 11 17:43:05.064021 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:43:05.064022    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5aa9157d-fd25-4e77-bb11-b0fd137bf739-tmp-dir\") pod \"embedding-w0-c7d192-kserve-5d5bfbb578-22zmh\" (UID: \"5aa9157d-fd25-4e77-bb11-b0fd137bf739\") " pod="e2e-models-e2e-worker-w0-c7d192/embedding-w0-c7d192-kserve-5d5bfbb578-22zmh"
Sep 11 17:43:05.064308 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:43:05.064057    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-65gcl\" (UniqueName: \"kubernetes.io/projected/5aa9157d-fd25-4e77-bb11-b0fd137bf739-kube-api-access-65gcl\") pod \"embedding-w0-c7d192-kserve-5d5bfbb578-22zmh\" (UID: \"5aa9157d-fd25-4e77-bb11-b0fd137bf739\") " pod="e2e-models-e2e-worker-w0-c7d192/embedding-w0-c7d192-kserve-5d5bfbb578-22zmh"
Sep 11 17:43:05.064308 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:43:05.064084    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5aa9157d-fd25-4e77-bb11-b0fd137bf739-home\") pod \"embedding-w0-c7d192-kserve-5d5bfbb578-22zmh\" (UID: \"5aa9157d-fd25-4e77-bb11-b0fd137bf739\") " pod="e2e-models-e2e-worker-w0-c7d192/embedding-w0-c7d192-kserve-5d5bfbb578-22zmh"
Sep 11 17:43:05.064308 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:43:05.064172    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5aa9157d-fd25-4e77-bb11-b0fd137bf739-model-cache\") pod \"embedding-w0-c7d192-kserve-5d5bfbb578-22zmh\" (UID: \"5aa9157d-fd25-4e77-bb11-b0fd137bf739\") " pod="e2e-models-e2e-worker-w0-c7d192/embedding-w0-c7d192-kserve-5d5bfbb578-22zmh"
Sep 11 17:43:05.064308 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:43:05.064238    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5aa9157d-fd25-4e77-bb11-b0fd137bf739-tls-certs\") pod \"embedding-w0-c7d192-kserve-5d5bfbb578-22zmh\" (UID: \"5aa9157d-fd25-4e77-bb11-b0fd137bf739\") " pod="e2e-models-e2e-worker-w0-c7d192/embedding-w0-c7d192-kserve-5d5bfbb578-22zmh"
Sep 11 17:43:05.064571 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:43:05.064548    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5aa9157d-fd25-4e77-bb11-b0fd137bf739-tmp-dir\") pod \"embedding-w0-c7d192-kserve-5d5bfbb578-22zmh\" (UID: \"5aa9157d-fd25-4e77-bb11-b0fd137bf739\") " pod="e2e-models-e2e-worker-w0-c7d192/embedding-w0-c7d192-kserve-5d5bfbb578-22zmh"
Sep 11 17:43:05.064629 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:43:05.064608    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5aa9157d-fd25-4e77-bb11-b0fd137bf739-home\") pod \"embedding-w0-c7d192-kserve-5d5bfbb578-22zmh\" (UID: \"5aa9157d-fd25-4e77-bb11-b0fd137bf739\") " pod="e2e-models-e2e-worker-w0-c7d192/embedding-w0-c7d192-kserve-5d5bfbb578-22zmh"
Sep 11 17:43:05.064672 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:43:05.064631    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5aa9157d-fd25-4e77-bb11-b0fd137bf739-model-cache\") pod \"embedding-w0-c7d192-kserve-5d5bfbb578-22zmh\" (UID: \"5aa9157d-fd25-4e77-bb11-b0fd137bf739\") " pod="e2e-models-e2e-worker-w0-c7d192/embedding-w0-c7d192-kserve-5d5bfbb578-22zmh"
Sep 11 17:43:05.066537 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:43:05.066497    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5aa9157d-fd25-4e77-bb11-b0fd137bf739-dshm\") pod \"embedding-w0-c7d192-kserve-5d5bfbb578-22zmh\" (UID: \"5aa9157d-fd25-4e77-bb11-b0fd137bf739\") " pod="e2e-models-e2e-worker-w0-c7d192/embedding-w0-c7d192-kserve-5d5bfbb578-22zmh"
Sep 11 17:43:05.066809 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:43:05.066791    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5aa9157d-fd25-4e77-bb11-b0fd137bf739-tls-certs\") pod \"embedding-w0-c7d192-kserve-5d5bfbb578-22zmh\" (UID: \"5aa9157d-fd25-4e77-bb11-b0fd137bf739\") " pod="e2e-models-e2e-worker-w0-c7d192/embedding-w0-c7d192-kserve-5d5bfbb578-22zmh"
Sep 11 17:43:05.104883 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:43:05.104853    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-65gcl\" (UniqueName: \"kubernetes.io/projected/5aa9157d-fd25-4e77-bb11-b0fd137bf739-kube-api-access-65gcl\") pod \"embedding-w0-c7d192-kserve-5d5bfbb578-22zmh\" (UID: \"5aa9157d-fd25-4e77-bb11-b0fd137bf739\") " pod="e2e-models-e2e-worker-w0-c7d192/embedding-w0-c7d192-kserve-5d5bfbb578-22zmh"
Sep 11 17:43:05.222531 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:43:05.222474    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-c7d192/embedding-w0-c7d192-kserve-5d5bfbb578-22zmh"
Sep 11 17:43:05.509690 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:43:05.509661    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-c7d192/embedding-w0-c7d192-kserve-5d5bfbb578-22zmh"]
Sep 11 17:43:05.510682 ip-10-0-141-139 kubenswrapper[2577]: W0911 17:43:05.510647    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod5aa9157d_fd25_4e77_bb11_b0fd137bf739.slice/crio-c4ac32ba2955d670269293aaa8c252f382d6e4e5a210c86755819d02e4f440d8 WatchSource:0}: Error finding container c4ac32ba2955d670269293aaa8c252f382d6e4e5a210c86755819d02e4f440d8: Status 404 returned error can't find the container with id c4ac32ba2955d670269293aaa8c252f382d6e4e5a210c86755819d02e4f440d8
Sep 11 17:43:05.869233 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:43:05.869143    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-c7d192/embedding-w0-c7d192-kserve-5d5bfbb578-22zmh" event={"ID":"5aa9157d-fd25-4e77-bb11-b0fd137bf739","Type":"ContainerStarted","Data":"6b0342a195c4b24066cdc90cc0361ba917590668658224032685ff839ef05c78"}
Sep 11 17:43:05.869233 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:43:05.869189    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-c7d192/embedding-w0-c7d192-kserve-5d5bfbb578-22zmh" event={"ID":"5aa9157d-fd25-4e77-bb11-b0fd137bf739","Type":"ContainerStarted","Data":"c4ac32ba2955d670269293aaa8c252f382d6e4e5a210c86755819d02e4f440d8"}
Sep 11 17:43:05.890889 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:43:05.890840    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-c7d192/embedding-w0-c7d192-kserve-5d5bfbb578-22zmh" podStartSLOduration=1.890824856 podStartE2EDuration="1.890824856s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 17:43:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 17:43:05.890008892 +0000 UTC m=+1463.463229005" watchObservedRunningTime="2026-09-11 17:43:05.890824856 +0000 UTC m=+1463.464045006"
Sep 11 17:43:12.439813 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:43:12.439777    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-83a2bc-6477bf7d6-cfzgz"]
Sep 11 17:43:12.443632 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:43:12.443612    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-83a2bc-6477bf7d6-cfzgz"
Sep 11 17:43:12.446023 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:43:12.446000    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-b-83a2bc\""
Sep 11 17:43:12.453599 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:43:12.453574    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-83a2bc-6477bf7d6-cfzgz"]
Sep 11 17:43:12.545406 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:43:12.545370    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nj5hx\" (UniqueName: \"kubernetes.io/projected/7738b716-4cc0-4b1b-9831-c13d86fdbbab-kube-api-access-nj5hx\") pod \"payload-pre-processing-e2e-api-b-83a2bc-6477bf7d6-cfzgz\" (UID: \"7738b716-4cc0-4b1b-9831-c13d86fdbbab\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-83a2bc-6477bf7d6-cfzgz"
Sep 11 17:43:12.545616 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:43:12.545428    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7738b716-4cc0-4b1b-9831-c13d86fdbbab-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-83a2bc-6477bf7d6-cfzgz\" (UID: \"7738b716-4cc0-4b1b-9831-c13d86fdbbab\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-83a2bc-6477bf7d6-cfzgz"
Sep 11 17:43:12.646297 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:43:12.646260    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nj5hx\" (UniqueName: \"kubernetes.io/projected/7738b716-4cc0-4b1b-9831-c13d86fdbbab-kube-api-access-nj5hx\") pod \"payload-pre-processing-e2e-api-b-83a2bc-6477bf7d6-cfzgz\" (UID: \"7738b716-4cc0-4b1b-9831-c13d86fdbbab\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-83a2bc-6477bf7d6-cfzgz"
Sep 11 17:43:12.646498 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:43:12.646332    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7738b716-4cc0-4b1b-9831-c13d86fdbbab-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-83a2bc-6477bf7d6-cfzgz\" (UID: \"7738b716-4cc0-4b1b-9831-c13d86fdbbab\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-83a2bc-6477bf7d6-cfzgz"
Sep 11 17:43:12.647120 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:43:12.647093    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7738b716-4cc0-4b1b-9831-c13d86fdbbab-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-83a2bc-6477bf7d6-cfzgz\" (UID: \"7738b716-4cc0-4b1b-9831-c13d86fdbbab\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-83a2bc-6477bf7d6-cfzgz"
Sep 11 17:43:12.655849 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:43:12.655823    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nj5hx\" (UniqueName: \"kubernetes.io/projected/7738b716-4cc0-4b1b-9831-c13d86fdbbab-kube-api-access-nj5hx\") pod \"payload-pre-processing-e2e-api-b-83a2bc-6477bf7d6-cfzgz\" (UID: \"7738b716-4cc0-4b1b-9831-c13d86fdbbab\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-83a2bc-6477bf7d6-cfzgz"
Sep 11 17:43:12.755464 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:43:12.755425    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-83a2bc-6477bf7d6-cfzgz"
Sep 11 17:43:13.030888 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:43:13.030857    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-83a2bc-6477bf7d6-cfzgz"]
Sep 11 17:43:13.032738 ip-10-0-141-139 kubenswrapper[2577]: W0911 17:43:13.032714    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7738b716_4cc0_4b1b_9831_c13d86fdbbab.slice/crio-03eebf50c6dea765d73b50103f734e5797aa33e082ab2ce55785f93cd720168a WatchSource:0}: Error finding container 03eebf50c6dea765d73b50103f734e5797aa33e082ab2ce55785f93cd720168a: Status 404 returned error can't find the container with id 03eebf50c6dea765d73b50103f734e5797aa33e082ab2ce55785f93cd720168a
Sep 11 17:43:13.926855 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:43:13.926814    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-83a2bc-6477bf7d6-cfzgz" event={"ID":"7738b716-4cc0-4b1b-9831-c13d86fdbbab","Type":"ContainerStarted","Data":"201c9d660395ac262a413681d25c3b300ba73182628912bc8995cef6f750584e"}
Sep 11 17:43:13.927263 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:43:13.926882    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-83a2bc-6477bf7d6-cfzgz" event={"ID":"7738b716-4cc0-4b1b-9831-c13d86fdbbab","Type":"ContainerStarted","Data":"03eebf50c6dea765d73b50103f734e5797aa33e082ab2ce55785f93cd720168a"}
Sep 11 17:43:13.927263 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:43:13.926926    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-83a2bc-6477bf7d6-cfzgz"
Sep 11 17:43:13.949155 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:43:13.949101    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-b-83a2bc-6477bf7d6-cfzgz" podStartSLOduration=1.9490845220000002 podStartE2EDuration="1.949084522s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 17:43:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 17:43:13.946131617 +0000 UTC m=+1471.519351730" watchObservedRunningTime="2026-09-11 17:43:13.949084522 +0000 UTC m=+1471.522304682"
Sep 11 17:43:15.223644 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:43:15.223599    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-c7d192/embedding-w0-c7d192-kserve-5d5bfbb578-22zmh"
Sep 11 17:43:15.224111 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:43:15.223666    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-c7d192/embedding-w0-c7d192-kserve-5d5bfbb578-22zmh"
Sep 11 17:43:15.236416 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:43:15.236383    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-c7d192/embedding-w0-c7d192-kserve-5d5bfbb578-22zmh"
Sep 11 17:43:15.951991 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:43:15.951958    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-c7d192/embedding-w0-c7d192-kserve-5d5bfbb578-22zmh"
Sep 11 17:43:24.933674 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:43:24.933645    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-83a2bc-6477bf7d6-cfzgz"
Sep 11 17:43:30.751113 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:43:30.751081    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-nkmkm"]
Sep 11 17:43:40.960683 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:43:40.960647    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-15f6f2-dtcg5"]
Sep 11 17:43:40.964868 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:43:40.964840    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-15f6f2-dtcg5"
Sep 11 17:43:40.967230 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:43:40.967211    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-9n8gf\""
Sep 11 17:43:40.967679 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:43:40.967660    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 11 17:43:40.973204 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:43:40.973004    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-15f6f2-dtcg5"]
Sep 11 17:43:41.050164 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:43:41.050126    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8d583c19-b3d5-4b6a-babc-1ed834347744-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-a-15f6f2-dtcg5\" (UID: \"8d583c19-b3d5-4b6a-babc-1ed834347744\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-15f6f2-dtcg5"
Sep 11 17:43:41.151569 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:43:41.151529    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8d583c19-b3d5-4b6a-babc-1ed834347744-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-a-15f6f2-dtcg5\" (UID: \"8d583c19-b3d5-4b6a-babc-1ed834347744\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-15f6f2-dtcg5"
Sep 11 17:43:41.152155 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:43:41.152134    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8d583c19-b3d5-4b6a-babc-1ed834347744-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-a-15f6f2-dtcg5\" (UID: \"8d583c19-b3d5-4b6a-babc-1ed834347744\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-15f6f2-dtcg5"
Sep 11 17:43:41.276988 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:43:41.276952    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-15f6f2-dtcg5"
Sep 11 17:43:41.564813 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:43:41.564788    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-15f6f2-dtcg5"]
Sep 11 17:43:41.566756 ip-10-0-141-139 kubenswrapper[2577]: W0911 17:43:41.566725    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8d583c19_b3d5_4b6a_babc_1ed834347744.slice/crio-d04bfa2c923f5cc181011f283d61abffa0fe6731fe6537cc394ad099b311e48a WatchSource:0}: Error finding container d04bfa2c923f5cc181011f283d61abffa0fe6731fe6537cc394ad099b311e48a: Status 404 returned error can't find the container with id d04bfa2c923f5cc181011f283d61abffa0fe6731fe6537cc394ad099b311e48a
Sep 11 17:43:42.090638 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:43:42.090598    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-15f6f2-dtcg5" event={"ID":"8d583c19-b3d5-4b6a-babc-1ed834347744","Type":"ContainerStarted","Data":"9edcaf85402e7dc80930be6e1780903689131bee3fdd52b323b0d7d7da488eef"}
Sep 11 17:43:42.091014 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:43:42.090661    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-15f6f2-dtcg5" event={"ID":"8d583c19-b3d5-4b6a-babc-1ed834347744","Type":"ContainerStarted","Data":"d04bfa2c923f5cc181011f283d61abffa0fe6731fe6537cc394ad099b311e48a"}
Sep 11 17:43:42.109817 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:43:42.109720    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-15f6f2-dtcg5" podStartSLOduration=2.1097055 podStartE2EDuration="2.1097055s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 17:43:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 17:43:42.107840633 +0000 UTC m=+1499.681060746" watchObservedRunningTime="2026-09-11 17:43:42.1097055 +0000 UTC m=+1499.682925612"
Sep 11 17:43:43.098496 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:43:43.098461    2577 generic.go:356] "Generic (PLEG): container finished" podID="8d583c19-b3d5-4b6a-babc-1ed834347744" containerID="9edcaf85402e7dc80930be6e1780903689131bee3fdd52b323b0d7d7da488eef" exitCode=0
Sep 11 17:43:43.098892 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:43:43.098536    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-15f6f2-dtcg5" event={"ID":"8d583c19-b3d5-4b6a-babc-1ed834347744","Type":"ContainerDied","Data":"9edcaf85402e7dc80930be6e1780903689131bee3fdd52b323b0d7d7da488eef"}
Sep 11 17:43:43.143698 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:43:43.143665    2577 scope.go:117] "RemoveContainer" containerID="ba8bcea1c2f00505544e3ae214586029eec2b7c1ea7cf384ad1a2c8eb1c7109c"
Sep 11 17:43:44.243477 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:43:44.243445    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-15f6f2-dtcg5"
Sep 11 17:43:44.288813 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:43:44.288777    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8d583c19-b3d5-4b6a-babc-1ed834347744-maas-api-service-ca\") pod \"8d583c19-b3d5-4b6a-babc-1ed834347744\" (UID: \"8d583c19-b3d5-4b6a-babc-1ed834347744\") "
Sep 11 17:43:44.289135 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:43:44.289108    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8d583c19-b3d5-4b6a-babc-1ed834347744-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "8d583c19-b3d5-4b6a-babc-1ed834347744" (UID: "8d583c19-b3d5-4b6a-babc-1ed834347744"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 17:43:44.389693 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:43:44.389611    2577 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8d583c19-b3d5-4b6a-babc-1ed834347744-maas-api-service-ca\") on node \"ip-10-0-141-139.ec2.internal\" DevicePath \"\""
Sep 11 17:43:45.111667 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:43:45.111637    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-15f6f2-dtcg5"
Sep 11 17:43:45.111839 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:43:45.111663    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-15f6f2-dtcg5" event={"ID":"8d583c19-b3d5-4b6a-babc-1ed834347744","Type":"ContainerDied","Data":"d04bfa2c923f5cc181011f283d61abffa0fe6731fe6537cc394ad099b311e48a"}
Sep 11 17:43:45.111839 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:43:45.111712    2577 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d04bfa2c923f5cc181011f283d61abffa0fe6731fe6537cc394ad099b311e48a"
Sep 11 17:43:56.622062 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:43:56.622027    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-15f6f2-5db9d8d8b7-pm48s"]
Sep 11 17:43:56.622464 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:43:56.622327    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-a-15f6f2-5db9d8d8b7-pm48s" podUID="dfb94eb0-7d89-4c81-80a2-2a11116fdd1b" containerName="payload-processing" containerID="cri-o://a4334ff2442c5626fada39cc54b16cb59886a18a6168bfc64936db12b80b664b" gracePeriod=30
Sep 11 17:43:56.636296 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:43:56.636265    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-15f6f2-77875d44c4-zf4pp"]
Sep 11 17:43:56.636539 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:43:56.636515    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-a-15f6f2-77875d44c4-zf4pp" podUID="6a29c245-ea49-4787-9405-d58d6587f401" containerName="payload-pre-processing" containerID="cri-o://afe9f4d9c44dd95fda722bdfe27232b93f5e25d657cfc3d79dfdecdced2a17be" gracePeriod=30
Sep 11 17:43:56.879206 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:43:56.879121    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-15f6f2-dtcg5"]
Sep 11 17:43:56.882820 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:43:56.882789    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-15f6f2-dtcg5"]
Sep 11 17:43:56.914788 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:43:56.914768    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-15f6f2-5db9d8d8b7-pm48s"
Sep 11 17:43:56.917973 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:43:56.917951    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-15f6f2-77875d44c4-zf4pp"
Sep 11 17:43:56.926835 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:43:56.926808    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8d583c19-b3d5-4b6a-babc-1ed834347744" path="/var/lib/kubelet/pods/8d583c19-b3d5-4b6a-babc-1ed834347744/volumes"
Sep 11 17:43:57.037183 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:43:57.037142    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dfb94eb0-7d89-4c81-80a2-2a11116fdd1b-plugins-config-volume\") pod \"dfb94eb0-7d89-4c81-80a2-2a11116fdd1b\" (UID: \"dfb94eb0-7d89-4c81-80a2-2a11116fdd1b\") "
Sep 11 17:43:57.037183 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:43:57.037179    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hc7tk\" (UniqueName: \"kubernetes.io/projected/dfb94eb0-7d89-4c81-80a2-2a11116fdd1b-kube-api-access-hc7tk\") pod \"dfb94eb0-7d89-4c81-80a2-2a11116fdd1b\" (UID: \"dfb94eb0-7d89-4c81-80a2-2a11116fdd1b\") "
Sep 11 17:43:57.037405 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:43:57.037239    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6a29c245-ea49-4787-9405-d58d6587f401-plugins-config-volume\") pod \"6a29c245-ea49-4787-9405-d58d6587f401\" (UID: \"6a29c245-ea49-4787-9405-d58d6587f401\") "
Sep 11 17:43:57.037405 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:43:57.037277    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rhq2x\" (UniqueName: \"kubernetes.io/projected/6a29c245-ea49-4787-9405-d58d6587f401-kube-api-access-rhq2x\") pod \"6a29c245-ea49-4787-9405-d58d6587f401\" (UID: \"6a29c245-ea49-4787-9405-d58d6587f401\") "
Sep 11 17:43:57.037638 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:43:57.037612    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dfb94eb0-7d89-4c81-80a2-2a11116fdd1b-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "dfb94eb0-7d89-4c81-80a2-2a11116fdd1b" (UID: "dfb94eb0-7d89-4c81-80a2-2a11116fdd1b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 17:43:57.037695 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:43:57.037655    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6a29c245-ea49-4787-9405-d58d6587f401-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "6a29c245-ea49-4787-9405-d58d6587f401" (UID: "6a29c245-ea49-4787-9405-d58d6587f401"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 17:43:57.037831 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:43:57.037806    2577 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6a29c245-ea49-4787-9405-d58d6587f401-plugins-config-volume\") on node \"ip-10-0-141-139.ec2.internal\" DevicePath \"\""
Sep 11 17:43:57.038133 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:43:57.037848    2577 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dfb94eb0-7d89-4c81-80a2-2a11116fdd1b-plugins-config-volume\") on node \"ip-10-0-141-139.ec2.internal\" DevicePath \"\""
Sep 11 17:43:57.039644 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:43:57.039613    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6a29c245-ea49-4787-9405-d58d6587f401-kube-api-access-rhq2x" (OuterVolumeSpecName: "kube-api-access-rhq2x") pod "6a29c245-ea49-4787-9405-d58d6587f401" (UID: "6a29c245-ea49-4787-9405-d58d6587f401"). InnerVolumeSpecName "kube-api-access-rhq2x". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 17:43:57.039733 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:43:57.039688    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dfb94eb0-7d89-4c81-80a2-2a11116fdd1b-kube-api-access-hc7tk" (OuterVolumeSpecName: "kube-api-access-hc7tk") pod "dfb94eb0-7d89-4c81-80a2-2a11116fdd1b" (UID: "dfb94eb0-7d89-4c81-80a2-2a11116fdd1b"). InnerVolumeSpecName "kube-api-access-hc7tk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 17:43:57.138843 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:43:57.138769    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rhq2x\" (UniqueName: \"kubernetes.io/projected/6a29c245-ea49-4787-9405-d58d6587f401-kube-api-access-rhq2x\") on node \"ip-10-0-141-139.ec2.internal\" DevicePath \"\""
Sep 11 17:43:57.138843 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:43:57.138793    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hc7tk\" (UniqueName: \"kubernetes.io/projected/dfb94eb0-7d89-4c81-80a2-2a11116fdd1b-kube-api-access-hc7tk\") on node \"ip-10-0-141-139.ec2.internal\" DevicePath \"\""
Sep 11 17:43:57.191311 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:43:57.191273    2577 generic.go:356] "Generic (PLEG): container finished" podID="6a29c245-ea49-4787-9405-d58d6587f401" containerID="afe9f4d9c44dd95fda722bdfe27232b93f5e25d657cfc3d79dfdecdced2a17be" exitCode=0
Sep 11 17:43:57.191479 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:43:57.191331    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-15f6f2-77875d44c4-zf4pp"
Sep 11 17:43:57.191479 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:43:57.191359    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-15f6f2-77875d44c4-zf4pp" event={"ID":"6a29c245-ea49-4787-9405-d58d6587f401","Type":"ContainerDied","Data":"afe9f4d9c44dd95fda722bdfe27232b93f5e25d657cfc3d79dfdecdced2a17be"}
Sep 11 17:43:57.191479 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:43:57.191416    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-15f6f2-77875d44c4-zf4pp" event={"ID":"6a29c245-ea49-4787-9405-d58d6587f401","Type":"ContainerDied","Data":"1c72ac5813b212a7c024f298aeae1e0ecabb3c814c58ad6d2fb96fe380fab8ab"}
Sep 11 17:43:57.191479 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:43:57.191443    2577 scope.go:117] "RemoveContainer" containerID="afe9f4d9c44dd95fda722bdfe27232b93f5e25d657cfc3d79dfdecdced2a17be"
Sep 11 17:43:57.192951 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:43:57.192932    2577 generic.go:356] "Generic (PLEG): container finished" podID="dfb94eb0-7d89-4c81-80a2-2a11116fdd1b" containerID="a4334ff2442c5626fada39cc54b16cb59886a18a6168bfc64936db12b80b664b" exitCode=0
Sep 11 17:43:57.193050 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:43:57.192976    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-15f6f2-5db9d8d8b7-pm48s"
Sep 11 17:43:57.193050 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:43:57.193009    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-15f6f2-5db9d8d8b7-pm48s" event={"ID":"dfb94eb0-7d89-4c81-80a2-2a11116fdd1b","Type":"ContainerDied","Data":"a4334ff2442c5626fada39cc54b16cb59886a18a6168bfc64936db12b80b664b"}
Sep 11 17:43:57.193135 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:43:57.193050    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-15f6f2-5db9d8d8b7-pm48s" event={"ID":"dfb94eb0-7d89-4c81-80a2-2a11116fdd1b","Type":"ContainerDied","Data":"9ba618c0c6f65130210fa01b8234e4fbe9b974f5b58462a1659f2d094785e19f"}
Sep 11 17:43:57.204592 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:43:57.204576    2577 scope.go:117] "RemoveContainer" containerID="afe9f4d9c44dd95fda722bdfe27232b93f5e25d657cfc3d79dfdecdced2a17be"
Sep 11 17:43:57.204857 ip-10-0-141-139 kubenswrapper[2577]: E0911 17:43:57.204838    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"afe9f4d9c44dd95fda722bdfe27232b93f5e25d657cfc3d79dfdecdced2a17be\": container with ID starting with afe9f4d9c44dd95fda722bdfe27232b93f5e25d657cfc3d79dfdecdced2a17be not found: ID does not exist" containerID="afe9f4d9c44dd95fda722bdfe27232b93f5e25d657cfc3d79dfdecdced2a17be"
Sep 11 17:43:57.204925 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:43:57.204866    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"afe9f4d9c44dd95fda722bdfe27232b93f5e25d657cfc3d79dfdecdced2a17be"} err="failed to get container status \"afe9f4d9c44dd95fda722bdfe27232b93f5e25d657cfc3d79dfdecdced2a17be\": rpc error: code = NotFound desc = could not find container \"afe9f4d9c44dd95fda722bdfe27232b93f5e25d657cfc3d79dfdecdced2a17be\": container with ID starting with afe9f4d9c44dd95fda722bdfe27232b93f5e25d657cfc3d79dfdecdced2a17be not found: ID does not exist"
Sep 11 17:43:57.204925 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:43:57.204882    2577 scope.go:117] "RemoveContainer" containerID="a4334ff2442c5626fada39cc54b16cb59886a18a6168bfc64936db12b80b664b"
Sep 11 17:43:57.213079 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:43:57.213063    2577 scope.go:117] "RemoveContainer" containerID="a4334ff2442c5626fada39cc54b16cb59886a18a6168bfc64936db12b80b664b"
Sep 11 17:43:57.213357 ip-10-0-141-139 kubenswrapper[2577]: E0911 17:43:57.213329    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a4334ff2442c5626fada39cc54b16cb59886a18a6168bfc64936db12b80b664b\": container with ID starting with a4334ff2442c5626fada39cc54b16cb59886a18a6168bfc64936db12b80b664b not found: ID does not exist" containerID="a4334ff2442c5626fada39cc54b16cb59886a18a6168bfc64936db12b80b664b"
Sep 11 17:43:57.213403 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:43:57.213354    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a4334ff2442c5626fada39cc54b16cb59886a18a6168bfc64936db12b80b664b"} err="failed to get container status \"a4334ff2442c5626fada39cc54b16cb59886a18a6168bfc64936db12b80b664b\": rpc error: code = NotFound desc = could not find container \"a4334ff2442c5626fada39cc54b16cb59886a18a6168bfc64936db12b80b664b\": container with ID starting with a4334ff2442c5626fada39cc54b16cb59886a18a6168bfc64936db12b80b664b not found: ID does not exist"
Sep 11 17:43:57.221021 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:43:57.220998    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-15f6f2-5db9d8d8b7-pm48s"]
Sep 11 17:43:57.227376 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:43:57.227355    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-15f6f2-5db9d8d8b7-pm48s"]
Sep 11 17:43:57.239972 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:43:57.239946    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-15f6f2-77875d44c4-zf4pp"]
Sep 11 17:43:57.247238 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:43:57.247219    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-15f6f2-77875d44c4-zf4pp"]
Sep 11 17:43:58.927233 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:43:58.927196    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6a29c245-ea49-4787-9405-d58d6587f401" path="/var/lib/kubelet/pods/6a29c245-ea49-4787-9405-d58d6587f401/volumes"
Sep 11 17:43:58.927629 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:43:58.927615    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="dfb94eb0-7d89-4c81-80a2-2a11116fdd1b" path="/var/lib/kubelet/pods/dfb94eb0-7d89-4c81-80a2-2a11116fdd1b/volumes"
Sep 11 17:44:04.655329 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:44:04.655248    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-83a2bc-7rpmt"]
Sep 11 17:44:04.655728 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:44:04.655713    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6a29c245-ea49-4787-9405-d58d6587f401" containerName="payload-pre-processing"
Sep 11 17:44:04.655728 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:44:04.655729    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="6a29c245-ea49-4787-9405-d58d6587f401" containerName="payload-pre-processing"
Sep 11 17:44:04.655815 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:44:04.655750    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8d583c19-b3d5-4b6a-babc-1ed834347744" containerName="revoke-keys"
Sep 11 17:44:04.655815 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:44:04.655756    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="8d583c19-b3d5-4b6a-babc-1ed834347744" containerName="revoke-keys"
Sep 11 17:44:04.655815 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:44:04.655772    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dfb94eb0-7d89-4c81-80a2-2a11116fdd1b" containerName="payload-processing"
Sep 11 17:44:04.655815 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:44:04.655777    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="dfb94eb0-7d89-4c81-80a2-2a11116fdd1b" containerName="payload-processing"
Sep 11 17:44:04.655935 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:44:04.655847    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="dfb94eb0-7d89-4c81-80a2-2a11116fdd1b" containerName="payload-processing"
Sep 11 17:44:04.655935 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:44:04.655857    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="8d583c19-b3d5-4b6a-babc-1ed834347744" containerName="revoke-keys"
Sep 11 17:44:04.655935 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:44:04.655867    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="6a29c245-ea49-4787-9405-d58d6587f401" containerName="payload-pre-processing"
Sep 11 17:44:04.660296 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:44:04.660279    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-83a2bc-7rpmt"
Sep 11 17:44:04.663160 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:44:04.663131    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 11 17:44:04.680793 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:44:04.680765    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-9n8gf\""
Sep 11 17:44:04.722523 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:44:04.722466    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/35f636da-afaa-4384-9c81-1e34e8bf5e11-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-b-83a2bc-7rpmt\" (UID: \"35f636da-afaa-4384-9c81-1e34e8bf5e11\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-83a2bc-7rpmt"
Sep 11 17:44:04.727693 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:44:04.727655    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-83a2bc-7rpmt"]
Sep 11 17:44:04.823400 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:44:04.823357    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/35f636da-afaa-4384-9c81-1e34e8bf5e11-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-b-83a2bc-7rpmt\" (UID: \"35f636da-afaa-4384-9c81-1e34e8bf5e11\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-83a2bc-7rpmt"
Sep 11 17:44:04.824008 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:44:04.823986    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/35f636da-afaa-4384-9c81-1e34e8bf5e11-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-b-83a2bc-7rpmt\" (UID: \"35f636da-afaa-4384-9c81-1e34e8bf5e11\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-83a2bc-7rpmt"
Sep 11 17:44:04.971029 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:44:04.970998    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-83a2bc-7rpmt"
Sep 11 17:44:05.248714 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:44:05.248690    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-83a2bc-7rpmt"]
Sep 11 17:44:05.250906 ip-10-0-141-139 kubenswrapper[2577]: W0911 17:44:05.250880    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod35f636da_afaa_4384_9c81_1e34e8bf5e11.slice/crio-a26aed128c61b881435968a61f200947dc4f1bc78b4df9d52e2f5fb4b28cb6c1 WatchSource:0}: Error finding container a26aed128c61b881435968a61f200947dc4f1bc78b4df9d52e2f5fb4b28cb6c1: Status 404 returned error can't find the container with id a26aed128c61b881435968a61f200947dc4f1bc78b4df9d52e2f5fb4b28cb6c1
Sep 11 17:44:06.251919 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:44:06.251889    2577 generic.go:356] "Generic (PLEG): container finished" podID="35f636da-afaa-4384-9c81-1e34e8bf5e11" containerID="f651a270e33fbbb83ff86fb6ce01def91a28068d2411f4e44d93b42124c78667" exitCode=0
Sep 11 17:44:06.252343 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:44:06.251979    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-83a2bc-7rpmt" event={"ID":"35f636da-afaa-4384-9c81-1e34e8bf5e11","Type":"ContainerDied","Data":"f651a270e33fbbb83ff86fb6ce01def91a28068d2411f4e44d93b42124c78667"}
Sep 11 17:44:06.252343 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:44:06.252045    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-83a2bc-7rpmt" event={"ID":"35f636da-afaa-4384-9c81-1e34e8bf5e11","Type":"ContainerStarted","Data":"a26aed128c61b881435968a61f200947dc4f1bc78b4df9d52e2f5fb4b28cb6c1"}
Sep 11 17:44:07.398287 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:44:07.398260    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-83a2bc-7rpmt"
Sep 11 17:44:07.450288 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:44:07.450258    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/35f636da-afaa-4384-9c81-1e34e8bf5e11-maas-api-service-ca\") pod \"35f636da-afaa-4384-9c81-1e34e8bf5e11\" (UID: \"35f636da-afaa-4384-9c81-1e34e8bf5e11\") "
Sep 11 17:44:07.450681 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:44:07.450657    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/35f636da-afaa-4384-9c81-1e34e8bf5e11-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "35f636da-afaa-4384-9c81-1e34e8bf5e11" (UID: "35f636da-afaa-4384-9c81-1e34e8bf5e11"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 17:44:07.552216 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:44:07.552135    2577 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/35f636da-afaa-4384-9c81-1e34e8bf5e11-maas-api-service-ca\") on node \"ip-10-0-141-139.ec2.internal\" DevicePath \"\""
Sep 11 17:44:08.264388 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:44:08.264354    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-83a2bc-7rpmt"
Sep 11 17:44:08.264388 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:44:08.264384    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-83a2bc-7rpmt" event={"ID":"35f636da-afaa-4384-9c81-1e34e8bf5e11","Type":"ContainerDied","Data":"a26aed128c61b881435968a61f200947dc4f1bc78b4df9d52e2f5fb4b28cb6c1"}
Sep 11 17:44:08.266641 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:44:08.264434    2577 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a26aed128c61b881435968a61f200947dc4f1bc78b4df9d52e2f5fb4b28cb6c1"
Sep 11 17:44:20.788438 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:44:20.788402    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-83a2bc-6477bf7d6-cfzgz"]
Sep 11 17:44:20.788991 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:44:20.788654    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-b-83a2bc-6477bf7d6-cfzgz" podUID="7738b716-4cc0-4b1b-9831-c13d86fdbbab" containerName="payload-pre-processing" containerID="cri-o://201c9d660395ac262a413681d25c3b300ba73182628912bc8995cef6f750584e" gracePeriod=30
Sep 11 17:44:20.924427 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:44:20.924393    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-nkmkm"]
Sep 11 17:44:21.048963 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:44:21.048901    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-83a2bc-6477bf7d6-cfzgz"
Sep 11 17:44:21.110776 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:44:21.110735    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-83a2bc-7rpmt"]
Sep 11 17:44:21.114918 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:44:21.114890    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-83a2bc-7rpmt"]
Sep 11 17:44:21.196573 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:44:21.196544    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7738b716-4cc0-4b1b-9831-c13d86fdbbab-plugins-config-volume\") pod \"7738b716-4cc0-4b1b-9831-c13d86fdbbab\" (UID: \"7738b716-4cc0-4b1b-9831-c13d86fdbbab\") "
Sep 11 17:44:21.196734 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:44:21.196674    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nj5hx\" (UniqueName: \"kubernetes.io/projected/7738b716-4cc0-4b1b-9831-c13d86fdbbab-kube-api-access-nj5hx\") pod \"7738b716-4cc0-4b1b-9831-c13d86fdbbab\" (UID: \"7738b716-4cc0-4b1b-9831-c13d86fdbbab\") "
Sep 11 17:44:21.196889 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:44:21.196867    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7738b716-4cc0-4b1b-9831-c13d86fdbbab-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "7738b716-4cc0-4b1b-9831-c13d86fdbbab" (UID: "7738b716-4cc0-4b1b-9831-c13d86fdbbab"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 17:44:21.198624 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:44:21.198596    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7738b716-4cc0-4b1b-9831-c13d86fdbbab-kube-api-access-nj5hx" (OuterVolumeSpecName: "kube-api-access-nj5hx") pod "7738b716-4cc0-4b1b-9831-c13d86fdbbab" (UID: "7738b716-4cc0-4b1b-9831-c13d86fdbbab"). InnerVolumeSpecName "kube-api-access-nj5hx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 17:44:21.297581 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:44:21.297548    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nj5hx\" (UniqueName: \"kubernetes.io/projected/7738b716-4cc0-4b1b-9831-c13d86fdbbab-kube-api-access-nj5hx\") on node \"ip-10-0-141-139.ec2.internal\" DevicePath \"\""
Sep 11 17:44:21.297581 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:44:21.297574    2577 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7738b716-4cc0-4b1b-9831-c13d86fdbbab-plugins-config-volume\") on node \"ip-10-0-141-139.ec2.internal\" DevicePath \"\""
Sep 11 17:44:21.339760 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:44:21.339681    2577 generic.go:356] "Generic (PLEG): container finished" podID="7738b716-4cc0-4b1b-9831-c13d86fdbbab" containerID="201c9d660395ac262a413681d25c3b300ba73182628912bc8995cef6f750584e" exitCode=0
Sep 11 17:44:21.339760 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:44:21.339729    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-83a2bc-6477bf7d6-cfzgz"
Sep 11 17:44:21.339942 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:44:21.339772    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-83a2bc-6477bf7d6-cfzgz" event={"ID":"7738b716-4cc0-4b1b-9831-c13d86fdbbab","Type":"ContainerDied","Data":"201c9d660395ac262a413681d25c3b300ba73182628912bc8995cef6f750584e"}
Sep 11 17:44:21.339942 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:44:21.339829    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-83a2bc-6477bf7d6-cfzgz" event={"ID":"7738b716-4cc0-4b1b-9831-c13d86fdbbab","Type":"ContainerDied","Data":"03eebf50c6dea765d73b50103f734e5797aa33e082ab2ce55785f93cd720168a"}
Sep 11 17:44:21.339942 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:44:21.339856    2577 scope.go:117] "RemoveContainer" containerID="201c9d660395ac262a413681d25c3b300ba73182628912bc8995cef6f750584e"
Sep 11 17:44:21.350219 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:44:21.350198    2577 scope.go:117] "RemoveContainer" containerID="201c9d660395ac262a413681d25c3b300ba73182628912bc8995cef6f750584e"
Sep 11 17:44:21.350483 ip-10-0-141-139 kubenswrapper[2577]: E0911 17:44:21.350466    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"201c9d660395ac262a413681d25c3b300ba73182628912bc8995cef6f750584e\": container with ID starting with 201c9d660395ac262a413681d25c3b300ba73182628912bc8995cef6f750584e not found: ID does not exist" containerID="201c9d660395ac262a413681d25c3b300ba73182628912bc8995cef6f750584e"
Sep 11 17:44:21.350567 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:44:21.350492    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"201c9d660395ac262a413681d25c3b300ba73182628912bc8995cef6f750584e"} err="failed to get container status \"201c9d660395ac262a413681d25c3b300ba73182628912bc8995cef6f750584e\": rpc error: code = NotFound desc = could not find container \"201c9d660395ac262a413681d25c3b300ba73182628912bc8995cef6f750584e\": container with ID starting with 201c9d660395ac262a413681d25c3b300ba73182628912bc8995cef6f750584e not found: ID does not exist"
Sep 11 17:44:21.361548 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:44:21.361516    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-83a2bc-6477bf7d6-cfzgz"]
Sep 11 17:44:21.366117 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:44:21.366093    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-83a2bc-6477bf7d6-cfzgz"]
Sep 11 17:44:22.926920 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:44:22.926841    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="35f636da-afaa-4384-9c81-1e34e8bf5e11" path="/var/lib/kubelet/pods/35f636da-afaa-4384-9c81-1e34e8bf5e11/volumes"
Sep 11 17:44:22.927279 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:44:22.927189    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7738b716-4cc0-4b1b-9831-c13d86fdbbab" path="/var/lib/kubelet/pods/7738b716-4cc0-4b1b-9831-c13d86fdbbab/volumes"
Sep 11 17:44:29.716551 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:44:29.716517    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-290683-k6lkh"]
Sep 11 17:44:29.716966 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:44:29.716946    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7738b716-4cc0-4b1b-9831-c13d86fdbbab" containerName="payload-pre-processing"
Sep 11 17:44:29.716966 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:44:29.716957    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="7738b716-4cc0-4b1b-9831-c13d86fdbbab" containerName="payload-pre-processing"
Sep 11 17:44:29.716966 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:44:29.716966    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="35f636da-afaa-4384-9c81-1e34e8bf5e11" containerName="revoke-keys"
Sep 11 17:44:29.717070 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:44:29.716973    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="35f636da-afaa-4384-9c81-1e34e8bf5e11" containerName="revoke-keys"
Sep 11 17:44:29.717070 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:44:29.717062    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="7738b716-4cc0-4b1b-9831-c13d86fdbbab" containerName="payload-pre-processing"
Sep 11 17:44:29.717070 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:44:29.717070    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="35f636da-afaa-4384-9c81-1e34e8bf5e11" containerName="revoke-keys"
Sep 11 17:44:29.721642 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:44:29.721623    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-290683-k6lkh"
Sep 11 17:44:29.725684 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:44:29.725663    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-9n8gf\""
Sep 11 17:44:29.726714 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:44:29.726695    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 11 17:44:29.733851 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:44:29.733827    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-290683-k6lkh"]
Sep 11 17:44:29.790141 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:44:29.790107    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3c3b23ac-229d-40e4-9263-3f184726d148-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-290683-k6lkh\" (UID: \"3c3b23ac-229d-40e4-9263-3f184726d148\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-290683-k6lkh"
Sep 11 17:44:29.890845 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:44:29.890808    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3c3b23ac-229d-40e4-9263-3f184726d148-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-290683-k6lkh\" (UID: \"3c3b23ac-229d-40e4-9263-3f184726d148\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-290683-k6lkh"
Sep 11 17:44:29.891418 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:44:29.891399    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3c3b23ac-229d-40e4-9263-3f184726d148-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-290683-k6lkh\" (UID: \"3c3b23ac-229d-40e4-9263-3f184726d148\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-290683-k6lkh"
Sep 11 17:44:30.032442 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:44:30.032357    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-290683-k6lkh"
Sep 11 17:44:30.308534 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:44:30.308486    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-290683-k6lkh"]
Sep 11 17:44:30.310141 ip-10-0-141-139 kubenswrapper[2577]: W0911 17:44:30.310112    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3c3b23ac_229d_40e4_9263_3f184726d148.slice/crio-7ba5b2b0f2a668a314dc7519d7a2aaeb748474c2ae9a0e153875444119975222 WatchSource:0}: Error finding container 7ba5b2b0f2a668a314dc7519d7a2aaeb748474c2ae9a0e153875444119975222: Status 404 returned error can't find the container with id 7ba5b2b0f2a668a314dc7519d7a2aaeb748474c2ae9a0e153875444119975222
Sep 11 17:44:30.393456 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:44:30.393419    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-290683-k6lkh" event={"ID":"3c3b23ac-229d-40e4-9263-3f184726d148","Type":"ContainerStarted","Data":"7ba5b2b0f2a668a314dc7519d7a2aaeb748474c2ae9a0e153875444119975222"}
Sep 11 17:44:31.401209 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:44:31.401105    2577 generic.go:356] "Generic (PLEG): container finished" podID="3c3b23ac-229d-40e4-9263-3f184726d148" containerID="cbe3932f264bbbdc16ad30a5c6cefb2da8ce18580e15ef1e7a221ef62427a0c5" exitCode=0
Sep 11 17:44:31.401590 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:44:31.401205    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-290683-k6lkh" event={"ID":"3c3b23ac-229d-40e4-9263-3f184726d148","Type":"ContainerDied","Data":"cbe3932f264bbbdc16ad30a5c6cefb2da8ce18580e15ef1e7a221ef62427a0c5"}
Sep 11 17:44:32.541945 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:44:32.541919    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-290683-k6lkh"
Sep 11 17:44:32.619920 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:44:32.619888    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3c3b23ac-229d-40e4-9263-3f184726d148-maas-api-service-ca\") pod \"3c3b23ac-229d-40e4-9263-3f184726d148\" (UID: \"3c3b23ac-229d-40e4-9263-3f184726d148\") "
Sep 11 17:44:32.620270 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:44:32.620243    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3c3b23ac-229d-40e4-9263-3f184726d148-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "3c3b23ac-229d-40e4-9263-3f184726d148" (UID: "3c3b23ac-229d-40e4-9263-3f184726d148"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 17:44:32.721631 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:44:32.721604    2577 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3c3b23ac-229d-40e4-9263-3f184726d148-maas-api-service-ca\") on node \"ip-10-0-141-139.ec2.internal\" DevicePath \"\""
Sep 11 17:44:33.413921 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:44:33.413893    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-290683-k6lkh"
Sep 11 17:44:33.414089 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:44:33.413892    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-290683-k6lkh" event={"ID":"3c3b23ac-229d-40e4-9263-3f184726d148","Type":"ContainerDied","Data":"7ba5b2b0f2a668a314dc7519d7a2aaeb748474c2ae9a0e153875444119975222"}
Sep 11 17:44:33.414089 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:44:33.414038    2577 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7ba5b2b0f2a668a314dc7519d7a2aaeb748474c2ae9a0e153875444119975222"
Sep 11 17:44:40.528373 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:44:40.528336    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-290683-k6lkh"]
Sep 11 17:44:40.531522 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:44:40.531479    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-290683-k6lkh"]
Sep 11 17:44:40.927897 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:44:40.927812    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3c3b23ac-229d-40e4-9263-3f184726d148" path="/var/lib/kubelet/pods/3c3b23ac-229d-40e4-9263-3f184726d148/volumes"
Sep 11 17:44:41.991951 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:44:41.991918    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-c7d192-jd2t6"]
Sep 11 17:44:41.992382 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:44:41.992368    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3c3b23ac-229d-40e4-9263-3f184726d148" containerName="revoke-keys"
Sep 11 17:44:41.992382 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:44:41.992383    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="3c3b23ac-229d-40e4-9263-3f184726d148" containerName="revoke-keys"
Sep 11 17:44:41.992474 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:44:41.992465    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="3c3b23ac-229d-40e4-9263-3f184726d148" containerName="revoke-keys"
Sep 11 17:44:41.995518 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:44:41.995485    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-c7d192-jd2t6"
Sep 11 17:44:41.997790 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:44:41.997768    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-9n8gf\""
Sep 11 17:44:41.997925 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:44:41.997793    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 11 17:44:42.003695 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:44:42.003669    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-c7d192-jd2t6"]
Sep 11 17:44:42.120085 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:44:42.120049    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/809c3e06-efa5-4d23-a084-7691a8a72d76-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-c7d192-jd2t6\" (UID: \"809c3e06-efa5-4d23-a084-7691a8a72d76\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-c7d192-jd2t6"
Sep 11 17:44:42.221554 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:44:42.221513    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/809c3e06-efa5-4d23-a084-7691a8a72d76-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-c7d192-jd2t6\" (UID: \"809c3e06-efa5-4d23-a084-7691a8a72d76\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-c7d192-jd2t6"
Sep 11 17:44:42.222090 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:44:42.222070    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/809c3e06-efa5-4d23-a084-7691a8a72d76-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-c7d192-jd2t6\" (UID: \"809c3e06-efa5-4d23-a084-7691a8a72d76\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-c7d192-jd2t6"
Sep 11 17:44:42.307113 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:44:42.307035    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-c7d192-jd2t6"
Sep 11 17:44:42.598213 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:44:42.598184    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-c7d192-jd2t6"]
Sep 11 17:44:42.600496 ip-10-0-141-139 kubenswrapper[2577]: W0911 17:44:42.600471    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod809c3e06_efa5_4d23_a084_7691a8a72d76.slice/crio-9da84884a0654a2a50e7fdebef51e33adcba73d51e6869c19c19c312452bacdd WatchSource:0}: Error finding container 9da84884a0654a2a50e7fdebef51e33adcba73d51e6869c19c19c312452bacdd: Status 404 returned error can't find the container with id 9da84884a0654a2a50e7fdebef51e33adcba73d51e6869c19c19c312452bacdd
Sep 11 17:44:43.472834 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:44:43.472793    2577 generic.go:356] "Generic (PLEG): container finished" podID="809c3e06-efa5-4d23-a084-7691a8a72d76" containerID="4760c3a8c4189836dba326d4f883217ac662b335e6bb5aeb61a540ccbfc04927" exitCode=0
Sep 11 17:44:43.473269 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:44:43.472883    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-c7d192-jd2t6" event={"ID":"809c3e06-efa5-4d23-a084-7691a8a72d76","Type":"ContainerDied","Data":"4760c3a8c4189836dba326d4f883217ac662b335e6bb5aeb61a540ccbfc04927"}
Sep 11 17:44:43.473269 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:44:43.472951    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-c7d192-jd2t6" event={"ID":"809c3e06-efa5-4d23-a084-7691a8a72d76","Type":"ContainerStarted","Data":"9da84884a0654a2a50e7fdebef51e33adcba73d51e6869c19c19c312452bacdd"}
Sep 11 17:44:44.616251 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:44:44.616217    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-c7d192-jd2t6"
Sep 11 17:44:44.750571 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:44:44.750481    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/809c3e06-efa5-4d23-a084-7691a8a72d76-maas-api-service-ca\") pod \"809c3e06-efa5-4d23-a084-7691a8a72d76\" (UID: \"809c3e06-efa5-4d23-a084-7691a8a72d76\") "
Sep 11 17:44:44.750858 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:44:44.750836    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/809c3e06-efa5-4d23-a084-7691a8a72d76-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "809c3e06-efa5-4d23-a084-7691a8a72d76" (UID: "809c3e06-efa5-4d23-a084-7691a8a72d76"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 17:44:44.851684 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:44:44.851648    2577 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/809c3e06-efa5-4d23-a084-7691a8a72d76-maas-api-service-ca\") on node \"ip-10-0-141-139.ec2.internal\" DevicePath \"\""
Sep 11 17:44:45.485873 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:44:45.485839    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-c7d192-jd2t6" event={"ID":"809c3e06-efa5-4d23-a084-7691a8a72d76","Type":"ContainerDied","Data":"9da84884a0654a2a50e7fdebef51e33adcba73d51e6869c19c19c312452bacdd"}
Sep 11 17:44:45.486053 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:44:45.485876    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-c7d192-jd2t6"
Sep 11 17:44:45.486053 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:44:45.485886    2577 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9da84884a0654a2a50e7fdebef51e33adcba73d51e6869c19c19c312452bacdd"
Sep 11 17:44:46.814900 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:44:46.814861    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-c7d192/distinct-w0-c7d192-kserve-645f648f77-lxn5g"]
Sep 11 17:44:46.824679 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:44:46.824650    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-c7d192/embedding-w0-c7d192-kserve-5d5bfbb578-22zmh"]
Sep 11 17:44:46.838354 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:44:46.838324    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-c7d192/premium-w0-c7d192-kserve-6f977d4fcd-n9nhd"]
Sep 11 17:44:46.860008 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:44:46.859977    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-c7d192/simulator-w0-c7d192-kserve-6bfc765cf5-klnxw"]
Sep 11 17:44:49.201062 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:44:49.201022    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-2f0a78-j7rgn"]
Sep 11 17:44:49.201471 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:44:49.201460    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="809c3e06-efa5-4d23-a084-7691a8a72d76" containerName="revoke-keys"
Sep 11 17:44:49.201471 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:44:49.201471    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="809c3e06-efa5-4d23-a084-7691a8a72d76" containerName="revoke-keys"
Sep 11 17:44:49.201605 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:44:49.201561    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="809c3e06-efa5-4d23-a084-7691a8a72d76" containerName="revoke-keys"
Sep 11 17:44:49.207473 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:44:49.207456    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-2f0a78-j7rgn"
Sep 11 17:44:49.210160 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:44:49.210140    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 11 17:44:49.210272 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:44:49.210247    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-9n8gf\""
Sep 11 17:44:49.241444 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:44:49.241413    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-2f0a78-j7rgn"]
Sep 11 17:44:49.301366 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:44:49.301332    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/89a6c24b-c1fd-42c7-9a58-1745a3542f3f-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-2f0a78-j7rgn\" (UID: \"89a6c24b-c1fd-42c7-9a58-1745a3542f3f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-2f0a78-j7rgn"
Sep 11 17:44:49.402091 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:44:49.402054    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/89a6c24b-c1fd-42c7-9a58-1745a3542f3f-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-2f0a78-j7rgn\" (UID: \"89a6c24b-c1fd-42c7-9a58-1745a3542f3f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-2f0a78-j7rgn"
Sep 11 17:44:49.402778 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:44:49.402753    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/89a6c24b-c1fd-42c7-9a58-1745a3542f3f-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-2f0a78-j7rgn\" (UID: \"89a6c24b-c1fd-42c7-9a58-1745a3542f3f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-2f0a78-j7rgn"
Sep 11 17:44:49.518290 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:44:49.518247    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-2f0a78-j7rgn"
Sep 11 17:44:49.562119 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:44:49.562089    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-nkmkm"]
Sep 11 17:44:49.804096 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:44:49.804071    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-2f0a78-j7rgn"]
Sep 11 17:44:49.805741 ip-10-0-141-139 kubenswrapper[2577]: W0911 17:44:49.805718    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod89a6c24b_c1fd_42c7_9a58_1745a3542f3f.slice/crio-6f86b69c3f0083d78fb2af1f7e1469e043c977b63933284138bb7c22421576a4 WatchSource:0}: Error finding container 6f86b69c3f0083d78fb2af1f7e1469e043c977b63933284138bb7c22421576a4: Status 404 returned error can't find the container with id 6f86b69c3f0083d78fb2af1f7e1469e043c977b63933284138bb7c22421576a4
Sep 11 17:44:50.515608 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:44:50.515563    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-2f0a78-j7rgn" event={"ID":"89a6c24b-c1fd-42c7-9a58-1745a3542f3f","Type":"ContainerStarted","Data":"455969995a4c1f06b3052488a4f23d256c2b245f1e8603b3abfa7f9598879c91"}
Sep 11 17:44:50.516077 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:44:50.515633    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-2f0a78-j7rgn" event={"ID":"89a6c24b-c1fd-42c7-9a58-1745a3542f3f","Type":"ContainerStarted","Data":"6f86b69c3f0083d78fb2af1f7e1469e043c977b63933284138bb7c22421576a4"}
Sep 11 17:44:50.541015 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:44:50.540964    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-2f0a78-j7rgn" podStartSLOduration=1.540948045 podStartE2EDuration="1.540948045s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 17:44:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 17:44:50.539991493 +0000 UTC m=+1568.113211604" watchObservedRunningTime="2026-09-11 17:44:50.540948045 +0000 UTC m=+1568.114168156"
Sep 11 17:44:51.522669 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:44:51.522633    2577 generic.go:356] "Generic (PLEG): container finished" podID="89a6c24b-c1fd-42c7-9a58-1745a3542f3f" containerID="455969995a4c1f06b3052488a4f23d256c2b245f1e8603b3abfa7f9598879c91" exitCode=0
Sep 11 17:44:51.523054 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:44:51.522706    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-2f0a78-j7rgn" event={"ID":"89a6c24b-c1fd-42c7-9a58-1745a3542f3f","Type":"ContainerDied","Data":"455969995a4c1f06b3052488a4f23d256c2b245f1e8603b3abfa7f9598879c91"}
Sep 11 17:44:52.675768 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:44:52.675740    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-2f0a78-j7rgn"
Sep 11 17:44:52.838254 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:44:52.838164    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/89a6c24b-c1fd-42c7-9a58-1745a3542f3f-maas-api-service-ca\") pod \"89a6c24b-c1fd-42c7-9a58-1745a3542f3f\" (UID: \"89a6c24b-c1fd-42c7-9a58-1745a3542f3f\") "
Sep 11 17:44:52.838554 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:44:52.838527    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/89a6c24b-c1fd-42c7-9a58-1745a3542f3f-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "89a6c24b-c1fd-42c7-9a58-1745a3542f3f" (UID: "89a6c24b-c1fd-42c7-9a58-1745a3542f3f"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 17:44:52.939943 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:44:52.939909    2577 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/89a6c24b-c1fd-42c7-9a58-1745a3542f3f-maas-api-service-ca\") on node \"ip-10-0-141-139.ec2.internal\" DevicePath \"\""
Sep 11 17:44:53.536698 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:44:53.536666    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-2f0a78-j7rgn"
Sep 11 17:44:53.536880 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:44:53.536672    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-2f0a78-j7rgn" event={"ID":"89a6c24b-c1fd-42c7-9a58-1745a3542f3f","Type":"ContainerDied","Data":"6f86b69c3f0083d78fb2af1f7e1469e043c977b63933284138bb7c22421576a4"}
Sep 11 17:44:53.536880 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:44:53.536784    2577 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6f86b69c3f0083d78fb2af1f7e1469e043c977b63933284138bb7c22421576a4"
Sep 11 17:44:57.588233 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:44:57.588186    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-c7d192-67844f9ccf-5kttv"]
Sep 11 17:44:57.588718 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:44:57.588440    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w0-c7d192-67844f9ccf-5kttv" podUID="cb6ad0f4-dfc7-4a72-91f1-4612e23f1bbe" containerName="payload-processing" containerID="cri-o://d13559ac08f3a8f852fa18ed2bde7109545647482e12198a7848c632a9715b4b" gracePeriod=30
Sep 11 17:44:57.603472 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:44:57.603434    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-c7d192-fb57bbbcd-nkspc"]
Sep 11 17:44:57.603762 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:44:57.603735    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-c7d192-fb57bbbcd-nkspc" podUID="d8aafb09-db77-4dce-9138-fb92ed438794" containerName="payload-pre-processing" containerID="cri-o://e8a1d60d0cdccd32768b1f8cf3a352466610194311e32547ca40236c5850f67d" gracePeriod=30
Sep 11 17:44:57.852855 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:44:57.852770    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-c7d192-jd2t6"]
Sep 11 17:44:57.856269 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:44:57.856242    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-c7d192-jd2t6"]
Sep 11 17:44:57.893791 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:44:57.893753    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-c7d192-67844f9ccf-5kttv"
Sep 11 17:44:57.896786 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:44:57.896764    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-c7d192-fb57bbbcd-nkspc"
Sep 11 17:44:57.994877 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:44:57.994842    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cb6ad0f4-dfc7-4a72-91f1-4612e23f1bbe-plugins-config-volume\") pod \"cb6ad0f4-dfc7-4a72-91f1-4612e23f1bbe\" (UID: \"cb6ad0f4-dfc7-4a72-91f1-4612e23f1bbe\") "
Sep 11 17:44:57.995047 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:44:57.994910    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wg876\" (UniqueName: \"kubernetes.io/projected/cb6ad0f4-dfc7-4a72-91f1-4612e23f1bbe-kube-api-access-wg876\") pod \"cb6ad0f4-dfc7-4a72-91f1-4612e23f1bbe\" (UID: \"cb6ad0f4-dfc7-4a72-91f1-4612e23f1bbe\") "
Sep 11 17:44:57.995047 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:44:57.994995    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d8aafb09-db77-4dce-9138-fb92ed438794-plugins-config-volume\") pod \"d8aafb09-db77-4dce-9138-fb92ed438794\" (UID: \"d8aafb09-db77-4dce-9138-fb92ed438794\") "
Sep 11 17:44:57.995130 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:44:57.995086    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-k452z\" (UniqueName: \"kubernetes.io/projected/d8aafb09-db77-4dce-9138-fb92ed438794-kube-api-access-k452z\") pod \"d8aafb09-db77-4dce-9138-fb92ed438794\" (UID: \"d8aafb09-db77-4dce-9138-fb92ed438794\") "
Sep 11 17:44:57.995299 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:44:57.995276    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cb6ad0f4-dfc7-4a72-91f1-4612e23f1bbe-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "cb6ad0f4-dfc7-4a72-91f1-4612e23f1bbe" (UID: "cb6ad0f4-dfc7-4a72-91f1-4612e23f1bbe"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 17:44:57.995393 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:44:57.995352    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d8aafb09-db77-4dce-9138-fb92ed438794-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "d8aafb09-db77-4dce-9138-fb92ed438794" (UID: "d8aafb09-db77-4dce-9138-fb92ed438794"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 17:44:57.995476 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:44:57.995462    2577 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d8aafb09-db77-4dce-9138-fb92ed438794-plugins-config-volume\") on node \"ip-10-0-141-139.ec2.internal\" DevicePath \"\""
Sep 11 17:44:57.995536 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:44:57.995478    2577 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cb6ad0f4-dfc7-4a72-91f1-4612e23f1bbe-plugins-config-volume\") on node \"ip-10-0-141-139.ec2.internal\" DevicePath \"\""
Sep 11 17:44:57.997262 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:44:57.997239    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cb6ad0f4-dfc7-4a72-91f1-4612e23f1bbe-kube-api-access-wg876" (OuterVolumeSpecName: "kube-api-access-wg876") pod "cb6ad0f4-dfc7-4a72-91f1-4612e23f1bbe" (UID: "cb6ad0f4-dfc7-4a72-91f1-4612e23f1bbe"). InnerVolumeSpecName "kube-api-access-wg876". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 17:44:57.997336 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:44:57.997304    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d8aafb09-db77-4dce-9138-fb92ed438794-kube-api-access-k452z" (OuterVolumeSpecName: "kube-api-access-k452z") pod "d8aafb09-db77-4dce-9138-fb92ed438794" (UID: "d8aafb09-db77-4dce-9138-fb92ed438794"). InnerVolumeSpecName "kube-api-access-k452z". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 17:44:58.097084 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:44:58.097051    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k452z\" (UniqueName: \"kubernetes.io/projected/d8aafb09-db77-4dce-9138-fb92ed438794-kube-api-access-k452z\") on node \"ip-10-0-141-139.ec2.internal\" DevicePath \"\""
Sep 11 17:44:58.097084 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:44:58.097083    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wg876\" (UniqueName: \"kubernetes.io/projected/cb6ad0f4-dfc7-4a72-91f1-4612e23f1bbe-kube-api-access-wg876\") on node \"ip-10-0-141-139.ec2.internal\" DevicePath \"\""
Sep 11 17:44:58.566320 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:44:58.566279    2577 generic.go:356] "Generic (PLEG): container finished" podID="cb6ad0f4-dfc7-4a72-91f1-4612e23f1bbe" containerID="d13559ac08f3a8f852fa18ed2bde7109545647482e12198a7848c632a9715b4b" exitCode=0
Sep 11 17:44:58.566532 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:44:58.566346    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-c7d192-67844f9ccf-5kttv"
Sep 11 17:44:58.566532 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:44:58.566358    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-c7d192-67844f9ccf-5kttv" event={"ID":"cb6ad0f4-dfc7-4a72-91f1-4612e23f1bbe","Type":"ContainerDied","Data":"d13559ac08f3a8f852fa18ed2bde7109545647482e12198a7848c632a9715b4b"}
Sep 11 17:44:58.566532 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:44:58.566444    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-c7d192-67844f9ccf-5kttv" event={"ID":"cb6ad0f4-dfc7-4a72-91f1-4612e23f1bbe","Type":"ContainerDied","Data":"10c385af04a76f51c161878f82b8fbe0546cea20a3c27d27235b4febe23ce3b6"}
Sep 11 17:44:58.566532 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:44:58.566475    2577 scope.go:117] "RemoveContainer" containerID="d13559ac08f3a8f852fa18ed2bde7109545647482e12198a7848c632a9715b4b"
Sep 11 17:44:58.567898 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:44:58.567832    2577 generic.go:356] "Generic (PLEG): container finished" podID="d8aafb09-db77-4dce-9138-fb92ed438794" containerID="e8a1d60d0cdccd32768b1f8cf3a352466610194311e32547ca40236c5850f67d" exitCode=0
Sep 11 17:44:58.567898 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:44:58.567868    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-c7d192-fb57bbbcd-nkspc" event={"ID":"d8aafb09-db77-4dce-9138-fb92ed438794","Type":"ContainerDied","Data":"e8a1d60d0cdccd32768b1f8cf3a352466610194311e32547ca40236c5850f67d"}
Sep 11 17:44:58.568051 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:44:58.567906    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-c7d192-fb57bbbcd-nkspc"
Sep 11 17:44:58.568051 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:44:58.567922    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-c7d192-fb57bbbcd-nkspc" event={"ID":"d8aafb09-db77-4dce-9138-fb92ed438794","Type":"ContainerDied","Data":"2bd7bb9259da560b031f75c0c9e236b61d784c64e8e0df8f4aaeac5c6436b9d0"}
Sep 11 17:44:58.576467 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:44:58.576442    2577 scope.go:117] "RemoveContainer" containerID="d13559ac08f3a8f852fa18ed2bde7109545647482e12198a7848c632a9715b4b"
Sep 11 17:44:58.576869 ip-10-0-141-139 kubenswrapper[2577]: E0911 17:44:58.576842    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d13559ac08f3a8f852fa18ed2bde7109545647482e12198a7848c632a9715b4b\": container with ID starting with d13559ac08f3a8f852fa18ed2bde7109545647482e12198a7848c632a9715b4b not found: ID does not exist" containerID="d13559ac08f3a8f852fa18ed2bde7109545647482e12198a7848c632a9715b4b"
Sep 11 17:44:58.576955 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:44:58.576879    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d13559ac08f3a8f852fa18ed2bde7109545647482e12198a7848c632a9715b4b"} err="failed to get container status \"d13559ac08f3a8f852fa18ed2bde7109545647482e12198a7848c632a9715b4b\": rpc error: code = NotFound desc = could not find container \"d13559ac08f3a8f852fa18ed2bde7109545647482e12198a7848c632a9715b4b\": container with ID starting with d13559ac08f3a8f852fa18ed2bde7109545647482e12198a7848c632a9715b4b not found: ID does not exist"
Sep 11 17:44:58.576955 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:44:58.576904    2577 scope.go:117] "RemoveContainer" containerID="e8a1d60d0cdccd32768b1f8cf3a352466610194311e32547ca40236c5850f67d"
Sep 11 17:44:58.585840 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:44:58.585820    2577 scope.go:117] "RemoveContainer" containerID="e8a1d60d0cdccd32768b1f8cf3a352466610194311e32547ca40236c5850f67d"
Sep 11 17:44:58.586159 ip-10-0-141-139 kubenswrapper[2577]: E0911 17:44:58.586138    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e8a1d60d0cdccd32768b1f8cf3a352466610194311e32547ca40236c5850f67d\": container with ID starting with e8a1d60d0cdccd32768b1f8cf3a352466610194311e32547ca40236c5850f67d not found: ID does not exist" containerID="e8a1d60d0cdccd32768b1f8cf3a352466610194311e32547ca40236c5850f67d"
Sep 11 17:44:58.586210 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:44:58.586168    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e8a1d60d0cdccd32768b1f8cf3a352466610194311e32547ca40236c5850f67d"} err="failed to get container status \"e8a1d60d0cdccd32768b1f8cf3a352466610194311e32547ca40236c5850f67d\": rpc error: code = NotFound desc = could not find container \"e8a1d60d0cdccd32768b1f8cf3a352466610194311e32547ca40236c5850f67d\": container with ID starting with e8a1d60d0cdccd32768b1f8cf3a352466610194311e32547ca40236c5850f67d not found: ID does not exist"
Sep 11 17:44:58.594854 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:44:58.594828    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-c7d192-67844f9ccf-5kttv"]
Sep 11 17:44:58.601514 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:44:58.601472    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-c7d192-67844f9ccf-5kttv"]
Sep 11 17:44:58.616339 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:44:58.616310    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-c7d192-fb57bbbcd-nkspc"]
Sep 11 17:44:58.620139 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:44:58.620113    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-c7d192-fb57bbbcd-nkspc"]
Sep 11 17:44:58.927345 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:44:58.927245    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="809c3e06-efa5-4d23-a084-7691a8a72d76" path="/var/lib/kubelet/pods/809c3e06-efa5-4d23-a084-7691a8a72d76/volumes"
Sep 11 17:44:58.927768 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:44:58.927748    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="cb6ad0f4-dfc7-4a72-91f1-4612e23f1bbe" path="/var/lib/kubelet/pods/cb6ad0f4-dfc7-4a72-91f1-4612e23f1bbe/volumes"
Sep 11 17:44:58.928148 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:44:58.928129    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d8aafb09-db77-4dce-9138-fb92ed438794" path="/var/lib/kubelet/pods/d8aafb09-db77-4dce-9138-fb92ed438794/volumes"
Sep 11 17:45:00.154172 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:45:00.154144    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-144bfa-29819145-r5fn7"]
Sep 11 17:45:00.154725 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:45:00.154617    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="89a6c24b-c1fd-42c7-9a58-1745a3542f3f" containerName="revoke-keys"
Sep 11 17:45:00.154725 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:45:00.154632    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="89a6c24b-c1fd-42c7-9a58-1745a3542f3f" containerName="revoke-keys"
Sep 11 17:45:00.154725 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:45:00.154643    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d8aafb09-db77-4dce-9138-fb92ed438794" containerName="payload-pre-processing"
Sep 11 17:45:00.154725 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:45:00.154650    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="d8aafb09-db77-4dce-9138-fb92ed438794" containerName="payload-pre-processing"
Sep 11 17:45:00.154725 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:45:00.154666    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cb6ad0f4-dfc7-4a72-91f1-4612e23f1bbe" containerName="payload-processing"
Sep 11 17:45:00.154725 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:45:00.154671    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="cb6ad0f4-dfc7-4a72-91f1-4612e23f1bbe" containerName="payload-processing"
Sep 11 17:45:00.155060 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:45:00.154736    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="d8aafb09-db77-4dce-9138-fb92ed438794" containerName="payload-pre-processing"
Sep 11 17:45:00.155060 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:45:00.154746    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="cb6ad0f4-dfc7-4a72-91f1-4612e23f1bbe" containerName="payload-processing"
Sep 11 17:45:00.155060 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:45:00.154758    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="89a6c24b-c1fd-42c7-9a58-1745a3542f3f" containerName="revoke-keys"
Sep 11 17:45:00.159339 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:45:00.159317    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29819145-k7xm4"]
Sep 11 17:45:00.159477 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:45:00.159446    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-144bfa-29819145-r5fn7"
Sep 11 17:45:00.162257 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:45:00.162235    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-9n8gf\""
Sep 11 17:45:00.162707 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:45:00.162692    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29819145-k7xm4"
Sep 11 17:45:00.168005 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:45:00.167981    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-144bfa-29819145-r5fn7"]
Sep 11 17:45:00.173784 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:45:00.173757    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-144bfa-29819145-r5fn7"
Sep 11 17:45:00.180924 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:45:00.180900    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29819145-k7xm4"
Sep 11 17:45:00.186252 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:45:00.186229    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29819145-k7xm4"]
Sep 11 17:45:00.402124 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:45:00.402088    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-2f0a78-j7rgn"]
Sep 11 17:45:00.408148 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:45:00.408075    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-2f0a78-j7rgn"]
Sep 11 17:45:00.461693 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:45:00.461663    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-144bfa-29819145-r5fn7"]
Sep 11 17:45:00.463321 ip-10-0-141-139 kubenswrapper[2577]: W0911 17:45:00.463300    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0c347dc7_e953_4728_a52d_702a4e0291a8.slice/crio-aabc7dfa1bdc07936acc5691ec9089663e372406028bda3fe0444a4dfa1ebfa9 WatchSource:0}: Error finding container aabc7dfa1bdc07936acc5691ec9089663e372406028bda3fe0444a4dfa1ebfa9: Status 404 returned error can't find the container with id aabc7dfa1bdc07936acc5691ec9089663e372406028bda3fe0444a4dfa1ebfa9
Sep 11 17:45:00.470343 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:45:00.470323    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29819145-k7xm4"]
Sep 11 17:45:00.472849 ip-10-0-141-139 kubenswrapper[2577]: W0911 17:45:00.472823    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podce07d5c0_d2f5_40b5_a2d7_1cf5ddf1242d.slice/crio-931ae1fe94bdc26c3fc38401d80dc18c61a6e52bdc12d4e1c6f3b87cfaab164f WatchSource:0}: Error finding container 931ae1fe94bdc26c3fc38401d80dc18c61a6e52bdc12d4e1c6f3b87cfaab164f: Status 404 returned error can't find the container with id 931ae1fe94bdc26c3fc38401d80dc18c61a6e52bdc12d4e1c6f3b87cfaab164f
Sep 11 17:45:00.585234 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:45:00.585197    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-144bfa-29819145-r5fn7" event={"ID":"0c347dc7-e953-4728-a52d-702a4e0291a8","Type":"ContainerStarted","Data":"aabc7dfa1bdc07936acc5691ec9089663e372406028bda3fe0444a4dfa1ebfa9"}
Sep 11 17:45:00.586529 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:45:00.586471    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29819145-k7xm4" event={"ID":"ce07d5c0-d2f5-40b5-a2d7-1cf5ddf1242d","Type":"ContainerStarted","Data":"931ae1fe94bdc26c3fc38401d80dc18c61a6e52bdc12d4e1c6f3b87cfaab164f"}
Sep 11 17:45:00.928965 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:45:00.928882    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="89a6c24b-c1fd-42c7-9a58-1745a3542f3f" path="/var/lib/kubelet/pods/89a6c24b-c1fd-42c7-9a58-1745a3542f3f/volumes"
Sep 11 17:45:01.594345 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:45:01.594302    2577 generic.go:356] "Generic (PLEG): container finished" podID="0c347dc7-e953-4728-a52d-702a4e0291a8" containerID="1b90bdaf9b8019047386e1929065eee3c082aef16ac0c68c7362d04969a586d9" exitCode=0
Sep 11 17:45:01.594802 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:45:01.594391    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-144bfa-29819145-r5fn7" event={"ID":"0c347dc7-e953-4728-a52d-702a4e0291a8","Type":"ContainerDied","Data":"1b90bdaf9b8019047386e1929065eee3c082aef16ac0c68c7362d04969a586d9"}
Sep 11 17:45:01.596129 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:45:01.596106    2577 generic.go:356] "Generic (PLEG): container finished" podID="ce07d5c0-d2f5-40b5-a2d7-1cf5ddf1242d" containerID="de8f24cf1cef5c776232f4db713ec2323820d68104f2d1b92a3994c4e2911872" exitCode=0
Sep 11 17:45:01.596253 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:45:01.596150    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29819145-k7xm4" event={"ID":"ce07d5c0-d2f5-40b5-a2d7-1cf5ddf1242d","Type":"ContainerDied","Data":"de8f24cf1cef5c776232f4db713ec2323820d68104f2d1b92a3994c4e2911872"}
Sep 11 17:45:01.835078 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:45:01.835043    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-c7d192/distinct-w0-c7d192-kserve-645f648f77-lxn5g" podUID="66e0a151-0ad9-46ce-9d4e-d0f2f8b03d4d" containerName="main" containerID="cri-o://8fb5e9789ab3a1fbf975b73256bd830adb7948376aadb7d182f466e765ec6ba7" gracePeriod=45
Sep 11 17:45:01.844524 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:45:01.844466    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-c7d192/embedding-w0-c7d192-kserve-5d5bfbb578-22zmh" podUID="5aa9157d-fd25-4e77-bb11-b0fd137bf739" containerName="main" containerID="cri-o://6b0342a195c4b24066cdc90cc0361ba917590668658224032685ff839ef05c78" gracePeriod=45
Sep 11 17:45:01.857704 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:45:01.857657    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-c7d192/premium-w0-c7d192-kserve-6f977d4fcd-n9nhd" podUID="55806c17-765a-43a1-babc-46540c0423bd" containerName="main" containerID="cri-o://d07916d97da58fa3f509688c4b9efe0ccdf7310d10eb8e8f733ef1bc63967c60" gracePeriod=45
Sep 11 17:45:01.878770 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:45:01.878730    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-c7d192/simulator-w0-c7d192-kserve-6bfc765cf5-klnxw" podUID="c30aa08c-b2d4-4edc-8561-e78ad422fd1f" containerName="main" containerID="cri-o://4967c83f7832a12d0f739396188f205f644df6605ff208a482fbc3255a33323a" gracePeriod=45
Sep 11 17:45:02.143566 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:45:02.143466    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-c7d192/embedding-w0-c7d192-kserve-5d5bfbb578-22zmh"
Sep 11 17:45:02.234635 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:45:02.234594    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-c7d192/premium-w0-c7d192-kserve-6f977d4fcd-n9nhd"
Sep 11 17:45:02.237903 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:45:02.237880    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-c7d192/simulator-w0-c7d192-kserve-6bfc765cf5-klnxw"
Sep 11 17:45:02.239781 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:45:02.239758    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5aa9157d-fd25-4e77-bb11-b0fd137bf739-tls-certs\") pod \"5aa9157d-fd25-4e77-bb11-b0fd137bf739\" (UID: \"5aa9157d-fd25-4e77-bb11-b0fd137bf739\") "
Sep 11 17:45:02.239887 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:45:02.239799    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-65gcl\" (UniqueName: \"kubernetes.io/projected/5aa9157d-fd25-4e77-bb11-b0fd137bf739-kube-api-access-65gcl\") pod \"5aa9157d-fd25-4e77-bb11-b0fd137bf739\" (UID: \"5aa9157d-fd25-4e77-bb11-b0fd137bf739\") "
Sep 11 17:45:02.239887 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:45:02.239855    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5aa9157d-fd25-4e77-bb11-b0fd137bf739-model-cache\") pod \"5aa9157d-fd25-4e77-bb11-b0fd137bf739\" (UID: \"5aa9157d-fd25-4e77-bb11-b0fd137bf739\") "
Sep 11 17:45:02.240009 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:45:02.239901    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5aa9157d-fd25-4e77-bb11-b0fd137bf739-dshm\") pod \"5aa9157d-fd25-4e77-bb11-b0fd137bf739\" (UID: \"5aa9157d-fd25-4e77-bb11-b0fd137bf739\") "
Sep 11 17:45:02.240009 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:45:02.239997    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5aa9157d-fd25-4e77-bb11-b0fd137bf739-home\") pod \"5aa9157d-fd25-4e77-bb11-b0fd137bf739\" (UID: \"5aa9157d-fd25-4e77-bb11-b0fd137bf739\") "
Sep 11 17:45:02.240122 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:45:02.240047    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5aa9157d-fd25-4e77-bb11-b0fd137bf739-tmp-dir\") pod \"5aa9157d-fd25-4e77-bb11-b0fd137bf739\" (UID: \"5aa9157d-fd25-4e77-bb11-b0fd137bf739\") "
Sep 11 17:45:02.240302 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:45:02.240270    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5aa9157d-fd25-4e77-bb11-b0fd137bf739-home" (OuterVolumeSpecName: "home") pod "5aa9157d-fd25-4e77-bb11-b0fd137bf739" (UID: "5aa9157d-fd25-4e77-bb11-b0fd137bf739"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 17:45:02.240411 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:45:02.240382    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5aa9157d-fd25-4e77-bb11-b0fd137bf739-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "5aa9157d-fd25-4e77-bb11-b0fd137bf739" (UID: "5aa9157d-fd25-4e77-bb11-b0fd137bf739"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 17:45:02.240462 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:45:02.240165    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5aa9157d-fd25-4e77-bb11-b0fd137bf739-model-cache" (OuterVolumeSpecName: "model-cache") pod "5aa9157d-fd25-4e77-bb11-b0fd137bf739" (UID: "5aa9157d-fd25-4e77-bb11-b0fd137bf739"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 17:45:02.240546 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:45:02.240525    2577 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5aa9157d-fd25-4e77-bb11-b0fd137bf739-tmp-dir\") on node \"ip-10-0-141-139.ec2.internal\" DevicePath \"\""
Sep 11 17:45:02.240615 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:45:02.240574    2577 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5aa9157d-fd25-4e77-bb11-b0fd137bf739-model-cache\") on node \"ip-10-0-141-139.ec2.internal\" DevicePath \"\""
Sep 11 17:45:02.240615 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:45:02.240587    2577 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5aa9157d-fd25-4e77-bb11-b0fd137bf739-home\") on node \"ip-10-0-141-139.ec2.internal\" DevicePath \"\""
Sep 11 17:45:02.241743 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:45:02.241722    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-c7d192/distinct-w0-c7d192-kserve-645f648f77-lxn5g"
Sep 11 17:45:02.242424 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:45:02.242403    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5aa9157d-fd25-4e77-bb11-b0fd137bf739-dshm" (OuterVolumeSpecName: "dshm") pod "5aa9157d-fd25-4e77-bb11-b0fd137bf739" (UID: "5aa9157d-fd25-4e77-bb11-b0fd137bf739"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 17:45:02.242622 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:45:02.242602    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5aa9157d-fd25-4e77-bb11-b0fd137bf739-kube-api-access-65gcl" (OuterVolumeSpecName: "kube-api-access-65gcl") pod "5aa9157d-fd25-4e77-bb11-b0fd137bf739" (UID: "5aa9157d-fd25-4e77-bb11-b0fd137bf739"). InnerVolumeSpecName "kube-api-access-65gcl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 17:45:02.242769 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:45:02.242748    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5aa9157d-fd25-4e77-bb11-b0fd137bf739-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "5aa9157d-fd25-4e77-bb11-b0fd137bf739" (UID: "5aa9157d-fd25-4e77-bb11-b0fd137bf739"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 17:45:02.341644 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:45:02.341538    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/66e0a151-0ad9-46ce-9d4e-d0f2f8b03d4d-tls-certs\") pod \"66e0a151-0ad9-46ce-9d4e-d0f2f8b03d4d\" (UID: \"66e0a151-0ad9-46ce-9d4e-d0f2f8b03d4d\") "
Sep 11 17:45:02.341644 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:45:02.341588    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/66e0a151-0ad9-46ce-9d4e-d0f2f8b03d4d-home\") pod \"66e0a151-0ad9-46ce-9d4e-d0f2f8b03d4d\" (UID: \"66e0a151-0ad9-46ce-9d4e-d0f2f8b03d4d\") "
Sep 11 17:45:02.341644 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:45:02.341621    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c30aa08c-b2d4-4edc-8561-e78ad422fd1f-model-cache\") pod \"c30aa08c-b2d4-4edc-8561-e78ad422fd1f\" (UID: \"c30aa08c-b2d4-4edc-8561-e78ad422fd1f\") "
Sep 11 17:45:02.341926 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:45:02.341679    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7csn8\" (UniqueName: \"kubernetes.io/projected/66e0a151-0ad9-46ce-9d4e-d0f2f8b03d4d-kube-api-access-7csn8\") pod \"66e0a151-0ad9-46ce-9d4e-d0f2f8b03d4d\" (UID: \"66e0a151-0ad9-46ce-9d4e-d0f2f8b03d4d\") "
Sep 11 17:45:02.341926 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:45:02.341717    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/55806c17-765a-43a1-babc-46540c0423bd-model-cache\") pod \"55806c17-765a-43a1-babc-46540c0423bd\" (UID: \"55806c17-765a-43a1-babc-46540c0423bd\") "
Sep 11 17:45:02.341926 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:45:02.341744    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/55806c17-765a-43a1-babc-46540c0423bd-tmp-dir\") pod \"55806c17-765a-43a1-babc-46540c0423bd\" (UID: \"55806c17-765a-43a1-babc-46540c0423bd\") "
Sep 11 17:45:02.341926 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:45:02.341772    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/66e0a151-0ad9-46ce-9d4e-d0f2f8b03d4d-dshm\") pod \"66e0a151-0ad9-46ce-9d4e-d0f2f8b03d4d\" (UID: \"66e0a151-0ad9-46ce-9d4e-d0f2f8b03d4d\") "
Sep 11 17:45:02.341926 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:45:02.341796    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/66e0a151-0ad9-46ce-9d4e-d0f2f8b03d4d-model-cache\") pod \"66e0a151-0ad9-46ce-9d4e-d0f2f8b03d4d\" (UID: \"66e0a151-0ad9-46ce-9d4e-d0f2f8b03d4d\") "
Sep 11 17:45:02.341926 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:45:02.341829    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-m4drh\" (UniqueName: \"kubernetes.io/projected/c30aa08c-b2d4-4edc-8561-e78ad422fd1f-kube-api-access-m4drh\") pod \"c30aa08c-b2d4-4edc-8561-e78ad422fd1f\" (UID: \"c30aa08c-b2d4-4edc-8561-e78ad422fd1f\") "
Sep 11 17:45:02.341926 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:45:02.341866    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c30aa08c-b2d4-4edc-8561-e78ad422fd1f-home\") pod \"c30aa08c-b2d4-4edc-8561-e78ad422fd1f\" (UID: \"c30aa08c-b2d4-4edc-8561-e78ad422fd1f\") "
Sep 11 17:45:02.341926 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:45:02.341880    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c30aa08c-b2d4-4edc-8561-e78ad422fd1f-model-cache" (OuterVolumeSpecName: "model-cache") pod "c30aa08c-b2d4-4edc-8561-e78ad422fd1f" (UID: "c30aa08c-b2d4-4edc-8561-e78ad422fd1f"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 17:45:02.341926 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:45:02.341903    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fjnp9\" (UniqueName: \"kubernetes.io/projected/55806c17-765a-43a1-babc-46540c0423bd-kube-api-access-fjnp9\") pod \"55806c17-765a-43a1-babc-46540c0423bd\" (UID: \"55806c17-765a-43a1-babc-46540c0423bd\") "
Sep 11 17:45:02.341926 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:45:02.341880    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/66e0a151-0ad9-46ce-9d4e-d0f2f8b03d4d-home" (OuterVolumeSpecName: "home") pod "66e0a151-0ad9-46ce-9d4e-d0f2f8b03d4d" (UID: "66e0a151-0ad9-46ce-9d4e-d0f2f8b03d4d"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 17:45:02.342479 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:45:02.341984    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/66e0a151-0ad9-46ce-9d4e-d0f2f8b03d4d-tmp-dir\") pod \"66e0a151-0ad9-46ce-9d4e-d0f2f8b03d4d\" (UID: \"66e0a151-0ad9-46ce-9d4e-d0f2f8b03d4d\") "
Sep 11 17:45:02.342479 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:45:02.342003    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/55806c17-765a-43a1-babc-46540c0423bd-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "55806c17-765a-43a1-babc-46540c0423bd" (UID: "55806c17-765a-43a1-babc-46540c0423bd"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 17:45:02.342479 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:45:02.342030    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/55806c17-765a-43a1-babc-46540c0423bd-tls-certs\") pod \"55806c17-765a-43a1-babc-46540c0423bd\" (UID: \"55806c17-765a-43a1-babc-46540c0423bd\") "
Sep 11 17:45:02.342479 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:45:02.342062    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/55806c17-765a-43a1-babc-46540c0423bd-dshm\") pod \"55806c17-765a-43a1-babc-46540c0423bd\" (UID: \"55806c17-765a-43a1-babc-46540c0423bd\") "
Sep 11 17:45:02.342479 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:45:02.342088    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c30aa08c-b2d4-4edc-8561-e78ad422fd1f-tls-certs\") pod \"c30aa08c-b2d4-4edc-8561-e78ad422fd1f\" (UID: \"c30aa08c-b2d4-4edc-8561-e78ad422fd1f\") "
Sep 11 17:45:02.342479 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:45:02.342120    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c30aa08c-b2d4-4edc-8561-e78ad422fd1f-dshm\") pod \"c30aa08c-b2d4-4edc-8561-e78ad422fd1f\" (UID: \"c30aa08c-b2d4-4edc-8561-e78ad422fd1f\") "
Sep 11 17:45:02.342479 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:45:02.342126    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/66e0a151-0ad9-46ce-9d4e-d0f2f8b03d4d-model-cache" (OuterVolumeSpecName: "model-cache") pod "66e0a151-0ad9-46ce-9d4e-d0f2f8b03d4d" (UID: "66e0a151-0ad9-46ce-9d4e-d0f2f8b03d4d"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 17:45:02.342479 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:45:02.342161    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c30aa08c-b2d4-4edc-8561-e78ad422fd1f-tmp-dir\") pod \"c30aa08c-b2d4-4edc-8561-e78ad422fd1f\" (UID: \"c30aa08c-b2d4-4edc-8561-e78ad422fd1f\") "
Sep 11 17:45:02.342479 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:45:02.342205    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/55806c17-765a-43a1-babc-46540c0423bd-home\") pod \"55806c17-765a-43a1-babc-46540c0423bd\" (UID: \"55806c17-765a-43a1-babc-46540c0423bd\") "
Sep 11 17:45:02.342479 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:45:02.342213    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/66e0a151-0ad9-46ce-9d4e-d0f2f8b03d4d-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "66e0a151-0ad9-46ce-9d4e-d0f2f8b03d4d" (UID: "66e0a151-0ad9-46ce-9d4e-d0f2f8b03d4d"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 17:45:02.342479 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:45:02.342240    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/55806c17-765a-43a1-babc-46540c0423bd-model-cache" (OuterVolumeSpecName: "model-cache") pod "55806c17-765a-43a1-babc-46540c0423bd" (UID: "55806c17-765a-43a1-babc-46540c0423bd"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 17:45:02.342479 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:45:02.342407    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c30aa08c-b2d4-4edc-8561-e78ad422fd1f-home" (OuterVolumeSpecName: "home") pod "c30aa08c-b2d4-4edc-8561-e78ad422fd1f" (UID: "c30aa08c-b2d4-4edc-8561-e78ad422fd1f"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 17:45:02.343157 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:45:02.342566    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/55806c17-765a-43a1-babc-46540c0423bd-home" (OuterVolumeSpecName: "home") pod "55806c17-765a-43a1-babc-46540c0423bd" (UID: "55806c17-765a-43a1-babc-46540c0423bd"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 17:45:02.343157 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:45:02.342629    2577 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c30aa08c-b2d4-4edc-8561-e78ad422fd1f-home\") on node \"ip-10-0-141-139.ec2.internal\" DevicePath \"\""
Sep 11 17:45:02.343157 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:45:02.342649    2577 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/66e0a151-0ad9-46ce-9d4e-d0f2f8b03d4d-tmp-dir\") on node \"ip-10-0-141-139.ec2.internal\" DevicePath \"\""
Sep 11 17:45:02.343157 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:45:02.342666    2577 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5aa9157d-fd25-4e77-bb11-b0fd137bf739-tls-certs\") on node \"ip-10-0-141-139.ec2.internal\" DevicePath \"\""
Sep 11 17:45:02.343157 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:45:02.342680    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-65gcl\" (UniqueName: \"kubernetes.io/projected/5aa9157d-fd25-4e77-bb11-b0fd137bf739-kube-api-access-65gcl\") on node \"ip-10-0-141-139.ec2.internal\" DevicePath \"\""
Sep 11 17:45:02.343157 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:45:02.342693    2577 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/66e0a151-0ad9-46ce-9d4e-d0f2f8b03d4d-home\") on node \"ip-10-0-141-139.ec2.internal\" DevicePath \"\""
Sep 11 17:45:02.343157 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:45:02.342706    2577 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c30aa08c-b2d4-4edc-8561-e78ad422fd1f-model-cache\") on node \"ip-10-0-141-139.ec2.internal\" DevicePath \"\""
Sep 11 17:45:02.343157 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:45:02.342720    2577 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5aa9157d-fd25-4e77-bb11-b0fd137bf739-dshm\") on node \"ip-10-0-141-139.ec2.internal\" DevicePath \"\""
Sep 11 17:45:02.343157 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:45:02.342732    2577 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/55806c17-765a-43a1-babc-46540c0423bd-model-cache\") on node \"ip-10-0-141-139.ec2.internal\" DevicePath \"\""
Sep 11 17:45:02.343157 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:45:02.342744    2577 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/55806c17-765a-43a1-babc-46540c0423bd-tmp-dir\") on node \"ip-10-0-141-139.ec2.internal\" DevicePath \"\""
Sep 11 17:45:02.343157 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:45:02.342756    2577 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/66e0a151-0ad9-46ce-9d4e-d0f2f8b03d4d-model-cache\") on node \"ip-10-0-141-139.ec2.internal\" DevicePath \"\""
Sep 11 17:45:02.344262 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:45:02.344035    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c30aa08c-b2d4-4edc-8561-e78ad422fd1f-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "c30aa08c-b2d4-4edc-8561-e78ad422fd1f" (UID: "c30aa08c-b2d4-4edc-8561-e78ad422fd1f"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 17:45:02.345120 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:45:02.345029    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/55806c17-765a-43a1-babc-46540c0423bd-kube-api-access-fjnp9" (OuterVolumeSpecName: "kube-api-access-fjnp9") pod "55806c17-765a-43a1-babc-46540c0423bd" (UID: "55806c17-765a-43a1-babc-46540c0423bd"). InnerVolumeSpecName "kube-api-access-fjnp9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 17:45:02.345349 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:45:02.345322    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c30aa08c-b2d4-4edc-8561-e78ad422fd1f-kube-api-access-m4drh" (OuterVolumeSpecName: "kube-api-access-m4drh") pod "c30aa08c-b2d4-4edc-8561-e78ad422fd1f" (UID: "c30aa08c-b2d4-4edc-8561-e78ad422fd1f"). InnerVolumeSpecName "kube-api-access-m4drh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 17:45:02.345705 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:45:02.345678    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/55806c17-765a-43a1-babc-46540c0423bd-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "55806c17-765a-43a1-babc-46540c0423bd" (UID: "55806c17-765a-43a1-babc-46540c0423bd"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 17:45:02.345880 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:45:02.345856    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/66e0a151-0ad9-46ce-9d4e-d0f2f8b03d4d-dshm" (OuterVolumeSpecName: "dshm") pod "66e0a151-0ad9-46ce-9d4e-d0f2f8b03d4d" (UID: "66e0a151-0ad9-46ce-9d4e-d0f2f8b03d4d"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 17:45:02.346112 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:45:02.346089    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c30aa08c-b2d4-4edc-8561-e78ad422fd1f-dshm" (OuterVolumeSpecName: "dshm") pod "c30aa08c-b2d4-4edc-8561-e78ad422fd1f" (UID: "c30aa08c-b2d4-4edc-8561-e78ad422fd1f"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 17:45:02.346271 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:45:02.346249    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/55806c17-765a-43a1-babc-46540c0423bd-dshm" (OuterVolumeSpecName: "dshm") pod "55806c17-765a-43a1-babc-46540c0423bd" (UID: "55806c17-765a-43a1-babc-46540c0423bd"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 17:45:02.346444 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:45:02.346427    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c30aa08c-b2d4-4edc-8561-e78ad422fd1f-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "c30aa08c-b2d4-4edc-8561-e78ad422fd1f" (UID: "c30aa08c-b2d4-4edc-8561-e78ad422fd1f"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 17:45:02.346719 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:45:02.346696    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/66e0a151-0ad9-46ce-9d4e-d0f2f8b03d4d-kube-api-access-7csn8" (OuterVolumeSpecName: "kube-api-access-7csn8") pod "66e0a151-0ad9-46ce-9d4e-d0f2f8b03d4d" (UID: "66e0a151-0ad9-46ce-9d4e-d0f2f8b03d4d"). InnerVolumeSpecName "kube-api-access-7csn8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 17:45:02.347150 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:45:02.347128    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/66e0a151-0ad9-46ce-9d4e-d0f2f8b03d4d-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "66e0a151-0ad9-46ce-9d4e-d0f2f8b03d4d" (UID: "66e0a151-0ad9-46ce-9d4e-d0f2f8b03d4d"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 17:45:02.443327 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:45:02.443290    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m4drh\" (UniqueName: \"kubernetes.io/projected/c30aa08c-b2d4-4edc-8561-e78ad422fd1f-kube-api-access-m4drh\") on node \"ip-10-0-141-139.ec2.internal\" DevicePath \"\""
Sep 11 17:45:02.443327 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:45:02.443323    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fjnp9\" (UniqueName: \"kubernetes.io/projected/55806c17-765a-43a1-babc-46540c0423bd-kube-api-access-fjnp9\") on node \"ip-10-0-141-139.ec2.internal\" DevicePath \"\""
Sep 11 17:45:02.443327 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:45:02.443333    2577 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/55806c17-765a-43a1-babc-46540c0423bd-tls-certs\") on node \"ip-10-0-141-139.ec2.internal\" DevicePath \"\""
Sep 11 17:45:02.443605 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:45:02.443342    2577 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/55806c17-765a-43a1-babc-46540c0423bd-dshm\") on node \"ip-10-0-141-139.ec2.internal\" DevicePath \"\""
Sep 11 17:45:02.443605 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:45:02.443350    2577 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c30aa08c-b2d4-4edc-8561-e78ad422fd1f-tls-certs\") on node \"ip-10-0-141-139.ec2.internal\" DevicePath \"\""
Sep 11 17:45:02.443605 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:45:02.443357    2577 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c30aa08c-b2d4-4edc-8561-e78ad422fd1f-dshm\") on node \"ip-10-0-141-139.ec2.internal\" DevicePath \"\""
Sep 11 17:45:02.443605 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:45:02.443365    2577 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c30aa08c-b2d4-4edc-8561-e78ad422fd1f-tmp-dir\") on node \"ip-10-0-141-139.ec2.internal\" DevicePath \"\""
Sep 11 17:45:02.443605 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:45:02.443371    2577 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/55806c17-765a-43a1-babc-46540c0423bd-home\") on node \"ip-10-0-141-139.ec2.internal\" DevicePath \"\""
Sep 11 17:45:02.443605 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:45:02.443380    2577 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/66e0a151-0ad9-46ce-9d4e-d0f2f8b03d4d-tls-certs\") on node \"ip-10-0-141-139.ec2.internal\" DevicePath \"\""
Sep 11 17:45:02.443605 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:45:02.443388    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7csn8\" (UniqueName: \"kubernetes.io/projected/66e0a151-0ad9-46ce-9d4e-d0f2f8b03d4d-kube-api-access-7csn8\") on node \"ip-10-0-141-139.ec2.internal\" DevicePath \"\""
Sep 11 17:45:02.443605 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:45:02.443396    2577 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/66e0a151-0ad9-46ce-9d4e-d0f2f8b03d4d-dshm\") on node \"ip-10-0-141-139.ec2.internal\" DevicePath \"\""
Sep 11 17:45:02.604940 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:45:02.604835    2577 generic.go:356] "Generic (PLEG): container finished" podID="c30aa08c-b2d4-4edc-8561-e78ad422fd1f" containerID="4967c83f7832a12d0f739396188f205f644df6605ff208a482fbc3255a33323a" exitCode=0
Sep 11 17:45:02.604940 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:45:02.604897    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-c7d192/simulator-w0-c7d192-kserve-6bfc765cf5-klnxw"
Sep 11 17:45:02.604940 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:45:02.604930    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-c7d192/simulator-w0-c7d192-kserve-6bfc765cf5-klnxw" event={"ID":"c30aa08c-b2d4-4edc-8561-e78ad422fd1f","Type":"ContainerDied","Data":"4967c83f7832a12d0f739396188f205f644df6605ff208a482fbc3255a33323a"}
Sep 11 17:45:02.605575 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:45:02.604982    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-c7d192/simulator-w0-c7d192-kserve-6bfc765cf5-klnxw" event={"ID":"c30aa08c-b2d4-4edc-8561-e78ad422fd1f","Type":"ContainerDied","Data":"1c9da7dd3c09e732d511a6da43370ce0d9a8827d33963f68c9f25d1b4629be42"}
Sep 11 17:45:02.605575 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:45:02.605009    2577 scope.go:117] "RemoveContainer" containerID="4967c83f7832a12d0f739396188f205f644df6605ff208a482fbc3255a33323a"
Sep 11 17:45:02.606346 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:45:02.606325    2577 generic.go:356] "Generic (PLEG): container finished" podID="55806c17-765a-43a1-babc-46540c0423bd" containerID="d07916d97da58fa3f509688c4b9efe0ccdf7310d10eb8e8f733ef1bc63967c60" exitCode=0
Sep 11 17:45:02.606435 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:45:02.606363    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-c7d192/premium-w0-c7d192-kserve-6f977d4fcd-n9nhd"
Sep 11 17:45:02.606435 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:45:02.606392    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-c7d192/premium-w0-c7d192-kserve-6f977d4fcd-n9nhd" event={"ID":"55806c17-765a-43a1-babc-46540c0423bd","Type":"ContainerDied","Data":"d07916d97da58fa3f509688c4b9efe0ccdf7310d10eb8e8f733ef1bc63967c60"}
Sep 11 17:45:02.606435 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:45:02.606431    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-c7d192/premium-w0-c7d192-kserve-6f977d4fcd-n9nhd" event={"ID":"55806c17-765a-43a1-babc-46540c0423bd","Type":"ContainerDied","Data":"f3eef0bef2b51f86e102732c1c4a39b09e989edbd4ceb160aecf57ba795512b5"}
Sep 11 17:45:02.608016 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:45:02.607989    2577 generic.go:356] "Generic (PLEG): container finished" podID="5aa9157d-fd25-4e77-bb11-b0fd137bf739" containerID="6b0342a195c4b24066cdc90cc0361ba917590668658224032685ff839ef05c78" exitCode=0
Sep 11 17:45:02.608140 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:45:02.608041    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-c7d192/embedding-w0-c7d192-kserve-5d5bfbb578-22zmh"
Sep 11 17:45:02.608140 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:45:02.608063    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-c7d192/embedding-w0-c7d192-kserve-5d5bfbb578-22zmh" event={"ID":"5aa9157d-fd25-4e77-bb11-b0fd137bf739","Type":"ContainerDied","Data":"6b0342a195c4b24066cdc90cc0361ba917590668658224032685ff839ef05c78"}
Sep 11 17:45:02.608140 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:45:02.608108    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-c7d192/embedding-w0-c7d192-kserve-5d5bfbb578-22zmh" event={"ID":"5aa9157d-fd25-4e77-bb11-b0fd137bf739","Type":"ContainerDied","Data":"c4ac32ba2955d670269293aaa8c252f382d6e4e5a210c86755819d02e4f440d8"}
Sep 11 17:45:02.609629 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:45:02.609606    2577 generic.go:356] "Generic (PLEG): container finished" podID="66e0a151-0ad9-46ce-9d4e-d0f2f8b03d4d" containerID="8fb5e9789ab3a1fbf975b73256bd830adb7948376aadb7d182f466e765ec6ba7" exitCode=0
Sep 11 17:45:02.609726 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:45:02.609688    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-c7d192/distinct-w0-c7d192-kserve-645f648f77-lxn5g" event={"ID":"66e0a151-0ad9-46ce-9d4e-d0f2f8b03d4d","Type":"ContainerDied","Data":"8fb5e9789ab3a1fbf975b73256bd830adb7948376aadb7d182f466e765ec6ba7"}
Sep 11 17:45:02.609726 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:45:02.609708    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-c7d192/distinct-w0-c7d192-kserve-645f648f77-lxn5g"
Sep 11 17:45:02.609726 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:45:02.609718    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-c7d192/distinct-w0-c7d192-kserve-645f648f77-lxn5g" event={"ID":"66e0a151-0ad9-46ce-9d4e-d0f2f8b03d4d","Type":"ContainerDied","Data":"767ed3d9c3fa1fec9a82eb8d4c695336055a352ce9981d6bbd05fff25ed8850a"}
Sep 11 17:45:02.619575 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:45:02.619142    2577 scope.go:117] "RemoveContainer" containerID="4967c83f7832a12d0f739396188f205f644df6605ff208a482fbc3255a33323a"
Sep 11 17:45:02.621577 ip-10-0-141-139 kubenswrapper[2577]: E0911 17:45:02.621419    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4967c83f7832a12d0f739396188f205f644df6605ff208a482fbc3255a33323a\": container with ID starting with 4967c83f7832a12d0f739396188f205f644df6605ff208a482fbc3255a33323a not found: ID does not exist" containerID="4967c83f7832a12d0f739396188f205f644df6605ff208a482fbc3255a33323a"
Sep 11 17:45:02.621577 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:45:02.621454    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4967c83f7832a12d0f739396188f205f644df6605ff208a482fbc3255a33323a"} err="failed to get container status \"4967c83f7832a12d0f739396188f205f644df6605ff208a482fbc3255a33323a\": rpc error: code = NotFound desc = could not find container \"4967c83f7832a12d0f739396188f205f644df6605ff208a482fbc3255a33323a\": container with ID starting with 4967c83f7832a12d0f739396188f205f644df6605ff208a482fbc3255a33323a not found: ID does not exist"
Sep 11 17:45:02.621577 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:45:02.621473    2577 scope.go:117] "RemoveContainer" containerID="d07916d97da58fa3f509688c4b9efe0ccdf7310d10eb8e8f733ef1bc63967c60"
Sep 11 17:45:02.634918 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:45:02.634894    2577 scope.go:117] "RemoveContainer" containerID="d07916d97da58fa3f509688c4b9efe0ccdf7310d10eb8e8f733ef1bc63967c60"
Sep 11 17:45:02.635420 ip-10-0-141-139 kubenswrapper[2577]: E0911 17:45:02.635398    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d07916d97da58fa3f509688c4b9efe0ccdf7310d10eb8e8f733ef1bc63967c60\": container with ID starting with d07916d97da58fa3f509688c4b9efe0ccdf7310d10eb8e8f733ef1bc63967c60 not found: ID does not exist" containerID="d07916d97da58fa3f509688c4b9efe0ccdf7310d10eb8e8f733ef1bc63967c60"
Sep 11 17:45:02.635538 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:45:02.635430    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d07916d97da58fa3f509688c4b9efe0ccdf7310d10eb8e8f733ef1bc63967c60"} err="failed to get container status \"d07916d97da58fa3f509688c4b9efe0ccdf7310d10eb8e8f733ef1bc63967c60\": rpc error: code = NotFound desc = could not find container \"d07916d97da58fa3f509688c4b9efe0ccdf7310d10eb8e8f733ef1bc63967c60\": container with ID starting with d07916d97da58fa3f509688c4b9efe0ccdf7310d10eb8e8f733ef1bc63967c60 not found: ID does not exist"
Sep 11 17:45:02.635538 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:45:02.635453    2577 scope.go:117] "RemoveContainer" containerID="6b0342a195c4b24066cdc90cc0361ba917590668658224032685ff839ef05c78"
Sep 11 17:45:02.648106 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:45:02.648033    2577 scope.go:117] "RemoveContainer" containerID="6b0342a195c4b24066cdc90cc0361ba917590668658224032685ff839ef05c78"
Sep 11 17:45:02.648819 ip-10-0-141-139 kubenswrapper[2577]: E0911 17:45:02.648790    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6b0342a195c4b24066cdc90cc0361ba917590668658224032685ff839ef05c78\": container with ID starting with 6b0342a195c4b24066cdc90cc0361ba917590668658224032685ff839ef05c78 not found: ID does not exist" containerID="6b0342a195c4b24066cdc90cc0361ba917590668658224032685ff839ef05c78"
Sep 11 17:45:02.648942 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:45:02.648825    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6b0342a195c4b24066cdc90cc0361ba917590668658224032685ff839ef05c78"} err="failed to get container status \"6b0342a195c4b24066cdc90cc0361ba917590668658224032685ff839ef05c78\": rpc error: code = NotFound desc = could not find container \"6b0342a195c4b24066cdc90cc0361ba917590668658224032685ff839ef05c78\": container with ID starting with 6b0342a195c4b24066cdc90cc0361ba917590668658224032685ff839ef05c78 not found: ID does not exist"
Sep 11 17:45:02.648942 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:45:02.648847    2577 scope.go:117] "RemoveContainer" containerID="8fb5e9789ab3a1fbf975b73256bd830adb7948376aadb7d182f466e765ec6ba7"
Sep 11 17:45:02.652725 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:45:02.652466    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-c7d192/simulator-w0-c7d192-kserve-6bfc765cf5-klnxw"]
Sep 11 17:45:02.656376 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:45:02.656351    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-c7d192/simulator-w0-c7d192-kserve-6bfc765cf5-klnxw"]
Sep 11 17:45:02.661454 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:45:02.661433    2577 scope.go:117] "RemoveContainer" containerID="8fb5e9789ab3a1fbf975b73256bd830adb7948376aadb7d182f466e765ec6ba7"
Sep 11 17:45:02.661871 ip-10-0-141-139 kubenswrapper[2577]: E0911 17:45:02.661843    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8fb5e9789ab3a1fbf975b73256bd830adb7948376aadb7d182f466e765ec6ba7\": container with ID starting with 8fb5e9789ab3a1fbf975b73256bd830adb7948376aadb7d182f466e765ec6ba7 not found: ID does not exist" containerID="8fb5e9789ab3a1fbf975b73256bd830adb7948376aadb7d182f466e765ec6ba7"
Sep 11 17:45:02.661960 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:45:02.661882    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8fb5e9789ab3a1fbf975b73256bd830adb7948376aadb7d182f466e765ec6ba7"} err="failed to get container status \"8fb5e9789ab3a1fbf975b73256bd830adb7948376aadb7d182f466e765ec6ba7\": rpc error: code = NotFound desc = could not find container \"8fb5e9789ab3a1fbf975b73256bd830adb7948376aadb7d182f466e765ec6ba7\": container with ID starting with 8fb5e9789ab3a1fbf975b73256bd830adb7948376aadb7d182f466e765ec6ba7 not found: ID does not exist"
Sep 11 17:45:02.676480 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:45:02.676448    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-c7d192/embedding-w0-c7d192-kserve-5d5bfbb578-22zmh"]
Sep 11 17:45:02.683058 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:45:02.683026    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-c7d192/embedding-w0-c7d192-kserve-5d5bfbb578-22zmh"]
Sep 11 17:45:02.697922 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:45:02.697892    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-c7d192/premium-w0-c7d192-kserve-6f977d4fcd-n9nhd"]
Sep 11 17:45:02.705260 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:45:02.705227    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-c7d192/premium-w0-c7d192-kserve-6f977d4fcd-n9nhd"]
Sep 11 17:45:02.717865 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:45:02.717832    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-c7d192/distinct-w0-c7d192-kserve-645f648f77-lxn5g"]
Sep 11 17:45:02.724261 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:45:02.724222    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-c7d192/distinct-w0-c7d192-kserve-645f648f77-lxn5g"]
Sep 11 17:45:02.760296 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:45:02.760274    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29819145-k7xm4"
Sep 11 17:45:02.762586 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:45:02.762560    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-144bfa-29819145-r5fn7"
Sep 11 17:45:02.930632 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:45:02.930535    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="55806c17-765a-43a1-babc-46540c0423bd" path="/var/lib/kubelet/pods/55806c17-765a-43a1-babc-46540c0423bd/volumes"
Sep 11 17:45:02.931101 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:45:02.931076    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5aa9157d-fd25-4e77-bb11-b0fd137bf739" path="/var/lib/kubelet/pods/5aa9157d-fd25-4e77-bb11-b0fd137bf739/volumes"
Sep 11 17:45:02.931543 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:45:02.931491    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="66e0a151-0ad9-46ce-9d4e-d0f2f8b03d4d" path="/var/lib/kubelet/pods/66e0a151-0ad9-46ce-9d4e-d0f2f8b03d4d/volumes"
Sep 11 17:45:02.932251 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:45:02.932221    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c30aa08c-b2d4-4edc-8561-e78ad422fd1f" path="/var/lib/kubelet/pods/c30aa08c-b2d4-4edc-8561-e78ad422fd1f/volumes"
Sep 11 17:45:03.619815 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:45:03.619788    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-144bfa-29819145-r5fn7"
Sep 11 17:45:03.620356 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:45:03.619829    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-144bfa-29819145-r5fn7" event={"ID":"0c347dc7-e953-4728-a52d-702a4e0291a8","Type":"ContainerDied","Data":"aabc7dfa1bdc07936acc5691ec9089663e372406028bda3fe0444a4dfa1ebfa9"}
Sep 11 17:45:03.620356 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:45:03.619878    2577 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="aabc7dfa1bdc07936acc5691ec9089663e372406028bda3fe0444a4dfa1ebfa9"
Sep 11 17:45:03.621194 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:45:03.621172    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29819145-k7xm4"
Sep 11 17:45:03.621194 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:45:03.621179    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29819145-k7xm4" event={"ID":"ce07d5c0-d2f5-40b5-a2d7-1cf5ddf1242d","Type":"ContainerDied","Data":"931ae1fe94bdc26c3fc38401d80dc18c61a6e52bdc12d4e1c6f3b87cfaab164f"}
Sep 11 17:45:03.621364 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:45:03.621221    2577 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="931ae1fe94bdc26c3fc38401d80dc18c61a6e52bdc12d4e1c6f3b87cfaab164f"
Sep 11 17:45:21.918690 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:45:21.918654    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-nkmkm"]
Sep 11 17:45:28.141453 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:45:28.141419    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-144bfa-bmzgz"]
Sep 11 17:45:28.142089 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:45:28.142066    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0c347dc7-e953-4728-a52d-702a4e0291a8" containerName="cleanup"
Sep 11 17:45:28.142189 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:45:28.142092    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="0c347dc7-e953-4728-a52d-702a4e0291a8" containerName="cleanup"
Sep 11 17:45:28.142189 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:45:28.142120    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5aa9157d-fd25-4e77-bb11-b0fd137bf739" containerName="main"
Sep 11 17:45:28.142189 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:45:28.142128    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="5aa9157d-fd25-4e77-bb11-b0fd137bf739" containerName="main"
Sep 11 17:45:28.142189 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:45:28.142154    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ce07d5c0-d2f5-40b5-a2d7-1cf5ddf1242d" containerName="cleanup"
Sep 11 17:45:28.142189 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:45:28.142162    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="ce07d5c0-d2f5-40b5-a2d7-1cf5ddf1242d" containerName="cleanup"
Sep 11 17:45:28.142189 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:45:28.142178    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c30aa08c-b2d4-4edc-8561-e78ad422fd1f" containerName="main"
Sep 11 17:45:28.142189 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:45:28.142186    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="c30aa08c-b2d4-4edc-8561-e78ad422fd1f" containerName="main"
Sep 11 17:45:28.142571 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:45:28.142197    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="55806c17-765a-43a1-babc-46540c0423bd" containerName="main"
Sep 11 17:45:28.142571 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:45:28.142204    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="55806c17-765a-43a1-babc-46540c0423bd" containerName="main"
Sep 11 17:45:28.142571 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:45:28.142219    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="66e0a151-0ad9-46ce-9d4e-d0f2f8b03d4d" containerName="main"
Sep 11 17:45:28.142571 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:45:28.142227    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="66e0a151-0ad9-46ce-9d4e-d0f2f8b03d4d" containerName="main"
Sep 11 17:45:28.142571 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:45:28.142332    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="ce07d5c0-d2f5-40b5-a2d7-1cf5ddf1242d" containerName="cleanup"
Sep 11 17:45:28.142571 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:45:28.142344    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="66e0a151-0ad9-46ce-9d4e-d0f2f8b03d4d" containerName="main"
Sep 11 17:45:28.142571 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:45:28.142355    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="5aa9157d-fd25-4e77-bb11-b0fd137bf739" containerName="main"
Sep 11 17:45:28.142571 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:45:28.142364    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="0c347dc7-e953-4728-a52d-702a4e0291a8" containerName="cleanup"
Sep 11 17:45:28.142571 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:45:28.142373    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="c30aa08c-b2d4-4edc-8561-e78ad422fd1f" containerName="main"
Sep 11 17:45:28.142571 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:45:28.142383    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="55806c17-765a-43a1-babc-46540c0423bd" containerName="main"
Sep 11 17:45:28.145899 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:45:28.145877    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-144bfa-bmzgz"
Sep 11 17:45:28.148205 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:45:28.148180    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 11 17:45:28.148326 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:45:28.148258    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-9n8gf\""
Sep 11 17:45:28.152402 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:45:28.152376    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-144bfa-bmzgz"]
Sep 11 17:45:28.309205 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:45:28.309158    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f5324d08-8464-43fc-a9c3-82960f5b1712-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-144bfa-bmzgz\" (UID: \"f5324d08-8464-43fc-a9c3-82960f5b1712\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-144bfa-bmzgz"
Sep 11 17:45:28.410082 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:45:28.409983    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f5324d08-8464-43fc-a9c3-82960f5b1712-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-144bfa-bmzgz\" (UID: \"f5324d08-8464-43fc-a9c3-82960f5b1712\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-144bfa-bmzgz"
Sep 11 17:45:28.410596 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:45:28.410577    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f5324d08-8464-43fc-a9c3-82960f5b1712-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-144bfa-bmzgz\" (UID: \"f5324d08-8464-43fc-a9c3-82960f5b1712\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-144bfa-bmzgz"
Sep 11 17:45:28.458565 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:45:28.458498    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-144bfa-bmzgz"
Sep 11 17:45:28.737380 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:45:28.737349    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-144bfa-bmzgz"]
Sep 11 17:45:28.739440 ip-10-0-141-139 kubenswrapper[2577]: W0911 17:45:28.739415    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf5324d08_8464_43fc_a9c3_82960f5b1712.slice/crio-233ddc1b4ddff81322c023105f24da9099f5923bd87e8785c9f0c591892080b1 WatchSource:0}: Error finding container 233ddc1b4ddff81322c023105f24da9099f5923bd87e8785c9f0c591892080b1: Status 404 returned error can't find the container with id 233ddc1b4ddff81322c023105f24da9099f5923bd87e8785c9f0c591892080b1
Sep 11 17:45:28.770826 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:45:28.770795    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-144bfa-bmzgz" event={"ID":"f5324d08-8464-43fc-a9c3-82960f5b1712","Type":"ContainerStarted","Data":"233ddc1b4ddff81322c023105f24da9099f5923bd87e8785c9f0c591892080b1"}
Sep 11 17:45:29.777708 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:45:29.777668    2577 generic.go:356] "Generic (PLEG): container finished" podID="f5324d08-8464-43fc-a9c3-82960f5b1712" containerID="61cf624eb0ffed828e65cd98b710c4ca7d8cdf6af8e29d1d604313cf6af060fa" exitCode=0
Sep 11 17:45:29.778095 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:45:29.777716    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-144bfa-bmzgz" event={"ID":"f5324d08-8464-43fc-a9c3-82960f5b1712","Type":"ContainerDied","Data":"61cf624eb0ffed828e65cd98b710c4ca7d8cdf6af8e29d1d604313cf6af060fa"}
Sep 11 17:45:30.919459 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:45:30.919427    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-144bfa-bmzgz"
Sep 11 17:45:30.934973 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:45:30.934938    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f5324d08-8464-43fc-a9c3-82960f5b1712-maas-api-service-ca\") pod \"f5324d08-8464-43fc-a9c3-82960f5b1712\" (UID: \"f5324d08-8464-43fc-a9c3-82960f5b1712\") "
Sep 11 17:45:30.935277 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:45:30.935255    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f5324d08-8464-43fc-a9c3-82960f5b1712-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "f5324d08-8464-43fc-a9c3-82960f5b1712" (UID: "f5324d08-8464-43fc-a9c3-82960f5b1712"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 17:45:31.036554 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:45:31.036432    2577 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f5324d08-8464-43fc-a9c3-82960f5b1712-maas-api-service-ca\") on node \"ip-10-0-141-139.ec2.internal\" DevicePath \"\""
Sep 11 17:45:31.790931 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:45:31.790887    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-144bfa-bmzgz" event={"ID":"f5324d08-8464-43fc-a9c3-82960f5b1712","Type":"ContainerDied","Data":"233ddc1b4ddff81322c023105f24da9099f5923bd87e8785c9f0c591892080b1"}
Sep 11 17:45:31.791120 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:45:31.790945    2577 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="233ddc1b4ddff81322c023105f24da9099f5923bd87e8785c9f0c591892080b1"
Sep 11 17:45:31.791120 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:45:31.790899    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-144bfa-bmzgz"
Sep 11 17:45:33.045821 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:45:33.045784    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-144bfa/distinct-w4-144bfa-kserve-648d4c7f9c-m2562"]
Sep 11 17:45:33.076686 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:45:33.076642    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-144bfa/simulator-w4-144bfa-kserve-6ddd858bc-4vs4s"]
Sep 11 17:45:43.193639 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:45:43.193547    2577 scope.go:117] "RemoveContainer" containerID="504109a314b0a7117a009b758797a7a7f77383cb9fbbf58029946c7b10ec74a7"
Sep 11 17:45:43.202386 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:45:43.202354    2577 scope.go:117] "RemoveContainer" containerID="ff5545207aeab7b03618f24f8ea7f22a66bf85861bf396df4e7e4e99d8c3c126"
Sep 11 17:45:43.705854 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:45:43.705820    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-nkmkm"]
Sep 11 17:45:43.929152 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:45:43.929116    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-144bfa-29819145-r5fn7"]
Sep 11 17:45:43.931650 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:45:43.931625    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-144bfa-29819145-r5fn7"]
Sep 11 17:45:43.959785 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:45:43.959698    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-144bfa-56f9948455-8n2pv"]
Sep 11 17:45:43.969747 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:45:43.969714    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-144bfa-7664866778-kh2p9"]
Sep 11 17:45:43.969996 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:45:43.969966    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-144bfa-7664866778-kh2p9" podUID="638cfdad-0c58-4a69-8cee-cc50f7ca2655" containerName="payload-pre-processing" containerID="cri-o://63d3bd8cc15b82ed68fcab6c03e7a4651bd8da07e8530aa5b0417ce80f0c1b6b" gracePeriod=30
Sep 11 17:45:44.155772 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:45:44.155725    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-144bfa-bmzgz"]
Sep 11 17:45:44.157799 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:45:44.157769    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-144bfa-bmzgz"]
Sep 11 17:45:44.228599 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:45:44.228572    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-144bfa-7664866778-kh2p9"
Sep 11 17:45:44.279010 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:45:44.278979    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-h25l5\" (UniqueName: \"kubernetes.io/projected/638cfdad-0c58-4a69-8cee-cc50f7ca2655-kube-api-access-h25l5\") pod \"638cfdad-0c58-4a69-8cee-cc50f7ca2655\" (UID: \"638cfdad-0c58-4a69-8cee-cc50f7ca2655\") "
Sep 11 17:45:44.279218 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:45:44.279139    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/638cfdad-0c58-4a69-8cee-cc50f7ca2655-plugins-config-volume\") pod \"638cfdad-0c58-4a69-8cee-cc50f7ca2655\" (UID: \"638cfdad-0c58-4a69-8cee-cc50f7ca2655\") "
Sep 11 17:45:44.279624 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:45:44.279593    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/638cfdad-0c58-4a69-8cee-cc50f7ca2655-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "638cfdad-0c58-4a69-8cee-cc50f7ca2655" (UID: "638cfdad-0c58-4a69-8cee-cc50f7ca2655"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 17:45:44.281281 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:45:44.281259    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/638cfdad-0c58-4a69-8cee-cc50f7ca2655-kube-api-access-h25l5" (OuterVolumeSpecName: "kube-api-access-h25l5") pod "638cfdad-0c58-4a69-8cee-cc50f7ca2655" (UID: "638cfdad-0c58-4a69-8cee-cc50f7ca2655"). InnerVolumeSpecName "kube-api-access-h25l5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 17:45:44.380007 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:45:44.379967    2577 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/638cfdad-0c58-4a69-8cee-cc50f7ca2655-plugins-config-volume\") on node \"ip-10-0-141-139.ec2.internal\" DevicePath \"\""
Sep 11 17:45:44.380007 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:45:44.379999    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h25l5\" (UniqueName: \"kubernetes.io/projected/638cfdad-0c58-4a69-8cee-cc50f7ca2655-kube-api-access-h25l5\") on node \"ip-10-0-141-139.ec2.internal\" DevicePath \"\""
Sep 11 17:45:44.866945 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:45:44.866904    2577 generic.go:356] "Generic (PLEG): container finished" podID="638cfdad-0c58-4a69-8cee-cc50f7ca2655" containerID="63d3bd8cc15b82ed68fcab6c03e7a4651bd8da07e8530aa5b0417ce80f0c1b6b" exitCode=0
Sep 11 17:45:44.866945 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:45:44.866945    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-144bfa-7664866778-kh2p9"
Sep 11 17:45:44.867169 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:45:44.866989    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-144bfa-7664866778-kh2p9" event={"ID":"638cfdad-0c58-4a69-8cee-cc50f7ca2655","Type":"ContainerDied","Data":"63d3bd8cc15b82ed68fcab6c03e7a4651bd8da07e8530aa5b0417ce80f0c1b6b"}
Sep 11 17:45:44.867169 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:45:44.867050    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-144bfa-7664866778-kh2p9" event={"ID":"638cfdad-0c58-4a69-8cee-cc50f7ca2655","Type":"ContainerDied","Data":"74d82bb671a5a99860f8cfe56f2152125f2956a50503a2261f6c4e0121996598"}
Sep 11 17:45:44.867169 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:45:44.867078    2577 scope.go:117] "RemoveContainer" containerID="63d3bd8cc15b82ed68fcab6c03e7a4651bd8da07e8530aa5b0417ce80f0c1b6b"
Sep 11 17:45:44.867311 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:45:44.867161    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w4-144bfa-56f9948455-8n2pv" podUID="401d5058-6997-4c1c-97b7-3dce976fbd62" containerName="payload-processing" containerID="cri-o://9592b808c1e0905d5e86370bfb2a3d30f41acbb6da93199b48c0031a67ddb26d" gracePeriod=30
Sep 11 17:45:44.877625 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:45:44.877534    2577 scope.go:117] "RemoveContainer" containerID="63d3bd8cc15b82ed68fcab6c03e7a4651bd8da07e8530aa5b0417ce80f0c1b6b"
Sep 11 17:45:44.877864 ip-10-0-141-139 kubenswrapper[2577]: E0911 17:45:44.877844    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"63d3bd8cc15b82ed68fcab6c03e7a4651bd8da07e8530aa5b0417ce80f0c1b6b\": container with ID starting with 63d3bd8cc15b82ed68fcab6c03e7a4651bd8da07e8530aa5b0417ce80f0c1b6b not found: ID does not exist" containerID="63d3bd8cc15b82ed68fcab6c03e7a4651bd8da07e8530aa5b0417ce80f0c1b6b"
Sep 11 17:45:44.877940 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:45:44.877876    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"63d3bd8cc15b82ed68fcab6c03e7a4651bd8da07e8530aa5b0417ce80f0c1b6b"} err="failed to get container status \"63d3bd8cc15b82ed68fcab6c03e7a4651bd8da07e8530aa5b0417ce80f0c1b6b\": rpc error: code = NotFound desc = could not find container \"63d3bd8cc15b82ed68fcab6c03e7a4651bd8da07e8530aa5b0417ce80f0c1b6b\": container with ID starting with 63d3bd8cc15b82ed68fcab6c03e7a4651bd8da07e8530aa5b0417ce80f0c1b6b not found: ID does not exist"
Sep 11 17:45:44.934405 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:45:44.934369    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0c347dc7-e953-4728-a52d-702a4e0291a8" path="/var/lib/kubelet/pods/0c347dc7-e953-4728-a52d-702a4e0291a8/volumes"
Sep 11 17:45:44.934749 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:45:44.934734    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f5324d08-8464-43fc-a9c3-82960f5b1712" path="/var/lib/kubelet/pods/f5324d08-8464-43fc-a9c3-82960f5b1712/volumes"
Sep 11 17:45:44.935028 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:45:44.935014    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-144bfa-7664866778-kh2p9"]
Sep 11 17:45:44.935078 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:45:44.935042    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-144bfa-7664866778-kh2p9"]
Sep 11 17:45:45.114153 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:45:45.114125    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w4-144bfa-56f9948455-8n2pv"
Sep 11 17:45:45.189024 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:45:45.188942    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wv6sj\" (UniqueName: \"kubernetes.io/projected/401d5058-6997-4c1c-97b7-3dce976fbd62-kube-api-access-wv6sj\") pod \"401d5058-6997-4c1c-97b7-3dce976fbd62\" (UID: \"401d5058-6997-4c1c-97b7-3dce976fbd62\") "
Sep 11 17:45:45.189185 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:45:45.189056    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/401d5058-6997-4c1c-97b7-3dce976fbd62-plugins-config-volume\") pod \"401d5058-6997-4c1c-97b7-3dce976fbd62\" (UID: \"401d5058-6997-4c1c-97b7-3dce976fbd62\") "
Sep 11 17:45:45.189424 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:45:45.189393    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/401d5058-6997-4c1c-97b7-3dce976fbd62-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "401d5058-6997-4c1c-97b7-3dce976fbd62" (UID: "401d5058-6997-4c1c-97b7-3dce976fbd62"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 17:45:45.191136 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:45:45.191110    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/401d5058-6997-4c1c-97b7-3dce976fbd62-kube-api-access-wv6sj" (OuterVolumeSpecName: "kube-api-access-wv6sj") pod "401d5058-6997-4c1c-97b7-3dce976fbd62" (UID: "401d5058-6997-4c1c-97b7-3dce976fbd62"). InnerVolumeSpecName "kube-api-access-wv6sj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 17:45:45.290716 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:45:45.290684    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wv6sj\" (UniqueName: \"kubernetes.io/projected/401d5058-6997-4c1c-97b7-3dce976fbd62-kube-api-access-wv6sj\") on node \"ip-10-0-141-139.ec2.internal\" DevicePath \"\""
Sep 11 17:45:45.290716 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:45:45.290711    2577 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/401d5058-6997-4c1c-97b7-3dce976fbd62-plugins-config-volume\") on node \"ip-10-0-141-139.ec2.internal\" DevicePath \"\""
Sep 11 17:45:45.875401 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:45:45.875370    2577 generic.go:356] "Generic (PLEG): container finished" podID="401d5058-6997-4c1c-97b7-3dce976fbd62" containerID="9592b808c1e0905d5e86370bfb2a3d30f41acbb6da93199b48c0031a67ddb26d" exitCode=0
Sep 11 17:45:45.875629 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:45:45.875417    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w4-144bfa-56f9948455-8n2pv"
Sep 11 17:45:45.875629 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:45:45.875446    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w4-144bfa-56f9948455-8n2pv" event={"ID":"401d5058-6997-4c1c-97b7-3dce976fbd62","Type":"ContainerDied","Data":"9592b808c1e0905d5e86370bfb2a3d30f41acbb6da93199b48c0031a67ddb26d"}
Sep 11 17:45:45.875629 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:45:45.875517    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w4-144bfa-56f9948455-8n2pv" event={"ID":"401d5058-6997-4c1c-97b7-3dce976fbd62","Type":"ContainerDied","Data":"0e99839d9216f3b884534a025515e47ad45e2cfe953fdae4079f01bc6e6e73e1"}
Sep 11 17:45:45.875629 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:45:45.875539    2577 scope.go:117] "RemoveContainer" containerID="9592b808c1e0905d5e86370bfb2a3d30f41acbb6da93199b48c0031a67ddb26d"
Sep 11 17:45:45.886379 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:45:45.886355    2577 scope.go:117] "RemoveContainer" containerID="9592b808c1e0905d5e86370bfb2a3d30f41acbb6da93199b48c0031a67ddb26d"
Sep 11 17:45:45.886730 ip-10-0-141-139 kubenswrapper[2577]: E0911 17:45:45.886709    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9592b808c1e0905d5e86370bfb2a3d30f41acbb6da93199b48c0031a67ddb26d\": container with ID starting with 9592b808c1e0905d5e86370bfb2a3d30f41acbb6da93199b48c0031a67ddb26d not found: ID does not exist" containerID="9592b808c1e0905d5e86370bfb2a3d30f41acbb6da93199b48c0031a67ddb26d"
Sep 11 17:45:45.886805 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:45:45.886740    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9592b808c1e0905d5e86370bfb2a3d30f41acbb6da93199b48c0031a67ddb26d"} err="failed to get container status \"9592b808c1e0905d5e86370bfb2a3d30f41acbb6da93199b48c0031a67ddb26d\": rpc error: code = NotFound desc = could not find container \"9592b808c1e0905d5e86370bfb2a3d30f41acbb6da93199b48c0031a67ddb26d\": container with ID starting with 9592b808c1e0905d5e86370bfb2a3d30f41acbb6da93199b48c0031a67ddb26d not found: ID does not exist"
Sep 11 17:45:45.901752 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:45:45.901723    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-144bfa-56f9948455-8n2pv"]
Sep 11 17:45:45.911378 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:45:45.911351    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-144bfa-56f9948455-8n2pv"]
Sep 11 17:45:46.926558 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:45:46.926495    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="401d5058-6997-4c1c-97b7-3dce976fbd62" path="/var/lib/kubelet/pods/401d5058-6997-4c1c-97b7-3dce976fbd62/volumes"
Sep 11 17:45:46.926951 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:45:46.926902    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="638cfdad-0c58-4a69-8cee-cc50f7ca2655" path="/var/lib/kubelet/pods/638cfdad-0c58-4a69-8cee-cc50f7ca2655/volumes"
Sep 11 17:45:48.065973 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:45:48.065930    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-144bfa/distinct-w4-144bfa-kserve-648d4c7f9c-m2562" podUID="44bbc11d-62da-41bb-90e3-37c721b01e2f" containerName="main" containerID="cri-o://fe6a7652f020a1ab2148e020c86e0f136ac1d4b4b815d4770a965c718c3d7ae0" gracePeriod=45
Sep 11 17:45:48.096769 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:45:48.096720    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-144bfa/simulator-w4-144bfa-kserve-6ddd858bc-4vs4s" podUID="1a60ac0d-485c-48b2-8ea4-6633cdbcd138" containerName="main" containerID="cri-o://1b1f22351c03b1e69349f6e8b5008c6a239520ea1e97a05676c19abf0fe06c85" gracePeriod=45
Sep 11 17:45:48.393096 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:45:48.393065    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-144bfa/distinct-w4-144bfa-kserve-648d4c7f9c-m2562"
Sep 11 17:45:48.396634 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:45:48.396607    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-144bfa/simulator-w4-144bfa-kserve-6ddd858bc-4vs4s"
Sep 11 17:45:48.525491 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:45:48.525441    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/44bbc11d-62da-41bb-90e3-37c721b01e2f-dshm\") pod \"44bbc11d-62da-41bb-90e3-37c721b01e2f\" (UID: \"44bbc11d-62da-41bb-90e3-37c721b01e2f\") "
Sep 11 17:45:48.525715 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:45:48.525532    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/44bbc11d-62da-41bb-90e3-37c721b01e2f-tmp-dir\") pod \"44bbc11d-62da-41bb-90e3-37c721b01e2f\" (UID: \"44bbc11d-62da-41bb-90e3-37c721b01e2f\") "
Sep 11 17:45:48.525715 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:45:48.525564    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1a60ac0d-485c-48b2-8ea4-6633cdbcd138-tmp-dir\") pod \"1a60ac0d-485c-48b2-8ea4-6633cdbcd138\" (UID: \"1a60ac0d-485c-48b2-8ea4-6633cdbcd138\") "
Sep 11 17:45:48.525715 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:45:48.525595    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1a60ac0d-485c-48b2-8ea4-6633cdbcd138-dshm\") pod \"1a60ac0d-485c-48b2-8ea4-6633cdbcd138\" (UID: \"1a60ac0d-485c-48b2-8ea4-6633cdbcd138\") "
Sep 11 17:45:48.525715 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:45:48.525613    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1a60ac0d-485c-48b2-8ea4-6633cdbcd138-model-cache\") pod \"1a60ac0d-485c-48b2-8ea4-6633cdbcd138\" (UID: \"1a60ac0d-485c-48b2-8ea4-6633cdbcd138\") "
Sep 11 17:45:48.525715 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:45:48.525633    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-r6vbp\" (UniqueName: \"kubernetes.io/projected/44bbc11d-62da-41bb-90e3-37c721b01e2f-kube-api-access-r6vbp\") pod \"44bbc11d-62da-41bb-90e3-37c721b01e2f\" (UID: \"44bbc11d-62da-41bb-90e3-37c721b01e2f\") "
Sep 11 17:45:48.525715 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:45:48.525661    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/44bbc11d-62da-41bb-90e3-37c721b01e2f-model-cache\") pod \"44bbc11d-62da-41bb-90e3-37c721b01e2f\" (UID: \"44bbc11d-62da-41bb-90e3-37c721b01e2f\") "
Sep 11 17:45:48.525715 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:45:48.525714    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1a60ac0d-485c-48b2-8ea4-6633cdbcd138-home\") pod \"1a60ac0d-485c-48b2-8ea4-6633cdbcd138\" (UID: \"1a60ac0d-485c-48b2-8ea4-6633cdbcd138\") "
Sep 11 17:45:48.526047 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:45:48.525754    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/44bbc11d-62da-41bb-90e3-37c721b01e2f-home\") pod \"44bbc11d-62da-41bb-90e3-37c721b01e2f\" (UID: \"44bbc11d-62da-41bb-90e3-37c721b01e2f\") "
Sep 11 17:45:48.526047 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:45:48.525770    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/44bbc11d-62da-41bb-90e3-37c721b01e2f-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "44bbc11d-62da-41bb-90e3-37c721b01e2f" (UID: "44bbc11d-62da-41bb-90e3-37c721b01e2f"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 17:45:48.526047 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:45:48.525789    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/44bbc11d-62da-41bb-90e3-37c721b01e2f-tls-certs\") pod \"44bbc11d-62da-41bb-90e3-37c721b01e2f\" (UID: \"44bbc11d-62da-41bb-90e3-37c721b01e2f\") "
Sep 11 17:45:48.526047 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:45:48.525834    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1a60ac0d-485c-48b2-8ea4-6633cdbcd138-tls-certs\") pod \"1a60ac0d-485c-48b2-8ea4-6633cdbcd138\" (UID: \"1a60ac0d-485c-48b2-8ea4-6633cdbcd138\") "
Sep 11 17:45:48.526047 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:45:48.525883    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rsw29\" (UniqueName: \"kubernetes.io/projected/1a60ac0d-485c-48b2-8ea4-6633cdbcd138-kube-api-access-rsw29\") pod \"1a60ac0d-485c-48b2-8ea4-6633cdbcd138\" (UID: \"1a60ac0d-485c-48b2-8ea4-6633cdbcd138\") "
Sep 11 17:45:48.526047 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:45:48.525947    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1a60ac0d-485c-48b2-8ea4-6633cdbcd138-home" (OuterVolumeSpecName: "home") pod "1a60ac0d-485c-48b2-8ea4-6633cdbcd138" (UID: "1a60ac0d-485c-48b2-8ea4-6633cdbcd138"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 17:45:48.526383 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:45:48.526072    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1a60ac0d-485c-48b2-8ea4-6633cdbcd138-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "1a60ac0d-485c-48b2-8ea4-6633cdbcd138" (UID: "1a60ac0d-485c-48b2-8ea4-6633cdbcd138"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 17:45:48.526383 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:45:48.526301    2577 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/44bbc11d-62da-41bb-90e3-37c721b01e2f-tmp-dir\") on node \"ip-10-0-141-139.ec2.internal\" DevicePath \"\""
Sep 11 17:45:48.526383 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:45:48.526323    2577 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1a60ac0d-485c-48b2-8ea4-6633cdbcd138-tmp-dir\") on node \"ip-10-0-141-139.ec2.internal\" DevicePath \"\""
Sep 11 17:45:48.526383 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:45:48.526335    2577 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1a60ac0d-485c-48b2-8ea4-6633cdbcd138-home\") on node \"ip-10-0-141-139.ec2.internal\" DevicePath \"\""
Sep 11 17:45:48.526383 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:45:48.526370    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1a60ac0d-485c-48b2-8ea4-6633cdbcd138-model-cache" (OuterVolumeSpecName: "model-cache") pod "1a60ac0d-485c-48b2-8ea4-6633cdbcd138" (UID: "1a60ac0d-485c-48b2-8ea4-6633cdbcd138"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 17:45:48.526811 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:45:48.526788    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/44bbc11d-62da-41bb-90e3-37c721b01e2f-model-cache" (OuterVolumeSpecName: "model-cache") pod "44bbc11d-62da-41bb-90e3-37c721b01e2f" (UID: "44bbc11d-62da-41bb-90e3-37c721b01e2f"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 17:45:48.526877 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:45:48.526862    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/44bbc11d-62da-41bb-90e3-37c721b01e2f-home" (OuterVolumeSpecName: "home") pod "44bbc11d-62da-41bb-90e3-37c721b01e2f" (UID: "44bbc11d-62da-41bb-90e3-37c721b01e2f"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 17:45:48.528679 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:45:48.528652    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/44bbc11d-62da-41bb-90e3-37c721b01e2f-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "44bbc11d-62da-41bb-90e3-37c721b01e2f" (UID: "44bbc11d-62da-41bb-90e3-37c721b01e2f"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 17:45:48.528911 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:45:48.528881    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/44bbc11d-62da-41bb-90e3-37c721b01e2f-dshm" (OuterVolumeSpecName: "dshm") pod "44bbc11d-62da-41bb-90e3-37c721b01e2f" (UID: "44bbc11d-62da-41bb-90e3-37c721b01e2f"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 17:45:48.529042 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:45:48.528909    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1a60ac0d-485c-48b2-8ea4-6633cdbcd138-dshm" (OuterVolumeSpecName: "dshm") pod "1a60ac0d-485c-48b2-8ea4-6633cdbcd138" (UID: "1a60ac0d-485c-48b2-8ea4-6633cdbcd138"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 17:45:48.529042 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:45:48.528912    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1a60ac0d-485c-48b2-8ea4-6633cdbcd138-kube-api-access-rsw29" (OuterVolumeSpecName: "kube-api-access-rsw29") pod "1a60ac0d-485c-48b2-8ea4-6633cdbcd138" (UID: "1a60ac0d-485c-48b2-8ea4-6633cdbcd138"). InnerVolumeSpecName "kube-api-access-rsw29". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 17:45:48.529170 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:45:48.529144    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1a60ac0d-485c-48b2-8ea4-6633cdbcd138-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "1a60ac0d-485c-48b2-8ea4-6633cdbcd138" (UID: "1a60ac0d-485c-48b2-8ea4-6633cdbcd138"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 17:45:48.529247 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:45:48.529226    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/44bbc11d-62da-41bb-90e3-37c721b01e2f-kube-api-access-r6vbp" (OuterVolumeSpecName: "kube-api-access-r6vbp") pod "44bbc11d-62da-41bb-90e3-37c721b01e2f" (UID: "44bbc11d-62da-41bb-90e3-37c721b01e2f"). InnerVolumeSpecName "kube-api-access-r6vbp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 17:45:48.627164 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:45:48.627063    2577 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1a60ac0d-485c-48b2-8ea4-6633cdbcd138-dshm\") on node \"ip-10-0-141-139.ec2.internal\" DevicePath \"\""
Sep 11 17:45:48.627164 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:45:48.627095    2577 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1a60ac0d-485c-48b2-8ea4-6633cdbcd138-model-cache\") on node \"ip-10-0-141-139.ec2.internal\" DevicePath \"\""
Sep 11 17:45:48.627164 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:45:48.627106    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r6vbp\" (UniqueName: \"kubernetes.io/projected/44bbc11d-62da-41bb-90e3-37c721b01e2f-kube-api-access-r6vbp\") on node \"ip-10-0-141-139.ec2.internal\" DevicePath \"\""
Sep 11 17:45:48.627164 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:45:48.627114    2577 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/44bbc11d-62da-41bb-90e3-37c721b01e2f-model-cache\") on node \"ip-10-0-141-139.ec2.internal\" DevicePath \"\""
Sep 11 17:45:48.627164 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:45:48.627122    2577 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/44bbc11d-62da-41bb-90e3-37c721b01e2f-home\") on node \"ip-10-0-141-139.ec2.internal\" DevicePath \"\""
Sep 11 17:45:48.627164 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:45:48.627131    2577 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/44bbc11d-62da-41bb-90e3-37c721b01e2f-tls-certs\") on node \"ip-10-0-141-139.ec2.internal\" DevicePath \"\""
Sep 11 17:45:48.627164 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:45:48.627139    2577 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1a60ac0d-485c-48b2-8ea4-6633cdbcd138-tls-certs\") on node \"ip-10-0-141-139.ec2.internal\" DevicePath \"\""
Sep 11 17:45:48.627164 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:45:48.627148    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rsw29\" (UniqueName: \"kubernetes.io/projected/1a60ac0d-485c-48b2-8ea4-6633cdbcd138-kube-api-access-rsw29\") on node \"ip-10-0-141-139.ec2.internal\" DevicePath \"\""
Sep 11 17:45:48.627164 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:45:48.627155    2577 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/44bbc11d-62da-41bb-90e3-37c721b01e2f-dshm\") on node \"ip-10-0-141-139.ec2.internal\" DevicePath \"\""
Sep 11 17:45:48.896072 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:45:48.895981    2577 generic.go:356] "Generic (PLEG): container finished" podID="1a60ac0d-485c-48b2-8ea4-6633cdbcd138" containerID="1b1f22351c03b1e69349f6e8b5008c6a239520ea1e97a05676c19abf0fe06c85" exitCode=0
Sep 11 17:45:48.896072 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:45:48.896043    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-144bfa/simulator-w4-144bfa-kserve-6ddd858bc-4vs4s"
Sep 11 17:45:48.896333 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:45:48.896069    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-144bfa/simulator-w4-144bfa-kserve-6ddd858bc-4vs4s" event={"ID":"1a60ac0d-485c-48b2-8ea4-6633cdbcd138","Type":"ContainerDied","Data":"1b1f22351c03b1e69349f6e8b5008c6a239520ea1e97a05676c19abf0fe06c85"}
Sep 11 17:45:48.896333 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:45:48.896122    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-144bfa/simulator-w4-144bfa-kserve-6ddd858bc-4vs4s" event={"ID":"1a60ac0d-485c-48b2-8ea4-6633cdbcd138","Type":"ContainerDied","Data":"8993064e6203d5e7848ede64b47cd1a9aa528e303c5453d2c7334ee9633d3649"}
Sep 11 17:45:48.896333 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:45:48.896144    2577 scope.go:117] "RemoveContainer" containerID="1b1f22351c03b1e69349f6e8b5008c6a239520ea1e97a05676c19abf0fe06c85"
Sep 11 17:45:48.897610 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:45:48.897586    2577 generic.go:356] "Generic (PLEG): container finished" podID="44bbc11d-62da-41bb-90e3-37c721b01e2f" containerID="fe6a7652f020a1ab2148e020c86e0f136ac1d4b4b815d4770a965c718c3d7ae0" exitCode=0
Sep 11 17:45:48.897705 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:45:48.897665    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-144bfa/distinct-w4-144bfa-kserve-648d4c7f9c-m2562"
Sep 11 17:45:48.897705 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:45:48.897676    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-144bfa/distinct-w4-144bfa-kserve-648d4c7f9c-m2562" event={"ID":"44bbc11d-62da-41bb-90e3-37c721b01e2f","Type":"ContainerDied","Data":"fe6a7652f020a1ab2148e020c86e0f136ac1d4b4b815d4770a965c718c3d7ae0"}
Sep 11 17:45:48.897800 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:45:48.897709    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-144bfa/distinct-w4-144bfa-kserve-648d4c7f9c-m2562" event={"ID":"44bbc11d-62da-41bb-90e3-37c721b01e2f","Type":"ContainerDied","Data":"4c25a1faa0ed80229963cbffaef70e2382b6c62a9e538e7b20df64e14431cf67"}
Sep 11 17:45:48.908393 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:45:48.908328    2577 scope.go:117] "RemoveContainer" containerID="1b1f22351c03b1e69349f6e8b5008c6a239520ea1e97a05676c19abf0fe06c85"
Sep 11 17:45:48.908868 ip-10-0-141-139 kubenswrapper[2577]: E0911 17:45:48.908844    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1b1f22351c03b1e69349f6e8b5008c6a239520ea1e97a05676c19abf0fe06c85\": container with ID starting with 1b1f22351c03b1e69349f6e8b5008c6a239520ea1e97a05676c19abf0fe06c85 not found: ID does not exist" containerID="1b1f22351c03b1e69349f6e8b5008c6a239520ea1e97a05676c19abf0fe06c85"
Sep 11 17:45:48.908989 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:45:48.908878    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1b1f22351c03b1e69349f6e8b5008c6a239520ea1e97a05676c19abf0fe06c85"} err="failed to get container status \"1b1f22351c03b1e69349f6e8b5008c6a239520ea1e97a05676c19abf0fe06c85\": rpc error: code = NotFound desc = could not find container \"1b1f22351c03b1e69349f6e8b5008c6a239520ea1e97a05676c19abf0fe06c85\": container with ID starting with 1b1f22351c03b1e69349f6e8b5008c6a239520ea1e97a05676c19abf0fe06c85 not found: ID does not exist"
Sep 11 17:45:48.908989 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:45:48.908896    2577 scope.go:117] "RemoveContainer" containerID="fe6a7652f020a1ab2148e020c86e0f136ac1d4b4b815d4770a965c718c3d7ae0"
Sep 11 17:45:48.918679 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:45:48.918652    2577 scope.go:117] "RemoveContainer" containerID="fe6a7652f020a1ab2148e020c86e0f136ac1d4b4b815d4770a965c718c3d7ae0"
Sep 11 17:45:48.919013 ip-10-0-141-139 kubenswrapper[2577]: E0911 17:45:48.918991    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fe6a7652f020a1ab2148e020c86e0f136ac1d4b4b815d4770a965c718c3d7ae0\": container with ID starting with fe6a7652f020a1ab2148e020c86e0f136ac1d4b4b815d4770a965c718c3d7ae0 not found: ID does not exist" containerID="fe6a7652f020a1ab2148e020c86e0f136ac1d4b4b815d4770a965c718c3d7ae0"
Sep 11 17:45:48.919086 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:45:48.919021    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fe6a7652f020a1ab2148e020c86e0f136ac1d4b4b815d4770a965c718c3d7ae0"} err="failed to get container status \"fe6a7652f020a1ab2148e020c86e0f136ac1d4b4b815d4770a965c718c3d7ae0\": rpc error: code = NotFound desc = could not find container \"fe6a7652f020a1ab2148e020c86e0f136ac1d4b4b815d4770a965c718c3d7ae0\": container with ID starting with fe6a7652f020a1ab2148e020c86e0f136ac1d4b4b815d4770a965c718c3d7ae0 not found: ID does not exist"
Sep 11 17:45:48.927827 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:45:48.927801    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-144bfa/simulator-w4-144bfa-kserve-6ddd858bc-4vs4s"]
Sep 11 17:45:48.927938 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:45:48.927843    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-144bfa/simulator-w4-144bfa-kserve-6ddd858bc-4vs4s"]
Sep 11 17:45:48.936283 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:45:48.936246    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-144bfa/distinct-w4-144bfa-kserve-648d4c7f9c-m2562"]
Sep 11 17:45:48.938048 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:45:48.938027    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-144bfa/distinct-w4-144bfa-kserve-648d4c7f9c-m2562"]
Sep 11 17:45:50.928333 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:45:50.928294    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1a60ac0d-485c-48b2-8ea4-6633cdbcd138" path="/var/lib/kubelet/pods/1a60ac0d-485c-48b2-8ea4-6633cdbcd138/volumes"
Sep 11 17:45:50.928749 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:45:50.928724    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="44bbc11d-62da-41bb-90e3-37c721b01e2f" path="/var/lib/kubelet/pods/44bbc11d-62da-41bb-90e3-37c721b01e2f/volumes"
Sep 11 17:46:03.721923 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:46:03.721882    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-nkmkm"]
Sep 11 17:46:17.794753 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:46:17.794716    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-nkmkm"]
Sep 11 17:46:27.400313 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:46:27.400272    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-nkmkm"]
Sep 11 17:46:35.302793 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:46:35.302757    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w3-2fac54/sub-test-model-2fac54-kserve-fd665846f-cnckr"]
Sep 11 17:46:35.303416 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:46:35.303400    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="44bbc11d-62da-41bb-90e3-37c721b01e2f" containerName="main"
Sep 11 17:46:35.303470 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:46:35.303420    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="44bbc11d-62da-41bb-90e3-37c721b01e2f" containerName="main"
Sep 11 17:46:35.303470 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:46:35.303437    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f5324d08-8464-43fc-a9c3-82960f5b1712" containerName="revoke-keys"
Sep 11 17:46:35.303470 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:46:35.303445    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="f5324d08-8464-43fc-a9c3-82960f5b1712" containerName="revoke-keys"
Sep 11 17:46:35.303470 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:46:35.303458    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="401d5058-6997-4c1c-97b7-3dce976fbd62" containerName="payload-processing"
Sep 11 17:46:35.303470 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:46:35.303467    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="401d5058-6997-4c1c-97b7-3dce976fbd62" containerName="payload-processing"
Sep 11 17:46:35.303665 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:46:35.303489    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="638cfdad-0c58-4a69-8cee-cc50f7ca2655" containerName="payload-pre-processing"
Sep 11 17:46:35.303665 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:46:35.303515    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="638cfdad-0c58-4a69-8cee-cc50f7ca2655" containerName="payload-pre-processing"
Sep 11 17:46:35.303665 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:46:35.303525    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1a60ac0d-485c-48b2-8ea4-6633cdbcd138" containerName="main"
Sep 11 17:46:35.303665 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:46:35.303533    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="1a60ac0d-485c-48b2-8ea4-6633cdbcd138" containerName="main"
Sep 11 17:46:35.303665 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:46:35.303623    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="1a60ac0d-485c-48b2-8ea4-6633cdbcd138" containerName="main"
Sep 11 17:46:35.303665 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:46:35.303636    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="401d5058-6997-4c1c-97b7-3dce976fbd62" containerName="payload-processing"
Sep 11 17:46:35.303665 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:46:35.303647    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="638cfdad-0c58-4a69-8cee-cc50f7ca2655" containerName="payload-pre-processing"
Sep 11 17:46:35.303665 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:46:35.303656    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="f5324d08-8464-43fc-a9c3-82960f5b1712" containerName="revoke-keys"
Sep 11 17:46:35.303665 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:46:35.303665    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="44bbc11d-62da-41bb-90e3-37c721b01e2f" containerName="main"
Sep 11 17:46:35.307413 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:46:35.307392    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-2fac54/sub-test-model-2fac54-kserve-fd665846f-cnckr"
Sep 11 17:46:35.310110 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:46:35.310088    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w3-2fac54\"/\"kube-root-ca.crt\""
Sep 11 17:46:35.311114 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:46:35.311093    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w3-2fac54\"/\"openshift-service-ca.crt\""
Sep 11 17:46:35.311239 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:46:35.311130    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-2fac54\"/\"sub-test-model-2fac54-kserve-self-signed-certs\""
Sep 11 17:46:35.311239 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:46:35.311114    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-2fac54\"/\"default-dockercfg-wsjxc\""
Sep 11 17:46:35.315106 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:46:35.315086    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-2fac54/sub-test-model-2fac54-kserve-fd665846f-cnckr"]
Sep 11 17:46:35.399042 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:46:35.399006    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1a4bb84a-d59d-4c83-aee2-d6bbb2fc67ca-tls-certs\") pod \"sub-test-model-2fac54-kserve-fd665846f-cnckr\" (UID: \"1a4bb84a-d59d-4c83-aee2-d6bbb2fc67ca\") " pod="ai-tenant-e2e-shared-a-w3-2fac54/sub-test-model-2fac54-kserve-fd665846f-cnckr"
Sep 11 17:46:35.399231 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:46:35.399109    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1a4bb84a-d59d-4c83-aee2-d6bbb2fc67ca-home\") pod \"sub-test-model-2fac54-kserve-fd665846f-cnckr\" (UID: \"1a4bb84a-d59d-4c83-aee2-d6bbb2fc67ca\") " pod="ai-tenant-e2e-shared-a-w3-2fac54/sub-test-model-2fac54-kserve-fd665846f-cnckr"
Sep 11 17:46:35.399231 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:46:35.399141    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4jg8t\" (UniqueName: \"kubernetes.io/projected/1a4bb84a-d59d-4c83-aee2-d6bbb2fc67ca-kube-api-access-4jg8t\") pod \"sub-test-model-2fac54-kserve-fd665846f-cnckr\" (UID: \"1a4bb84a-d59d-4c83-aee2-d6bbb2fc67ca\") " pod="ai-tenant-e2e-shared-a-w3-2fac54/sub-test-model-2fac54-kserve-fd665846f-cnckr"
Sep 11 17:46:35.399231 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:46:35.399170    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1a4bb84a-d59d-4c83-aee2-d6bbb2fc67ca-dshm\") pod \"sub-test-model-2fac54-kserve-fd665846f-cnckr\" (UID: \"1a4bb84a-d59d-4c83-aee2-d6bbb2fc67ca\") " pod="ai-tenant-e2e-shared-a-w3-2fac54/sub-test-model-2fac54-kserve-fd665846f-cnckr"
Sep 11 17:46:35.399231 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:46:35.399200    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1a4bb84a-d59d-4c83-aee2-d6bbb2fc67ca-model-cache\") pod \"sub-test-model-2fac54-kserve-fd665846f-cnckr\" (UID: \"1a4bb84a-d59d-4c83-aee2-d6bbb2fc67ca\") " pod="ai-tenant-e2e-shared-a-w3-2fac54/sub-test-model-2fac54-kserve-fd665846f-cnckr"
Sep 11 17:46:35.399231 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:46:35.399221    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1a4bb84a-d59d-4c83-aee2-d6bbb2fc67ca-tmp-dir\") pod \"sub-test-model-2fac54-kserve-fd665846f-cnckr\" (UID: \"1a4bb84a-d59d-4c83-aee2-d6bbb2fc67ca\") " pod="ai-tenant-e2e-shared-a-w3-2fac54/sub-test-model-2fac54-kserve-fd665846f-cnckr"
Sep 11 17:46:35.499845 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:46:35.499798    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1a4bb84a-d59d-4c83-aee2-d6bbb2fc67ca-home\") pod \"sub-test-model-2fac54-kserve-fd665846f-cnckr\" (UID: \"1a4bb84a-d59d-4c83-aee2-d6bbb2fc67ca\") " pod="ai-tenant-e2e-shared-a-w3-2fac54/sub-test-model-2fac54-kserve-fd665846f-cnckr"
Sep 11 17:46:35.499845 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:46:35.499846    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4jg8t\" (UniqueName: \"kubernetes.io/projected/1a4bb84a-d59d-4c83-aee2-d6bbb2fc67ca-kube-api-access-4jg8t\") pod \"sub-test-model-2fac54-kserve-fd665846f-cnckr\" (UID: \"1a4bb84a-d59d-4c83-aee2-d6bbb2fc67ca\") " pod="ai-tenant-e2e-shared-a-w3-2fac54/sub-test-model-2fac54-kserve-fd665846f-cnckr"
Sep 11 17:46:35.500118 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:46:35.499869    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1a4bb84a-d59d-4c83-aee2-d6bbb2fc67ca-dshm\") pod \"sub-test-model-2fac54-kserve-fd665846f-cnckr\" (UID: \"1a4bb84a-d59d-4c83-aee2-d6bbb2fc67ca\") " pod="ai-tenant-e2e-shared-a-w3-2fac54/sub-test-model-2fac54-kserve-fd665846f-cnckr"
Sep 11 17:46:35.500118 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:46:35.499899    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1a4bb84a-d59d-4c83-aee2-d6bbb2fc67ca-model-cache\") pod \"sub-test-model-2fac54-kserve-fd665846f-cnckr\" (UID: \"1a4bb84a-d59d-4c83-aee2-d6bbb2fc67ca\") " pod="ai-tenant-e2e-shared-a-w3-2fac54/sub-test-model-2fac54-kserve-fd665846f-cnckr"
Sep 11 17:46:35.500118 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:46:35.499922    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1a4bb84a-d59d-4c83-aee2-d6bbb2fc67ca-tmp-dir\") pod \"sub-test-model-2fac54-kserve-fd665846f-cnckr\" (UID: \"1a4bb84a-d59d-4c83-aee2-d6bbb2fc67ca\") " pod="ai-tenant-e2e-shared-a-w3-2fac54/sub-test-model-2fac54-kserve-fd665846f-cnckr"
Sep 11 17:46:35.500118 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:46:35.499991    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1a4bb84a-d59d-4c83-aee2-d6bbb2fc67ca-tls-certs\") pod \"sub-test-model-2fac54-kserve-fd665846f-cnckr\" (UID: \"1a4bb84a-d59d-4c83-aee2-d6bbb2fc67ca\") " pod="ai-tenant-e2e-shared-a-w3-2fac54/sub-test-model-2fac54-kserve-fd665846f-cnckr"
Sep 11 17:46:35.500335 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:46:35.500275    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1a4bb84a-d59d-4c83-aee2-d6bbb2fc67ca-home\") pod \"sub-test-model-2fac54-kserve-fd665846f-cnckr\" (UID: \"1a4bb84a-d59d-4c83-aee2-d6bbb2fc67ca\") " pod="ai-tenant-e2e-shared-a-w3-2fac54/sub-test-model-2fac54-kserve-fd665846f-cnckr"
Sep 11 17:46:35.500379 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:46:35.500362    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1a4bb84a-d59d-4c83-aee2-d6bbb2fc67ca-model-cache\") pod \"sub-test-model-2fac54-kserve-fd665846f-cnckr\" (UID: \"1a4bb84a-d59d-4c83-aee2-d6bbb2fc67ca\") " pod="ai-tenant-e2e-shared-a-w3-2fac54/sub-test-model-2fac54-kserve-fd665846f-cnckr"
Sep 11 17:46:35.500415 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:46:35.500386    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1a4bb84a-d59d-4c83-aee2-d6bbb2fc67ca-tmp-dir\") pod \"sub-test-model-2fac54-kserve-fd665846f-cnckr\" (UID: \"1a4bb84a-d59d-4c83-aee2-d6bbb2fc67ca\") " pod="ai-tenant-e2e-shared-a-w3-2fac54/sub-test-model-2fac54-kserve-fd665846f-cnckr"
Sep 11 17:46:35.502278 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:46:35.502246    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1a4bb84a-d59d-4c83-aee2-d6bbb2fc67ca-dshm\") pod \"sub-test-model-2fac54-kserve-fd665846f-cnckr\" (UID: \"1a4bb84a-d59d-4c83-aee2-d6bbb2fc67ca\") " pod="ai-tenant-e2e-shared-a-w3-2fac54/sub-test-model-2fac54-kserve-fd665846f-cnckr"
Sep 11 17:46:35.502516 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:46:35.502486    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1a4bb84a-d59d-4c83-aee2-d6bbb2fc67ca-tls-certs\") pod \"sub-test-model-2fac54-kserve-fd665846f-cnckr\" (UID: \"1a4bb84a-d59d-4c83-aee2-d6bbb2fc67ca\") " pod="ai-tenant-e2e-shared-a-w3-2fac54/sub-test-model-2fac54-kserve-fd665846f-cnckr"
Sep 11 17:46:35.509427 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:46:35.509400    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4jg8t\" (UniqueName: \"kubernetes.io/projected/1a4bb84a-d59d-4c83-aee2-d6bbb2fc67ca-kube-api-access-4jg8t\") pod \"sub-test-model-2fac54-kserve-fd665846f-cnckr\" (UID: \"1a4bb84a-d59d-4c83-aee2-d6bbb2fc67ca\") " pod="ai-tenant-e2e-shared-a-w3-2fac54/sub-test-model-2fac54-kserve-fd665846f-cnckr"
Sep 11 17:46:35.620363 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:46:35.620273    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-2fac54/sub-test-model-2fac54-kserve-fd665846f-cnckr"
Sep 11 17:46:35.897159 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:46:35.897093    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-2fac54/sub-test-model-2fac54-kserve-fd665846f-cnckr"]
Sep 11 17:46:35.899359 ip-10-0-141-139 kubenswrapper[2577]: W0911 17:46:35.899336    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod1a4bb84a_d59d_4c83_aee2_d6bbb2fc67ca.slice/crio-627d7fc808df358c2f61fe4f38ea116044c2008cc3eb1aad53d5d0a071a9e95d WatchSource:0}: Error finding container 627d7fc808df358c2f61fe4f38ea116044c2008cc3eb1aad53d5d0a071a9e95d: Status 404 returned error can't find the container with id 627d7fc808df358c2f61fe4f38ea116044c2008cc3eb1aad53d5d0a071a9e95d
Sep 11 17:46:35.999487 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:46:35.999450    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-nkmkm"]
Sep 11 17:46:36.163754 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:46:36.163653    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-2fac54/sub-test-model-2fac54-kserve-fd665846f-cnckr" event={"ID":"1a4bb84a-d59d-4c83-aee2-d6bbb2fc67ca","Type":"ContainerStarted","Data":"bf0627f455bd9f9963a34a8ddc5cb4b180d5629c0445605a5093625d4cefa869"}
Sep 11 17:46:36.163754 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:46:36.163704    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-2fac54/sub-test-model-2fac54-kserve-fd665846f-cnckr" event={"ID":"1a4bb84a-d59d-4c83-aee2-d6bbb2fc67ca","Type":"ContainerStarted","Data":"627d7fc808df358c2f61fe4f38ea116044c2008cc3eb1aad53d5d0a071a9e95d"}
Sep 11 17:46:36.183050 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:46:36.183000    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w3-2fac54/sub-test-model-2fac54-kserve-fd665846f-cnckr" podStartSLOduration=1.18298469 podStartE2EDuration="1.18298469s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 17:46:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 17:46:36.180440483 +0000 UTC m=+1673.753660595" watchObservedRunningTime="2026-09-11 17:46:36.18298469 +0000 UTC m=+1673.756204801"
Sep 11 17:46:43.255154 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:46:43.255127    2577 scope.go:117] "RemoveContainer" containerID="8f4ddcd41fe11ee377d709f3667e2d49a4697d87da721384c9f559ce5b2f7228"
Sep 11 17:46:44.699201 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:46:44.699158    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-nkmkm"]
Sep 11 17:46:45.620688 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:46:45.620646    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w3-2fac54/sub-test-model-2fac54-kserve-fd665846f-cnckr"
Sep 11 17:46:45.620910 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:46:45.620874    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w3-2fac54/sub-test-model-2fac54-kserve-fd665846f-cnckr"
Sep 11 17:46:45.633363 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:46:45.633339    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w3-2fac54/sub-test-model-2fac54-kserve-fd665846f-cnckr"
Sep 11 17:46:46.229740 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:46:46.229712    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w3-2fac54/sub-test-model-2fac54-kserve-fd665846f-cnckr"
Sep 11 17:47:12.810004 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:47:12.809970    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w3-6fc9e6/sub-test-model-6fc9e6-kserve-58558d44fc-kzz54"]
Sep 11 17:47:12.815214 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:47:12.815190    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-6fc9e6/sub-test-model-6fc9e6-kserve-58558d44fc-kzz54"
Sep 11 17:47:12.825384 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:47:12.825361    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-6fc9e6\"/\"kube-root-ca.crt\""
Sep 11 17:47:12.825532 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:47:12.825388    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-6fc9e6\"/\"default-dockercfg-kgvr7\""
Sep 11 17:47:12.825532 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:47:12.825398    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-6fc9e6\"/\"openshift-service-ca.crt\""
Sep 11 17:47:12.825532 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:47:12.825362    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-6fc9e6\"/\"sub-test-model-6fc9e6-kserve-self-signed-certs\""
Sep 11 17:47:12.850469 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:47:12.850441    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-6fc9e6/sub-test-model-6fc9e6-kserve-58558d44fc-kzz54"]
Sep 11 17:47:12.868444 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:47:12.868419    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/58031c8a-342e-4962-a843-0a404595f291-tls-certs\") pod \"sub-test-model-6fc9e6-kserve-58558d44fc-kzz54\" (UID: \"58031c8a-342e-4962-a843-0a404595f291\") " pod="ai-tenant-e2e-shared-b-w3-6fc9e6/sub-test-model-6fc9e6-kserve-58558d44fc-kzz54"
Sep 11 17:47:12.868583 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:47:12.868496    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/58031c8a-342e-4962-a843-0a404595f291-model-cache\") pod \"sub-test-model-6fc9e6-kserve-58558d44fc-kzz54\" (UID: \"58031c8a-342e-4962-a843-0a404595f291\") " pod="ai-tenant-e2e-shared-b-w3-6fc9e6/sub-test-model-6fc9e6-kserve-58558d44fc-kzz54"
Sep 11 17:47:12.868583 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:47:12.868550    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xf92s\" (UniqueName: \"kubernetes.io/projected/58031c8a-342e-4962-a843-0a404595f291-kube-api-access-xf92s\") pod \"sub-test-model-6fc9e6-kserve-58558d44fc-kzz54\" (UID: \"58031c8a-342e-4962-a843-0a404595f291\") " pod="ai-tenant-e2e-shared-b-w3-6fc9e6/sub-test-model-6fc9e6-kserve-58558d44fc-kzz54"
Sep 11 17:47:12.868583 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:47:12.868570    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/58031c8a-342e-4962-a843-0a404595f291-tmp-dir\") pod \"sub-test-model-6fc9e6-kserve-58558d44fc-kzz54\" (UID: \"58031c8a-342e-4962-a843-0a404595f291\") " pod="ai-tenant-e2e-shared-b-w3-6fc9e6/sub-test-model-6fc9e6-kserve-58558d44fc-kzz54"
Sep 11 17:47:12.868702 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:47:12.868586    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/58031c8a-342e-4962-a843-0a404595f291-dshm\") pod \"sub-test-model-6fc9e6-kserve-58558d44fc-kzz54\" (UID: \"58031c8a-342e-4962-a843-0a404595f291\") " pod="ai-tenant-e2e-shared-b-w3-6fc9e6/sub-test-model-6fc9e6-kserve-58558d44fc-kzz54"
Sep 11 17:47:12.868702 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:47:12.868607    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/58031c8a-342e-4962-a843-0a404595f291-home\") pod \"sub-test-model-6fc9e6-kserve-58558d44fc-kzz54\" (UID: \"58031c8a-342e-4962-a843-0a404595f291\") " pod="ai-tenant-e2e-shared-b-w3-6fc9e6/sub-test-model-6fc9e6-kserve-58558d44fc-kzz54"
Sep 11 17:47:12.969578 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:47:12.969537    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/58031c8a-342e-4962-a843-0a404595f291-model-cache\") pod \"sub-test-model-6fc9e6-kserve-58558d44fc-kzz54\" (UID: \"58031c8a-342e-4962-a843-0a404595f291\") " pod="ai-tenant-e2e-shared-b-w3-6fc9e6/sub-test-model-6fc9e6-kserve-58558d44fc-kzz54"
Sep 11 17:47:12.969781 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:47:12.969604    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xf92s\" (UniqueName: \"kubernetes.io/projected/58031c8a-342e-4962-a843-0a404595f291-kube-api-access-xf92s\") pod \"sub-test-model-6fc9e6-kserve-58558d44fc-kzz54\" (UID: \"58031c8a-342e-4962-a843-0a404595f291\") " pod="ai-tenant-e2e-shared-b-w3-6fc9e6/sub-test-model-6fc9e6-kserve-58558d44fc-kzz54"
Sep 11 17:47:12.969781 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:47:12.969654    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/58031c8a-342e-4962-a843-0a404595f291-tmp-dir\") pod \"sub-test-model-6fc9e6-kserve-58558d44fc-kzz54\" (UID: \"58031c8a-342e-4962-a843-0a404595f291\") " pod="ai-tenant-e2e-shared-b-w3-6fc9e6/sub-test-model-6fc9e6-kserve-58558d44fc-kzz54"
Sep 11 17:47:12.969928 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:47:12.969874    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/58031c8a-342e-4962-a843-0a404595f291-dshm\") pod \"sub-test-model-6fc9e6-kserve-58558d44fc-kzz54\" (UID: \"58031c8a-342e-4962-a843-0a404595f291\") " pod="ai-tenant-e2e-shared-b-w3-6fc9e6/sub-test-model-6fc9e6-kserve-58558d44fc-kzz54"
Sep 11 17:47:12.969928 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:47:12.969910    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/58031c8a-342e-4962-a843-0a404595f291-home\") pod \"sub-test-model-6fc9e6-kserve-58558d44fc-kzz54\" (UID: \"58031c8a-342e-4962-a843-0a404595f291\") " pod="ai-tenant-e2e-shared-b-w3-6fc9e6/sub-test-model-6fc9e6-kserve-58558d44fc-kzz54"
Sep 11 17:47:12.970045 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:47:12.969995    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/58031c8a-342e-4962-a843-0a404595f291-tls-certs\") pod \"sub-test-model-6fc9e6-kserve-58558d44fc-kzz54\" (UID: \"58031c8a-342e-4962-a843-0a404595f291\") " pod="ai-tenant-e2e-shared-b-w3-6fc9e6/sub-test-model-6fc9e6-kserve-58558d44fc-kzz54"
Sep 11 17:47:12.970106 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:47:12.970045    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/58031c8a-342e-4962-a843-0a404595f291-model-cache\") pod \"sub-test-model-6fc9e6-kserve-58558d44fc-kzz54\" (UID: \"58031c8a-342e-4962-a843-0a404595f291\") " pod="ai-tenant-e2e-shared-b-w3-6fc9e6/sub-test-model-6fc9e6-kserve-58558d44fc-kzz54"
Sep 11 17:47:12.970106 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:47:12.970083    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/58031c8a-342e-4962-a843-0a404595f291-tmp-dir\") pod \"sub-test-model-6fc9e6-kserve-58558d44fc-kzz54\" (UID: \"58031c8a-342e-4962-a843-0a404595f291\") " pod="ai-tenant-e2e-shared-b-w3-6fc9e6/sub-test-model-6fc9e6-kserve-58558d44fc-kzz54"
Sep 11 17:47:12.970448 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:47:12.970423    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/58031c8a-342e-4962-a843-0a404595f291-home\") pod \"sub-test-model-6fc9e6-kserve-58558d44fc-kzz54\" (UID: \"58031c8a-342e-4962-a843-0a404595f291\") " pod="ai-tenant-e2e-shared-b-w3-6fc9e6/sub-test-model-6fc9e6-kserve-58558d44fc-kzz54"
Sep 11 17:47:12.972542 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:47:12.972517    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/58031c8a-342e-4962-a843-0a404595f291-dshm\") pod \"sub-test-model-6fc9e6-kserve-58558d44fc-kzz54\" (UID: \"58031c8a-342e-4962-a843-0a404595f291\") " pod="ai-tenant-e2e-shared-b-w3-6fc9e6/sub-test-model-6fc9e6-kserve-58558d44fc-kzz54"
Sep 11 17:47:12.972646 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:47:12.972630    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/58031c8a-342e-4962-a843-0a404595f291-tls-certs\") pod \"sub-test-model-6fc9e6-kserve-58558d44fc-kzz54\" (UID: \"58031c8a-342e-4962-a843-0a404595f291\") " pod="ai-tenant-e2e-shared-b-w3-6fc9e6/sub-test-model-6fc9e6-kserve-58558d44fc-kzz54"
Sep 11 17:47:12.977657 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:47:12.977636    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xf92s\" (UniqueName: \"kubernetes.io/projected/58031c8a-342e-4962-a843-0a404595f291-kube-api-access-xf92s\") pod \"sub-test-model-6fc9e6-kserve-58558d44fc-kzz54\" (UID: \"58031c8a-342e-4962-a843-0a404595f291\") " pod="ai-tenant-e2e-shared-b-w3-6fc9e6/sub-test-model-6fc9e6-kserve-58558d44fc-kzz54"
Sep 11 17:47:13.125033 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:47:13.124942    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-6fc9e6/sub-test-model-6fc9e6-kserve-58558d44fc-kzz54"
Sep 11 17:47:13.401107 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:47:13.401081    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-6fc9e6/sub-test-model-6fc9e6-kserve-58558d44fc-kzz54"]
Sep 11 17:47:13.403252 ip-10-0-141-139 kubenswrapper[2577]: W0911 17:47:13.403222    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod58031c8a_342e_4962_a843_0a404595f291.slice/crio-9a1a8179184279915091ad2de22c41f06f5c49928893259228019e7f38191aac WatchSource:0}: Error finding container 9a1a8179184279915091ad2de22c41f06f5c49928893259228019e7f38191aac: Status 404 returned error can't find the container with id 9a1a8179184279915091ad2de22c41f06f5c49928893259228019e7f38191aac
Sep 11 17:47:14.369718 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:47:14.369673    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-6fc9e6/sub-test-model-6fc9e6-kserve-58558d44fc-kzz54" event={"ID":"58031c8a-342e-4962-a843-0a404595f291","Type":"ContainerStarted","Data":"63d73491a9b7842fe5f587c13e67a56bf6f87b7639e975443b0541103704e8b0"}
Sep 11 17:47:14.370126 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:47:14.369728    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-6fc9e6/sub-test-model-6fc9e6-kserve-58558d44fc-kzz54" event={"ID":"58031c8a-342e-4962-a843-0a404595f291","Type":"ContainerStarted","Data":"9a1a8179184279915091ad2de22c41f06f5c49928893259228019e7f38191aac"}
Sep 11 17:47:14.389713 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:47:14.389650    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w3-6fc9e6/sub-test-model-6fc9e6-kserve-58558d44fc-kzz54" podStartSLOduration=2.389631491 podStartE2EDuration="2.389631491s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 17:47:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 17:47:14.387167637 +0000 UTC m=+1711.960387751" watchObservedRunningTime="2026-09-11 17:47:14.389631491 +0000 UTC m=+1711.962851602"
Sep 11 17:47:16.438961 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:47:16.438927    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-nkmkm"]
Sep 11 17:47:23.125817 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:47:23.125771    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w3-6fc9e6/sub-test-model-6fc9e6-kserve-58558d44fc-kzz54"
Sep 11 17:47:23.126268 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:47:23.125844    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w3-6fc9e6/sub-test-model-6fc9e6-kserve-58558d44fc-kzz54"
Sep 11 17:47:23.138742 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:47:23.138715    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w3-6fc9e6/sub-test-model-6fc9e6-kserve-58558d44fc-kzz54"
Sep 11 17:47:23.429667 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:47:23.429592    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w3-6fc9e6/sub-test-model-6fc9e6-kserve-58558d44fc-kzz54"
Sep 11 17:47:31.092050 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:47:31.092011    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-nkmkm"]
Sep 11 17:47:44.394863 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:47:44.394820    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-nkmkm"]
Sep 11 17:47:47.353209 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:47:47.353172    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-2fac54/sub-test-model-2fac54-kserve-fd665846f-cnckr"]
Sep 11 17:47:48.312301 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:47:48.312266    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-6fc9e6/sub-test-model-6fc9e6-kserve-58558d44fc-kzz54"]
Sep 11 17:47:56.195117 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:47:56.195082    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-nkmkm"]
Sep 11 17:48:02.373680 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:48:02.373622    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w3-2fac54/sub-test-model-2fac54-kserve-fd665846f-cnckr" podUID="1a4bb84a-d59d-4c83-aee2-d6bbb2fc67ca" containerName="main" containerID="cri-o://bf0627f455bd9f9963a34a8ddc5cb4b180d5629c0445605a5093625d4cefa869" gracePeriod=45
Sep 11 17:48:02.627841 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:48:02.627775    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-2fac54/sub-test-model-2fac54-kserve-fd665846f-cnckr"
Sep 11 17:48:02.647638 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:48:02.647599    2577 generic.go:356] "Generic (PLEG): container finished" podID="1a4bb84a-d59d-4c83-aee2-d6bbb2fc67ca" containerID="bf0627f455bd9f9963a34a8ddc5cb4b180d5629c0445605a5093625d4cefa869" exitCode=0
Sep 11 17:48:02.647820 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:48:02.647657    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-2fac54/sub-test-model-2fac54-kserve-fd665846f-cnckr" event={"ID":"1a4bb84a-d59d-4c83-aee2-d6bbb2fc67ca","Type":"ContainerDied","Data":"bf0627f455bd9f9963a34a8ddc5cb4b180d5629c0445605a5093625d4cefa869"}
Sep 11 17:48:02.647820 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:48:02.647692    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-2fac54/sub-test-model-2fac54-kserve-fd665846f-cnckr" event={"ID":"1a4bb84a-d59d-4c83-aee2-d6bbb2fc67ca","Type":"ContainerDied","Data":"627d7fc808df358c2f61fe4f38ea116044c2008cc3eb1aad53d5d0a071a9e95d"}
Sep 11 17:48:02.647820 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:48:02.647713    2577 scope.go:117] "RemoveContainer" containerID="bf0627f455bd9f9963a34a8ddc5cb4b180d5629c0445605a5093625d4cefa869"
Sep 11 17:48:02.647984 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:48:02.647872    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-2fac54/sub-test-model-2fac54-kserve-fd665846f-cnckr"
Sep 11 17:48:02.659214 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:48:02.659183    2577 scope.go:117] "RemoveContainer" containerID="bf0627f455bd9f9963a34a8ddc5cb4b180d5629c0445605a5093625d4cefa869"
Sep 11 17:48:02.659562 ip-10-0-141-139 kubenswrapper[2577]: E0911 17:48:02.659540    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bf0627f455bd9f9963a34a8ddc5cb4b180d5629c0445605a5093625d4cefa869\": container with ID starting with bf0627f455bd9f9963a34a8ddc5cb4b180d5629c0445605a5093625d4cefa869 not found: ID does not exist" containerID="bf0627f455bd9f9963a34a8ddc5cb4b180d5629c0445605a5093625d4cefa869"
Sep 11 17:48:02.659649 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:48:02.659575    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bf0627f455bd9f9963a34a8ddc5cb4b180d5629c0445605a5093625d4cefa869"} err="failed to get container status \"bf0627f455bd9f9963a34a8ddc5cb4b180d5629c0445605a5093625d4cefa869\": rpc error: code = NotFound desc = could not find container \"bf0627f455bd9f9963a34a8ddc5cb4b180d5629c0445605a5093625d4cefa869\": container with ID starting with bf0627f455bd9f9963a34a8ddc5cb4b180d5629c0445605a5093625d4cefa869 not found: ID does not exist"
Sep 11 17:48:02.667849 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:48:02.667829    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1a4bb84a-d59d-4c83-aee2-d6bbb2fc67ca-tls-certs\") pod \"1a4bb84a-d59d-4c83-aee2-d6bbb2fc67ca\" (UID: \"1a4bb84a-d59d-4c83-aee2-d6bbb2fc67ca\") "
Sep 11 17:48:02.667933 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:48:02.667875    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4jg8t\" (UniqueName: \"kubernetes.io/projected/1a4bb84a-d59d-4c83-aee2-d6bbb2fc67ca-kube-api-access-4jg8t\") pod \"1a4bb84a-d59d-4c83-aee2-d6bbb2fc67ca\" (UID: \"1a4bb84a-d59d-4c83-aee2-d6bbb2fc67ca\") "
Sep 11 17:48:02.668031 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:48:02.668009    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1a4bb84a-d59d-4c83-aee2-d6bbb2fc67ca-home\") pod \"1a4bb84a-d59d-4c83-aee2-d6bbb2fc67ca\" (UID: \"1a4bb84a-d59d-4c83-aee2-d6bbb2fc67ca\") "
Sep 11 17:48:02.668086 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:48:02.668056    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1a4bb84a-d59d-4c83-aee2-d6bbb2fc67ca-dshm\") pod \"1a4bb84a-d59d-4c83-aee2-d6bbb2fc67ca\" (UID: \"1a4bb84a-d59d-4c83-aee2-d6bbb2fc67ca\") "
Sep 11 17:48:02.668145 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:48:02.668104    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1a4bb84a-d59d-4c83-aee2-d6bbb2fc67ca-tmp-dir\") pod \"1a4bb84a-d59d-4c83-aee2-d6bbb2fc67ca\" (UID: \"1a4bb84a-d59d-4c83-aee2-d6bbb2fc67ca\") "
Sep 11 17:48:02.668262 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:48:02.668244    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1a4bb84a-d59d-4c83-aee2-d6bbb2fc67ca-model-cache\") pod \"1a4bb84a-d59d-4c83-aee2-d6bbb2fc67ca\" (UID: \"1a4bb84a-d59d-4c83-aee2-d6bbb2fc67ca\") "
Sep 11 17:48:02.668331 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:48:02.668284    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1a4bb84a-d59d-4c83-aee2-d6bbb2fc67ca-home" (OuterVolumeSpecName: "home") pod "1a4bb84a-d59d-4c83-aee2-d6bbb2fc67ca" (UID: "1a4bb84a-d59d-4c83-aee2-d6bbb2fc67ca"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 17:48:02.668498 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:48:02.668472    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1a4bb84a-d59d-4c83-aee2-d6bbb2fc67ca-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "1a4bb84a-d59d-4c83-aee2-d6bbb2fc67ca" (UID: "1a4bb84a-d59d-4c83-aee2-d6bbb2fc67ca"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 17:48:02.668617 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:48:02.668585    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1a4bb84a-d59d-4c83-aee2-d6bbb2fc67ca-model-cache" (OuterVolumeSpecName: "model-cache") pod "1a4bb84a-d59d-4c83-aee2-d6bbb2fc67ca" (UID: "1a4bb84a-d59d-4c83-aee2-d6bbb2fc67ca"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 17:48:02.668866 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:48:02.668850    2577 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1a4bb84a-d59d-4c83-aee2-d6bbb2fc67ca-home\") on node \"ip-10-0-141-139.ec2.internal\" DevicePath \"\""
Sep 11 17:48:02.668933 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:48:02.668874    2577 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1a4bb84a-d59d-4c83-aee2-d6bbb2fc67ca-tmp-dir\") on node \"ip-10-0-141-139.ec2.internal\" DevicePath \"\""
Sep 11 17:48:02.668933 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:48:02.668890    2577 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1a4bb84a-d59d-4c83-aee2-d6bbb2fc67ca-model-cache\") on node \"ip-10-0-141-139.ec2.internal\" DevicePath \"\""
Sep 11 17:48:02.670312 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:48:02.670285    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1a4bb84a-d59d-4c83-aee2-d6bbb2fc67ca-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "1a4bb84a-d59d-4c83-aee2-d6bbb2fc67ca" (UID: "1a4bb84a-d59d-4c83-aee2-d6bbb2fc67ca"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 17:48:02.670754 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:48:02.670728    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1a4bb84a-d59d-4c83-aee2-d6bbb2fc67ca-dshm" (OuterVolumeSpecName: "dshm") pod "1a4bb84a-d59d-4c83-aee2-d6bbb2fc67ca" (UID: "1a4bb84a-d59d-4c83-aee2-d6bbb2fc67ca"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 17:48:02.670885 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:48:02.670765    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1a4bb84a-d59d-4c83-aee2-d6bbb2fc67ca-kube-api-access-4jg8t" (OuterVolumeSpecName: "kube-api-access-4jg8t") pod "1a4bb84a-d59d-4c83-aee2-d6bbb2fc67ca" (UID: "1a4bb84a-d59d-4c83-aee2-d6bbb2fc67ca"). InnerVolumeSpecName "kube-api-access-4jg8t". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 17:48:02.770179 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:48:02.770143    2577 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1a4bb84a-d59d-4c83-aee2-d6bbb2fc67ca-tls-certs\") on node \"ip-10-0-141-139.ec2.internal\" DevicePath \"\""
Sep 11 17:48:02.770179 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:48:02.770172    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4jg8t\" (UniqueName: \"kubernetes.io/projected/1a4bb84a-d59d-4c83-aee2-d6bbb2fc67ca-kube-api-access-4jg8t\") on node \"ip-10-0-141-139.ec2.internal\" DevicePath \"\""
Sep 11 17:48:02.770179 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:48:02.770182    2577 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1a4bb84a-d59d-4c83-aee2-d6bbb2fc67ca-dshm\") on node \"ip-10-0-141-139.ec2.internal\" DevicePath \"\""
Sep 11 17:48:02.966237 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:48:02.966197    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-2fac54/sub-test-model-2fac54-kserve-fd665846f-cnckr"]
Sep 11 17:48:02.970129 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:48:02.970102    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w3-2fac54/sub-test-model-2fac54-kserve-fd665846f-cnckr"]
Sep 11 17:48:03.332182 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:48:03.332141    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w3-6fc9e6/sub-test-model-6fc9e6-kserve-58558d44fc-kzz54" podUID="58031c8a-342e-4962-a843-0a404595f291" containerName="main" containerID="cri-o://63d73491a9b7842fe5f587c13e67a56bf6f87b7639e975443b0541103704e8b0" gracePeriod=45
Sep 11 17:48:03.419056 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:48:03.419018    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-b-w3-6fc9e6/sub-test-model-6fc9e6-kserve-58558d44fc-kzz54" podUID="58031c8a-342e-4962-a843-0a404595f291" containerName="main" probeResult="failure" output="Get \"https://10.134.0.82:8000/ready\": dial tcp 10.134.0.82:8000: connect: connection refused"
Sep 11 17:48:03.584944 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:48:03.584875    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-6fc9e6/sub-test-model-6fc9e6-kserve-58558d44fc-kzz54"
Sep 11 17:48:03.656012 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:48:03.655972    2577 generic.go:356] "Generic (PLEG): container finished" podID="58031c8a-342e-4962-a843-0a404595f291" containerID="63d73491a9b7842fe5f587c13e67a56bf6f87b7639e975443b0541103704e8b0" exitCode=0
Sep 11 17:48:03.656203 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:48:03.656028    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-6fc9e6/sub-test-model-6fc9e6-kserve-58558d44fc-kzz54"
Sep 11 17:48:03.656203 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:48:03.656032    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-6fc9e6/sub-test-model-6fc9e6-kserve-58558d44fc-kzz54" event={"ID":"58031c8a-342e-4962-a843-0a404595f291","Type":"ContainerDied","Data":"63d73491a9b7842fe5f587c13e67a56bf6f87b7639e975443b0541103704e8b0"}
Sep 11 17:48:03.656203 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:48:03.656073    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-6fc9e6/sub-test-model-6fc9e6-kserve-58558d44fc-kzz54" event={"ID":"58031c8a-342e-4962-a843-0a404595f291","Type":"ContainerDied","Data":"9a1a8179184279915091ad2de22c41f06f5c49928893259228019e7f38191aac"}
Sep 11 17:48:03.656203 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:48:03.656095    2577 scope.go:117] "RemoveContainer" containerID="63d73491a9b7842fe5f587c13e67a56bf6f87b7639e975443b0541103704e8b0"
Sep 11 17:48:03.665045 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:48:03.665026    2577 scope.go:117] "RemoveContainer" containerID="63d73491a9b7842fe5f587c13e67a56bf6f87b7639e975443b0541103704e8b0"
Sep 11 17:48:03.665338 ip-10-0-141-139 kubenswrapper[2577]: E0911 17:48:03.665317    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"63d73491a9b7842fe5f587c13e67a56bf6f87b7639e975443b0541103704e8b0\": container with ID starting with 63d73491a9b7842fe5f587c13e67a56bf6f87b7639e975443b0541103704e8b0 not found: ID does not exist" containerID="63d73491a9b7842fe5f587c13e67a56bf6f87b7639e975443b0541103704e8b0"
Sep 11 17:48:03.665384 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:48:03.665347    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"63d73491a9b7842fe5f587c13e67a56bf6f87b7639e975443b0541103704e8b0"} err="failed to get container status \"63d73491a9b7842fe5f587c13e67a56bf6f87b7639e975443b0541103704e8b0\": rpc error: code = NotFound desc = could not find container \"63d73491a9b7842fe5f587c13e67a56bf6f87b7639e975443b0541103704e8b0\": container with ID starting with 63d73491a9b7842fe5f587c13e67a56bf6f87b7639e975443b0541103704e8b0 not found: ID does not exist"
Sep 11 17:48:03.684763 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:48:03.684725    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/58031c8a-342e-4962-a843-0a404595f291-tls-certs\") pod \"58031c8a-342e-4962-a843-0a404595f291\" (UID: \"58031c8a-342e-4962-a843-0a404595f291\") "
Sep 11 17:48:03.684891 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:48:03.684789    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/58031c8a-342e-4962-a843-0a404595f291-home\") pod \"58031c8a-342e-4962-a843-0a404595f291\" (UID: \"58031c8a-342e-4962-a843-0a404595f291\") "
Sep 11 17:48:03.684891 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:48:03.684814    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/58031c8a-342e-4962-a843-0a404595f291-tmp-dir\") pod \"58031c8a-342e-4962-a843-0a404595f291\" (UID: \"58031c8a-342e-4962-a843-0a404595f291\") "
Sep 11 17:48:03.684984 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:48:03.684912    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/58031c8a-342e-4962-a843-0a404595f291-model-cache\") pod \"58031c8a-342e-4962-a843-0a404595f291\" (UID: \"58031c8a-342e-4962-a843-0a404595f291\") "
Sep 11 17:48:03.684984 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:48:03.684954    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xf92s\" (UniqueName: \"kubernetes.io/projected/58031c8a-342e-4962-a843-0a404595f291-kube-api-access-xf92s\") pod \"58031c8a-342e-4962-a843-0a404595f291\" (UID: \"58031c8a-342e-4962-a843-0a404595f291\") "
Sep 11 17:48:03.685071 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:48:03.685000    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/58031c8a-342e-4962-a843-0a404595f291-dshm\") pod \"58031c8a-342e-4962-a843-0a404595f291\" (UID: \"58031c8a-342e-4962-a843-0a404595f291\") "
Sep 11 17:48:03.685071 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:48:03.685025    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/58031c8a-342e-4962-a843-0a404595f291-home" (OuterVolumeSpecName: "home") pod "58031c8a-342e-4962-a843-0a404595f291" (UID: "58031c8a-342e-4962-a843-0a404595f291"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 17:48:03.685071 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:48:03.685051    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/58031c8a-342e-4962-a843-0a404595f291-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "58031c8a-342e-4962-a843-0a404595f291" (UID: "58031c8a-342e-4962-a843-0a404595f291"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 17:48:03.685210 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:48:03.685179    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/58031c8a-342e-4962-a843-0a404595f291-model-cache" (OuterVolumeSpecName: "model-cache") pod "58031c8a-342e-4962-a843-0a404595f291" (UID: "58031c8a-342e-4962-a843-0a404595f291"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 17:48:03.685392 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:48:03.685374    2577 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/58031c8a-342e-4962-a843-0a404595f291-model-cache\") on node \"ip-10-0-141-139.ec2.internal\" DevicePath \"\""
Sep 11 17:48:03.685392 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:48:03.685391    2577 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/58031c8a-342e-4962-a843-0a404595f291-home\") on node \"ip-10-0-141-139.ec2.internal\" DevicePath \"\""
Sep 11 17:48:03.685527 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:48:03.685401    2577 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/58031c8a-342e-4962-a843-0a404595f291-tmp-dir\") on node \"ip-10-0-141-139.ec2.internal\" DevicePath \"\""
Sep 11 17:48:03.687263 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:48:03.687242    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/58031c8a-342e-4962-a843-0a404595f291-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "58031c8a-342e-4962-a843-0a404595f291" (UID: "58031c8a-342e-4962-a843-0a404595f291"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 17:48:03.687355 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:48:03.687268    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/58031c8a-342e-4962-a843-0a404595f291-kube-api-access-xf92s" (OuterVolumeSpecName: "kube-api-access-xf92s") pod "58031c8a-342e-4962-a843-0a404595f291" (UID: "58031c8a-342e-4962-a843-0a404595f291"). InnerVolumeSpecName "kube-api-access-xf92s". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 17:48:03.687355 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:48:03.687293    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/58031c8a-342e-4962-a843-0a404595f291-dshm" (OuterVolumeSpecName: "dshm") pod "58031c8a-342e-4962-a843-0a404595f291" (UID: "58031c8a-342e-4962-a843-0a404595f291"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 17:48:03.786929 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:48:03.786889    2577 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/58031c8a-342e-4962-a843-0a404595f291-dshm\") on node \"ip-10-0-141-139.ec2.internal\" DevicePath \"\""
Sep 11 17:48:03.786929 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:48:03.786920    2577 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/58031c8a-342e-4962-a843-0a404595f291-tls-certs\") on node \"ip-10-0-141-139.ec2.internal\" DevicePath \"\""
Sep 11 17:48:03.786929 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:48:03.786930    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xf92s\" (UniqueName: \"kubernetes.io/projected/58031c8a-342e-4962-a843-0a404595f291-kube-api-access-xf92s\") on node \"ip-10-0-141-139.ec2.internal\" DevicePath \"\""
Sep 11 17:48:03.977417 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:48:03.977370    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-6fc9e6/sub-test-model-6fc9e6-kserve-58558d44fc-kzz54"]
Sep 11 17:48:03.981146 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:48:03.981116    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w3-6fc9e6/sub-test-model-6fc9e6-kserve-58558d44fc-kzz54"]
Sep 11 17:48:04.926562 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:48:04.926521    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1a4bb84a-d59d-4c83-aee2-d6bbb2fc67ca" path="/var/lib/kubelet/pods/1a4bb84a-d59d-4c83-aee2-d6bbb2fc67ca/volumes"
Sep 11 17:48:04.927116 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:48:04.927096    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="58031c8a-342e-4962-a843-0a404595f291" path="/var/lib/kubelet/pods/58031c8a-342e-4962-a843-0a404595f291/volumes"
Sep 11 17:48:13.504467 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:48:13.504434    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w3-2fac54/rate-test-model-2fac54-kserve-85f5f8cb6c-kvmch"]
Sep 11 17:48:13.504959 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:48:13.504878    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1a4bb84a-d59d-4c83-aee2-d6bbb2fc67ca" containerName="main"
Sep 11 17:48:13.504959 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:48:13.504891    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="1a4bb84a-d59d-4c83-aee2-d6bbb2fc67ca" containerName="main"
Sep 11 17:48:13.504959 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:48:13.504900    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="58031c8a-342e-4962-a843-0a404595f291" containerName="main"
Sep 11 17:48:13.504959 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:48:13.504906    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="58031c8a-342e-4962-a843-0a404595f291" containerName="main"
Sep 11 17:48:13.505115 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:48:13.504979    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="1a4bb84a-d59d-4c83-aee2-d6bbb2fc67ca" containerName="main"
Sep 11 17:48:13.505115 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:48:13.504988    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="58031c8a-342e-4962-a843-0a404595f291" containerName="main"
Sep 11 17:48:13.508081 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:48:13.508062    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-2fac54/rate-test-model-2fac54-kserve-85f5f8cb6c-kvmch"
Sep 11 17:48:13.510658 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:48:13.510638    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w3-2fac54\"/\"openshift-service-ca.crt\""
Sep 11 17:48:13.510995 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:48:13.510979    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w3-2fac54\"/\"kube-root-ca.crt\""
Sep 11 17:48:13.511940 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:48:13.511921    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-2fac54\"/\"default-dockercfg-wsjxc\""
Sep 11 17:48:13.512039 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:48:13.511954    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-2fac54\"/\"rate-test-model-2fac54-kserve-self-signed-certs\""
Sep 11 17:48:13.523329 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:48:13.523292    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-2fac54/rate-test-model-2fac54-kserve-85f5f8cb6c-kvmch"]
Sep 11 17:48:13.584280 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:48:13.584240    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4fbefcb1-b54a-4367-9026-e6a42536f21a-model-cache\") pod \"rate-test-model-2fac54-kserve-85f5f8cb6c-kvmch\" (UID: \"4fbefcb1-b54a-4367-9026-e6a42536f21a\") " pod="ai-tenant-e2e-shared-a-w3-2fac54/rate-test-model-2fac54-kserve-85f5f8cb6c-kvmch"
Sep 11 17:48:13.584470 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:48:13.584291    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4fbefcb1-b54a-4367-9026-e6a42536f21a-home\") pod \"rate-test-model-2fac54-kserve-85f5f8cb6c-kvmch\" (UID: \"4fbefcb1-b54a-4367-9026-e6a42536f21a\") " pod="ai-tenant-e2e-shared-a-w3-2fac54/rate-test-model-2fac54-kserve-85f5f8cb6c-kvmch"
Sep 11 17:48:13.584470 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:48:13.584347    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4fbefcb1-b54a-4367-9026-e6a42536f21a-tmp-dir\") pod \"rate-test-model-2fac54-kserve-85f5f8cb6c-kvmch\" (UID: \"4fbefcb1-b54a-4367-9026-e6a42536f21a\") " pod="ai-tenant-e2e-shared-a-w3-2fac54/rate-test-model-2fac54-kserve-85f5f8cb6c-kvmch"
Sep 11 17:48:13.584470 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:48:13.584373    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p4bx6\" (UniqueName: \"kubernetes.io/projected/4fbefcb1-b54a-4367-9026-e6a42536f21a-kube-api-access-p4bx6\") pod \"rate-test-model-2fac54-kserve-85f5f8cb6c-kvmch\" (UID: \"4fbefcb1-b54a-4367-9026-e6a42536f21a\") " pod="ai-tenant-e2e-shared-a-w3-2fac54/rate-test-model-2fac54-kserve-85f5f8cb6c-kvmch"
Sep 11 17:48:13.584470 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:48:13.584420    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4fbefcb1-b54a-4367-9026-e6a42536f21a-tls-certs\") pod \"rate-test-model-2fac54-kserve-85f5f8cb6c-kvmch\" (UID: \"4fbefcb1-b54a-4367-9026-e6a42536f21a\") " pod="ai-tenant-e2e-shared-a-w3-2fac54/rate-test-model-2fac54-kserve-85f5f8cb6c-kvmch"
Sep 11 17:48:13.584470 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:48:13.584462    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4fbefcb1-b54a-4367-9026-e6a42536f21a-dshm\") pod \"rate-test-model-2fac54-kserve-85f5f8cb6c-kvmch\" (UID: \"4fbefcb1-b54a-4367-9026-e6a42536f21a\") " pod="ai-tenant-e2e-shared-a-w3-2fac54/rate-test-model-2fac54-kserve-85f5f8cb6c-kvmch"
Sep 11 17:48:13.685389 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:48:13.685335    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4fbefcb1-b54a-4367-9026-e6a42536f21a-tmp-dir\") pod \"rate-test-model-2fac54-kserve-85f5f8cb6c-kvmch\" (UID: \"4fbefcb1-b54a-4367-9026-e6a42536f21a\") " pod="ai-tenant-e2e-shared-a-w3-2fac54/rate-test-model-2fac54-kserve-85f5f8cb6c-kvmch"
Sep 11 17:48:13.685389 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:48:13.685397    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p4bx6\" (UniqueName: \"kubernetes.io/projected/4fbefcb1-b54a-4367-9026-e6a42536f21a-kube-api-access-p4bx6\") pod \"rate-test-model-2fac54-kserve-85f5f8cb6c-kvmch\" (UID: \"4fbefcb1-b54a-4367-9026-e6a42536f21a\") " pod="ai-tenant-e2e-shared-a-w3-2fac54/rate-test-model-2fac54-kserve-85f5f8cb6c-kvmch"
Sep 11 17:48:13.685680 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:48:13.685428    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4fbefcb1-b54a-4367-9026-e6a42536f21a-tls-certs\") pod \"rate-test-model-2fac54-kserve-85f5f8cb6c-kvmch\" (UID: \"4fbefcb1-b54a-4367-9026-e6a42536f21a\") " pod="ai-tenant-e2e-shared-a-w3-2fac54/rate-test-model-2fac54-kserve-85f5f8cb6c-kvmch"
Sep 11 17:48:13.685680 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:48:13.685468    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4fbefcb1-b54a-4367-9026-e6a42536f21a-dshm\") pod \"rate-test-model-2fac54-kserve-85f5f8cb6c-kvmch\" (UID: \"4fbefcb1-b54a-4367-9026-e6a42536f21a\") " pod="ai-tenant-e2e-shared-a-w3-2fac54/rate-test-model-2fac54-kserve-85f5f8cb6c-kvmch"
Sep 11 17:48:13.685680 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:48:13.685498    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4fbefcb1-b54a-4367-9026-e6a42536f21a-model-cache\") pod \"rate-test-model-2fac54-kserve-85f5f8cb6c-kvmch\" (UID: \"4fbefcb1-b54a-4367-9026-e6a42536f21a\") " pod="ai-tenant-e2e-shared-a-w3-2fac54/rate-test-model-2fac54-kserve-85f5f8cb6c-kvmch"
Sep 11 17:48:13.685680 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:48:13.685559    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4fbefcb1-b54a-4367-9026-e6a42536f21a-home\") pod \"rate-test-model-2fac54-kserve-85f5f8cb6c-kvmch\" (UID: \"4fbefcb1-b54a-4367-9026-e6a42536f21a\") " pod="ai-tenant-e2e-shared-a-w3-2fac54/rate-test-model-2fac54-kserve-85f5f8cb6c-kvmch"
Sep 11 17:48:13.685890 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:48:13.685839    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4fbefcb1-b54a-4367-9026-e6a42536f21a-tmp-dir\") pod \"rate-test-model-2fac54-kserve-85f5f8cb6c-kvmch\" (UID: \"4fbefcb1-b54a-4367-9026-e6a42536f21a\") " pod="ai-tenant-e2e-shared-a-w3-2fac54/rate-test-model-2fac54-kserve-85f5f8cb6c-kvmch"
Sep 11 17:48:13.685890 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:48:13.685880    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4fbefcb1-b54a-4367-9026-e6a42536f21a-home\") pod \"rate-test-model-2fac54-kserve-85f5f8cb6c-kvmch\" (UID: \"4fbefcb1-b54a-4367-9026-e6a42536f21a\") " pod="ai-tenant-e2e-shared-a-w3-2fac54/rate-test-model-2fac54-kserve-85f5f8cb6c-kvmch"
Sep 11 17:48:13.686005 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:48:13.685968    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4fbefcb1-b54a-4367-9026-e6a42536f21a-model-cache\") pod \"rate-test-model-2fac54-kserve-85f5f8cb6c-kvmch\" (UID: \"4fbefcb1-b54a-4367-9026-e6a42536f21a\") " pod="ai-tenant-e2e-shared-a-w3-2fac54/rate-test-model-2fac54-kserve-85f5f8cb6c-kvmch"
Sep 11 17:48:13.687836 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:48:13.687818    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4fbefcb1-b54a-4367-9026-e6a42536f21a-dshm\") pod \"rate-test-model-2fac54-kserve-85f5f8cb6c-kvmch\" (UID: \"4fbefcb1-b54a-4367-9026-e6a42536f21a\") " pod="ai-tenant-e2e-shared-a-w3-2fac54/rate-test-model-2fac54-kserve-85f5f8cb6c-kvmch"
Sep 11 17:48:13.688158 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:48:13.688140    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4fbefcb1-b54a-4367-9026-e6a42536f21a-tls-certs\") pod \"rate-test-model-2fac54-kserve-85f5f8cb6c-kvmch\" (UID: \"4fbefcb1-b54a-4367-9026-e6a42536f21a\") " pod="ai-tenant-e2e-shared-a-w3-2fac54/rate-test-model-2fac54-kserve-85f5f8cb6c-kvmch"
Sep 11 17:48:13.694894 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:48:13.694875    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-p4bx6\" (UniqueName: \"kubernetes.io/projected/4fbefcb1-b54a-4367-9026-e6a42536f21a-kube-api-access-p4bx6\") pod \"rate-test-model-2fac54-kserve-85f5f8cb6c-kvmch\" (UID: \"4fbefcb1-b54a-4367-9026-e6a42536f21a\") " pod="ai-tenant-e2e-shared-a-w3-2fac54/rate-test-model-2fac54-kserve-85f5f8cb6c-kvmch"
Sep 11 17:48:13.817884 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:48:13.817791    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-2fac54/rate-test-model-2fac54-kserve-85f5f8cb6c-kvmch"
Sep 11 17:48:14.099933 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:48:14.099846    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-2fac54/rate-test-model-2fac54-kserve-85f5f8cb6c-kvmch"]
Sep 11 17:48:14.104776 ip-10-0-141-139 kubenswrapper[2577]: W0911 17:48:14.104731    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod4fbefcb1_b54a_4367_9026_e6a42536f21a.slice/crio-ea08d77dec7a93ba7e08b26023fb7e48f9bce87dd3e901c9aea4ec3e1d85ab21 WatchSource:0}: Error finding container ea08d77dec7a93ba7e08b26023fb7e48f9bce87dd3e901c9aea4ec3e1d85ab21: Status 404 returned error can't find the container with id ea08d77dec7a93ba7e08b26023fb7e48f9bce87dd3e901c9aea4ec3e1d85ab21
Sep 11 17:48:14.716927 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:48:14.716887    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-2fac54/rate-test-model-2fac54-kserve-85f5f8cb6c-kvmch" event={"ID":"4fbefcb1-b54a-4367-9026-e6a42536f21a","Type":"ContainerStarted","Data":"055b4c3a14f6110a25a15ef6fa7e325923ffda54acd032b41ef5558bcc1745b5"}
Sep 11 17:48:14.716927 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:48:14.716930    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-2fac54/rate-test-model-2fac54-kserve-85f5f8cb6c-kvmch" event={"ID":"4fbefcb1-b54a-4367-9026-e6a42536f21a","Type":"ContainerStarted","Data":"ea08d77dec7a93ba7e08b26023fb7e48f9bce87dd3e901c9aea4ec3e1d85ab21"}
Sep 11 17:48:16.809265 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:48:16.809211    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w3-2fac54/rate-test-model-2fac54-kserve-85f5f8cb6c-kvmch" podStartSLOduration=3.80919046 podStartE2EDuration="3.80919046s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 17:48:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 17:48:14.753052561 +0000 UTC m=+1772.326272683" watchObservedRunningTime="2026-09-11 17:48:16.80919046 +0000 UTC m=+1774.382410571"
Sep 11 17:48:16.810733 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:48:16.810713    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-nkmkm"]
Sep 11 17:48:23.818410 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:48:23.818364    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w3-2fac54/rate-test-model-2fac54-kserve-85f5f8cb6c-kvmch"
Sep 11 17:48:23.818410 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:48:23.818421    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w3-2fac54/rate-test-model-2fac54-kserve-85f5f8cb6c-kvmch"
Sep 11 17:48:23.831387 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:48:23.831327    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w3-2fac54/rate-test-model-2fac54-kserve-85f5f8cb6c-kvmch"
Sep 11 17:48:24.780743 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:48:24.780709    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w3-2fac54/rate-test-model-2fac54-kserve-85f5f8cb6c-kvmch"
Sep 11 17:49:18.607113 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:49:18.607078    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w3-6fc9e6/rate-test-model-6fc9e6-kserve-9fc6ff-r2bwl"]
Sep 11 17:49:18.610648 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:49:18.610630    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-6fc9e6/rate-test-model-6fc9e6-kserve-9fc6ff-r2bwl"
Sep 11 17:49:18.613158 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:49:18.613140    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-6fc9e6\"/\"kube-root-ca.crt\""
Sep 11 17:49:18.614091 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:49:18.614068    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-6fc9e6\"/\"default-dockercfg-kgvr7\""
Sep 11 17:49:18.614091 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:49:18.614084    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-6fc9e6\"/\"rate-test-model-6fc9e6-kserve-self-signed-certs\""
Sep 11 17:49:18.614262 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:49:18.614127    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-6fc9e6\"/\"openshift-service-ca.crt\""
Sep 11 17:49:18.622342 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:49:18.622323    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-6fc9e6/rate-test-model-6fc9e6-kserve-9fc6ff-r2bwl"]
Sep 11 17:49:18.742263 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:49:18.742229    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c13822c9-0bb4-4617-9bff-8ee18f4b5f5b-tls-certs\") pod \"rate-test-model-6fc9e6-kserve-9fc6ff-r2bwl\" (UID: \"c13822c9-0bb4-4617-9bff-8ee18f4b5f5b\") " pod="ai-tenant-e2e-shared-b-w3-6fc9e6/rate-test-model-6fc9e6-kserve-9fc6ff-r2bwl"
Sep 11 17:49:18.742441 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:49:18.742268    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pzj6g\" (UniqueName: \"kubernetes.io/projected/c13822c9-0bb4-4617-9bff-8ee18f4b5f5b-kube-api-access-pzj6g\") pod \"rate-test-model-6fc9e6-kserve-9fc6ff-r2bwl\" (UID: \"c13822c9-0bb4-4617-9bff-8ee18f4b5f5b\") " pod="ai-tenant-e2e-shared-b-w3-6fc9e6/rate-test-model-6fc9e6-kserve-9fc6ff-r2bwl"
Sep 11 17:49:18.742441 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:49:18.742292    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c13822c9-0bb4-4617-9bff-8ee18f4b5f5b-dshm\") pod \"rate-test-model-6fc9e6-kserve-9fc6ff-r2bwl\" (UID: \"c13822c9-0bb4-4617-9bff-8ee18f4b5f5b\") " pod="ai-tenant-e2e-shared-b-w3-6fc9e6/rate-test-model-6fc9e6-kserve-9fc6ff-r2bwl"
Sep 11 17:49:18.742441 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:49:18.742360    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c13822c9-0bb4-4617-9bff-8ee18f4b5f5b-model-cache\") pod \"rate-test-model-6fc9e6-kserve-9fc6ff-r2bwl\" (UID: \"c13822c9-0bb4-4617-9bff-8ee18f4b5f5b\") " pod="ai-tenant-e2e-shared-b-w3-6fc9e6/rate-test-model-6fc9e6-kserve-9fc6ff-r2bwl"
Sep 11 17:49:18.742441 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:49:18.742402    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c13822c9-0bb4-4617-9bff-8ee18f4b5f5b-home\") pod \"rate-test-model-6fc9e6-kserve-9fc6ff-r2bwl\" (UID: \"c13822c9-0bb4-4617-9bff-8ee18f4b5f5b\") " pod="ai-tenant-e2e-shared-b-w3-6fc9e6/rate-test-model-6fc9e6-kserve-9fc6ff-r2bwl"
Sep 11 17:49:18.742604 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:49:18.742458    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c13822c9-0bb4-4617-9bff-8ee18f4b5f5b-tmp-dir\") pod \"rate-test-model-6fc9e6-kserve-9fc6ff-r2bwl\" (UID: \"c13822c9-0bb4-4617-9bff-8ee18f4b5f5b\") " pod="ai-tenant-e2e-shared-b-w3-6fc9e6/rate-test-model-6fc9e6-kserve-9fc6ff-r2bwl"
Sep 11 17:49:18.843832 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:49:18.843796    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c13822c9-0bb4-4617-9bff-8ee18f4b5f5b-home\") pod \"rate-test-model-6fc9e6-kserve-9fc6ff-r2bwl\" (UID: \"c13822c9-0bb4-4617-9bff-8ee18f4b5f5b\") " pod="ai-tenant-e2e-shared-b-w3-6fc9e6/rate-test-model-6fc9e6-kserve-9fc6ff-r2bwl"
Sep 11 17:49:18.844025 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:49:18.843866    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c13822c9-0bb4-4617-9bff-8ee18f4b5f5b-tmp-dir\") pod \"rate-test-model-6fc9e6-kserve-9fc6ff-r2bwl\" (UID: \"c13822c9-0bb4-4617-9bff-8ee18f4b5f5b\") " pod="ai-tenant-e2e-shared-b-w3-6fc9e6/rate-test-model-6fc9e6-kserve-9fc6ff-r2bwl"
Sep 11 17:49:18.844025 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:49:18.843941    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c13822c9-0bb4-4617-9bff-8ee18f4b5f5b-tls-certs\") pod \"rate-test-model-6fc9e6-kserve-9fc6ff-r2bwl\" (UID: \"c13822c9-0bb4-4617-9bff-8ee18f4b5f5b\") " pod="ai-tenant-e2e-shared-b-w3-6fc9e6/rate-test-model-6fc9e6-kserve-9fc6ff-r2bwl"
Sep 11 17:49:18.844025 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:49:18.843961    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pzj6g\" (UniqueName: \"kubernetes.io/projected/c13822c9-0bb4-4617-9bff-8ee18f4b5f5b-kube-api-access-pzj6g\") pod \"rate-test-model-6fc9e6-kserve-9fc6ff-r2bwl\" (UID: \"c13822c9-0bb4-4617-9bff-8ee18f4b5f5b\") " pod="ai-tenant-e2e-shared-b-w3-6fc9e6/rate-test-model-6fc9e6-kserve-9fc6ff-r2bwl"
Sep 11 17:49:18.844025 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:49:18.843979    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c13822c9-0bb4-4617-9bff-8ee18f4b5f5b-dshm\") pod \"rate-test-model-6fc9e6-kserve-9fc6ff-r2bwl\" (UID: \"c13822c9-0bb4-4617-9bff-8ee18f4b5f5b\") " pod="ai-tenant-e2e-shared-b-w3-6fc9e6/rate-test-model-6fc9e6-kserve-9fc6ff-r2bwl"
Sep 11 17:49:18.844025 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:49:18.844000    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c13822c9-0bb4-4617-9bff-8ee18f4b5f5b-model-cache\") pod \"rate-test-model-6fc9e6-kserve-9fc6ff-r2bwl\" (UID: \"c13822c9-0bb4-4617-9bff-8ee18f4b5f5b\") " pod="ai-tenant-e2e-shared-b-w3-6fc9e6/rate-test-model-6fc9e6-kserve-9fc6ff-r2bwl"
Sep 11 17:49:18.844381 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:49:18.844271    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c13822c9-0bb4-4617-9bff-8ee18f4b5f5b-tmp-dir\") pod \"rate-test-model-6fc9e6-kserve-9fc6ff-r2bwl\" (UID: \"c13822c9-0bb4-4617-9bff-8ee18f4b5f5b\") " pod="ai-tenant-e2e-shared-b-w3-6fc9e6/rate-test-model-6fc9e6-kserve-9fc6ff-r2bwl"
Sep 11 17:49:18.844381 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:49:18.844287    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c13822c9-0bb4-4617-9bff-8ee18f4b5f5b-home\") pod \"rate-test-model-6fc9e6-kserve-9fc6ff-r2bwl\" (UID: \"c13822c9-0bb4-4617-9bff-8ee18f4b5f5b\") " pod="ai-tenant-e2e-shared-b-w3-6fc9e6/rate-test-model-6fc9e6-kserve-9fc6ff-r2bwl"
Sep 11 17:49:18.844463 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:49:18.844426    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c13822c9-0bb4-4617-9bff-8ee18f4b5f5b-model-cache\") pod \"rate-test-model-6fc9e6-kserve-9fc6ff-r2bwl\" (UID: \"c13822c9-0bb4-4617-9bff-8ee18f4b5f5b\") " pod="ai-tenant-e2e-shared-b-w3-6fc9e6/rate-test-model-6fc9e6-kserve-9fc6ff-r2bwl"
Sep 11 17:49:18.846375 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:49:18.846354    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c13822c9-0bb4-4617-9bff-8ee18f4b5f5b-dshm\") pod \"rate-test-model-6fc9e6-kserve-9fc6ff-r2bwl\" (UID: \"c13822c9-0bb4-4617-9bff-8ee18f4b5f5b\") " pod="ai-tenant-e2e-shared-b-w3-6fc9e6/rate-test-model-6fc9e6-kserve-9fc6ff-r2bwl"
Sep 11 17:49:18.846551 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:49:18.846536    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c13822c9-0bb4-4617-9bff-8ee18f4b5f5b-tls-certs\") pod \"rate-test-model-6fc9e6-kserve-9fc6ff-r2bwl\" (UID: \"c13822c9-0bb4-4617-9bff-8ee18f4b5f5b\") " pod="ai-tenant-e2e-shared-b-w3-6fc9e6/rate-test-model-6fc9e6-kserve-9fc6ff-r2bwl"
Sep 11 17:49:18.851810 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:49:18.851786    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pzj6g\" (UniqueName: \"kubernetes.io/projected/c13822c9-0bb4-4617-9bff-8ee18f4b5f5b-kube-api-access-pzj6g\") pod \"rate-test-model-6fc9e6-kserve-9fc6ff-r2bwl\" (UID: \"c13822c9-0bb4-4617-9bff-8ee18f4b5f5b\") " pod="ai-tenant-e2e-shared-b-w3-6fc9e6/rate-test-model-6fc9e6-kserve-9fc6ff-r2bwl"
Sep 11 17:49:18.920666 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:49:18.920578    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-6fc9e6/rate-test-model-6fc9e6-kserve-9fc6ff-r2bwl"
Sep 11 17:49:19.201951 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:49:19.201920    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-6fc9e6/rate-test-model-6fc9e6-kserve-9fc6ff-r2bwl"]
Sep 11 17:49:19.204020 ip-10-0-141-139 kubenswrapper[2577]: W0911 17:49:19.203993    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc13822c9_0bb4_4617_9bff_8ee18f4b5f5b.slice/crio-f0a987cf175541821eed17b71c11331e806ce6131318586a66ba9223ed64d6b1 WatchSource:0}: Error finding container f0a987cf175541821eed17b71c11331e806ce6131318586a66ba9223ed64d6b1: Status 404 returned error can't find the container with id f0a987cf175541821eed17b71c11331e806ce6131318586a66ba9223ed64d6b1
Sep 11 17:49:20.064220 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:49:20.064178    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-6fc9e6/rate-test-model-6fc9e6-kserve-9fc6ff-r2bwl" event={"ID":"c13822c9-0bb4-4617-9bff-8ee18f4b5f5b","Type":"ContainerStarted","Data":"4d69c57091d297456c6bc44fda13893a78d9048f19c2572b3aab4693240d38cc"}
Sep 11 17:49:20.064622 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:49:20.064241    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-6fc9e6/rate-test-model-6fc9e6-kserve-9fc6ff-r2bwl" event={"ID":"c13822c9-0bb4-4617-9bff-8ee18f4b5f5b","Type":"ContainerStarted","Data":"f0a987cf175541821eed17b71c11331e806ce6131318586a66ba9223ed64d6b1"}
Sep 11 17:49:20.082779 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:49:20.082729    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w3-6fc9e6/rate-test-model-6fc9e6-kserve-9fc6ff-r2bwl" podStartSLOduration=2.082712638 podStartE2EDuration="2.082712638s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 17:49:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 17:49:20.080458861 +0000 UTC m=+1837.653678972" watchObservedRunningTime="2026-09-11 17:49:20.082712638 +0000 UTC m=+1837.655932749"
Sep 11 17:49:22.207167 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:49:22.207130    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-nkmkm"]
Sep 11 17:49:28.920719 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:49:28.920672    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w3-6fc9e6/rate-test-model-6fc9e6-kserve-9fc6ff-r2bwl"
Sep 11 17:49:28.927638 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:49:28.927613    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w3-6fc9e6/rate-test-model-6fc9e6-kserve-9fc6ff-r2bwl"
Sep 11 17:49:28.933904 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:49:28.933880    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w3-6fc9e6/rate-test-model-6fc9e6-kserve-9fc6ff-r2bwl"
Sep 11 17:49:29.123414 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:49:29.123385    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w3-6fc9e6/rate-test-model-6fc9e6-kserve-9fc6ff-r2bwl"
Sep 11 17:49:42.299265 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:49:42.299230    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-nkmkm"]
Sep 11 17:49:43.317746 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:49:43.317713    2577 scope.go:117] "RemoveContainer" containerID="9edcaf85402e7dc80930be6e1780903689131bee3fdd52b323b0d7d7da488eef"
Sep 11 17:49:52.193657 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:49:52.193620    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-nkmkm"]
Sep 11 17:50:04.790604 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:50:04.790521    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-nkmkm"]
Sep 11 17:50:08.432893 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:50:08.432856    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-2fac54/rate-test-model-2fac54-kserve-85f5f8cb6c-kvmch"]
Sep 11 17:50:09.386488 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:50:09.386441    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-6fc9e6/rate-test-model-6fc9e6-kserve-9fc6ff-r2bwl"]
Sep 11 17:50:16.793525 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:50:16.793470    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-nkmkm"]
Sep 11 17:50:23.455291 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:50:23.455250    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w3-2fac54/rate-test-model-2fac54-kserve-85f5f8cb6c-kvmch" podUID="4fbefcb1-b54a-4367-9026-e6a42536f21a" containerName="main" containerID="cri-o://055b4c3a14f6110a25a15ef6fa7e325923ffda54acd032b41ef5558bcc1745b5" gracePeriod=45
Sep 11 17:50:23.708990 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:50:23.708914    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-2fac54/rate-test-model-2fac54-kserve-85f5f8cb6c-kvmch"
Sep 11 17:50:23.804084 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:50:23.804040    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4fbefcb1-b54a-4367-9026-e6a42536f21a-home\") pod \"4fbefcb1-b54a-4367-9026-e6a42536f21a\" (UID: \"4fbefcb1-b54a-4367-9026-e6a42536f21a\") "
Sep 11 17:50:23.804305 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:50:23.804103    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-p4bx6\" (UniqueName: \"kubernetes.io/projected/4fbefcb1-b54a-4367-9026-e6a42536f21a-kube-api-access-p4bx6\") pod \"4fbefcb1-b54a-4367-9026-e6a42536f21a\" (UID: \"4fbefcb1-b54a-4367-9026-e6a42536f21a\") "
Sep 11 17:50:23.804305 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:50:23.804136    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4fbefcb1-b54a-4367-9026-e6a42536f21a-model-cache\") pod \"4fbefcb1-b54a-4367-9026-e6a42536f21a\" (UID: \"4fbefcb1-b54a-4367-9026-e6a42536f21a\") "
Sep 11 17:50:23.804305 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:50:23.804199    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4fbefcb1-b54a-4367-9026-e6a42536f21a-tmp-dir\") pod \"4fbefcb1-b54a-4367-9026-e6a42536f21a\" (UID: \"4fbefcb1-b54a-4367-9026-e6a42536f21a\") "
Sep 11 17:50:23.804305 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:50:23.804248    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4fbefcb1-b54a-4367-9026-e6a42536f21a-dshm\") pod \"4fbefcb1-b54a-4367-9026-e6a42536f21a\" (UID: \"4fbefcb1-b54a-4367-9026-e6a42536f21a\") "
Sep 11 17:50:23.804558 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:50:23.804339    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4fbefcb1-b54a-4367-9026-e6a42536f21a-tls-certs\") pod \"4fbefcb1-b54a-4367-9026-e6a42536f21a\" (UID: \"4fbefcb1-b54a-4367-9026-e6a42536f21a\") "
Sep 11 17:50:23.804558 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:50:23.804362    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4fbefcb1-b54a-4367-9026-e6a42536f21a-home" (OuterVolumeSpecName: "home") pod "4fbefcb1-b54a-4367-9026-e6a42536f21a" (UID: "4fbefcb1-b54a-4367-9026-e6a42536f21a"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 17:50:23.804558 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:50:23.804443    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4fbefcb1-b54a-4367-9026-e6a42536f21a-model-cache" (OuterVolumeSpecName: "model-cache") pod "4fbefcb1-b54a-4367-9026-e6a42536f21a" (UID: "4fbefcb1-b54a-4367-9026-e6a42536f21a"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 17:50:23.804558 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:50:23.804537    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4fbefcb1-b54a-4367-9026-e6a42536f21a-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "4fbefcb1-b54a-4367-9026-e6a42536f21a" (UID: "4fbefcb1-b54a-4367-9026-e6a42536f21a"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 17:50:23.804727 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:50:23.804701    2577 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4fbefcb1-b54a-4367-9026-e6a42536f21a-tmp-dir\") on node \"ip-10-0-141-139.ec2.internal\" DevicePath \"\""
Sep 11 17:50:23.804727 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:50:23.804715    2577 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4fbefcb1-b54a-4367-9026-e6a42536f21a-home\") on node \"ip-10-0-141-139.ec2.internal\" DevicePath \"\""
Sep 11 17:50:23.804844 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:50:23.804727    2577 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4fbefcb1-b54a-4367-9026-e6a42536f21a-model-cache\") on node \"ip-10-0-141-139.ec2.internal\" DevicePath \"\""
Sep 11 17:50:23.806551 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:50:23.806524    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4fbefcb1-b54a-4367-9026-e6a42536f21a-dshm" (OuterVolumeSpecName: "dshm") pod "4fbefcb1-b54a-4367-9026-e6a42536f21a" (UID: "4fbefcb1-b54a-4367-9026-e6a42536f21a"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 17:50:23.806660 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:50:23.806572    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4fbefcb1-b54a-4367-9026-e6a42536f21a-kube-api-access-p4bx6" (OuterVolumeSpecName: "kube-api-access-p4bx6") pod "4fbefcb1-b54a-4367-9026-e6a42536f21a" (UID: "4fbefcb1-b54a-4367-9026-e6a42536f21a"). InnerVolumeSpecName "kube-api-access-p4bx6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 17:50:23.806660 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:50:23.806585    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4fbefcb1-b54a-4367-9026-e6a42536f21a-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "4fbefcb1-b54a-4367-9026-e6a42536f21a" (UID: "4fbefcb1-b54a-4367-9026-e6a42536f21a"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 17:50:23.905885 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:50:23.905836    2577 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4fbefcb1-b54a-4367-9026-e6a42536f21a-dshm\") on node \"ip-10-0-141-139.ec2.internal\" DevicePath \"\""
Sep 11 17:50:23.905885 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:50:23.905876    2577 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4fbefcb1-b54a-4367-9026-e6a42536f21a-tls-certs\") on node \"ip-10-0-141-139.ec2.internal\" DevicePath \"\""
Sep 11 17:50:23.905885 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:50:23.905891    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p4bx6\" (UniqueName: \"kubernetes.io/projected/4fbefcb1-b54a-4367-9026-e6a42536f21a-kube-api-access-p4bx6\") on node \"ip-10-0-141-139.ec2.internal\" DevicePath \"\""
Sep 11 17:50:24.406383 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:50:24.406318    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w3-6fc9e6/rate-test-model-6fc9e6-kserve-9fc6ff-r2bwl" podUID="c13822c9-0bb4-4617-9bff-8ee18f4b5f5b" containerName="main" containerID="cri-o://4d69c57091d297456c6bc44fda13893a78d9048f19c2572b3aab4693240d38cc" gracePeriod=45
Sep 11 17:50:24.414214 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:50:24.414183    2577 generic.go:356] "Generic (PLEG): container finished" podID="4fbefcb1-b54a-4367-9026-e6a42536f21a" containerID="055b4c3a14f6110a25a15ef6fa7e325923ffda54acd032b41ef5558bcc1745b5" exitCode=0
Sep 11 17:50:24.414335 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:50:24.414231    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-2fac54/rate-test-model-2fac54-kserve-85f5f8cb6c-kvmch"
Sep 11 17:50:24.414335 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:50:24.414259    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-2fac54/rate-test-model-2fac54-kserve-85f5f8cb6c-kvmch" event={"ID":"4fbefcb1-b54a-4367-9026-e6a42536f21a","Type":"ContainerDied","Data":"055b4c3a14f6110a25a15ef6fa7e325923ffda54acd032b41ef5558bcc1745b5"}
Sep 11 17:50:24.414335 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:50:24.414304    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-2fac54/rate-test-model-2fac54-kserve-85f5f8cb6c-kvmch" event={"ID":"4fbefcb1-b54a-4367-9026-e6a42536f21a","Type":"ContainerDied","Data":"ea08d77dec7a93ba7e08b26023fb7e48f9bce87dd3e901c9aea4ec3e1d85ab21"}
Sep 11 17:50:24.414335 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:50:24.414325    2577 scope.go:117] "RemoveContainer" containerID="055b4c3a14f6110a25a15ef6fa7e325923ffda54acd032b41ef5558bcc1745b5"
Sep 11 17:50:24.489201 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:50:24.489164    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-2fac54/rate-test-model-2fac54-kserve-85f5f8cb6c-kvmch"]
Sep 11 17:50:24.493233 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:50:24.493199    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w3-2fac54/rate-test-model-2fac54-kserve-85f5f8cb6c-kvmch"]
Sep 11 17:50:24.518273 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:50:24.518135    2577 scope.go:117] "RemoveContainer" containerID="055b4c3a14f6110a25a15ef6fa7e325923ffda54acd032b41ef5558bcc1745b5"
Sep 11 17:50:24.518590 ip-10-0-141-139 kubenswrapper[2577]: E0911 17:50:24.518567    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"055b4c3a14f6110a25a15ef6fa7e325923ffda54acd032b41ef5558bcc1745b5\": container with ID starting with 055b4c3a14f6110a25a15ef6fa7e325923ffda54acd032b41ef5558bcc1745b5 not found: ID does not exist" containerID="055b4c3a14f6110a25a15ef6fa7e325923ffda54acd032b41ef5558bcc1745b5"
Sep 11 17:50:24.518653 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:50:24.518598    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"055b4c3a14f6110a25a15ef6fa7e325923ffda54acd032b41ef5558bcc1745b5"} err="failed to get container status \"055b4c3a14f6110a25a15ef6fa7e325923ffda54acd032b41ef5558bcc1745b5\": rpc error: code = NotFound desc = could not find container \"055b4c3a14f6110a25a15ef6fa7e325923ffda54acd032b41ef5558bcc1745b5\": container with ID starting with 055b4c3a14f6110a25a15ef6fa7e325923ffda54acd032b41ef5558bcc1745b5 not found: ID does not exist"
Sep 11 17:50:24.667592 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:50:24.667523    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-6fc9e6/rate-test-model-6fc9e6-kserve-9fc6ff-r2bwl"
Sep 11 17:50:24.813531 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:50:24.813461    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c13822c9-0bb4-4617-9bff-8ee18f4b5f5b-home\") pod \"c13822c9-0bb4-4617-9bff-8ee18f4b5f5b\" (UID: \"c13822c9-0bb4-4617-9bff-8ee18f4b5f5b\") "
Sep 11 17:50:24.813698 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:50:24.813549    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c13822c9-0bb4-4617-9bff-8ee18f4b5f5b-model-cache\") pod \"c13822c9-0bb4-4617-9bff-8ee18f4b5f5b\" (UID: \"c13822c9-0bb4-4617-9bff-8ee18f4b5f5b\") "
Sep 11 17:50:24.813698 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:50:24.813593    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pzj6g\" (UniqueName: \"kubernetes.io/projected/c13822c9-0bb4-4617-9bff-8ee18f4b5f5b-kube-api-access-pzj6g\") pod \"c13822c9-0bb4-4617-9bff-8ee18f4b5f5b\" (UID: \"c13822c9-0bb4-4617-9bff-8ee18f4b5f5b\") "
Sep 11 17:50:24.813698 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:50:24.813619    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c13822c9-0bb4-4617-9bff-8ee18f4b5f5b-dshm\") pod \"c13822c9-0bb4-4617-9bff-8ee18f4b5f5b\" (UID: \"c13822c9-0bb4-4617-9bff-8ee18f4b5f5b\") "
Sep 11 17:50:24.813698 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:50:24.813660    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c13822c9-0bb4-4617-9bff-8ee18f4b5f5b-tmp-dir\") pod \"c13822c9-0bb4-4617-9bff-8ee18f4b5f5b\" (UID: \"c13822c9-0bb4-4617-9bff-8ee18f4b5f5b\") "
Sep 11 17:50:24.813935 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:50:24.813705    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c13822c9-0bb4-4617-9bff-8ee18f4b5f5b-tls-certs\") pod \"c13822c9-0bb4-4617-9bff-8ee18f4b5f5b\" (UID: \"c13822c9-0bb4-4617-9bff-8ee18f4b5f5b\") "
Sep 11 17:50:24.813935 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:50:24.813809    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c13822c9-0bb4-4617-9bff-8ee18f4b5f5b-home" (OuterVolumeSpecName: "home") pod "c13822c9-0bb4-4617-9bff-8ee18f4b5f5b" (UID: "c13822c9-0bb4-4617-9bff-8ee18f4b5f5b"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 17:50:24.813935 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:50:24.813825    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c13822c9-0bb4-4617-9bff-8ee18f4b5f5b-model-cache" (OuterVolumeSpecName: "model-cache") pod "c13822c9-0bb4-4617-9bff-8ee18f4b5f5b" (UID: "c13822c9-0bb4-4617-9bff-8ee18f4b5f5b"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 17:50:24.813935 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:50:24.813926    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c13822c9-0bb4-4617-9bff-8ee18f4b5f5b-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "c13822c9-0bb4-4617-9bff-8ee18f4b5f5b" (UID: "c13822c9-0bb4-4617-9bff-8ee18f4b5f5b"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 17:50:24.814131 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:50:24.814050    2577 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c13822c9-0bb4-4617-9bff-8ee18f4b5f5b-home\") on node \"ip-10-0-141-139.ec2.internal\" DevicePath \"\""
Sep 11 17:50:24.814131 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:50:24.814067    2577 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c13822c9-0bb4-4617-9bff-8ee18f4b5f5b-model-cache\") on node \"ip-10-0-141-139.ec2.internal\" DevicePath \"\""
Sep 11 17:50:24.814131 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:50:24.814082    2577 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c13822c9-0bb4-4617-9bff-8ee18f4b5f5b-tmp-dir\") on node \"ip-10-0-141-139.ec2.internal\" DevicePath \"\""
Sep 11 17:50:24.815951 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:50:24.815928    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c13822c9-0bb4-4617-9bff-8ee18f4b5f5b-dshm" (OuterVolumeSpecName: "dshm") pod "c13822c9-0bb4-4617-9bff-8ee18f4b5f5b" (UID: "c13822c9-0bb4-4617-9bff-8ee18f4b5f5b"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 17:50:24.816043 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:50:24.815952    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c13822c9-0bb4-4617-9bff-8ee18f4b5f5b-kube-api-access-pzj6g" (OuterVolumeSpecName: "kube-api-access-pzj6g") pod "c13822c9-0bb4-4617-9bff-8ee18f4b5f5b" (UID: "c13822c9-0bb4-4617-9bff-8ee18f4b5f5b"). InnerVolumeSpecName "kube-api-access-pzj6g". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 17:50:24.816043 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:50:24.815982    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c13822c9-0bb4-4617-9bff-8ee18f4b5f5b-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "c13822c9-0bb4-4617-9bff-8ee18f4b5f5b" (UID: "c13822c9-0bb4-4617-9bff-8ee18f4b5f5b"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 17:50:24.914785 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:50:24.914743    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pzj6g\" (UniqueName: \"kubernetes.io/projected/c13822c9-0bb4-4617-9bff-8ee18f4b5f5b-kube-api-access-pzj6g\") on node \"ip-10-0-141-139.ec2.internal\" DevicePath \"\""
Sep 11 17:50:24.914785 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:50:24.914777    2577 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c13822c9-0bb4-4617-9bff-8ee18f4b5f5b-dshm\") on node \"ip-10-0-141-139.ec2.internal\" DevicePath \"\""
Sep 11 17:50:24.914785 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:50:24.914789    2577 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c13822c9-0bb4-4617-9bff-8ee18f4b5f5b-tls-certs\") on node \"ip-10-0-141-139.ec2.internal\" DevicePath \"\""
Sep 11 17:50:24.927163 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:50:24.927077    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4fbefcb1-b54a-4367-9026-e6a42536f21a" path="/var/lib/kubelet/pods/4fbefcb1-b54a-4367-9026-e6a42536f21a/volumes"
Sep 11 17:50:25.421873 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:50:25.421831    2577 generic.go:356] "Generic (PLEG): container finished" podID="c13822c9-0bb4-4617-9bff-8ee18f4b5f5b" containerID="4d69c57091d297456c6bc44fda13893a78d9048f19c2572b3aab4693240d38cc" exitCode=0
Sep 11 17:50:25.422069 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:50:25.421878    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-6fc9e6/rate-test-model-6fc9e6-kserve-9fc6ff-r2bwl"
Sep 11 17:50:25.422069 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:50:25.421893    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-6fc9e6/rate-test-model-6fc9e6-kserve-9fc6ff-r2bwl" event={"ID":"c13822c9-0bb4-4617-9bff-8ee18f4b5f5b","Type":"ContainerDied","Data":"4d69c57091d297456c6bc44fda13893a78d9048f19c2572b3aab4693240d38cc"}
Sep 11 17:50:25.422069 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:50:25.421925    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-6fc9e6/rate-test-model-6fc9e6-kserve-9fc6ff-r2bwl" event={"ID":"c13822c9-0bb4-4617-9bff-8ee18f4b5f5b","Type":"ContainerDied","Data":"f0a987cf175541821eed17b71c11331e806ce6131318586a66ba9223ed64d6b1"}
Sep 11 17:50:25.428529 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:50:25.421941    2577 scope.go:117] "RemoveContainer" containerID="4d69c57091d297456c6bc44fda13893a78d9048f19c2572b3aab4693240d38cc"
Sep 11 17:50:25.437762 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:50:25.437742    2577 scope.go:117] "RemoveContainer" containerID="4d69c57091d297456c6bc44fda13893a78d9048f19c2572b3aab4693240d38cc"
Sep 11 17:50:25.438092 ip-10-0-141-139 kubenswrapper[2577]: E0911 17:50:25.438071    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4d69c57091d297456c6bc44fda13893a78d9048f19c2572b3aab4693240d38cc\": container with ID starting with 4d69c57091d297456c6bc44fda13893a78d9048f19c2572b3aab4693240d38cc not found: ID does not exist" containerID="4d69c57091d297456c6bc44fda13893a78d9048f19c2572b3aab4693240d38cc"
Sep 11 17:50:25.438183 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:50:25.438102    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4d69c57091d297456c6bc44fda13893a78d9048f19c2572b3aab4693240d38cc"} err="failed to get container status \"4d69c57091d297456c6bc44fda13893a78d9048f19c2572b3aab4693240d38cc\": rpc error: code = NotFound desc = could not find container \"4d69c57091d297456c6bc44fda13893a78d9048f19c2572b3aab4693240d38cc\": container with ID starting with 4d69c57091d297456c6bc44fda13893a78d9048f19c2572b3aab4693240d38cc not found: ID does not exist"
Sep 11 17:50:25.444139 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:50:25.444108    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-6fc9e6/rate-test-model-6fc9e6-kserve-9fc6ff-r2bwl"]
Sep 11 17:50:25.447852 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:50:25.447829    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w3-6fc9e6/rate-test-model-6fc9e6-kserve-9fc6ff-r2bwl"]
Sep 11 17:50:25.606307 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:50:25.606270    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-nkmkm"]
Sep 11 17:50:26.926439 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:50:26.926400    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c13822c9-0bb4-4617-9bff-8ee18f4b5f5b" path="/var/lib/kubelet/pods/c13822c9-0bb4-4617-9bff-8ee18f4b5f5b/volumes"
Sep 11 17:50:43.339546 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:50:43.339485    2577 scope.go:117] "RemoveContainer" containerID="f651a270e33fbbb83ff86fb6ce01def91a28068d2411f4e44d93b42124c78667"
Sep 11 17:50:43.348786 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:50:43.348762    2577 scope.go:117] "RemoveContainer" containerID="4760c3a8c4189836dba326d4f883217ac662b335e6bb5aeb61a540ccbfc04927"
Sep 11 17:50:43.357779 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:50:43.357755    2577 scope.go:117] "RemoveContainer" containerID="cbe3932f264bbbdc16ad30a5c6cefb2da8ce18580e15ef1e7a221ef62427a0c5"
Sep 11 17:50:46.135089 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:50:46.135056    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-10bbea-c56d6578-ltjgg"]
Sep 11 17:50:46.135535 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:50:46.135456    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c13822c9-0bb4-4617-9bff-8ee18f4b5f5b" containerName="main"
Sep 11 17:50:46.135535 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:50:46.135466    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="c13822c9-0bb4-4617-9bff-8ee18f4b5f5b" containerName="main"
Sep 11 17:50:46.135535 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:50:46.135490    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4fbefcb1-b54a-4367-9026-e6a42536f21a" containerName="main"
Sep 11 17:50:46.135535 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:50:46.135495    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="4fbefcb1-b54a-4367-9026-e6a42536f21a" containerName="main"
Sep 11 17:50:46.135783 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:50:46.135563    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="c13822c9-0bb4-4617-9bff-8ee18f4b5f5b" containerName="main"
Sep 11 17:50:46.135783 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:50:46.135576    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="4fbefcb1-b54a-4367-9026-e6a42536f21a" containerName="main"
Sep 11 17:50:46.138794 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:50:46.138769    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-10bbea-c56d6578-ltjgg"
Sep 11 17:50:46.141352 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:50:46.141329    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-10bbea\""
Sep 11 17:50:46.141440 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:50:46.141329    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-10bbea-dockercfg-4ftsm\""
Sep 11 17:50:46.149127 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:50:46.149105    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-10bbea-c56d6578-ltjgg"]
Sep 11 17:50:46.153740 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:50:46.153716    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-10bbea-5c9b98f589-bxj2n"]
Sep 11 17:50:46.161832 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:50:46.161813    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-10bbea-5c9b98f589-bxj2n"
Sep 11 17:50:46.170258 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:50:46.170232    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-10bbea-5c9b98f589-bxj2n"]
Sep 11 17:50:46.213478 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:50:46.213444    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5ab31ffa-01d9-4749-a409-e2d17dcae79c-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-10bbea-c56d6578-ltjgg\" (UID: \"5ab31ffa-01d9-4749-a409-e2d17dcae79c\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-10bbea-c56d6578-ltjgg"
Sep 11 17:50:46.213478 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:50:46.213483    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7hwcr\" (UniqueName: \"kubernetes.io/projected/5ab31ffa-01d9-4749-a409-e2d17dcae79c-kube-api-access-7hwcr\") pod \"payload-pre-processing-e2e-ipp-b-10bbea-c56d6578-ltjgg\" (UID: \"5ab31ffa-01d9-4749-a409-e2d17dcae79c\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-10bbea-c56d6578-ltjgg"
Sep 11 17:50:46.213746 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:50:46.213568    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7x68b\" (UniqueName: \"kubernetes.io/projected/948e1e6f-0b40-4057-b996-590541520ac1-kube-api-access-7x68b\") pod \"payload-processing-e2e-ipp-b-10bbea-5c9b98f589-bxj2n\" (UID: \"948e1e6f-0b40-4057-b996-590541520ac1\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-10bbea-5c9b98f589-bxj2n"
Sep 11 17:50:46.213746 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:50:46.213611    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/948e1e6f-0b40-4057-b996-590541520ac1-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-10bbea-5c9b98f589-bxj2n\" (UID: \"948e1e6f-0b40-4057-b996-590541520ac1\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-10bbea-5c9b98f589-bxj2n"
Sep 11 17:50:46.314386 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:50:46.314350    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7x68b\" (UniqueName: \"kubernetes.io/projected/948e1e6f-0b40-4057-b996-590541520ac1-kube-api-access-7x68b\") pod \"payload-processing-e2e-ipp-b-10bbea-5c9b98f589-bxj2n\" (UID: \"948e1e6f-0b40-4057-b996-590541520ac1\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-10bbea-5c9b98f589-bxj2n"
Sep 11 17:50:46.314386 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:50:46.314391    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/948e1e6f-0b40-4057-b996-590541520ac1-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-10bbea-5c9b98f589-bxj2n\" (UID: \"948e1e6f-0b40-4057-b996-590541520ac1\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-10bbea-5c9b98f589-bxj2n"
Sep 11 17:50:46.314666 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:50:46.314467    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5ab31ffa-01d9-4749-a409-e2d17dcae79c-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-10bbea-c56d6578-ltjgg\" (UID: \"5ab31ffa-01d9-4749-a409-e2d17dcae79c\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-10bbea-c56d6578-ltjgg"
Sep 11 17:50:46.314666 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:50:46.314485    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7hwcr\" (UniqueName: \"kubernetes.io/projected/5ab31ffa-01d9-4749-a409-e2d17dcae79c-kube-api-access-7hwcr\") pod \"payload-pre-processing-e2e-ipp-b-10bbea-c56d6578-ltjgg\" (UID: \"5ab31ffa-01d9-4749-a409-e2d17dcae79c\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-10bbea-c56d6578-ltjgg"
Sep 11 17:50:46.315404 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:50:46.315379    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/948e1e6f-0b40-4057-b996-590541520ac1-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-10bbea-5c9b98f589-bxj2n\" (UID: \"948e1e6f-0b40-4057-b996-590541520ac1\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-10bbea-5c9b98f589-bxj2n"
Sep 11 17:50:46.315449 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:50:46.315379    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5ab31ffa-01d9-4749-a409-e2d17dcae79c-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-10bbea-c56d6578-ltjgg\" (UID: \"5ab31ffa-01d9-4749-a409-e2d17dcae79c\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-10bbea-c56d6578-ltjgg"
Sep 11 17:50:46.322807 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:50:46.322778    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7x68b\" (UniqueName: \"kubernetes.io/projected/948e1e6f-0b40-4057-b996-590541520ac1-kube-api-access-7x68b\") pod \"payload-processing-e2e-ipp-b-10bbea-5c9b98f589-bxj2n\" (UID: \"948e1e6f-0b40-4057-b996-590541520ac1\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-10bbea-5c9b98f589-bxj2n"
Sep 11 17:50:46.322909 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:50:46.322834    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7hwcr\" (UniqueName: \"kubernetes.io/projected/5ab31ffa-01d9-4749-a409-e2d17dcae79c-kube-api-access-7hwcr\") pod \"payload-pre-processing-e2e-ipp-b-10bbea-c56d6578-ltjgg\" (UID: \"5ab31ffa-01d9-4749-a409-e2d17dcae79c\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-10bbea-c56d6578-ltjgg"
Sep 11 17:50:46.450728 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:50:46.450630    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-10bbea-c56d6578-ltjgg"
Sep 11 17:50:46.474027 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:50:46.473979    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-10bbea-5c9b98f589-bxj2n"
Sep 11 17:50:46.728924 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:50:46.728892    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-10bbea-c56d6578-ltjgg"]
Sep 11 17:50:46.730933 ip-10-0-141-139 kubenswrapper[2577]: W0911 17:50:46.730905    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5ab31ffa_01d9_4749_a409_e2d17dcae79c.slice/crio-9e7da8072b03d1eab8ac7d6dbf627e0f95c07c3b7158b2369d642cf6edb9c990 WatchSource:0}: Error finding container 9e7da8072b03d1eab8ac7d6dbf627e0f95c07c3b7158b2369d642cf6edb9c990: Status 404 returned error can't find the container with id 9e7da8072b03d1eab8ac7d6dbf627e0f95c07c3b7158b2369d642cf6edb9c990
Sep 11 17:50:46.759014 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:50:46.758887    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-10bbea-5c9b98f589-bxj2n"]
Sep 11 17:50:46.766845 ip-10-0-141-139 kubenswrapper[2577]: W0911 17:50:46.766814    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod948e1e6f_0b40_4057_b996_590541520ac1.slice/crio-7c334f09e58613bdca905a96fd58680ff5b2ac8a625044397e6570cf53e3305a WatchSource:0}: Error finding container 7c334f09e58613bdca905a96fd58680ff5b2ac8a625044397e6570cf53e3305a: Status 404 returned error can't find the container with id 7c334f09e58613bdca905a96fd58680ff5b2ac8a625044397e6570cf53e3305a
Sep 11 17:50:47.542413 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:50:47.542373    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-10bbea-5c9b98f589-bxj2n" event={"ID":"948e1e6f-0b40-4057-b996-590541520ac1","Type":"ContainerStarted","Data":"d05ced7f6b59271673d1dd21bb4196e27fd21622611b9aa02c0f57c91d06c7d7"}
Sep 11 17:50:47.542914 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:50:47.542448    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-10bbea-5c9b98f589-bxj2n"
Sep 11 17:50:47.542914 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:50:47.542472    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-10bbea-5c9b98f589-bxj2n" event={"ID":"948e1e6f-0b40-4057-b996-590541520ac1","Type":"ContainerStarted","Data":"7c334f09e58613bdca905a96fd58680ff5b2ac8a625044397e6570cf53e3305a"}
Sep 11 17:50:47.544149 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:50:47.544124    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-10bbea-c56d6578-ltjgg" event={"ID":"5ab31ffa-01d9-4749-a409-e2d17dcae79c","Type":"ContainerStarted","Data":"ec9e21ab4aa04779bfee74c2d99dd6bc508233b266c3e125a1ad0a798d27d61f"}
Sep 11 17:50:47.544295 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:50:47.544169    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-10bbea-c56d6578-ltjgg" event={"ID":"5ab31ffa-01d9-4749-a409-e2d17dcae79c","Type":"ContainerStarted","Data":"9e7da8072b03d1eab8ac7d6dbf627e0f95c07c3b7158b2369d642cf6edb9c990"}
Sep 11 17:50:47.544295 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:50:47.544202    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-10bbea-c56d6578-ltjgg"
Sep 11 17:50:47.559609 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:50:47.559547    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-10bbea-5c9b98f589-bxj2n" podStartSLOduration=1.559529516 podStartE2EDuration="1.559529516s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 17:50:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 17:50:47.558635218 +0000 UTC m=+1925.131855332" watchObservedRunningTime="2026-09-11 17:50:47.559529516 +0000 UTC m=+1925.132749629"
Sep 11 17:50:47.576987 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:50:47.576924    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-10bbea-c56d6578-ltjgg" podStartSLOduration=1.576901145 podStartE2EDuration="1.576901145s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 17:50:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 17:50:47.573756546 +0000 UTC m=+1925.146976669" watchObservedRunningTime="2026-09-11 17:50:47.576901145 +0000 UTC m=+1925.150121258"
Sep 11 17:50:50.400068 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:50:50.400017    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-nkmkm"]
Sep 11 17:50:58.552440 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:50:58.552401    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-10bbea-c56d6578-ltjgg"
Sep 11 17:50:58.552870 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:50:58.552473    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-10bbea-5c9b98f589-bxj2n"
Sep 11 17:51:13.004490 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:51:13.004447    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-b-10bbea-5c9b98f589-bxj2n_948e1e6f-0b40-4057-b996-590541520ac1/payload-processing/0.log"
Sep 11 17:51:14.294545 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:51:14.294487    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-ipp-a-027edd/ipp-route-027edd-kserve-7cd6f577c9-qqtlz"]
Sep 11 17:51:14.298258 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:51:14.298233    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-027edd/ipp-route-027edd-kserve-7cd6f577c9-qqtlz"
Sep 11 17:51:14.300969 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:51:14.300942    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-ipp-a-027edd\"/\"openshift-service-ca.crt\""
Sep 11 17:51:14.302119 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:51:14.302093    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-ipp-a-027edd\"/\"kube-root-ca.crt\""
Sep 11 17:51:14.302545 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:51:14.302526    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-ipp-a-027edd\"/\"default-dockercfg-nfk45\""
Sep 11 17:51:14.302652 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:51:14.302573    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-ipp-a-027edd\"/\"ipp-route-027edd-kserve-self-signed-certs\""
Sep 11 17:51:14.308090 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:51:14.308061    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-ipp-a-027edd/ipp-route-027edd-kserve-7cd6f577c9-qqtlz"]
Sep 11 17:51:14.400098 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:51:14.400054    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/654d7a14-7a93-44d0-903d-8af82e7864f5-home\") pod \"ipp-route-027edd-kserve-7cd6f577c9-qqtlz\" (UID: \"654d7a14-7a93-44d0-903d-8af82e7864f5\") " pod="ai-tenant-e2e-ipp-a-027edd/ipp-route-027edd-kserve-7cd6f577c9-qqtlz"
Sep 11 17:51:14.400293 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:51:14.400116    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/654d7a14-7a93-44d0-903d-8af82e7864f5-dshm\") pod \"ipp-route-027edd-kserve-7cd6f577c9-qqtlz\" (UID: \"654d7a14-7a93-44d0-903d-8af82e7864f5\") " pod="ai-tenant-e2e-ipp-a-027edd/ipp-route-027edd-kserve-7cd6f577c9-qqtlz"
Sep 11 17:51:14.400293 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:51:14.400143    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/654d7a14-7a93-44d0-903d-8af82e7864f5-model-cache\") pod \"ipp-route-027edd-kserve-7cd6f577c9-qqtlz\" (UID: \"654d7a14-7a93-44d0-903d-8af82e7864f5\") " pod="ai-tenant-e2e-ipp-a-027edd/ipp-route-027edd-kserve-7cd6f577c9-qqtlz"
Sep 11 17:51:14.400293 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:51:14.400169    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v5l9x\" (UniqueName: \"kubernetes.io/projected/654d7a14-7a93-44d0-903d-8af82e7864f5-kube-api-access-v5l9x\") pod \"ipp-route-027edd-kserve-7cd6f577c9-qqtlz\" (UID: \"654d7a14-7a93-44d0-903d-8af82e7864f5\") " pod="ai-tenant-e2e-ipp-a-027edd/ipp-route-027edd-kserve-7cd6f577c9-qqtlz"
Sep 11 17:51:14.400293 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:51:14.400237    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/654d7a14-7a93-44d0-903d-8af82e7864f5-tmp-dir\") pod \"ipp-route-027edd-kserve-7cd6f577c9-qqtlz\" (UID: \"654d7a14-7a93-44d0-903d-8af82e7864f5\") " pod="ai-tenant-e2e-ipp-a-027edd/ipp-route-027edd-kserve-7cd6f577c9-qqtlz"
Sep 11 17:51:14.400439 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:51:14.400334    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/654d7a14-7a93-44d0-903d-8af82e7864f5-tls-certs\") pod \"ipp-route-027edd-kserve-7cd6f577c9-qqtlz\" (UID: \"654d7a14-7a93-44d0-903d-8af82e7864f5\") " pod="ai-tenant-e2e-ipp-a-027edd/ipp-route-027edd-kserve-7cd6f577c9-qqtlz"
Sep 11 17:51:14.501059 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:51:14.501013    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/654d7a14-7a93-44d0-903d-8af82e7864f5-tls-certs\") pod \"ipp-route-027edd-kserve-7cd6f577c9-qqtlz\" (UID: \"654d7a14-7a93-44d0-903d-8af82e7864f5\") " pod="ai-tenant-e2e-ipp-a-027edd/ipp-route-027edd-kserve-7cd6f577c9-qqtlz"
Sep 11 17:51:14.501228 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:51:14.501083    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/654d7a14-7a93-44d0-903d-8af82e7864f5-home\") pod \"ipp-route-027edd-kserve-7cd6f577c9-qqtlz\" (UID: \"654d7a14-7a93-44d0-903d-8af82e7864f5\") " pod="ai-tenant-e2e-ipp-a-027edd/ipp-route-027edd-kserve-7cd6f577c9-qqtlz"
Sep 11 17:51:14.501228 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:51:14.501102    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/654d7a14-7a93-44d0-903d-8af82e7864f5-dshm\") pod \"ipp-route-027edd-kserve-7cd6f577c9-qqtlz\" (UID: \"654d7a14-7a93-44d0-903d-8af82e7864f5\") " pod="ai-tenant-e2e-ipp-a-027edd/ipp-route-027edd-kserve-7cd6f577c9-qqtlz"
Sep 11 17:51:14.501228 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:51:14.501122    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/654d7a14-7a93-44d0-903d-8af82e7864f5-model-cache\") pod \"ipp-route-027edd-kserve-7cd6f577c9-qqtlz\" (UID: \"654d7a14-7a93-44d0-903d-8af82e7864f5\") " pod="ai-tenant-e2e-ipp-a-027edd/ipp-route-027edd-kserve-7cd6f577c9-qqtlz"
Sep 11 17:51:14.501228 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:51:14.501144    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v5l9x\" (UniqueName: \"kubernetes.io/projected/654d7a14-7a93-44d0-903d-8af82e7864f5-kube-api-access-v5l9x\") pod \"ipp-route-027edd-kserve-7cd6f577c9-qqtlz\" (UID: \"654d7a14-7a93-44d0-903d-8af82e7864f5\") " pod="ai-tenant-e2e-ipp-a-027edd/ipp-route-027edd-kserve-7cd6f577c9-qqtlz"
Sep 11 17:51:14.501228 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:51:14.501187    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/654d7a14-7a93-44d0-903d-8af82e7864f5-tmp-dir\") pod \"ipp-route-027edd-kserve-7cd6f577c9-qqtlz\" (UID: \"654d7a14-7a93-44d0-903d-8af82e7864f5\") " pod="ai-tenant-e2e-ipp-a-027edd/ipp-route-027edd-kserve-7cd6f577c9-qqtlz"
Sep 11 17:51:14.501633 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:51:14.501611    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/654d7a14-7a93-44d0-903d-8af82e7864f5-tmp-dir\") pod \"ipp-route-027edd-kserve-7cd6f577c9-qqtlz\" (UID: \"654d7a14-7a93-44d0-903d-8af82e7864f5\") " pod="ai-tenant-e2e-ipp-a-027edd/ipp-route-027edd-kserve-7cd6f577c9-qqtlz"
Sep 11 17:51:14.501695 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:51:14.501665    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/654d7a14-7a93-44d0-903d-8af82e7864f5-model-cache\") pod \"ipp-route-027edd-kserve-7cd6f577c9-qqtlz\" (UID: \"654d7a14-7a93-44d0-903d-8af82e7864f5\") " pod="ai-tenant-e2e-ipp-a-027edd/ipp-route-027edd-kserve-7cd6f577c9-qqtlz"
Sep 11 17:51:14.501695 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:51:14.501680    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/654d7a14-7a93-44d0-903d-8af82e7864f5-home\") pod \"ipp-route-027edd-kserve-7cd6f577c9-qqtlz\" (UID: \"654d7a14-7a93-44d0-903d-8af82e7864f5\") " pod="ai-tenant-e2e-ipp-a-027edd/ipp-route-027edd-kserve-7cd6f577c9-qqtlz"
Sep 11 17:51:14.503719 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:51:14.503682    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/654d7a14-7a93-44d0-903d-8af82e7864f5-dshm\") pod \"ipp-route-027edd-kserve-7cd6f577c9-qqtlz\" (UID: \"654d7a14-7a93-44d0-903d-8af82e7864f5\") " pod="ai-tenant-e2e-ipp-a-027edd/ipp-route-027edd-kserve-7cd6f577c9-qqtlz"
Sep 11 17:51:14.503989 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:51:14.503968    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/654d7a14-7a93-44d0-903d-8af82e7864f5-tls-certs\") pod \"ipp-route-027edd-kserve-7cd6f577c9-qqtlz\" (UID: \"654d7a14-7a93-44d0-903d-8af82e7864f5\") " pod="ai-tenant-e2e-ipp-a-027edd/ipp-route-027edd-kserve-7cd6f577c9-qqtlz"
Sep 11 17:51:14.509966 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:51:14.509940    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-v5l9x\" (UniqueName: \"kubernetes.io/projected/654d7a14-7a93-44d0-903d-8af82e7864f5-kube-api-access-v5l9x\") pod \"ipp-route-027edd-kserve-7cd6f577c9-qqtlz\" (UID: \"654d7a14-7a93-44d0-903d-8af82e7864f5\") " pod="ai-tenant-e2e-ipp-a-027edd/ipp-route-027edd-kserve-7cd6f577c9-qqtlz"
Sep 11 17:51:14.611173 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:51:14.611082    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-027edd/ipp-route-027edd-kserve-7cd6f577c9-qqtlz"
Sep 11 17:51:14.885723 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:51:14.885696    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-ipp-a-027edd/ipp-route-027edd-kserve-7cd6f577c9-qqtlz"]
Sep 11 17:51:14.887621 ip-10-0-141-139 kubenswrapper[2577]: W0911 17:51:14.887591    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod654d7a14_7a93_44d0_903d_8af82e7864f5.slice/crio-3c90c6ed59f8ba917a7d4584bc0e4373ca55dbe8514f5f1b2ed51f3d06791596 WatchSource:0}: Error finding container 3c90c6ed59f8ba917a7d4584bc0e4373ca55dbe8514f5f1b2ed51f3d06791596: Status 404 returned error can't find the container with id 3c90c6ed59f8ba917a7d4584bc0e4373ca55dbe8514f5f1b2ed51f3d06791596
Sep 11 17:51:15.703758 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:51:15.703723    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-027edd/ipp-route-027edd-kserve-7cd6f577c9-qqtlz" event={"ID":"654d7a14-7a93-44d0-903d-8af82e7864f5","Type":"ContainerStarted","Data":"9e753385e87a5146260e5c6d27c31ed309b8169f299075a46b0d2b6d079ed255"}
Sep 11 17:51:15.703758 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:51:15.703766    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-027edd/ipp-route-027edd-kserve-7cd6f577c9-qqtlz" event={"ID":"654d7a14-7a93-44d0-903d-8af82e7864f5","Type":"ContainerStarted","Data":"3c90c6ed59f8ba917a7d4584bc0e4373ca55dbe8514f5f1b2ed51f3d06791596"}
Sep 11 17:51:15.721680 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:51:15.721623    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-ipp-a-027edd/ipp-route-027edd-kserve-7cd6f577c9-qqtlz" podStartSLOduration=1.721607063 podStartE2EDuration="1.721607063s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 17:51:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 17:51:15.720897139 +0000 UTC m=+1953.294117252" watchObservedRunningTime="2026-09-11 17:51:15.721607063 +0000 UTC m=+1953.294827158"
Sep 11 17:51:18.108876 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:51:18.108840    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-nkmkm"]
Sep 11 17:51:24.612260 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:51:24.612197    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-ipp-a-027edd/ipp-route-027edd-kserve-7cd6f577c9-qqtlz"
Sep 11 17:51:24.612763 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:51:24.612286    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-ipp-a-027edd/ipp-route-027edd-kserve-7cd6f577c9-qqtlz"
Sep 11 17:51:24.625359 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:51:24.625330    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-ipp-a-027edd/ipp-route-027edd-kserve-7cd6f577c9-qqtlz"
Sep 11 17:51:24.763651 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:51:24.763618    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-ipp-a-027edd/ipp-route-027edd-kserve-7cd6f577c9-qqtlz"
Sep 11 17:51:37.395219 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:51:37.395137    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-nkmkm"]
Sep 11 17:51:43.396876 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:51:43.396834    2577 scope.go:117] "RemoveContainer" containerID="61cf624eb0ffed828e65cd98b710c4ca7d8cdf6af8e29d1d604313cf6af060fa"
Sep 11 17:51:43.406367 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:51:43.406343    2577 scope.go:117] "RemoveContainer" containerID="1b90bdaf9b8019047386e1929065eee3c082aef16ac0c68c7362d04969a586d9"
Sep 11 17:51:43.415543 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:51:43.415498    2577 scope.go:117] "RemoveContainer" containerID="455969995a4c1f06b3052488a4f23d256c2b245f1e8603b3abfa7f9598879c91"
Sep 11 17:51:43.932890 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:51:43.932841    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-b-10bbea-5c9b98f589-bxj2n_948e1e6f-0b40-4057-b996-590541520ac1/payload-processing/0.log"
Sep 11 17:51:52.873993 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:51:52.873957    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-574f8b-7f9fbdc4bc-hf8f8"]
Sep 11 17:51:52.877760 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:51:52.877739    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-574f8b-7f9fbdc4bc-hf8f8"
Sep 11 17:51:52.880324 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:51:52.880297    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-cleanup-574f8b\""
Sep 11 17:51:52.888441 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:51:52.888410    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c4fkt\" (UniqueName: \"kubernetes.io/projected/5feb91ff-cc02-4b41-9e2d-ece6b50c4736-kube-api-access-c4fkt\") pod \"payload-processing-e2e-ipp-cleanup-574f8b-7f9fbdc4bc-hf8f8\" (UID: \"5feb91ff-cc02-4b41-9e2d-ece6b50c4736\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-574f8b-7f9fbdc4bc-hf8f8"
Sep 11 17:51:52.888585 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:51:52.888454    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5feb91ff-cc02-4b41-9e2d-ece6b50c4736-plugins-config-volume\") pod \"payload-processing-e2e-ipp-cleanup-574f8b-7f9fbdc4bc-hf8f8\" (UID: \"5feb91ff-cc02-4b41-9e2d-ece6b50c4736\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-574f8b-7f9fbdc4bc-hf8f8"
Sep 11 17:51:52.889657 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:51:52.889631    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-574f8b-7f9fbdc4bc-hf8f8"]
Sep 11 17:51:52.937888 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:51:52.937853    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-574f8b-f96c9bc74-2twgm"]
Sep 11 17:51:52.946067 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:51:52.946029    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-574f8b-f96c9bc74-2twgm"
Sep 11 17:51:52.949868 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:51:52.949845    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-cleanup-574f8b-dockercfg-hsslw\""
Sep 11 17:51:52.952473 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:51:52.952446    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-574f8b-f96c9bc74-2twgm"]
Sep 11 17:51:52.989305 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:51:52.989261    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5feb91ff-cc02-4b41-9e2d-ece6b50c4736-plugins-config-volume\") pod \"payload-processing-e2e-ipp-cleanup-574f8b-7f9fbdc4bc-hf8f8\" (UID: \"5feb91ff-cc02-4b41-9e2d-ece6b50c4736\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-574f8b-7f9fbdc4bc-hf8f8"
Sep 11 17:51:52.989490 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:51:52.989314    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9fbf1aa3-dc84-46d4-bb19-8a98aaf15aca-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-cleanup-574f8b-f96c9bc74-2twgm\" (UID: \"9fbf1aa3-dc84-46d4-bb19-8a98aaf15aca\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-574f8b-f96c9bc74-2twgm"
Sep 11 17:51:52.989490 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:51:52.989382    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m4nvv\" (UniqueName: \"kubernetes.io/projected/9fbf1aa3-dc84-46d4-bb19-8a98aaf15aca-kube-api-access-m4nvv\") pod \"payload-pre-processing-e2e-ipp-cleanup-574f8b-f96c9bc74-2twgm\" (UID: \"9fbf1aa3-dc84-46d4-bb19-8a98aaf15aca\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-574f8b-f96c9bc74-2twgm"
Sep 11 17:51:52.989490 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:51:52.989420    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c4fkt\" (UniqueName: \"kubernetes.io/projected/5feb91ff-cc02-4b41-9e2d-ece6b50c4736-kube-api-access-c4fkt\") pod \"payload-processing-e2e-ipp-cleanup-574f8b-7f9fbdc4bc-hf8f8\" (UID: \"5feb91ff-cc02-4b41-9e2d-ece6b50c4736\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-574f8b-7f9fbdc4bc-hf8f8"
Sep 11 17:51:52.989981 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:51:52.989962    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5feb91ff-cc02-4b41-9e2d-ece6b50c4736-plugins-config-volume\") pod \"payload-processing-e2e-ipp-cleanup-574f8b-7f9fbdc4bc-hf8f8\" (UID: \"5feb91ff-cc02-4b41-9e2d-ece6b50c4736\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-574f8b-7f9fbdc4bc-hf8f8"
Sep 11 17:51:53.017261 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:51:53.017234    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-c4fkt\" (UniqueName: \"kubernetes.io/projected/5feb91ff-cc02-4b41-9e2d-ece6b50c4736-kube-api-access-c4fkt\") pod \"payload-processing-e2e-ipp-cleanup-574f8b-7f9fbdc4bc-hf8f8\" (UID: \"5feb91ff-cc02-4b41-9e2d-ece6b50c4736\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-574f8b-7f9fbdc4bc-hf8f8"
Sep 11 17:51:53.090333 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:51:53.090294    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m4nvv\" (UniqueName: \"kubernetes.io/projected/9fbf1aa3-dc84-46d4-bb19-8a98aaf15aca-kube-api-access-m4nvv\") pod \"payload-pre-processing-e2e-ipp-cleanup-574f8b-f96c9bc74-2twgm\" (UID: \"9fbf1aa3-dc84-46d4-bb19-8a98aaf15aca\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-574f8b-f96c9bc74-2twgm"
Sep 11 17:51:53.090532 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:51:53.090374    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9fbf1aa3-dc84-46d4-bb19-8a98aaf15aca-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-cleanup-574f8b-f96c9bc74-2twgm\" (UID: \"9fbf1aa3-dc84-46d4-bb19-8a98aaf15aca\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-574f8b-f96c9bc74-2twgm"
Sep 11 17:51:53.091082 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:51:53.091060    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9fbf1aa3-dc84-46d4-bb19-8a98aaf15aca-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-cleanup-574f8b-f96c9bc74-2twgm\" (UID: \"9fbf1aa3-dc84-46d4-bb19-8a98aaf15aca\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-574f8b-f96c9bc74-2twgm"
Sep 11 17:51:53.107608 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:51:53.107575    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-m4nvv\" (UniqueName: \"kubernetes.io/projected/9fbf1aa3-dc84-46d4-bb19-8a98aaf15aca-kube-api-access-m4nvv\") pod \"payload-pre-processing-e2e-ipp-cleanup-574f8b-f96c9bc74-2twgm\" (UID: \"9fbf1aa3-dc84-46d4-bb19-8a98aaf15aca\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-574f8b-f96c9bc74-2twgm"
Sep 11 17:51:53.190839 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:51:53.190746    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-574f8b-7f9fbdc4bc-hf8f8"
Sep 11 17:51:53.259269 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:51:53.259237    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-574f8b-f96c9bc74-2twgm"
Sep 11 17:51:53.468892 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:51:53.468863    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-574f8b-7f9fbdc4bc-hf8f8"]
Sep 11 17:51:53.470404 ip-10-0-141-139 kubenswrapper[2577]: W0911 17:51:53.470373    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5feb91ff_cc02_4b41_9e2d_ece6b50c4736.slice/crio-632654b69f41834b34b5c732d5b009747941d8cebdcdf0c7df1337dadee6b290 WatchSource:0}: Error finding container 632654b69f41834b34b5c732d5b009747941d8cebdcdf0c7df1337dadee6b290: Status 404 returned error can't find the container with id 632654b69f41834b34b5c732d5b009747941d8cebdcdf0c7df1337dadee6b290
Sep 11 17:51:53.537866 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:51:53.537825    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-574f8b-f96c9bc74-2twgm"]
Sep 11 17:51:53.539710 ip-10-0-141-139 kubenswrapper[2577]: W0911 17:51:53.539681    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9fbf1aa3_dc84_46d4_bb19_8a98aaf15aca.slice/crio-a689a9495bada2dc09b0b1155af3827faa2282073c9bde9673f43812b3d8f7d1 WatchSource:0}: Error finding container a689a9495bada2dc09b0b1155af3827faa2282073c9bde9673f43812b3d8f7d1: Status 404 returned error can't find the container with id a689a9495bada2dc09b0b1155af3827faa2282073c9bde9673f43812b3d8f7d1
Sep 11 17:51:53.912310 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:51:53.912204    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-574f8b-7f9fbdc4bc-hf8f8" event={"ID":"5feb91ff-cc02-4b41-9e2d-ece6b50c4736","Type":"ContainerStarted","Data":"d9232bb1428310582926bb4bf75462e4cb2c0c9dac61b351ccb1de7c3fe864cf"}
Sep 11 17:51:53.912310 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:51:53.912276    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-574f8b-7f9fbdc4bc-hf8f8"
Sep 11 17:51:53.912310 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:51:53.912304    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-574f8b-7f9fbdc4bc-hf8f8" event={"ID":"5feb91ff-cc02-4b41-9e2d-ece6b50c4736","Type":"ContainerStarted","Data":"632654b69f41834b34b5c732d5b009747941d8cebdcdf0c7df1337dadee6b290"}
Sep 11 17:51:53.913897 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:51:53.913875    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-574f8b-f96c9bc74-2twgm" event={"ID":"9fbf1aa3-dc84-46d4-bb19-8a98aaf15aca","Type":"ContainerStarted","Data":"3c5d757b589b0cf95531e1fdcf06ab65e230d5083e715ea9cae501ff7f2cd75c"}
Sep 11 17:51:53.913987 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:51:53.913915    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-574f8b-f96c9bc74-2twgm" event={"ID":"9fbf1aa3-dc84-46d4-bb19-8a98aaf15aca","Type":"ContainerStarted","Data":"a689a9495bada2dc09b0b1155af3827faa2282073c9bde9673f43812b3d8f7d1"}
Sep 11 17:51:53.913987 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:51:53.913935    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-574f8b-f96c9bc74-2twgm"
Sep 11 17:51:53.931933 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:51:53.931886    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-574f8b-7f9fbdc4bc-hf8f8" podStartSLOduration=1.931870391 podStartE2EDuration="1.931870391s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 17:51:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 17:51:53.928712157 +0000 UTC m=+1991.501932284" watchObservedRunningTime="2026-09-11 17:51:53.931870391 +0000 UTC m=+1991.505090503"
Sep 11 17:51:53.951013 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:51:53.950954    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-574f8b-f96c9bc74-2twgm" podStartSLOduration=1.950938575 podStartE2EDuration="1.950938575s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 17:51:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 17:51:53.948237408 +0000 UTC m=+1991.521457520" watchObservedRunningTime="2026-09-11 17:51:53.950938575 +0000 UTC m=+1991.524158687"
Sep 11 17:51:57.670298 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:51:57.670262    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-nkmkm"]
Sep 11 17:52:04.926978 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:52:04.926947    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-574f8b-f96c9bc74-2twgm"
Sep 11 17:52:04.927386 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:52:04.927013    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-574f8b-7f9fbdc4bc-hf8f8"
Sep 11 17:52:18.415139 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:52:18.415102    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-574f8b-dgcfd"]
Sep 11 17:52:18.418762 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:52:18.418744    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-574f8b-dgcfd"
Sep 11 17:52:18.423957 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:52:18.423937    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-9n8gf\""
Sep 11 17:52:18.424351 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:52:18.424335    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 11 17:52:18.439327 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:52:18.439290    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-574f8b-dgcfd"]
Sep 11 17:52:18.540068 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:52:18.540030    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fadd17d8-cef9-42b8-946e-09cafe8f4c34-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-cleanup-574f8b-dgcfd\" (UID: \"fadd17d8-cef9-42b8-946e-09cafe8f4c34\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-574f8b-dgcfd"
Sep 11 17:52:18.641064 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:52:18.641028    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fadd17d8-cef9-42b8-946e-09cafe8f4c34-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-cleanup-574f8b-dgcfd\" (UID: \"fadd17d8-cef9-42b8-946e-09cafe8f4c34\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-574f8b-dgcfd"
Sep 11 17:52:18.641696 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:52:18.641674    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fadd17d8-cef9-42b8-946e-09cafe8f4c34-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-cleanup-574f8b-dgcfd\" (UID: \"fadd17d8-cef9-42b8-946e-09cafe8f4c34\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-574f8b-dgcfd"
Sep 11 17:52:18.730243 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:52:18.730207    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-574f8b-dgcfd"
Sep 11 17:52:19.015372 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:52:19.015343    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-574f8b-dgcfd"]
Sep 11 17:52:19.016199 ip-10-0-141-139 kubenswrapper[2577]: W0911 17:52:19.016177    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfadd17d8_cef9_42b8_946e_09cafe8f4c34.slice/crio-5936bcb49d18005df432dadb6dd286c484dba570bc2b98d73b1ebb21514c3669 WatchSource:0}: Error finding container 5936bcb49d18005df432dadb6dd286c484dba570bc2b98d73b1ebb21514c3669: Status 404 returned error can't find the container with id 5936bcb49d18005df432dadb6dd286c484dba570bc2b98d73b1ebb21514c3669
Sep 11 17:52:19.056946 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:52:19.056897    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-574f8b-dgcfd" event={"ID":"fadd17d8-cef9-42b8-946e-09cafe8f4c34","Type":"ContainerStarted","Data":"5936bcb49d18005df432dadb6dd286c484dba570bc2b98d73b1ebb21514c3669"}
Sep 11 17:52:20.064141 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:52:20.064104    2577 generic.go:356] "Generic (PLEG): container finished" podID="fadd17d8-cef9-42b8-946e-09cafe8f4c34" containerID="472c2bf8a41441e04af93eb7785668c57e964284892b5417d0961f4166f44c0a" exitCode=0
Sep 11 17:52:20.064593 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:52:20.064185    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-574f8b-dgcfd" event={"ID":"fadd17d8-cef9-42b8-946e-09cafe8f4c34","Type":"ContainerDied","Data":"472c2bf8a41441e04af93eb7785668c57e964284892b5417d0961f4166f44c0a"}
Sep 11 17:52:21.207569 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:52:21.207543    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-574f8b-dgcfd"
Sep 11 17:52:21.368681 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:52:21.368578    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fadd17d8-cef9-42b8-946e-09cafe8f4c34-maas-api-service-ca\") pod \"fadd17d8-cef9-42b8-946e-09cafe8f4c34\" (UID: \"fadd17d8-cef9-42b8-946e-09cafe8f4c34\") "
Sep 11 17:52:21.368954 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:52:21.368930    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fadd17d8-cef9-42b8-946e-09cafe8f4c34-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "fadd17d8-cef9-42b8-946e-09cafe8f4c34" (UID: "fadd17d8-cef9-42b8-946e-09cafe8f4c34"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 17:52:21.469764 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:52:21.469727    2577 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fadd17d8-cef9-42b8-946e-09cafe8f4c34-maas-api-service-ca\") on node \"ip-10-0-141-139.ec2.internal\" DevicePath \"\""
Sep 11 17:52:22.076474 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:52:22.076444    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-574f8b-dgcfd"
Sep 11 17:52:22.076672 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:52:22.076491    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-574f8b-dgcfd" event={"ID":"fadd17d8-cef9-42b8-946e-09cafe8f4c34","Type":"ContainerDied","Data":"5936bcb49d18005df432dadb6dd286c484dba570bc2b98d73b1ebb21514c3669"}
Sep 11 17:52:22.076672 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:52:22.076555    2577 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5936bcb49d18005df432dadb6dd286c484dba570bc2b98d73b1ebb21514c3669"
Sep 11 17:52:29.669706 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:52:29.669672    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-574f8b-7f9fbdc4bc-hf8f8"]
Sep 11 17:52:29.670316 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:52:29.669918    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-574f8b-7f9fbdc4bc-hf8f8" podUID="5feb91ff-cc02-4b41-9e2d-ece6b50c4736" containerName="payload-processing" containerID="cri-o://d9232bb1428310582926bb4bf75462e4cb2c0c9dac61b351ccb1de7c3fe864cf" gracePeriod=30
Sep 11 17:52:29.680990 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:52:29.680799    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-574f8b-f96c9bc74-2twgm"]
Sep 11 17:52:29.681111 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:52:29.681029    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-574f8b-f96c9bc74-2twgm" podUID="9fbf1aa3-dc84-46d4-bb19-8a98aaf15aca" containerName="payload-pre-processing" containerID="cri-o://3c5d757b589b0cf95531e1fdcf06ab65e230d5083e715ea9cae501ff7f2cd75c" gracePeriod=30
Sep 11 17:52:29.871543 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:52:29.871482    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-574f8b-dgcfd"]
Sep 11 17:52:29.876124 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:52:29.876095    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-574f8b-dgcfd"]
Sep 11 17:52:29.953941 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:52:29.953918    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-574f8b-7f9fbdc4bc-hf8f8"
Sep 11 17:52:29.957520 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:52:29.957465    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-574f8b-f96c9bc74-2twgm"
Sep 11 17:52:30.054403 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:52:30.054371    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-c4fkt\" (UniqueName: \"kubernetes.io/projected/5feb91ff-cc02-4b41-9e2d-ece6b50c4736-kube-api-access-c4fkt\") pod \"5feb91ff-cc02-4b41-9e2d-ece6b50c4736\" (UID: \"5feb91ff-cc02-4b41-9e2d-ece6b50c4736\") "
Sep 11 17:52:30.054641 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:52:30.054430    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-m4nvv\" (UniqueName: \"kubernetes.io/projected/9fbf1aa3-dc84-46d4-bb19-8a98aaf15aca-kube-api-access-m4nvv\") pod \"9fbf1aa3-dc84-46d4-bb19-8a98aaf15aca\" (UID: \"9fbf1aa3-dc84-46d4-bb19-8a98aaf15aca\") "
Sep 11 17:52:30.054641 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:52:30.054450    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5feb91ff-cc02-4b41-9e2d-ece6b50c4736-plugins-config-volume\") pod \"5feb91ff-cc02-4b41-9e2d-ece6b50c4736\" (UID: \"5feb91ff-cc02-4b41-9e2d-ece6b50c4736\") "
Sep 11 17:52:30.054641 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:52:30.054482    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9fbf1aa3-dc84-46d4-bb19-8a98aaf15aca-plugins-config-volume\") pod \"9fbf1aa3-dc84-46d4-bb19-8a98aaf15aca\" (UID: \"9fbf1aa3-dc84-46d4-bb19-8a98aaf15aca\") "
Sep 11 17:52:30.054917 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:52:30.054886    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5feb91ff-cc02-4b41-9e2d-ece6b50c4736-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "5feb91ff-cc02-4b41-9e2d-ece6b50c4736" (UID: "5feb91ff-cc02-4b41-9e2d-ece6b50c4736"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 17:52:30.054917 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:52:30.054907    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9fbf1aa3-dc84-46d4-bb19-8a98aaf15aca-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "9fbf1aa3-dc84-46d4-bb19-8a98aaf15aca" (UID: "9fbf1aa3-dc84-46d4-bb19-8a98aaf15aca"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 17:52:30.056662 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:52:30.056640    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5feb91ff-cc02-4b41-9e2d-ece6b50c4736-kube-api-access-c4fkt" (OuterVolumeSpecName: "kube-api-access-c4fkt") pod "5feb91ff-cc02-4b41-9e2d-ece6b50c4736" (UID: "5feb91ff-cc02-4b41-9e2d-ece6b50c4736"). InnerVolumeSpecName "kube-api-access-c4fkt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 17:52:30.056721 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:52:30.056676    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9fbf1aa3-dc84-46d4-bb19-8a98aaf15aca-kube-api-access-m4nvv" (OuterVolumeSpecName: "kube-api-access-m4nvv") pod "9fbf1aa3-dc84-46d4-bb19-8a98aaf15aca" (UID: "9fbf1aa3-dc84-46d4-bb19-8a98aaf15aca"). InnerVolumeSpecName "kube-api-access-m4nvv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 17:52:30.121875 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:52:30.121835    2577 generic.go:356] "Generic (PLEG): container finished" podID="5feb91ff-cc02-4b41-9e2d-ece6b50c4736" containerID="d9232bb1428310582926bb4bf75462e4cb2c0c9dac61b351ccb1de7c3fe864cf" exitCode=0
Sep 11 17:52:30.122074 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:52:30.121882    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-574f8b-7f9fbdc4bc-hf8f8"
Sep 11 17:52:30.122074 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:52:30.121912    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-574f8b-7f9fbdc4bc-hf8f8" event={"ID":"5feb91ff-cc02-4b41-9e2d-ece6b50c4736","Type":"ContainerDied","Data":"d9232bb1428310582926bb4bf75462e4cb2c0c9dac61b351ccb1de7c3fe864cf"}
Sep 11 17:52:30.122074 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:52:30.121961    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-574f8b-7f9fbdc4bc-hf8f8" event={"ID":"5feb91ff-cc02-4b41-9e2d-ece6b50c4736","Type":"ContainerDied","Data":"632654b69f41834b34b5c732d5b009747941d8cebdcdf0c7df1337dadee6b290"}
Sep 11 17:52:30.122074 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:52:30.121985    2577 scope.go:117] "RemoveContainer" containerID="d9232bb1428310582926bb4bf75462e4cb2c0c9dac61b351ccb1de7c3fe864cf"
Sep 11 17:52:30.123244 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:52:30.123224    2577 generic.go:356] "Generic (PLEG): container finished" podID="9fbf1aa3-dc84-46d4-bb19-8a98aaf15aca" containerID="3c5d757b589b0cf95531e1fdcf06ab65e230d5083e715ea9cae501ff7f2cd75c" exitCode=0
Sep 11 17:52:30.123339 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:52:30.123273    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-574f8b-f96c9bc74-2twgm"
Sep 11 17:52:30.123339 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:52:30.123299    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-574f8b-f96c9bc74-2twgm" event={"ID":"9fbf1aa3-dc84-46d4-bb19-8a98aaf15aca","Type":"ContainerDied","Data":"3c5d757b589b0cf95531e1fdcf06ab65e230d5083e715ea9cae501ff7f2cd75c"}
Sep 11 17:52:30.123423 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:52:30.123338    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-574f8b-f96c9bc74-2twgm" event={"ID":"9fbf1aa3-dc84-46d4-bb19-8a98aaf15aca","Type":"ContainerDied","Data":"a689a9495bada2dc09b0b1155af3827faa2282073c9bde9673f43812b3d8f7d1"}
Sep 11 17:52:30.131577 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:52:30.131559    2577 scope.go:117] "RemoveContainer" containerID="d9232bb1428310582926bb4bf75462e4cb2c0c9dac61b351ccb1de7c3fe864cf"
Sep 11 17:52:30.131905 ip-10-0-141-139 kubenswrapper[2577]: E0911 17:52:30.131879    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d9232bb1428310582926bb4bf75462e4cb2c0c9dac61b351ccb1de7c3fe864cf\": container with ID starting with d9232bb1428310582926bb4bf75462e4cb2c0c9dac61b351ccb1de7c3fe864cf not found: ID does not exist" containerID="d9232bb1428310582926bb4bf75462e4cb2c0c9dac61b351ccb1de7c3fe864cf"
Sep 11 17:52:30.132018 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:52:30.131914    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d9232bb1428310582926bb4bf75462e4cb2c0c9dac61b351ccb1de7c3fe864cf"} err="failed to get container status \"d9232bb1428310582926bb4bf75462e4cb2c0c9dac61b351ccb1de7c3fe864cf\": rpc error: code = NotFound desc = could not find container \"d9232bb1428310582926bb4bf75462e4cb2c0c9dac61b351ccb1de7c3fe864cf\": container with ID starting with d9232bb1428310582926bb4bf75462e4cb2c0c9dac61b351ccb1de7c3fe864cf not found: ID does not exist"
Sep 11 17:52:30.132018 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:52:30.131936    2577 scope.go:117] "RemoveContainer" containerID="3c5d757b589b0cf95531e1fdcf06ab65e230d5083e715ea9cae501ff7f2cd75c"
Sep 11 17:52:30.141465 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:52:30.141449    2577 scope.go:117] "RemoveContainer" containerID="3c5d757b589b0cf95531e1fdcf06ab65e230d5083e715ea9cae501ff7f2cd75c"
Sep 11 17:52:30.141779 ip-10-0-141-139 kubenswrapper[2577]: E0911 17:52:30.141762    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3c5d757b589b0cf95531e1fdcf06ab65e230d5083e715ea9cae501ff7f2cd75c\": container with ID starting with 3c5d757b589b0cf95531e1fdcf06ab65e230d5083e715ea9cae501ff7f2cd75c not found: ID does not exist" containerID="3c5d757b589b0cf95531e1fdcf06ab65e230d5083e715ea9cae501ff7f2cd75c"
Sep 11 17:52:30.141827 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:52:30.141788    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3c5d757b589b0cf95531e1fdcf06ab65e230d5083e715ea9cae501ff7f2cd75c"} err="failed to get container status \"3c5d757b589b0cf95531e1fdcf06ab65e230d5083e715ea9cae501ff7f2cd75c\": rpc error: code = NotFound desc = could not find container \"3c5d757b589b0cf95531e1fdcf06ab65e230d5083e715ea9cae501ff7f2cd75c\": container with ID starting with 3c5d757b589b0cf95531e1fdcf06ab65e230d5083e715ea9cae501ff7f2cd75c not found: ID does not exist"
Sep 11 17:52:30.155341 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:52:30.155321    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c4fkt\" (UniqueName: \"kubernetes.io/projected/5feb91ff-cc02-4b41-9e2d-ece6b50c4736-kube-api-access-c4fkt\") on node \"ip-10-0-141-139.ec2.internal\" DevicePath \"\""
Sep 11 17:52:30.155439 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:52:30.155343    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m4nvv\" (UniqueName: \"kubernetes.io/projected/9fbf1aa3-dc84-46d4-bb19-8a98aaf15aca-kube-api-access-m4nvv\") on node \"ip-10-0-141-139.ec2.internal\" DevicePath \"\""
Sep 11 17:52:30.155439 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:52:30.155357    2577 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5feb91ff-cc02-4b41-9e2d-ece6b50c4736-plugins-config-volume\") on node \"ip-10-0-141-139.ec2.internal\" DevicePath \"\""
Sep 11 17:52:30.155439 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:52:30.155370    2577 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9fbf1aa3-dc84-46d4-bb19-8a98aaf15aca-plugins-config-volume\") on node \"ip-10-0-141-139.ec2.internal\" DevicePath \"\""
Sep 11 17:52:30.161518 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:52:30.161482    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-574f8b-7f9fbdc4bc-hf8f8"]
Sep 11 17:52:30.172755 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:52:30.172734    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-574f8b-7f9fbdc4bc-hf8f8"]
Sep 11 17:52:30.189602 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:52:30.189527    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-574f8b-f96c9bc74-2twgm"]
Sep 11 17:52:30.195486 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:52:30.195461    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-574f8b-f96c9bc74-2twgm"]
Sep 11 17:52:30.926848 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:52:30.926815    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5feb91ff-cc02-4b41-9e2d-ece6b50c4736" path="/var/lib/kubelet/pods/5feb91ff-cc02-4b41-9e2d-ece6b50c4736/volumes"
Sep 11 17:52:30.927203 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:52:30.927160    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9fbf1aa3-dc84-46d4-bb19-8a98aaf15aca" path="/var/lib/kubelet/pods/9fbf1aa3-dc84-46d4-bb19-8a98aaf15aca/volumes"
Sep 11 17:52:30.927449 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:52:30.927436    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fadd17d8-cef9-42b8-946e-09cafe8f4c34" path="/var/lib/kubelet/pods/fadd17d8-cef9-42b8-946e-09cafe8f4c34/volumes"
Sep 11 17:52:33.921838 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:52:33.921807    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-nkmkm"]
Sep 11 17:52:40.122019 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:52:40.121980    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-027edd-wzs7g"]
Sep 11 17:52:40.122420 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:52:40.122407    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5feb91ff-cc02-4b41-9e2d-ece6b50c4736" containerName="payload-processing"
Sep 11 17:52:40.122462 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:52:40.122421    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="5feb91ff-cc02-4b41-9e2d-ece6b50c4736" containerName="payload-processing"
Sep 11 17:52:40.122462 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:52:40.122433    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9fbf1aa3-dc84-46d4-bb19-8a98aaf15aca" containerName="payload-pre-processing"
Sep 11 17:52:40.122462 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:52:40.122439    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="9fbf1aa3-dc84-46d4-bb19-8a98aaf15aca" containerName="payload-pre-processing"
Sep 11 17:52:40.122462 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:52:40.122445    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fadd17d8-cef9-42b8-946e-09cafe8f4c34" containerName="revoke-keys"
Sep 11 17:52:40.122462 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:52:40.122451    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="fadd17d8-cef9-42b8-946e-09cafe8f4c34" containerName="revoke-keys"
Sep 11 17:52:40.122655 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:52:40.122530    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="5feb91ff-cc02-4b41-9e2d-ece6b50c4736" containerName="payload-processing"
Sep 11 17:52:40.122655 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:52:40.122540    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="9fbf1aa3-dc84-46d4-bb19-8a98aaf15aca" containerName="payload-pre-processing"
Sep 11 17:52:40.122655 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:52:40.122548    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="fadd17d8-cef9-42b8-946e-09cafe8f4c34" containerName="revoke-keys"
Sep 11 17:52:40.127023 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:52:40.127006    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-027edd-wzs7g"
Sep 11 17:52:40.129447 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:52:40.129427    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-9n8gf\""
Sep 11 17:52:40.129447 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:52:40.129429    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 11 17:52:40.133955 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:52:40.133929    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-027edd-wzs7g"]
Sep 11 17:52:40.264477 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:52:40.264434    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/80a0ba8b-1801-4ce9-a1e4-19800c7f7e4d-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-027edd-wzs7g\" (UID: \"80a0ba8b-1801-4ce9-a1e4-19800c7f7e4d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-027edd-wzs7g"
Sep 11 17:52:40.365574 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:52:40.365536    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/80a0ba8b-1801-4ce9-a1e4-19800c7f7e4d-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-027edd-wzs7g\" (UID: \"80a0ba8b-1801-4ce9-a1e4-19800c7f7e4d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-027edd-wzs7g"
Sep 11 17:52:40.366188 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:52:40.366163    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/80a0ba8b-1801-4ce9-a1e4-19800c7f7e4d-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-027edd-wzs7g\" (UID: \"80a0ba8b-1801-4ce9-a1e4-19800c7f7e4d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-027edd-wzs7g"
Sep 11 17:52:40.439203 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:52:40.439112    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-027edd-wzs7g"
Sep 11 17:52:40.717684 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:52:40.717653    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-027edd-wzs7g"]
Sep 11 17:52:40.719078 ip-10-0-141-139 kubenswrapper[2577]: W0911 17:52:40.719055    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod80a0ba8b_1801_4ce9_a1e4_19800c7f7e4d.slice/crio-9a7510c949a5bbafb17b59efb9f97a395a349c5f8a932f8647028b5670476a5e WatchSource:0}: Error finding container 9a7510c949a5bbafb17b59efb9f97a395a349c5f8a932f8647028b5670476a5e: Status 404 returned error can't find the container with id 9a7510c949a5bbafb17b59efb9f97a395a349c5f8a932f8647028b5670476a5e
Sep 11 17:52:41.185088 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:52:41.185055    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-027edd-wzs7g" event={"ID":"80a0ba8b-1801-4ce9-a1e4-19800c7f7e4d","Type":"ContainerStarted","Data":"262e6d21eeba3ff9db80a6ce47049f1e708d114203fa39bfa2c9f39ae7c07bd0"}
Sep 11 17:52:41.185580 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:52:41.185115    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-027edd-wzs7g" event={"ID":"80a0ba8b-1801-4ce9-a1e4-19800c7f7e4d","Type":"ContainerStarted","Data":"9a7510c949a5bbafb17b59efb9f97a395a349c5f8a932f8647028b5670476a5e"}
Sep 11 17:52:41.201216 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:52:41.201167    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-027edd-wzs7g" podStartSLOduration=1.201150457 podStartE2EDuration="1.201150457s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 17:52:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 17:52:41.20057283 +0000 UTC m=+2038.773792953" watchObservedRunningTime="2026-09-11 17:52:41.201150457 +0000 UTC m=+2038.774370571"
Sep 11 17:52:42.192527 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:52:42.192475    2577 generic.go:356] "Generic (PLEG): container finished" podID="80a0ba8b-1801-4ce9-a1e4-19800c7f7e4d" containerID="262e6d21eeba3ff9db80a6ce47049f1e708d114203fa39bfa2c9f39ae7c07bd0" exitCode=0
Sep 11 17:52:42.192934 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:52:42.192546    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-027edd-wzs7g" event={"ID":"80a0ba8b-1801-4ce9-a1e4-19800c7f7e4d","Type":"ContainerDied","Data":"262e6d21eeba3ff9db80a6ce47049f1e708d114203fa39bfa2c9f39ae7c07bd0"}
Sep 11 17:52:43.331701 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:52:43.331673    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-027edd-wzs7g"
Sep 11 17:52:43.398515 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:52:43.398479    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/80a0ba8b-1801-4ce9-a1e4-19800c7f7e4d-maas-api-service-ca\") pod \"80a0ba8b-1801-4ce9-a1e4-19800c7f7e4d\" (UID: \"80a0ba8b-1801-4ce9-a1e4-19800c7f7e4d\") "
Sep 11 17:52:43.400529 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:52:43.399170    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/80a0ba8b-1801-4ce9-a1e4-19800c7f7e4d-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "80a0ba8b-1801-4ce9-a1e4-19800c7f7e4d" (UID: "80a0ba8b-1801-4ce9-a1e4-19800c7f7e4d"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 17:52:43.400529 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:52:43.399529    2577 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/80a0ba8b-1801-4ce9-a1e4-19800c7f7e4d-maas-api-service-ca\") on node \"ip-10-0-141-139.ec2.internal\" DevicePath \"\""
Sep 11 17:52:44.204312 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:52:44.204277    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-027edd-wzs7g"
Sep 11 17:52:44.204312 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:52:44.204307    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-027edd-wzs7g" event={"ID":"80a0ba8b-1801-4ce9-a1e4-19800c7f7e4d","Type":"ContainerDied","Data":"9a7510c949a5bbafb17b59efb9f97a395a349c5f8a932f8647028b5670476a5e"}
Sep 11 17:52:44.204561 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:52:44.204359    2577 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9a7510c949a5bbafb17b59efb9f97a395a349c5f8a932f8647028b5670476a5e"
Sep 11 17:52:53.391788 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:52:53.391747    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-nkmkm"]
Sep 11 17:52:55.946565 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:52:55.946515    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-027edd-wzs7g"]
Sep 11 17:52:55.949414 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:52:55.949381    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-027edd-wzs7g"]
Sep 11 17:52:56.927416 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:52:56.927374    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="80a0ba8b-1801-4ce9-a1e4-19800c7f7e4d" path="/var/lib/kubelet/pods/80a0ba8b-1801-4ce9-a1e4-19800c7f7e4d/volumes"
Sep 11 17:53:01.295290 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:53:01.295251    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-ipp-a-027edd/ipp-route-027edd-kserve-7cd6f577c9-qqtlz"]
Sep 11 17:53:06.403140 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:53:06.403054    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-nkmkm"]
Sep 11 17:53:16.317066 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:53:16.317000    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-ipp-a-027edd/ipp-route-027edd-kserve-7cd6f577c9-qqtlz" podUID="654d7a14-7a93-44d0-903d-8af82e7864f5" containerName="main" containerID="cri-o://9e753385e87a5146260e5c6d27c31ed309b8169f299075a46b0d2b6d079ed255" gracePeriod=45
Sep 11 17:53:16.577321 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:53:16.577259    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-027edd/ipp-route-027edd-kserve-7cd6f577c9-qqtlz"
Sep 11 17:53:16.641549 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:53:16.641493    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/654d7a14-7a93-44d0-903d-8af82e7864f5-home\") pod \"654d7a14-7a93-44d0-903d-8af82e7864f5\" (UID: \"654d7a14-7a93-44d0-903d-8af82e7864f5\") "
Sep 11 17:53:16.641738 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:53:16.641592    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/654d7a14-7a93-44d0-903d-8af82e7864f5-model-cache\") pod \"654d7a14-7a93-44d0-903d-8af82e7864f5\" (UID: \"654d7a14-7a93-44d0-903d-8af82e7864f5\") "
Sep 11 17:53:16.641738 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:53:16.641613    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/654d7a14-7a93-44d0-903d-8af82e7864f5-dshm\") pod \"654d7a14-7a93-44d0-903d-8af82e7864f5\" (UID: \"654d7a14-7a93-44d0-903d-8af82e7864f5\") "
Sep 11 17:53:16.641738 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:53:16.641638    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-v5l9x\" (UniqueName: \"kubernetes.io/projected/654d7a14-7a93-44d0-903d-8af82e7864f5-kube-api-access-v5l9x\") pod \"654d7a14-7a93-44d0-903d-8af82e7864f5\" (UID: \"654d7a14-7a93-44d0-903d-8af82e7864f5\") "
Sep 11 17:53:16.641738 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:53:16.641663    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/654d7a14-7a93-44d0-903d-8af82e7864f5-tmp-dir\") pod \"654d7a14-7a93-44d0-903d-8af82e7864f5\" (UID: \"654d7a14-7a93-44d0-903d-8af82e7864f5\") "
Sep 11 17:53:16.641944 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:53:16.641755    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/654d7a14-7a93-44d0-903d-8af82e7864f5-tls-certs\") pod \"654d7a14-7a93-44d0-903d-8af82e7864f5\" (UID: \"654d7a14-7a93-44d0-903d-8af82e7864f5\") "
Sep 11 17:53:16.641944 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:53:16.641777    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/654d7a14-7a93-44d0-903d-8af82e7864f5-home" (OuterVolumeSpecName: "home") pod "654d7a14-7a93-44d0-903d-8af82e7864f5" (UID: "654d7a14-7a93-44d0-903d-8af82e7864f5"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 17:53:16.642081 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:53:16.642043    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/654d7a14-7a93-44d0-903d-8af82e7864f5-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "654d7a14-7a93-44d0-903d-8af82e7864f5" (UID: "654d7a14-7a93-44d0-903d-8af82e7864f5"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 17:53:16.642216 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:53:16.642196    2577 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/654d7a14-7a93-44d0-903d-8af82e7864f5-tmp-dir\") on node \"ip-10-0-141-139.ec2.internal\" DevicePath \"\""
Sep 11 17:53:16.642278 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:53:16.642221    2577 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/654d7a14-7a93-44d0-903d-8af82e7864f5-home\") on node \"ip-10-0-141-139.ec2.internal\" DevicePath \"\""
Sep 11 17:53:16.642321 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:53:16.642275    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/654d7a14-7a93-44d0-903d-8af82e7864f5-model-cache" (OuterVolumeSpecName: "model-cache") pod "654d7a14-7a93-44d0-903d-8af82e7864f5" (UID: "654d7a14-7a93-44d0-903d-8af82e7864f5"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 17:53:16.644153 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:53:16.644119    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/654d7a14-7a93-44d0-903d-8af82e7864f5-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "654d7a14-7a93-44d0-903d-8af82e7864f5" (UID: "654d7a14-7a93-44d0-903d-8af82e7864f5"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 17:53:16.644422 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:53:16.644405    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/654d7a14-7a93-44d0-903d-8af82e7864f5-dshm" (OuterVolumeSpecName: "dshm") pod "654d7a14-7a93-44d0-903d-8af82e7864f5" (UID: "654d7a14-7a93-44d0-903d-8af82e7864f5"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 17:53:16.644563 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:53:16.644424    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/654d7a14-7a93-44d0-903d-8af82e7864f5-kube-api-access-v5l9x" (OuterVolumeSpecName: "kube-api-access-v5l9x") pod "654d7a14-7a93-44d0-903d-8af82e7864f5" (UID: "654d7a14-7a93-44d0-903d-8af82e7864f5"). InnerVolumeSpecName "kube-api-access-v5l9x". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 17:53:16.743123 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:53:16.743087    2577 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/654d7a14-7a93-44d0-903d-8af82e7864f5-model-cache\") on node \"ip-10-0-141-139.ec2.internal\" DevicePath \"\""
Sep 11 17:53:16.743123 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:53:16.743117    2577 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/654d7a14-7a93-44d0-903d-8af82e7864f5-dshm\") on node \"ip-10-0-141-139.ec2.internal\" DevicePath \"\""
Sep 11 17:53:16.743123 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:53:16.743126    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v5l9x\" (UniqueName: \"kubernetes.io/projected/654d7a14-7a93-44d0-903d-8af82e7864f5-kube-api-access-v5l9x\") on node \"ip-10-0-141-139.ec2.internal\" DevicePath \"\""
Sep 11 17:53:16.743388 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:53:16.743137    2577 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/654d7a14-7a93-44d0-903d-8af82e7864f5-tls-certs\") on node \"ip-10-0-141-139.ec2.internal\" DevicePath \"\""
Sep 11 17:53:17.387261 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:53:17.387223    2577 generic.go:356] "Generic (PLEG): container finished" podID="654d7a14-7a93-44d0-903d-8af82e7864f5" containerID="9e753385e87a5146260e5c6d27c31ed309b8169f299075a46b0d2b6d079ed255" exitCode=0
Sep 11 17:53:17.387797 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:53:17.387279    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-027edd/ipp-route-027edd-kserve-7cd6f577c9-qqtlz"
Sep 11 17:53:17.387797 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:53:17.387310    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-027edd/ipp-route-027edd-kserve-7cd6f577c9-qqtlz" event={"ID":"654d7a14-7a93-44d0-903d-8af82e7864f5","Type":"ContainerDied","Data":"9e753385e87a5146260e5c6d27c31ed309b8169f299075a46b0d2b6d079ed255"}
Sep 11 17:53:17.387797 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:53:17.387362    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-027edd/ipp-route-027edd-kserve-7cd6f577c9-qqtlz" event={"ID":"654d7a14-7a93-44d0-903d-8af82e7864f5","Type":"ContainerDied","Data":"3c90c6ed59f8ba917a7d4584bc0e4373ca55dbe8514f5f1b2ed51f3d06791596"}
Sep 11 17:53:17.387797 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:53:17.387385    2577 scope.go:117] "RemoveContainer" containerID="9e753385e87a5146260e5c6d27c31ed309b8169f299075a46b0d2b6d079ed255"
Sep 11 17:53:17.397182 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:53:17.397162    2577 scope.go:117] "RemoveContainer" containerID="9e753385e87a5146260e5c6d27c31ed309b8169f299075a46b0d2b6d079ed255"
Sep 11 17:53:17.397470 ip-10-0-141-139 kubenswrapper[2577]: E0911 17:53:17.397453    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9e753385e87a5146260e5c6d27c31ed309b8169f299075a46b0d2b6d079ed255\": container with ID starting with 9e753385e87a5146260e5c6d27c31ed309b8169f299075a46b0d2b6d079ed255 not found: ID does not exist" containerID="9e753385e87a5146260e5c6d27c31ed309b8169f299075a46b0d2b6d079ed255"
Sep 11 17:53:17.397562 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:53:17.397478    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9e753385e87a5146260e5c6d27c31ed309b8169f299075a46b0d2b6d079ed255"} err="failed to get container status \"9e753385e87a5146260e5c6d27c31ed309b8169f299075a46b0d2b6d079ed255\": rpc error: code = NotFound desc = could not find container \"9e753385e87a5146260e5c6d27c31ed309b8169f299075a46b0d2b6d079ed255\": container with ID starting with 9e753385e87a5146260e5c6d27c31ed309b8169f299075a46b0d2b6d079ed255 not found: ID does not exist"
Sep 11 17:53:17.406582 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:53:17.406540    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-ipp-a-027edd/ipp-route-027edd-kserve-7cd6f577c9-qqtlz"]
Sep 11 17:53:17.410224 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:53:17.410185    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-ipp-a-027edd/ipp-route-027edd-kserve-7cd6f577c9-qqtlz"]
Sep 11 17:53:18.927230 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:53:18.927190    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="654d7a14-7a93-44d0-903d-8af82e7864f5" path="/var/lib/kubelet/pods/654d7a14-7a93-44d0-903d-8af82e7864f5/volumes"
Sep 11 17:53:50.946689 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:53:50.946651    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-10bbea-d6wq8"]
Sep 11 17:53:50.947326 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:53:50.947112    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="654d7a14-7a93-44d0-903d-8af82e7864f5" containerName="main"
Sep 11 17:53:50.947326 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:53:50.947125    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="654d7a14-7a93-44d0-903d-8af82e7864f5" containerName="main"
Sep 11 17:53:50.947326 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:53:50.947153    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="80a0ba8b-1801-4ce9-a1e4-19800c7f7e4d" containerName="revoke-keys"
Sep 11 17:53:50.947326 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:53:50.947159    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="80a0ba8b-1801-4ce9-a1e4-19800c7f7e4d" containerName="revoke-keys"
Sep 11 17:53:50.947326 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:53:50.947229    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="80a0ba8b-1801-4ce9-a1e4-19800c7f7e4d" containerName="revoke-keys"
Sep 11 17:53:50.947326 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:53:50.947237    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="654d7a14-7a93-44d0-903d-8af82e7864f5" containerName="main"
Sep 11 17:53:50.950445 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:53:50.950426    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-10bbea-d6wq8"
Sep 11 17:53:50.952721 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:53:50.952700    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 11 17:53:50.953052 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:53:50.953039    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-9n8gf\""
Sep 11 17:53:50.959693 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:53:50.959664    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-10bbea-d6wq8"]
Sep 11 17:53:51.063221 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:53:51.063175    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/63a4c1cd-92ce-4ef3-9723-e57746b4e030-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-10bbea-d6wq8\" (UID: \"63a4c1cd-92ce-4ef3-9723-e57746b4e030\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-10bbea-d6wq8"
Sep 11 17:53:51.164300 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:53:51.164247    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/63a4c1cd-92ce-4ef3-9723-e57746b4e030-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-10bbea-d6wq8\" (UID: \"63a4c1cd-92ce-4ef3-9723-e57746b4e030\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-10bbea-d6wq8"
Sep 11 17:53:51.164982 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:53:51.164953    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/63a4c1cd-92ce-4ef3-9723-e57746b4e030-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-10bbea-d6wq8\" (UID: \"63a4c1cd-92ce-4ef3-9723-e57746b4e030\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-10bbea-d6wq8"
Sep 11 17:53:51.262960 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:53:51.262900    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-10bbea-d6wq8"
Sep 11 17:53:51.549210 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:53:51.549178    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-10bbea-d6wq8"]
Sep 11 17:53:51.576789 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:53:51.576754    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-10bbea-d6wq8" event={"ID":"63a4c1cd-92ce-4ef3-9723-e57746b4e030","Type":"ContainerStarted","Data":"3de1e428e131eb590fe2f88e391983924c696e555303ede9162a63f7a570968f"}
Sep 11 17:53:52.584404 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:53:52.584367    2577 generic.go:356] "Generic (PLEG): container finished" podID="63a4c1cd-92ce-4ef3-9723-e57746b4e030" containerID="9de4b37650d4a2a865ac321b436dbd8e40876568e2db84fdf17469d95b40116a" exitCode=0
Sep 11 17:53:52.584896 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:53:52.584446    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-10bbea-d6wq8" event={"ID":"63a4c1cd-92ce-4ef3-9723-e57746b4e030","Type":"ContainerDied","Data":"9de4b37650d4a2a865ac321b436dbd8e40876568e2db84fdf17469d95b40116a"}
Sep 11 17:53:53.725844 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:53:53.725819    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-10bbea-d6wq8"
Sep 11 17:53:53.789567 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:53:53.789531    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/63a4c1cd-92ce-4ef3-9723-e57746b4e030-maas-api-service-ca\") pod \"63a4c1cd-92ce-4ef3-9723-e57746b4e030\" (UID: \"63a4c1cd-92ce-4ef3-9723-e57746b4e030\") "
Sep 11 17:53:53.789920 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:53:53.789891    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/63a4c1cd-92ce-4ef3-9723-e57746b4e030-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "63a4c1cd-92ce-4ef3-9723-e57746b4e030" (UID: "63a4c1cd-92ce-4ef3-9723-e57746b4e030"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 17:53:53.890478 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:53:53.890390    2577 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/63a4c1cd-92ce-4ef3-9723-e57746b4e030-maas-api-service-ca\") on node \"ip-10-0-141-139.ec2.internal\" DevicePath \"\""
Sep 11 17:53:54.597036 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:53:54.597001    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-10bbea-d6wq8"
Sep 11 17:53:54.597214 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:53:54.597009    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-10bbea-d6wq8" event={"ID":"63a4c1cd-92ce-4ef3-9723-e57746b4e030","Type":"ContainerDied","Data":"3de1e428e131eb590fe2f88e391983924c696e555303ede9162a63f7a570968f"}
Sep 11 17:53:54.597214 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:53:54.597129    2577 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3de1e428e131eb590fe2f88e391983924c696e555303ede9162a63f7a570968f"
Sep 11 17:54:01.482409 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:54:01.482370    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-10bbea-5c9b98f589-bxj2n"]
Sep 11 17:54:01.482961 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:54:01.482675    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-10bbea-5c9b98f589-bxj2n" podUID="948e1e6f-0b40-4057-b996-590541520ac1" containerName="payload-processing" containerID="cri-o://d05ced7f6b59271673d1dd21bb4196e27fd21622611b9aa02c0f57c91d06c7d7" gracePeriod=30
Sep 11 17:54:01.491287 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:54:01.491258    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-10bbea-c56d6578-ltjgg"]
Sep 11 17:54:01.491692 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:54:01.491575    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-10bbea-c56d6578-ltjgg" podUID="5ab31ffa-01d9-4749-a409-e2d17dcae79c" containerName="payload-pre-processing" containerID="cri-o://ec9e21ab4aa04779bfee74c2d99dd6bc508233b266c3e125a1ad0a798d27d61f" gracePeriod=30
Sep 11 17:54:01.644523 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:54:01.644471    2577 generic.go:356] "Generic (PLEG): container finished" podID="5ab31ffa-01d9-4749-a409-e2d17dcae79c" containerID="ec9e21ab4aa04779bfee74c2d99dd6bc508233b266c3e125a1ad0a798d27d61f" exitCode=0
Sep 11 17:54:01.644713 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:54:01.644630    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-10bbea-c56d6578-ltjgg" event={"ID":"5ab31ffa-01d9-4749-a409-e2d17dcae79c","Type":"ContainerDied","Data":"ec9e21ab4aa04779bfee74c2d99dd6bc508233b266c3e125a1ad0a798d27d61f"}
Sep 11 17:54:01.646527 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:54:01.646480    2577 generic.go:356] "Generic (PLEG): container finished" podID="948e1e6f-0b40-4057-b996-590541520ac1" containerID="d05ced7f6b59271673d1dd21bb4196e27fd21622611b9aa02c0f57c91d06c7d7" exitCode=0
Sep 11 17:54:01.646603 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:54:01.646587    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-10bbea-5c9b98f589-bxj2n" event={"ID":"948e1e6f-0b40-4057-b996-590541520ac1","Type":"ContainerDied","Data":"d05ced7f6b59271673d1dd21bb4196e27fd21622611b9aa02c0f57c91d06c7d7"}
Sep 11 17:54:01.740399 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:54:01.740304    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-10bbea-d6wq8"]
Sep 11 17:54:01.752281 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:54:01.752244    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-10bbea-d6wq8"]
Sep 11 17:54:01.776086 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:54:01.776058    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-10bbea-5c9b98f589-bxj2n"
Sep 11 17:54:01.779265 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:54:01.779239    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-10bbea-c56d6578-ltjgg"
Sep 11 17:54:01.866581 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:54:01.866541    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/948e1e6f-0b40-4057-b996-590541520ac1-plugins-config-volume\") pod \"948e1e6f-0b40-4057-b996-590541520ac1\" (UID: \"948e1e6f-0b40-4057-b996-590541520ac1\") "
Sep 11 17:54:01.866790 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:54:01.866656    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7hwcr\" (UniqueName: \"kubernetes.io/projected/5ab31ffa-01d9-4749-a409-e2d17dcae79c-kube-api-access-7hwcr\") pod \"5ab31ffa-01d9-4749-a409-e2d17dcae79c\" (UID: \"5ab31ffa-01d9-4749-a409-e2d17dcae79c\") "
Sep 11 17:54:01.866790 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:54:01.866736    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7x68b\" (UniqueName: \"kubernetes.io/projected/948e1e6f-0b40-4057-b996-590541520ac1-kube-api-access-7x68b\") pod \"948e1e6f-0b40-4057-b996-590541520ac1\" (UID: \"948e1e6f-0b40-4057-b996-590541520ac1\") "
Sep 11 17:54:01.866915 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:54:01.866802    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5ab31ffa-01d9-4749-a409-e2d17dcae79c-plugins-config-volume\") pod \"5ab31ffa-01d9-4749-a409-e2d17dcae79c\" (UID: \"5ab31ffa-01d9-4749-a409-e2d17dcae79c\") "
Sep 11 17:54:01.866991 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:54:01.866961    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/948e1e6f-0b40-4057-b996-590541520ac1-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "948e1e6f-0b40-4057-b996-590541520ac1" (UID: "948e1e6f-0b40-4057-b996-590541520ac1"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 17:54:01.867161 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:54:01.867135    2577 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/948e1e6f-0b40-4057-b996-590541520ac1-plugins-config-volume\") on node \"ip-10-0-141-139.ec2.internal\" DevicePath \"\""
Sep 11 17:54:01.867284 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:54:01.867166    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5ab31ffa-01d9-4749-a409-e2d17dcae79c-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "5ab31ffa-01d9-4749-a409-e2d17dcae79c" (UID: "5ab31ffa-01d9-4749-a409-e2d17dcae79c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 17:54:01.869094 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:54:01.869065    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/948e1e6f-0b40-4057-b996-590541520ac1-kube-api-access-7x68b" (OuterVolumeSpecName: "kube-api-access-7x68b") pod "948e1e6f-0b40-4057-b996-590541520ac1" (UID: "948e1e6f-0b40-4057-b996-590541520ac1"). InnerVolumeSpecName "kube-api-access-7x68b". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 17:54:01.869189 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:54:01.869141    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5ab31ffa-01d9-4749-a409-e2d17dcae79c-kube-api-access-7hwcr" (OuterVolumeSpecName: "kube-api-access-7hwcr") pod "5ab31ffa-01d9-4749-a409-e2d17dcae79c" (UID: "5ab31ffa-01d9-4749-a409-e2d17dcae79c"). InnerVolumeSpecName "kube-api-access-7hwcr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 17:54:01.968341 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:54:01.968300    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7hwcr\" (UniqueName: \"kubernetes.io/projected/5ab31ffa-01d9-4749-a409-e2d17dcae79c-kube-api-access-7hwcr\") on node \"ip-10-0-141-139.ec2.internal\" DevicePath \"\""
Sep 11 17:54:01.968341 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:54:01.968329    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7x68b\" (UniqueName: \"kubernetes.io/projected/948e1e6f-0b40-4057-b996-590541520ac1-kube-api-access-7x68b\") on node \"ip-10-0-141-139.ec2.internal\" DevicePath \"\""
Sep 11 17:54:01.968341 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:54:01.968340    2577 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5ab31ffa-01d9-4749-a409-e2d17dcae79c-plugins-config-volume\") on node \"ip-10-0-141-139.ec2.internal\" DevicePath \"\""
Sep 11 17:54:02.652891 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:54:02.652853    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-10bbea-c56d6578-ltjgg"
Sep 11 17:54:02.653375 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:54:02.652892    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-10bbea-c56d6578-ltjgg" event={"ID":"5ab31ffa-01d9-4749-a409-e2d17dcae79c","Type":"ContainerDied","Data":"9e7da8072b03d1eab8ac7d6dbf627e0f95c07c3b7158b2369d642cf6edb9c990"}
Sep 11 17:54:02.653375 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:54:02.652960    2577 scope.go:117] "RemoveContainer" containerID="ec9e21ab4aa04779bfee74c2d99dd6bc508233b266c3e125a1ad0a798d27d61f"
Sep 11 17:54:02.654591 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:54:02.654563    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-10bbea-5c9b98f589-bxj2n"
Sep 11 17:54:02.654691 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:54:02.654563    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-10bbea-5c9b98f589-bxj2n" event={"ID":"948e1e6f-0b40-4057-b996-590541520ac1","Type":"ContainerDied","Data":"7c334f09e58613bdca905a96fd58680ff5b2ac8a625044397e6570cf53e3305a"}
Sep 11 17:54:02.663061 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:54:02.663037    2577 scope.go:117] "RemoveContainer" containerID="d05ced7f6b59271673d1dd21bb4196e27fd21622611b9aa02c0f57c91d06c7d7"
Sep 11 17:54:02.691927 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:54:02.691887    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-10bbea-c56d6578-ltjgg"]
Sep 11 17:54:02.702682 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:54:02.702631    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-10bbea-c56d6578-ltjgg"]
Sep 11 17:54:02.711554 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:54:02.711486    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-10bbea-5c9b98f589-bxj2n"]
Sep 11 17:54:02.714312 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:54:02.714285    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-10bbea-5c9b98f589-bxj2n"]
Sep 11 17:54:02.932740 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:54:02.932653    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5ab31ffa-01d9-4749-a409-e2d17dcae79c" path="/var/lib/kubelet/pods/5ab31ffa-01d9-4749-a409-e2d17dcae79c/volumes"
Sep 11 17:54:02.933004 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:54:02.932991    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="63a4c1cd-92ce-4ef3-9723-e57746b4e030" path="/var/lib/kubelet/pods/63a4c1cd-92ce-4ef3-9723-e57746b4e030/volumes"
Sep 11 17:54:02.933309 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:54:02.933297    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="948e1e6f-0b40-4057-b996-590541520ac1" path="/var/lib/kubelet/pods/948e1e6f-0b40-4057-b996-590541520ac1/volumes"
Sep 11 17:54:04.899272 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:54:04.899237    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-nkmkm"]
Sep 11 17:54:08.945276 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:54:08.945241    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-2fac54-w4qrc"]
Sep 11 17:54:08.945718 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:54:08.945666    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="948e1e6f-0b40-4057-b996-590541520ac1" containerName="payload-processing"
Sep 11 17:54:08.945718 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:54:08.945679    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="948e1e6f-0b40-4057-b996-590541520ac1" containerName="payload-processing"
Sep 11 17:54:08.945718 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:54:08.945696    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="63a4c1cd-92ce-4ef3-9723-e57746b4e030" containerName="revoke-keys"
Sep 11 17:54:08.945718 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:54:08.945702    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="63a4c1cd-92ce-4ef3-9723-e57746b4e030" containerName="revoke-keys"
Sep 11 17:54:08.945718 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:54:08.945708    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5ab31ffa-01d9-4749-a409-e2d17dcae79c" containerName="payload-pre-processing"
Sep 11 17:54:08.945718 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:54:08.945714    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="5ab31ffa-01d9-4749-a409-e2d17dcae79c" containerName="payload-pre-processing"
Sep 11 17:54:08.945929 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:54:08.945779    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="63a4c1cd-92ce-4ef3-9723-e57746b4e030" containerName="revoke-keys"
Sep 11 17:54:08.945929 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:54:08.945786    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="948e1e6f-0b40-4057-b996-590541520ac1" containerName="payload-processing"
Sep 11 17:54:08.945929 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:54:08.945798    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="5ab31ffa-01d9-4749-a409-e2d17dcae79c" containerName="payload-pre-processing"
Sep 11 17:54:08.950662 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:54:08.950633    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-2fac54-w4qrc"
Sep 11 17:54:08.953100 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:54:08.953065    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 11 17:54:08.953238 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:54:08.953168    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-9n8gf\""
Sep 11 17:54:08.958110 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:54:08.958072    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-2fac54-w4qrc"]
Sep 11 17:54:09.037069 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:54:09.037024    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7ef174c7-4a64-49ae-b9b0-8cc5ca48a1aa-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-2fac54-w4qrc\" (UID: \"7ef174c7-4a64-49ae-b9b0-8cc5ca48a1aa\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-2fac54-w4qrc"
Sep 11 17:54:09.138096 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:54:09.138052    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7ef174c7-4a64-49ae-b9b0-8cc5ca48a1aa-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-2fac54-w4qrc\" (UID: \"7ef174c7-4a64-49ae-b9b0-8cc5ca48a1aa\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-2fac54-w4qrc"
Sep 11 17:54:09.138744 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:54:09.138723    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7ef174c7-4a64-49ae-b9b0-8cc5ca48a1aa-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-2fac54-w4qrc\" (UID: \"7ef174c7-4a64-49ae-b9b0-8cc5ca48a1aa\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-2fac54-w4qrc"
Sep 11 17:54:09.263313 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:54:09.263259    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-2fac54-w4qrc"
Sep 11 17:54:09.535740 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:54:09.535712    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-2fac54-w4qrc"]
Sep 11 17:54:09.538624 ip-10-0-141-139 kubenswrapper[2577]: W0911 17:54:09.538591    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7ef174c7_4a64_49ae_b9b0_8cc5ca48a1aa.slice/crio-ccd7ba48e006e5da00fda5795e54f3fa3638c0f965c7d489c62ec8536670b9af WatchSource:0}: Error finding container ccd7ba48e006e5da00fda5795e54f3fa3638c0f965c7d489c62ec8536670b9af: Status 404 returned error can't find the container with id ccd7ba48e006e5da00fda5795e54f3fa3638c0f965c7d489c62ec8536670b9af
Sep 11 17:54:09.698259 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:54:09.698215    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-2fac54-w4qrc" event={"ID":"7ef174c7-4a64-49ae-b9b0-8cc5ca48a1aa","Type":"ContainerStarted","Data":"ccd7ba48e006e5da00fda5795e54f3fa3638c0f965c7d489c62ec8536670b9af"}
Sep 11 17:54:10.705676 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:54:10.705583    2577 generic.go:356] "Generic (PLEG): container finished" podID="7ef174c7-4a64-49ae-b9b0-8cc5ca48a1aa" containerID="56e5b31353b6590d3f1e9049bef8d015223b6d717211158ddf0832ae96d484f4" exitCode=0
Sep 11 17:54:10.706033 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:54:10.705675    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-2fac54-w4qrc" event={"ID":"7ef174c7-4a64-49ae-b9b0-8cc5ca48a1aa","Type":"ContainerDied","Data":"56e5b31353b6590d3f1e9049bef8d015223b6d717211158ddf0832ae96d484f4"}
Sep 11 17:54:11.847117 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:54:11.847086    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-2fac54-w4qrc"
Sep 11 17:54:11.975443 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:54:11.975401    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7ef174c7-4a64-49ae-b9b0-8cc5ca48a1aa-maas-api-service-ca\") pod \"7ef174c7-4a64-49ae-b9b0-8cc5ca48a1aa\" (UID: \"7ef174c7-4a64-49ae-b9b0-8cc5ca48a1aa\") "
Sep 11 17:54:11.975880 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:54:11.975850    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7ef174c7-4a64-49ae-b9b0-8cc5ca48a1aa-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "7ef174c7-4a64-49ae-b9b0-8cc5ca48a1aa" (UID: "7ef174c7-4a64-49ae-b9b0-8cc5ca48a1aa"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 17:54:12.076458 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:54:12.076418    2577 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7ef174c7-4a64-49ae-b9b0-8cc5ca48a1aa-maas-api-service-ca\") on node \"ip-10-0-141-139.ec2.internal\" DevicePath \"\""
Sep 11 17:54:12.720664 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:54:12.720631    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-2fac54-w4qrc"
Sep 11 17:54:12.720845 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:54:12.720631    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-2fac54-w4qrc" event={"ID":"7ef174c7-4a64-49ae-b9b0-8cc5ca48a1aa","Type":"ContainerDied","Data":"ccd7ba48e006e5da00fda5795e54f3fa3638c0f965c7d489c62ec8536670b9af"}
Sep 11 17:54:12.720845 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:54:12.720761    2577 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ccd7ba48e006e5da00fda5795e54f3fa3638c0f965c7d489c62ec8536670b9af"
Sep 11 17:54:19.501219 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:54:19.501185    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-2fac54-5796b4b6c7-c29rj"]
Sep 11 17:54:19.503723 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:54:19.501403    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-2fac54-5796b4b6c7-c29rj" podUID="c5842af6-6103-4759-951c-0fb4717aff70" containerName="payload-processing" containerID="cri-o://e93da037698548f38d28624bad4e6de8af9dda61926acf27becf0099acf043af" gracePeriod=30
Sep 11 17:54:19.667948 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:54:19.667906    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-2fac54-w4qrc"]
Sep 11 17:54:19.671326 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:54:19.671278    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-2fac54-w4qrc"]
Sep 11 17:54:19.760324 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:54:19.760237    2577 generic.go:356] "Generic (PLEG): container finished" podID="c5842af6-6103-4759-951c-0fb4717aff70" containerID="e93da037698548f38d28624bad4e6de8af9dda61926acf27becf0099acf043af" exitCode=0
Sep 11 17:54:19.760474 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:54:19.760312    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-2fac54-5796b4b6c7-c29rj" event={"ID":"c5842af6-6103-4759-951c-0fb4717aff70","Type":"ContainerDied","Data":"e93da037698548f38d28624bad4e6de8af9dda61926acf27becf0099acf043af"}
Sep 11 17:54:19.760474 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:54:19.760366    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-2fac54-5796b4b6c7-c29rj" event={"ID":"c5842af6-6103-4759-951c-0fb4717aff70","Type":"ContainerDied","Data":"2afefeb2ed0de0a1a24ebcf3082d143f7010860610f7271b5124329d4a11208d"}
Sep 11 17:54:19.760474 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:54:19.760387    2577 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2afefeb2ed0de0a1a24ebcf3082d143f7010860610f7271b5124329d4a11208d"
Sep 11 17:54:19.767728 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:54:19.767707    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-2fac54-5796b4b6c7-c29rj"
Sep 11 17:54:19.846106 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:54:19.846071    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c5842af6-6103-4759-951c-0fb4717aff70-plugins-config-volume\") pod \"c5842af6-6103-4759-951c-0fb4717aff70\" (UID: \"c5842af6-6103-4759-951c-0fb4717aff70\") "
Sep 11 17:54:19.846310 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:54:19.846147    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hq2wf\" (UniqueName: \"kubernetes.io/projected/c5842af6-6103-4759-951c-0fb4717aff70-kube-api-access-hq2wf\") pod \"c5842af6-6103-4759-951c-0fb4717aff70\" (UID: \"c5842af6-6103-4759-951c-0fb4717aff70\") "
Sep 11 17:54:19.849006 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:54:19.846691    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c5842af6-6103-4759-951c-0fb4717aff70-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "c5842af6-6103-4759-951c-0fb4717aff70" (UID: "c5842af6-6103-4759-951c-0fb4717aff70"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 17:54:19.849006 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:54:19.846870    2577 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c5842af6-6103-4759-951c-0fb4717aff70-plugins-config-volume\") on node \"ip-10-0-141-139.ec2.internal\" DevicePath \"\""
Sep 11 17:54:19.854644 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:54:19.854612    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c5842af6-6103-4759-951c-0fb4717aff70-kube-api-access-hq2wf" (OuterVolumeSpecName: "kube-api-access-hq2wf") pod "c5842af6-6103-4759-951c-0fb4717aff70" (UID: "c5842af6-6103-4759-951c-0fb4717aff70"). InnerVolumeSpecName "kube-api-access-hq2wf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 17:54:19.947839 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:54:19.947801    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hq2wf\" (UniqueName: \"kubernetes.io/projected/c5842af6-6103-4759-951c-0fb4717aff70-kube-api-access-hq2wf\") on node \"ip-10-0-141-139.ec2.internal\" DevicePath \"\""
Sep 11 17:54:20.765617 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:54:20.765581    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-2fac54-5796b4b6c7-c29rj"
Sep 11 17:54:20.786652 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:54:20.786622    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-2fac54-5796b4b6c7-c29rj"]
Sep 11 17:54:20.790338 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:54:20.790312    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-2fac54-5796b4b6c7-c29rj"]
Sep 11 17:54:20.926291 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:54:20.926250    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7ef174c7-4a64-49ae-b9b0-8cc5ca48a1aa" path="/var/lib/kubelet/pods/7ef174c7-4a64-49ae-b9b0-8cc5ca48a1aa/volumes"
Sep 11 17:54:20.926661 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:54:20.926647    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c5842af6-6103-4759-951c-0fb4717aff70" path="/var/lib/kubelet/pods/c5842af6-6103-4759-951c-0fb4717aff70/volumes"
Sep 11 17:54:22.790845 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:54:22.790807    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-nkmkm"]
Sep 11 17:54:26.695356 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:54:26.695319    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-6fc9e6-kfhpk"]
Sep 11 17:54:26.695788 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:54:26.695755    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c5842af6-6103-4759-951c-0fb4717aff70" containerName="payload-processing"
Sep 11 17:54:26.695788 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:54:26.695768    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="c5842af6-6103-4759-951c-0fb4717aff70" containerName="payload-processing"
Sep 11 17:54:26.695788 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:54:26.695784    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7ef174c7-4a64-49ae-b9b0-8cc5ca48a1aa" containerName="revoke-keys"
Sep 11 17:54:26.695788 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:54:26.695789    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="7ef174c7-4a64-49ae-b9b0-8cc5ca48a1aa" containerName="revoke-keys"
Sep 11 17:54:26.695925 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:54:26.695869    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="c5842af6-6103-4759-951c-0fb4717aff70" containerName="payload-processing"
Sep 11 17:54:26.695925 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:54:26.695882    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="7ef174c7-4a64-49ae-b9b0-8cc5ca48a1aa" containerName="revoke-keys"
Sep 11 17:54:26.700158 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:54:26.700139    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-6fc9e6-kfhpk"
Sep 11 17:54:26.702465 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:54:26.702444    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-9n8gf\""
Sep 11 17:54:26.702620 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:54:26.702450    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 11 17:54:26.706262 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:54:26.706235    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-6fc9e6-kfhpk"]
Sep 11 17:54:26.807847 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:54:26.807808    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c61b03af-a7d2-409a-a170-d5288bf5f9bf-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-6fc9e6-kfhpk\" (UID: \"c61b03af-a7d2-409a-a170-d5288bf5f9bf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-6fc9e6-kfhpk"
Sep 11 17:54:26.909193 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:54:26.909150    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c61b03af-a7d2-409a-a170-d5288bf5f9bf-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-6fc9e6-kfhpk\" (UID: \"c61b03af-a7d2-409a-a170-d5288bf5f9bf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-6fc9e6-kfhpk"
Sep 11 17:54:26.909826 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:54:26.909799    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c61b03af-a7d2-409a-a170-d5288bf5f9bf-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-6fc9e6-kfhpk\" (UID: \"c61b03af-a7d2-409a-a170-d5288bf5f9bf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-6fc9e6-kfhpk"
Sep 11 17:54:27.011834 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:54:27.011795    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-6fc9e6-kfhpk"
Sep 11 17:54:27.286178 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:54:27.286151    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-6fc9e6-kfhpk"]
Sep 11 17:54:27.288104 ip-10-0-141-139 kubenswrapper[2577]: W0911 17:54:27.288074    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc61b03af_a7d2_409a_a170_d5288bf5f9bf.slice/crio-3ffcef1509bf4768ba3e5cac37186dcf85d5754d4cc6a11021db226593f31b76 WatchSource:0}: Error finding container 3ffcef1509bf4768ba3e5cac37186dcf85d5754d4cc6a11021db226593f31b76: Status 404 returned error can't find the container with id 3ffcef1509bf4768ba3e5cac37186dcf85d5754d4cc6a11021db226593f31b76
Sep 11 17:54:27.803464 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:54:27.803363    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-6fc9e6-kfhpk" event={"ID":"c61b03af-a7d2-409a-a170-d5288bf5f9bf","Type":"ContainerStarted","Data":"ee52c930412e4a9f68d34ecc80fd4ae4d3c686b12590e244b561736313731fdd"}
Sep 11 17:54:27.803464 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:54:27.803433    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-6fc9e6-kfhpk" event={"ID":"c61b03af-a7d2-409a-a170-d5288bf5f9bf","Type":"ContainerStarted","Data":"3ffcef1509bf4768ba3e5cac37186dcf85d5754d4cc6a11021db226593f31b76"}
Sep 11 17:54:27.821948 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:54:27.821891    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-6fc9e6-kfhpk" podStartSLOduration=1.821874634 podStartE2EDuration="1.821874634s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 17:54:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 17:54:27.820370543 +0000 UTC m=+2145.393590656" watchObservedRunningTime="2026-09-11 17:54:27.821874634 +0000 UTC m=+2145.395094746"
Sep 11 17:54:28.810121 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:54:28.810085    2577 generic.go:356] "Generic (PLEG): container finished" podID="c61b03af-a7d2-409a-a170-d5288bf5f9bf" containerID="ee52c930412e4a9f68d34ecc80fd4ae4d3c686b12590e244b561736313731fdd" exitCode=0
Sep 11 17:54:28.810566 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:54:28.810139    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-6fc9e6-kfhpk" event={"ID":"c61b03af-a7d2-409a-a170-d5288bf5f9bf","Type":"ContainerDied","Data":"ee52c930412e4a9f68d34ecc80fd4ae4d3c686b12590e244b561736313731fdd"}
Sep 11 17:54:29.947147 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:54:29.947120    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-6fc9e6-kfhpk"
Sep 11 17:54:30.040458 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:54:30.040422    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c61b03af-a7d2-409a-a170-d5288bf5f9bf-maas-api-service-ca\") pod \"c61b03af-a7d2-409a-a170-d5288bf5f9bf\" (UID: \"c61b03af-a7d2-409a-a170-d5288bf5f9bf\") "
Sep 11 17:54:30.040892 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:54:30.040867    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c61b03af-a7d2-409a-a170-d5288bf5f9bf-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "c61b03af-a7d2-409a-a170-d5288bf5f9bf" (UID: "c61b03af-a7d2-409a-a170-d5288bf5f9bf"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 17:54:30.141732 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:54:30.141645    2577 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c61b03af-a7d2-409a-a170-d5288bf5f9bf-maas-api-service-ca\") on node \"ip-10-0-141-139.ec2.internal\" DevicePath \"\""
Sep 11 17:54:30.822400 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:54:30.822369    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-6fc9e6-kfhpk"
Sep 11 17:54:30.822593 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:54:30.822366    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-6fc9e6-kfhpk" event={"ID":"c61b03af-a7d2-409a-a170-d5288bf5f9bf","Type":"ContainerDied","Data":"3ffcef1509bf4768ba3e5cac37186dcf85d5754d4cc6a11021db226593f31b76"}
Sep 11 17:54:30.822593 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:54:30.822520    2577 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3ffcef1509bf4768ba3e5cac37186dcf85d5754d4cc6a11021db226593f31b76"
Sep 11 17:54:37.402310 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:54:37.402230    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-6fc9e6-6447f977cd-rrrsv"]
Sep 11 17:54:37.402875 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:54:37.402460    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-6fc9e6-6447f977cd-rrrsv" podUID="931af6e1-ab7b-46e8-83ab-ae9329e32a8f" containerName="payload-pre-processing" containerID="cri-o://df965416545f4f31401bed82fc54e6ca8b2c5d095b5382ee2f1e6fc7f5f6c9a0" gracePeriod=30
Sep 11 17:54:37.576193 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:54:37.576156    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-6fc9e6-kfhpk"]
Sep 11 17:54:37.587140 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:54:37.587069    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-6fc9e6-kfhpk"]
Sep 11 17:54:37.656961 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:54:37.656880    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-6fc9e6-6447f977cd-rrrsv"
Sep 11 17:54:37.712123 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:54:37.712089    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nd5ql\" (UniqueName: \"kubernetes.io/projected/931af6e1-ab7b-46e8-83ab-ae9329e32a8f-kube-api-access-nd5ql\") pod \"931af6e1-ab7b-46e8-83ab-ae9329e32a8f\" (UID: \"931af6e1-ab7b-46e8-83ab-ae9329e32a8f\") "
Sep 11 17:54:37.712123 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:54:37.712128    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/931af6e1-ab7b-46e8-83ab-ae9329e32a8f-plugins-config-volume\") pod \"931af6e1-ab7b-46e8-83ab-ae9329e32a8f\" (UID: \"931af6e1-ab7b-46e8-83ab-ae9329e32a8f\") "
Sep 11 17:54:37.712557 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:54:37.712531    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/931af6e1-ab7b-46e8-83ab-ae9329e32a8f-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "931af6e1-ab7b-46e8-83ab-ae9329e32a8f" (UID: "931af6e1-ab7b-46e8-83ab-ae9329e32a8f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 17:54:37.714412 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:54:37.714379    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/931af6e1-ab7b-46e8-83ab-ae9329e32a8f-kube-api-access-nd5ql" (OuterVolumeSpecName: "kube-api-access-nd5ql") pod "931af6e1-ab7b-46e8-83ab-ae9329e32a8f" (UID: "931af6e1-ab7b-46e8-83ab-ae9329e32a8f"). InnerVolumeSpecName "kube-api-access-nd5ql". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 17:54:37.813657 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:54:37.813623    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nd5ql\" (UniqueName: \"kubernetes.io/projected/931af6e1-ab7b-46e8-83ab-ae9329e32a8f-kube-api-access-nd5ql\") on node \"ip-10-0-141-139.ec2.internal\" DevicePath \"\""
Sep 11 17:54:37.813657 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:54:37.813653    2577 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/931af6e1-ab7b-46e8-83ab-ae9329e32a8f-plugins-config-volume\") on node \"ip-10-0-141-139.ec2.internal\" DevicePath \"\""
Sep 11 17:54:37.859699 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:54:37.859667    2577 generic.go:356] "Generic (PLEG): container finished" podID="931af6e1-ab7b-46e8-83ab-ae9329e32a8f" containerID="df965416545f4f31401bed82fc54e6ca8b2c5d095b5382ee2f1e6fc7f5f6c9a0" exitCode=0
Sep 11 17:54:37.859875 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:54:37.859717    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-6fc9e6-6447f977cd-rrrsv"
Sep 11 17:54:37.859875 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:54:37.859730    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-6fc9e6-6447f977cd-rrrsv" event={"ID":"931af6e1-ab7b-46e8-83ab-ae9329e32a8f","Type":"ContainerDied","Data":"df965416545f4f31401bed82fc54e6ca8b2c5d095b5382ee2f1e6fc7f5f6c9a0"}
Sep 11 17:54:37.859875 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:54:37.859797    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-6fc9e6-6447f977cd-rrrsv" event={"ID":"931af6e1-ab7b-46e8-83ab-ae9329e32a8f","Type":"ContainerDied","Data":"45e126b5f30ffaea090a1a3bcecd04173f3417356b79fd5f4d838b9ac5e5f60e"}
Sep 11 17:54:37.859875 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:54:37.859818    2577 scope.go:117] "RemoveContainer" containerID="df965416545f4f31401bed82fc54e6ca8b2c5d095b5382ee2f1e6fc7f5f6c9a0"
Sep 11 17:54:37.871903 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:54:37.871877    2577 scope.go:117] "RemoveContainer" containerID="df965416545f4f31401bed82fc54e6ca8b2c5d095b5382ee2f1e6fc7f5f6c9a0"
Sep 11 17:54:37.872234 ip-10-0-141-139 kubenswrapper[2577]: E0911 17:54:37.872214    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"df965416545f4f31401bed82fc54e6ca8b2c5d095b5382ee2f1e6fc7f5f6c9a0\": container with ID starting with df965416545f4f31401bed82fc54e6ca8b2c5d095b5382ee2f1e6fc7f5f6c9a0 not found: ID does not exist" containerID="df965416545f4f31401bed82fc54e6ca8b2c5d095b5382ee2f1e6fc7f5f6c9a0"
Sep 11 17:54:37.872285 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:54:37.872245    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"df965416545f4f31401bed82fc54e6ca8b2c5d095b5382ee2f1e6fc7f5f6c9a0"} err="failed to get container status \"df965416545f4f31401bed82fc54e6ca8b2c5d095b5382ee2f1e6fc7f5f6c9a0\": rpc error: code = NotFound desc = could not find container \"df965416545f4f31401bed82fc54e6ca8b2c5d095b5382ee2f1e6fc7f5f6c9a0\": container with ID starting with df965416545f4f31401bed82fc54e6ca8b2c5d095b5382ee2f1e6fc7f5f6c9a0 not found: ID does not exist"
Sep 11 17:54:37.904779 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:54:37.904741    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-6fc9e6-6447f977cd-rrrsv"]
Sep 11 17:54:37.909035 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:54:37.908963    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-6fc9e6-6447f977cd-rrrsv"]
Sep 11 17:54:38.927105 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:54:38.927058    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="931af6e1-ab7b-46e8-83ab-ae9329e32a8f" path="/var/lib/kubelet/pods/931af6e1-ab7b-46e8-83ab-ae9329e32a8f/volumes"
Sep 11 17:54:38.927620 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:54:38.927577    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c61b03af-a7d2-409a-a170-d5288bf5f9bf" path="/var/lib/kubelet/pods/c61b03af-a7d2-409a-a170-d5288bf5f9bf/volumes"
Sep 11 17:54:40.142565 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:54:40.142524    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-nkmkm"]
Sep 11 17:54:43.472866 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:54:43.472832    2577 scope.go:117] "RemoveContainer" containerID="e93da037698548f38d28624bad4e6de8af9dda61926acf27becf0099acf043af"
Sep 11 17:54:51.349249 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:54:51.349209    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-nkmkm"]
Sep 11 17:56:11.793677 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:56:11.793632    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-nkmkm"]
Sep 11 17:56:38.808711 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:56:38.808671    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-nkmkm"]
Sep 11 17:56:47.104035 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:56:47.104000    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-nkmkm"]
Sep 11 17:56:54.991036 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:56:54.990996    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-nkmkm"]
Sep 11 17:57:08.992570 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:57:08.992527    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-nkmkm"]
Sep 11 17:57:14.899662 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:57:14.899619    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-nkmkm"]
Sep 11 17:58:41.815950 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:58:41.815910    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-nkmkm"]
Sep 11 17:58:43.532362 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:58:43.532330    2577 scope.go:117] "RemoveContainer" containerID="262e6d21eeba3ff9db80a6ce47049f1e708d114203fa39bfa2c9f39ae7c07bd0"
Sep 11 17:58:43.541103 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:58:43.541080    2577 scope.go:117] "RemoveContainer" containerID="472c2bf8a41441e04af93eb7785668c57e964284892b5417d0961f4166f44c0a"
Sep 11 17:58:50.498001 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:58:50.497959    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-nkmkm"]
Sep 11 17:58:55.119986 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:58:55.119950    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-same-id-a-31388ae6-kserve-97bdc6947-v65zt"]
Sep 11 17:58:55.120484 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:58:55.120432    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c61b03af-a7d2-409a-a170-d5288bf5f9bf" containerName="revoke-keys"
Sep 11 17:58:55.120484 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:58:55.120447    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="c61b03af-a7d2-409a-a170-d5288bf5f9bf" containerName="revoke-keys"
Sep 11 17:58:55.120484 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:58:55.120469    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="931af6e1-ab7b-46e8-83ab-ae9329e32a8f" containerName="payload-pre-processing"
Sep 11 17:58:55.120484 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:58:55.120475    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="931af6e1-ab7b-46e8-83ab-ae9329e32a8f" containerName="payload-pre-processing"
Sep 11 17:58:55.120682 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:58:55.120561    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="931af6e1-ab7b-46e8-83ab-ae9329e32a8f" containerName="payload-pre-processing"
Sep 11 17:58:55.120682 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:58:55.120576    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="c61b03af-a7d2-409a-a170-d5288bf5f9bf" containerName="revoke-keys"
Sep 11 17:58:55.123770 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:58:55.123746    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-31388ae6-kserve-97bdc6947-v65zt"
Sep 11 17:58:55.127051 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:58:55.127027    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-same-id-a-31388ae6-kserve-self-signed-certs\""
Sep 11 17:58:55.135877 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:58:55.135847    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-a-31388ae6-kserve-97bdc6947-v65zt"]
Sep 11 17:58:55.220928 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:58:55.220891    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w6276\" (UniqueName: \"kubernetes.io/projected/4095004f-6853-49a8-b07e-1986d2e68f8f-kube-api-access-w6276\") pod \"e2e-same-id-a-31388ae6-kserve-97bdc6947-v65zt\" (UID: \"4095004f-6853-49a8-b07e-1986d2e68f8f\") " pod="llm/e2e-same-id-a-31388ae6-kserve-97bdc6947-v65zt"
Sep 11 17:58:55.221112 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:58:55.220942    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4095004f-6853-49a8-b07e-1986d2e68f8f-dshm\") pod \"e2e-same-id-a-31388ae6-kserve-97bdc6947-v65zt\" (UID: \"4095004f-6853-49a8-b07e-1986d2e68f8f\") " pod="llm/e2e-same-id-a-31388ae6-kserve-97bdc6947-v65zt"
Sep 11 17:58:55.221112 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:58:55.221075    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4095004f-6853-49a8-b07e-1986d2e68f8f-tmp-dir\") pod \"e2e-same-id-a-31388ae6-kserve-97bdc6947-v65zt\" (UID: \"4095004f-6853-49a8-b07e-1986d2e68f8f\") " pod="llm/e2e-same-id-a-31388ae6-kserve-97bdc6947-v65zt"
Sep 11 17:58:55.221112 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:58:55.221107    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4095004f-6853-49a8-b07e-1986d2e68f8f-home\") pod \"e2e-same-id-a-31388ae6-kserve-97bdc6947-v65zt\" (UID: \"4095004f-6853-49a8-b07e-1986d2e68f8f\") " pod="llm/e2e-same-id-a-31388ae6-kserve-97bdc6947-v65zt"
Sep 11 17:58:55.221225 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:58:55.221133    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4095004f-6853-49a8-b07e-1986d2e68f8f-model-cache\") pod \"e2e-same-id-a-31388ae6-kserve-97bdc6947-v65zt\" (UID: \"4095004f-6853-49a8-b07e-1986d2e68f8f\") " pod="llm/e2e-same-id-a-31388ae6-kserve-97bdc6947-v65zt"
Sep 11 17:58:55.221225 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:58:55.221210    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4095004f-6853-49a8-b07e-1986d2e68f8f-tls-certs\") pod \"e2e-same-id-a-31388ae6-kserve-97bdc6947-v65zt\" (UID: \"4095004f-6853-49a8-b07e-1986d2e68f8f\") " pod="llm/e2e-same-id-a-31388ae6-kserve-97bdc6947-v65zt"
Sep 11 17:58:55.322678 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:58:55.322626    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4095004f-6853-49a8-b07e-1986d2e68f8f-tmp-dir\") pod \"e2e-same-id-a-31388ae6-kserve-97bdc6947-v65zt\" (UID: \"4095004f-6853-49a8-b07e-1986d2e68f8f\") " pod="llm/e2e-same-id-a-31388ae6-kserve-97bdc6947-v65zt"
Sep 11 17:58:55.322678 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:58:55.322676    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4095004f-6853-49a8-b07e-1986d2e68f8f-home\") pod \"e2e-same-id-a-31388ae6-kserve-97bdc6947-v65zt\" (UID: \"4095004f-6853-49a8-b07e-1986d2e68f8f\") " pod="llm/e2e-same-id-a-31388ae6-kserve-97bdc6947-v65zt"
Sep 11 17:58:55.323004 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:58:55.322718    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4095004f-6853-49a8-b07e-1986d2e68f8f-model-cache\") pod \"e2e-same-id-a-31388ae6-kserve-97bdc6947-v65zt\" (UID: \"4095004f-6853-49a8-b07e-1986d2e68f8f\") " pod="llm/e2e-same-id-a-31388ae6-kserve-97bdc6947-v65zt"
Sep 11 17:58:55.323004 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:58:55.322747    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4095004f-6853-49a8-b07e-1986d2e68f8f-tls-certs\") pod \"e2e-same-id-a-31388ae6-kserve-97bdc6947-v65zt\" (UID: \"4095004f-6853-49a8-b07e-1986d2e68f8f\") " pod="llm/e2e-same-id-a-31388ae6-kserve-97bdc6947-v65zt"
Sep 11 17:58:55.323004 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:58:55.322774    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w6276\" (UniqueName: \"kubernetes.io/projected/4095004f-6853-49a8-b07e-1986d2e68f8f-kube-api-access-w6276\") pod \"e2e-same-id-a-31388ae6-kserve-97bdc6947-v65zt\" (UID: \"4095004f-6853-49a8-b07e-1986d2e68f8f\") " pod="llm/e2e-same-id-a-31388ae6-kserve-97bdc6947-v65zt"
Sep 11 17:58:55.323004 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:58:55.322809    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4095004f-6853-49a8-b07e-1986d2e68f8f-dshm\") pod \"e2e-same-id-a-31388ae6-kserve-97bdc6947-v65zt\" (UID: \"4095004f-6853-49a8-b07e-1986d2e68f8f\") " pod="llm/e2e-same-id-a-31388ae6-kserve-97bdc6947-v65zt"
Sep 11 17:58:55.323232 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:58:55.323188    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4095004f-6853-49a8-b07e-1986d2e68f8f-tmp-dir\") pod \"e2e-same-id-a-31388ae6-kserve-97bdc6947-v65zt\" (UID: \"4095004f-6853-49a8-b07e-1986d2e68f8f\") " pod="llm/e2e-same-id-a-31388ae6-kserve-97bdc6947-v65zt"
Sep 11 17:58:55.323232 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:58:55.323216    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4095004f-6853-49a8-b07e-1986d2e68f8f-home\") pod \"e2e-same-id-a-31388ae6-kserve-97bdc6947-v65zt\" (UID: \"4095004f-6853-49a8-b07e-1986d2e68f8f\") " pod="llm/e2e-same-id-a-31388ae6-kserve-97bdc6947-v65zt"
Sep 11 17:58:55.323321 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:58:55.323245    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4095004f-6853-49a8-b07e-1986d2e68f8f-model-cache\") pod \"e2e-same-id-a-31388ae6-kserve-97bdc6947-v65zt\" (UID: \"4095004f-6853-49a8-b07e-1986d2e68f8f\") " pod="llm/e2e-same-id-a-31388ae6-kserve-97bdc6947-v65zt"
Sep 11 17:58:55.325700 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:58:55.325674    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4095004f-6853-49a8-b07e-1986d2e68f8f-dshm\") pod \"e2e-same-id-a-31388ae6-kserve-97bdc6947-v65zt\" (UID: \"4095004f-6853-49a8-b07e-1986d2e68f8f\") " pod="llm/e2e-same-id-a-31388ae6-kserve-97bdc6947-v65zt"
Sep 11 17:58:55.325867 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:58:55.325847    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4095004f-6853-49a8-b07e-1986d2e68f8f-tls-certs\") pod \"e2e-same-id-a-31388ae6-kserve-97bdc6947-v65zt\" (UID: \"4095004f-6853-49a8-b07e-1986d2e68f8f\") " pod="llm/e2e-same-id-a-31388ae6-kserve-97bdc6947-v65zt"
Sep 11 17:58:55.332087 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:58:55.332059    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-w6276\" (UniqueName: \"kubernetes.io/projected/4095004f-6853-49a8-b07e-1986d2e68f8f-kube-api-access-w6276\") pod \"e2e-same-id-a-31388ae6-kserve-97bdc6947-v65zt\" (UID: \"4095004f-6853-49a8-b07e-1986d2e68f8f\") " pod="llm/e2e-same-id-a-31388ae6-kserve-97bdc6947-v65zt"
Sep 11 17:58:55.434730 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:58:55.434629    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-31388ae6-kserve-97bdc6947-v65zt"
Sep 11 17:58:55.711492 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:58:55.711402    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-a-31388ae6-kserve-97bdc6947-v65zt"]
Sep 11 17:58:55.714686 ip-10-0-141-139 kubenswrapper[2577]: W0911 17:58:55.714656    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod4095004f_6853_49a8_b07e_1986d2e68f8f.slice/crio-062fd262b4135a0d51c8571ed3cefc255476aa9d1d243ef97cc6173432b6c4e8 WatchSource:0}: Error finding container 062fd262b4135a0d51c8571ed3cefc255476aa9d1d243ef97cc6173432b6c4e8: Status 404 returned error can't find the container with id 062fd262b4135a0d51c8571ed3cefc255476aa9d1d243ef97cc6173432b6c4e8
Sep 11 17:58:56.211367 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:58:56.211323    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-31388ae6-kserve-97bdc6947-v65zt" event={"ID":"4095004f-6853-49a8-b07e-1986d2e68f8f","Type":"ContainerStarted","Data":"a7cb59dd0d7a1ea1a95996a843ddf8f763f2800b50798630ee91eb68a5c25cf1"}
Sep 11 17:58:56.211367 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:58:56.211381    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-31388ae6-kserve-97bdc6947-v65zt" event={"ID":"4095004f-6853-49a8-b07e-1986d2e68f8f","Type":"ContainerStarted","Data":"062fd262b4135a0d51c8571ed3cefc255476aa9d1d243ef97cc6173432b6c4e8"}
Sep 11 17:58:56.229996 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:58:56.229942    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-same-id-a-31388ae6-kserve-97bdc6947-v65zt" podStartSLOduration=1.229920506 podStartE2EDuration="1.229920506s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 17:58:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 17:58:56.229572989 +0000 UTC m=+2413.802793098" watchObservedRunningTime="2026-09-11 17:58:56.229920506 +0000 UTC m=+2413.803140620"
Sep 11 17:58:58.112642 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:58:58.112604    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-nkmkm"]
Sep 11 17:59:05.435138 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:59:05.435053    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-same-id-a-31388ae6-kserve-97bdc6947-v65zt"
Sep 11 17:59:05.435138 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:59:05.435107    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-same-id-a-31388ae6-kserve-97bdc6947-v65zt"
Sep 11 17:59:05.447622 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:59:05.447598    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-same-id-a-31388ae6-kserve-97bdc6947-v65zt"
Sep 11 17:59:06.270436 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:59:06.270409    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-same-id-a-31388ae6-kserve-97bdc6947-v65zt"
Sep 11 17:59:14.497170 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:59:14.497134    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-nkmkm"]
Sep 11 17:59:46.365450 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:59:46.365414    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-a-31388ae6-kserve-97bdc6947-v65zt"]
Sep 11 17:59:48.608715 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:59:48.608676    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-nkmkm"]
Sep 11 17:59:57.600877 ip-10-0-141-139 kubenswrapper[2577]: I0911 17:59:57.600837    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-nkmkm"]
Sep 11 18:00:01.385401 ip-10-0-141-139 kubenswrapper[2577]: I0911 18:00:01.385358    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="llm/e2e-same-id-a-31388ae6-kserve-97bdc6947-v65zt" podUID="4095004f-6853-49a8-b07e-1986d2e68f8f" containerName="main" containerID="cri-o://a7cb59dd0d7a1ea1a95996a843ddf8f763f2800b50798630ee91eb68a5c25cf1" gracePeriod=45
Sep 11 18:00:01.545762 ip-10-0-141-139 kubenswrapper[2577]: I0911 18:00:01.545722    2577 generic.go:356] "Generic (PLEG): container finished" podID="4095004f-6853-49a8-b07e-1986d2e68f8f" containerID="a7cb59dd0d7a1ea1a95996a843ddf8f763f2800b50798630ee91eb68a5c25cf1" exitCode=0
Sep 11 18:00:01.545944 ip-10-0-141-139 kubenswrapper[2577]: I0911 18:00:01.545777    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-31388ae6-kserve-97bdc6947-v65zt" event={"ID":"4095004f-6853-49a8-b07e-1986d2e68f8f","Type":"ContainerDied","Data":"a7cb59dd0d7a1ea1a95996a843ddf8f763f2800b50798630ee91eb68a5c25cf1"}
Sep 11 18:00:01.645266 ip-10-0-141-139 kubenswrapper[2577]: I0911 18:00:01.645197    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-31388ae6-kserve-97bdc6947-v65zt"
Sep 11 18:00:01.688069 ip-10-0-141-139 kubenswrapper[2577]: I0911 18:00:01.688038    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4095004f-6853-49a8-b07e-1986d2e68f8f-dshm\") pod \"4095004f-6853-49a8-b07e-1986d2e68f8f\" (UID: \"4095004f-6853-49a8-b07e-1986d2e68f8f\") "
Sep 11 18:00:01.688256 ip-10-0-141-139 kubenswrapper[2577]: I0911 18:00:01.688153    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4095004f-6853-49a8-b07e-1986d2e68f8f-tmp-dir\") pod \"4095004f-6853-49a8-b07e-1986d2e68f8f\" (UID: \"4095004f-6853-49a8-b07e-1986d2e68f8f\") "
Sep 11 18:00:01.688256 ip-10-0-141-139 kubenswrapper[2577]: I0911 18:00:01.688186    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4095004f-6853-49a8-b07e-1986d2e68f8f-home\") pod \"4095004f-6853-49a8-b07e-1986d2e68f8f\" (UID: \"4095004f-6853-49a8-b07e-1986d2e68f8f\") "
Sep 11 18:00:01.688256 ip-10-0-141-139 kubenswrapper[2577]: I0911 18:00:01.688246    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4095004f-6853-49a8-b07e-1986d2e68f8f-tls-certs\") pod \"4095004f-6853-49a8-b07e-1986d2e68f8f\" (UID: \"4095004f-6853-49a8-b07e-1986d2e68f8f\") "
Sep 11 18:00:01.688416 ip-10-0-141-139 kubenswrapper[2577]: I0911 18:00:01.688282    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4095004f-6853-49a8-b07e-1986d2e68f8f-model-cache\") pod \"4095004f-6853-49a8-b07e-1986d2e68f8f\" (UID: \"4095004f-6853-49a8-b07e-1986d2e68f8f\") "
Sep 11 18:00:01.688416 ip-10-0-141-139 kubenswrapper[2577]: I0911 18:00:01.688320    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-w6276\" (UniqueName: \"kubernetes.io/projected/4095004f-6853-49a8-b07e-1986d2e68f8f-kube-api-access-w6276\") pod \"4095004f-6853-49a8-b07e-1986d2e68f8f\" (UID: \"4095004f-6853-49a8-b07e-1986d2e68f8f\") "
Sep 11 18:00:01.688540 ip-10-0-141-139 kubenswrapper[2577]: I0911 18:00:01.688435    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4095004f-6853-49a8-b07e-1986d2e68f8f-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "4095004f-6853-49a8-b07e-1986d2e68f8f" (UID: "4095004f-6853-49a8-b07e-1986d2e68f8f"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 18:00:01.688540 ip-10-0-141-139 kubenswrapper[2577]: I0911 18:00:01.688474    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4095004f-6853-49a8-b07e-1986d2e68f8f-home" (OuterVolumeSpecName: "home") pod "4095004f-6853-49a8-b07e-1986d2e68f8f" (UID: "4095004f-6853-49a8-b07e-1986d2e68f8f"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 18:00:01.688659 ip-10-0-141-139 kubenswrapper[2577]: I0911 18:00:01.688588    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4095004f-6853-49a8-b07e-1986d2e68f8f-model-cache" (OuterVolumeSpecName: "model-cache") pod "4095004f-6853-49a8-b07e-1986d2e68f8f" (UID: "4095004f-6853-49a8-b07e-1986d2e68f8f"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 18:00:01.689087 ip-10-0-141-139 kubenswrapper[2577]: I0911 18:00:01.688988    2577 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4095004f-6853-49a8-b07e-1986d2e68f8f-model-cache\") on node \"ip-10-0-141-139.ec2.internal\" DevicePath \"\""
Sep 11 18:00:01.689087 ip-10-0-141-139 kubenswrapper[2577]: I0911 18:00:01.689028    2577 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4095004f-6853-49a8-b07e-1986d2e68f8f-tmp-dir\") on node \"ip-10-0-141-139.ec2.internal\" DevicePath \"\""
Sep 11 18:00:01.689087 ip-10-0-141-139 kubenswrapper[2577]: I0911 18:00:01.689042    2577 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4095004f-6853-49a8-b07e-1986d2e68f8f-home\") on node \"ip-10-0-141-139.ec2.internal\" DevicePath \"\""
Sep 11 18:00:01.690680 ip-10-0-141-139 kubenswrapper[2577]: I0911 18:00:01.690651    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4095004f-6853-49a8-b07e-1986d2e68f8f-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "4095004f-6853-49a8-b07e-1986d2e68f8f" (UID: "4095004f-6853-49a8-b07e-1986d2e68f8f"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 18:00:01.690808 ip-10-0-141-139 kubenswrapper[2577]: I0911 18:00:01.690728    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4095004f-6853-49a8-b07e-1986d2e68f8f-kube-api-access-w6276" (OuterVolumeSpecName: "kube-api-access-w6276") pod "4095004f-6853-49a8-b07e-1986d2e68f8f" (UID: "4095004f-6853-49a8-b07e-1986d2e68f8f"). InnerVolumeSpecName "kube-api-access-w6276". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 18:00:01.690808 ip-10-0-141-139 kubenswrapper[2577]: I0911 18:00:01.690763    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4095004f-6853-49a8-b07e-1986d2e68f8f-dshm" (OuterVolumeSpecName: "dshm") pod "4095004f-6853-49a8-b07e-1986d2e68f8f" (UID: "4095004f-6853-49a8-b07e-1986d2e68f8f"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 18:00:01.790277 ip-10-0-141-139 kubenswrapper[2577]: I0911 18:00:01.790237    2577 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4095004f-6853-49a8-b07e-1986d2e68f8f-tls-certs\") on node \"ip-10-0-141-139.ec2.internal\" DevicePath \"\""
Sep 11 18:00:01.790277 ip-10-0-141-139 kubenswrapper[2577]: I0911 18:00:01.790270    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w6276\" (UniqueName: \"kubernetes.io/projected/4095004f-6853-49a8-b07e-1986d2e68f8f-kube-api-access-w6276\") on node \"ip-10-0-141-139.ec2.internal\" DevicePath \"\""
Sep 11 18:00:01.790277 ip-10-0-141-139 kubenswrapper[2577]: I0911 18:00:01.790283    2577 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4095004f-6853-49a8-b07e-1986d2e68f8f-dshm\") on node \"ip-10-0-141-139.ec2.internal\" DevicePath \"\""
Sep 11 18:00:02.554963 ip-10-0-141-139 kubenswrapper[2577]: I0911 18:00:02.554931    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-31388ae6-kserve-97bdc6947-v65zt" event={"ID":"4095004f-6853-49a8-b07e-1986d2e68f8f","Type":"ContainerDied","Data":"062fd262b4135a0d51c8571ed3cefc255476aa9d1d243ef97cc6173432b6c4e8"}
Sep 11 18:00:02.554963 ip-10-0-141-139 kubenswrapper[2577]: I0911 18:00:02.554953    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-31388ae6-kserve-97bdc6947-v65zt"
Sep 11 18:00:02.555598 ip-10-0-141-139 kubenswrapper[2577]: I0911 18:00:02.554976    2577 scope.go:117] "RemoveContainer" containerID="a7cb59dd0d7a1ea1a95996a843ddf8f763f2800b50798630ee91eb68a5c25cf1"
Sep 11 18:00:02.580605 ip-10-0-141-139 kubenswrapper[2577]: I0911 18:00:02.580574    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-a-31388ae6-kserve-97bdc6947-v65zt"]
Sep 11 18:00:02.584831 ip-10-0-141-139 kubenswrapper[2577]: I0911 18:00:02.584805    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["llm/e2e-same-id-a-31388ae6-kserve-97bdc6947-v65zt"]
Sep 11 18:00:02.926523 ip-10-0-141-139 kubenswrapper[2577]: I0911 18:00:02.926419    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4095004f-6853-49a8-b07e-1986d2e68f8f" path="/var/lib/kubelet/pods/4095004f-6853-49a8-b07e-1986d2e68f8f/volumes"
Sep 11 18:00:05.149377 ip-10-0-141-139 kubenswrapper[2577]: I0911 18:00:05.149343    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-nkmkm"]
Sep 11 18:00:11.497634 ip-10-0-141-139 kubenswrapper[2577]: I0911 18:00:11.497589    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-nkmkm"]
Sep 11 18:00:19.024844 ip-10-0-141-139 kubenswrapper[2577]: I0911 18:00:19.024801    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-nkmkm"]
Sep 11 18:00:26.398358 ip-10-0-141-139 kubenswrapper[2577]: I0911 18:00:26.398323    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-nkmkm"]
Sep 11 18:00:34.495315 ip-10-0-141-139 kubenswrapper[2577]: I0911 18:00:34.495231    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-nkmkm"]
Sep 11 18:00:40.202440 ip-10-0-141-139 kubenswrapper[2577]: I0911 18:00:40.202406    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-nkmkm"]
Sep 11 18:00:43.575457 ip-10-0-141-139 kubenswrapper[2577]: I0911 18:00:43.575420    2577 scope.go:117] "RemoveContainer" containerID="56e5b31353b6590d3f1e9049bef8d015223b6d717211158ddf0832ae96d484f4"
Sep 11 18:00:43.584041 ip-10-0-141-139 kubenswrapper[2577]: I0911 18:00:43.584023    2577 scope.go:117] "RemoveContainer" containerID="9de4b37650d4a2a865ac321b436dbd8e40876568e2db84fdf17469d95b40116a"
Sep 11 18:00:43.591752 ip-10-0-141-139 kubenswrapper[2577]: I0911 18:00:43.591731    2577 scope.go:117] "RemoveContainer" containerID="ee52c930412e4a9f68d34ecc80fd4ae4d3c686b12590e244b561736313731fdd"
Sep 11 18:00:49.041663 ip-10-0-141-139 kubenswrapper[2577]: I0911 18:00:49.041633    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-b887fdd9f-jc4h9_65106181-ae0e-493b-b7d7-b6707f6f1bab/copy-manifests/0.log"
Sep 11 18:00:49.051481 ip-10-0-141-139 kubenswrapper[2577]: I0911 18:00:49.051455    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-b887fdd9f-jc4h9_65106181-ae0e-493b-b7d7-b6707f6f1bab/manager/0.log"
Sep 11 18:00:49.938950 ip-10-0-141-139 kubenswrapper[2577]: I0911 18:00:49.938915    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-key-cleanup-29819145-k7xm4_ce07d5c0-d2f5-40b5-a2d7-1cf5ddf1242d/cleanup/0.log"
Sep 11 18:00:51.476090 ip-10-0-141-139 kubenswrapper[2577]: I0911 18:00:51.476063    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-fbdd4f445-z6f86_972596b5-0a5e-4145-be4e-dfb8cbda2dfe/kuadrant-console-plugin/0.log"
Sep 11 18:00:51.744333 ip-10-0-141-139 kubenswrapper[2577]: I0911 18:00:51.744252    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-84994f7ffc-nkmkm_a0d4403a-011b-4bf2-ae88-a27c8ee0fa86/limitador/0.log"
Sep 11 18:00:52.572811 ip-10-0-141-139 kubenswrapper[2577]: I0911 18:00:52.572781    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-7d6df6ff46-hkdnw_78758cdb-2c93-44c5-9039-ff181899b15f/kube-auth-proxy/0.log"
Sep 11 18:00:52.919641 ip-10-0-141-139 kubenswrapper[2577]: I0911 18:00:52.919559    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-59fbc47bf8-r7l5l_c779770c-0017-4b67-bf14-415f9ec89cc2/payload-pre-processing/0.log"
Sep 11 18:00:53.520430 ip-10-0-141-139 kubenswrapper[2577]: I0911 18:00:53.520396    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-2-simulated-kserve-67d8d5d6d-jrgvr_53d4883a-1f41-4db0-a78d-fc4d2fcf1857/main/0.log"
Sep 11 18:00:53.884494 ip-10-0-141-139 kubenswrapper[2577]: I0911 18:00:53.884418    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-trlp-test-simulated-kserve-598466548d-6vpzl_334bafcb-51cd-4cc6-b9ea-da39d516395b/main/0.log"
Sep 11 18:00:54.138876 ip-10-0-141-139 kubenswrapper[2577]: I0911 18:00:54.138791    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_facebook-opt-125m-simulated-kserve-74b68447df-xjqr7_cf363870-c91b-4c09-8a71-c1daa17229c8/main/0.log"
Sep 11 18:00:54.268117 ip-10-0-141-139 kubenswrapper[2577]: I0911 18:00:54.268088    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_premium-simulated-simulated-premium-kserve-57fc66f697-lrzlk_647175ee-c483-455d-8bd0-987ca40fb2bd/main/0.log"
Sep 11 18:01:03.693388 ip-10-0-141-139 kubenswrapper[2577]: I0911 18:01:03.693355    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-fsh8r_73dd9a34-fac4-44c9-96d3-b2c8968ed271/global-pull-secret-syncer/0.log"
Sep 11 18:01:03.831442 ip-10-0-141-139 kubenswrapper[2577]: I0911 18:01:03.831416    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-g6gcb_337b4304-78e3-4abd-afbf-b866f7104056/konnectivity-agent/0.log"
Sep 11 18:01:03.961807 ip-10-0-141-139 kubenswrapper[2577]: I0911 18:01:03.961737    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-141-139.ec2.internal_8b76f1b2a16574ce38927f7f5da97679/haproxy/0.log"
Sep 11 18:01:09.200818 ip-10-0-141-139 kubenswrapper[2577]: I0911 18:01:09.200790    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-fbdd4f445-z6f86_972596b5-0a5e-4145-be4e-dfb8cbda2dfe/kuadrant-console-plugin/0.log"
Sep 11 18:01:09.283134 ip-10-0-141-139 kubenswrapper[2577]: I0911 18:01:09.283102    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-84994f7ffc-nkmkm_a0d4403a-011b-4bf2-ae88-a27c8ee0fa86/limitador/0.log"
Sep 11 18:01:10.736048 ip-10-0-141-139 kubenswrapper[2577]: I0911 18:01:10.736014    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_cc806362-6764-4116-b742-c516e267f9cc/alertmanager/0.log"
Sep 11 18:01:10.764915 ip-10-0-141-139 kubenswrapper[2577]: I0911 18:01:10.764893    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_cc806362-6764-4116-b742-c516e267f9cc/config-reloader/0.log"
Sep 11 18:01:10.787408 ip-10-0-141-139 kubenswrapper[2577]: I0911 18:01:10.787379    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_cc806362-6764-4116-b742-c516e267f9cc/kube-rbac-proxy-web/0.log"
Sep 11 18:01:10.815804 ip-10-0-141-139 kubenswrapper[2577]: I0911 18:01:10.815778    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_cc806362-6764-4116-b742-c516e267f9cc/kube-rbac-proxy/0.log"
Sep 11 18:01:10.841680 ip-10-0-141-139 kubenswrapper[2577]: I0911 18:01:10.841653    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_cc806362-6764-4116-b742-c516e267f9cc/kube-rbac-proxy-metric/0.log"
Sep 11 18:01:10.866277 ip-10-0-141-139 kubenswrapper[2577]: I0911 18:01:10.866250    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_cc806362-6764-4116-b742-c516e267f9cc/prom-label-proxy/0.log"
Sep 11 18:01:10.894161 ip-10-0-141-139 kubenswrapper[2577]: I0911 18:01:10.894134    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_cc806362-6764-4116-b742-c516e267f9cc/init-config-reloader/0.log"
Sep 11 18:01:10.975762 ip-10-0-141-139 kubenswrapper[2577]: I0911 18:01:10.975730    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-59fbdd77d-bfz6b_e41207d9-a802-4537-a459-01deb5c8fb16/kube-state-metrics/0.log"
Sep 11 18:01:10.999434 ip-10-0-141-139 kubenswrapper[2577]: I0911 18:01:10.999365    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-59fbdd77d-bfz6b_e41207d9-a802-4537-a459-01deb5c8fb16/kube-rbac-proxy-main/0.log"
Sep 11 18:01:11.031172 ip-10-0-141-139 kubenswrapper[2577]: I0911 18:01:11.031150    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-59fbdd77d-bfz6b_e41207d9-a802-4537-a459-01deb5c8fb16/kube-rbac-proxy-self/0.log"
Sep 11 18:01:11.069454 ip-10-0-141-139 kubenswrapper[2577]: I0911 18:01:11.069423    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_metrics-server-7f9b5d9fd4-wnwpt_7b9f4b24-4fe6-4fc8-8917-08f335a75844/metrics-server/0.log"
Sep 11 18:01:11.116519 ip-10-0-141-139 kubenswrapper[2577]: I0911 18:01:11.116469    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_monitoring-plugin-5d85778b9d-cqvpv_071e2825-99c5-4ccf-9443-e0cdf8f52df3/monitoring-plugin/0.log"
Sep 11 18:01:11.356894 ip-10-0-141-139 kubenswrapper[2577]: I0911 18:01:11.356867    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-mthpt_df65734a-e4e6-4b4e-9114-35af7b635044/node-exporter/0.log"
Sep 11 18:01:11.382835 ip-10-0-141-139 kubenswrapper[2577]: I0911 18:01:11.382809    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-mthpt_df65734a-e4e6-4b4e-9114-35af7b635044/kube-rbac-proxy/0.log"
Sep 11 18:01:11.410645 ip-10-0-141-139 kubenswrapper[2577]: I0911 18:01:11.410617    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-mthpt_df65734a-e4e6-4b4e-9114-35af7b635044/init-textfile/0.log"
Sep 11 18:01:11.440778 ip-10-0-141-139 kubenswrapper[2577]: I0911 18:01:11.440746    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-67dcf8c979-fwdgr_f1b075b6-9bae-41d6-b0c0-1c8b81c3207a/kube-rbac-proxy-main/0.log"
Sep 11 18:01:11.466049 ip-10-0-141-139 kubenswrapper[2577]: I0911 18:01:11.466021    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-67dcf8c979-fwdgr_f1b075b6-9bae-41d6-b0c0-1c8b81c3207a/kube-rbac-proxy-self/0.log"
Sep 11 18:01:11.490528 ip-10-0-141-139 kubenswrapper[2577]: I0911 18:01:11.490475    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-67dcf8c979-fwdgr_f1b075b6-9bae-41d6-b0c0-1c8b81c3207a/openshift-state-metrics/0.log"
Sep 11 18:01:11.539015 ip-10-0-141-139 kubenswrapper[2577]: I0911 18:01:11.538934    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_d669874a-e27f-4a06-af94-6cbb47fb7b57/prometheus/0.log"
Sep 11 18:01:11.569357 ip-10-0-141-139 kubenswrapper[2577]: I0911 18:01:11.569332    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_d669874a-e27f-4a06-af94-6cbb47fb7b57/config-reloader/0.log"
Sep 11 18:01:11.586143 ip-10-0-141-139 kubenswrapper[2577]: I0911 18:01:11.586114    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_d669874a-e27f-4a06-af94-6cbb47fb7b57/thanos-sidecar/0.log"
Sep 11 18:01:11.612806 ip-10-0-141-139 kubenswrapper[2577]: I0911 18:01:11.612781    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_d669874a-e27f-4a06-af94-6cbb47fb7b57/kube-rbac-proxy-web/0.log"
Sep 11 18:01:11.639379 ip-10-0-141-139 kubenswrapper[2577]: I0911 18:01:11.639350    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_d669874a-e27f-4a06-af94-6cbb47fb7b57/kube-rbac-proxy/0.log"
Sep 11 18:01:11.659990 ip-10-0-141-139 kubenswrapper[2577]: I0911 18:01:11.659963    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_d669874a-e27f-4a06-af94-6cbb47fb7b57/kube-rbac-proxy-thanos/0.log"
Sep 11 18:01:11.683458 ip-10-0-141-139 kubenswrapper[2577]: I0911 18:01:11.683433    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_d669874a-e27f-4a06-af94-6cbb47fb7b57/init-config-reloader/0.log"
Sep 11 18:01:11.713993 ip-10-0-141-139 kubenswrapper[2577]: I0911 18:01:11.713963    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-68bcd7f987-j6qnv_f066e3ca-d446-4516-bc85-d3e0b841ba58/prometheus-operator/0.log"
Sep 11 18:01:11.734300 ip-10-0-141-139 kubenswrapper[2577]: I0911 18:01:11.734272    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-68bcd7f987-j6qnv_f066e3ca-d446-4516-bc85-d3e0b841ba58/kube-rbac-proxy/0.log"
Sep 11 18:01:11.805643 ip-10-0-141-139 kubenswrapper[2577]: I0911 18:01:11.805563    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-76875c59d5-d247c_addf52ab-b262-4b3a-be3d-5ad1e772090f/telemeter-client/0.log"
Sep 11 18:01:11.835215 ip-10-0-141-139 kubenswrapper[2577]: I0911 18:01:11.835188    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-76875c59d5-d247c_addf52ab-b262-4b3a-be3d-5ad1e772090f/reload/0.log"
Sep 11 18:01:11.860259 ip-10-0-141-139 kubenswrapper[2577]: I0911 18:01:11.860227    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-76875c59d5-d247c_addf52ab-b262-4b3a-be3d-5ad1e772090f/kube-rbac-proxy/0.log"
Sep 11 18:01:11.893264 ip-10-0-141-139 kubenswrapper[2577]: I0911 18:01:11.893228    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-7775c558d6-95wpp_44e112d2-e31f-4dfc-abc3-17f9c80ae9b4/thanos-query/0.log"
Sep 11 18:01:11.920686 ip-10-0-141-139 kubenswrapper[2577]: I0911 18:01:11.920659    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-7775c558d6-95wpp_44e112d2-e31f-4dfc-abc3-17f9c80ae9b4/kube-rbac-proxy-web/0.log"
Sep 11 18:01:11.953809 ip-10-0-141-139 kubenswrapper[2577]: I0911 18:01:11.953781    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-7775c558d6-95wpp_44e112d2-e31f-4dfc-abc3-17f9c80ae9b4/kube-rbac-proxy/0.log"
Sep 11 18:01:11.974822 ip-10-0-141-139 kubenswrapper[2577]: I0911 18:01:11.974798    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-7775c558d6-95wpp_44e112d2-e31f-4dfc-abc3-17f9c80ae9b4/prom-label-proxy/0.log"
Sep 11 18:01:12.006196 ip-10-0-141-139 kubenswrapper[2577]: I0911 18:01:12.006169    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-7775c558d6-95wpp_44e112d2-e31f-4dfc-abc3-17f9c80ae9b4/kube-rbac-proxy-rules/0.log"
Sep 11 18:01:12.040300 ip-10-0-141-139 kubenswrapper[2577]: I0911 18:01:12.040274    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-7775c558d6-95wpp_44e112d2-e31f-4dfc-abc3-17f9c80ae9b4/kube-rbac-proxy-metrics/0.log"
Sep 11 18:01:13.991959 ip-10-0-141-139 kubenswrapper[2577]: I0911 18:01:13.991929    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-tsftd/perf-node-gather-daemonset-pggj8"]
Sep 11 18:01:13.992372 ip-10-0-141-139 kubenswrapper[2577]: I0911 18:01:13.992331    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4095004f-6853-49a8-b07e-1986d2e68f8f" containerName="main"
Sep 11 18:01:13.992372 ip-10-0-141-139 kubenswrapper[2577]: I0911 18:01:13.992343    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="4095004f-6853-49a8-b07e-1986d2e68f8f" containerName="main"
Sep 11 18:01:13.992456 ip-10-0-141-139 kubenswrapper[2577]: I0911 18:01:13.992404    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="4095004f-6853-49a8-b07e-1986d2e68f8f" containerName="main"
Sep 11 18:01:13.995966 ip-10-0-141-139 kubenswrapper[2577]: I0911 18:01:13.995950    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-tsftd/perf-node-gather-daemonset-pggj8"
Sep 11 18:01:13.999967 ip-10-0-141-139 kubenswrapper[2577]: I0911 18:01:13.999941    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-tsftd\"/\"default-dockercfg-cw6vd\""
Sep 11 18:01:13.999967 ip-10-0-141-139 kubenswrapper[2577]: I0911 18:01:13.999958    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-tsftd\"/\"kube-root-ca.crt\""
Sep 11 18:01:14.000165 ip-10-0-141-139 kubenswrapper[2577]: I0911 18:01:13.999967    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-tsftd\"/\"openshift-service-ca.crt\""
Sep 11 18:01:14.005068 ip-10-0-141-139 kubenswrapper[2577]: I0911 18:01:14.005044    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-tsftd/perf-node-gather-daemonset-pggj8"]
Sep 11 18:01:14.091225 ip-10-0-141-139 kubenswrapper[2577]: I0911 18:01:14.091186    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8n8nt\" (UniqueName: \"kubernetes.io/projected/f1089a14-817d-45de-a8fc-e0d3aaf05f33-kube-api-access-8n8nt\") pod \"perf-node-gather-daemonset-pggj8\" (UID: \"f1089a14-817d-45de-a8fc-e0d3aaf05f33\") " pod="openshift-must-gather-tsftd/perf-node-gather-daemonset-pggj8"
Sep 11 18:01:14.091225 ip-10-0-141-139 kubenswrapper[2577]: I0911 18:01:14.091241    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/f1089a14-817d-45de-a8fc-e0d3aaf05f33-proc\") pod \"perf-node-gather-daemonset-pggj8\" (UID: \"f1089a14-817d-45de-a8fc-e0d3aaf05f33\") " pod="openshift-must-gather-tsftd/perf-node-gather-daemonset-pggj8"
Sep 11 18:01:14.091495 ip-10-0-141-139 kubenswrapper[2577]: I0911 18:01:14.091269    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/f1089a14-817d-45de-a8fc-e0d3aaf05f33-sys\") pod \"perf-node-gather-daemonset-pggj8\" (UID: \"f1089a14-817d-45de-a8fc-e0d3aaf05f33\") " pod="openshift-must-gather-tsftd/perf-node-gather-daemonset-pggj8"
Sep 11 18:01:14.091495 ip-10-0-141-139 kubenswrapper[2577]: I0911 18:01:14.091296    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/f1089a14-817d-45de-a8fc-e0d3aaf05f33-podres\") pod \"perf-node-gather-daemonset-pggj8\" (UID: \"f1089a14-817d-45de-a8fc-e0d3aaf05f33\") " pod="openshift-must-gather-tsftd/perf-node-gather-daemonset-pggj8"
Sep 11 18:01:14.091495 ip-10-0-141-139 kubenswrapper[2577]: I0911 18:01:14.091355    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/f1089a14-817d-45de-a8fc-e0d3aaf05f33-lib-modules\") pod \"perf-node-gather-daemonset-pggj8\" (UID: \"f1089a14-817d-45de-a8fc-e0d3aaf05f33\") " pod="openshift-must-gather-tsftd/perf-node-gather-daemonset-pggj8"
Sep 11 18:01:14.192113 ip-10-0-141-139 kubenswrapper[2577]: I0911 18:01:14.192079    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8n8nt\" (UniqueName: \"kubernetes.io/projected/f1089a14-817d-45de-a8fc-e0d3aaf05f33-kube-api-access-8n8nt\") pod \"perf-node-gather-daemonset-pggj8\" (UID: \"f1089a14-817d-45de-a8fc-e0d3aaf05f33\") " pod="openshift-must-gather-tsftd/perf-node-gather-daemonset-pggj8"
Sep 11 18:01:14.192316 ip-10-0-141-139 kubenswrapper[2577]: I0911 18:01:14.192119    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/f1089a14-817d-45de-a8fc-e0d3aaf05f33-proc\") pod \"perf-node-gather-daemonset-pggj8\" (UID: \"f1089a14-817d-45de-a8fc-e0d3aaf05f33\") " pod="openshift-must-gather-tsftd/perf-node-gather-daemonset-pggj8"
Sep 11 18:01:14.192316 ip-10-0-141-139 kubenswrapper[2577]: I0911 18:01:14.192139    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/f1089a14-817d-45de-a8fc-e0d3aaf05f33-sys\") pod \"perf-node-gather-daemonset-pggj8\" (UID: \"f1089a14-817d-45de-a8fc-e0d3aaf05f33\") " pod="openshift-must-gather-tsftd/perf-node-gather-daemonset-pggj8"
Sep 11 18:01:14.192316 ip-10-0-141-139 kubenswrapper[2577]: I0911 18:01:14.192165    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/f1089a14-817d-45de-a8fc-e0d3aaf05f33-podres\") pod \"perf-node-gather-daemonset-pggj8\" (UID: \"f1089a14-817d-45de-a8fc-e0d3aaf05f33\") " pod="openshift-must-gather-tsftd/perf-node-gather-daemonset-pggj8"
Sep 11 18:01:14.192316 ip-10-0-141-139 kubenswrapper[2577]: I0911 18:01:14.192194    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/f1089a14-817d-45de-a8fc-e0d3aaf05f33-lib-modules\") pod \"perf-node-gather-daemonset-pggj8\" (UID: \"f1089a14-817d-45de-a8fc-e0d3aaf05f33\") " pod="openshift-must-gather-tsftd/perf-node-gather-daemonset-pggj8"
Sep 11 18:01:14.192316 ip-10-0-141-139 kubenswrapper[2577]: I0911 18:01:14.192230    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/f1089a14-817d-45de-a8fc-e0d3aaf05f33-sys\") pod \"perf-node-gather-daemonset-pggj8\" (UID: \"f1089a14-817d-45de-a8fc-e0d3aaf05f33\") " pod="openshift-must-gather-tsftd/perf-node-gather-daemonset-pggj8"
Sep 11 18:01:14.192316 ip-10-0-141-139 kubenswrapper[2577]: I0911 18:01:14.192225    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/f1089a14-817d-45de-a8fc-e0d3aaf05f33-proc\") pod \"perf-node-gather-daemonset-pggj8\" (UID: \"f1089a14-817d-45de-a8fc-e0d3aaf05f33\") " pod="openshift-must-gather-tsftd/perf-node-gather-daemonset-pggj8"
Sep 11 18:01:14.192598 ip-10-0-141-139 kubenswrapper[2577]: I0911 18:01:14.192332    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/f1089a14-817d-45de-a8fc-e0d3aaf05f33-podres\") pod \"perf-node-gather-daemonset-pggj8\" (UID: \"f1089a14-817d-45de-a8fc-e0d3aaf05f33\") " pod="openshift-must-gather-tsftd/perf-node-gather-daemonset-pggj8"
Sep 11 18:01:14.192598 ip-10-0-141-139 kubenswrapper[2577]: I0911 18:01:14.192358    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/f1089a14-817d-45de-a8fc-e0d3aaf05f33-lib-modules\") pod \"perf-node-gather-daemonset-pggj8\" (UID: \"f1089a14-817d-45de-a8fc-e0d3aaf05f33\") " pod="openshift-must-gather-tsftd/perf-node-gather-daemonset-pggj8"
Sep 11 18:01:14.200811 ip-10-0-141-139 kubenswrapper[2577]: I0911 18:01:14.200788    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8n8nt\" (UniqueName: \"kubernetes.io/projected/f1089a14-817d-45de-a8fc-e0d3aaf05f33-kube-api-access-8n8nt\") pod \"perf-node-gather-daemonset-pggj8\" (UID: \"f1089a14-817d-45de-a8fc-e0d3aaf05f33\") " pod="openshift-must-gather-tsftd/perf-node-gather-daemonset-pggj8"
Sep 11 18:01:14.306993 ip-10-0-141-139 kubenswrapper[2577]: I0911 18:01:14.306884    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-tsftd/perf-node-gather-daemonset-pggj8"
Sep 11 18:01:14.580682 ip-10-0-141-139 kubenswrapper[2577]: I0911 18:01:14.580655    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-tsftd/perf-node-gather-daemonset-pggj8"]
Sep 11 18:01:14.582919 ip-10-0-141-139 kubenswrapper[2577]: W0911 18:01:14.582892    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-podf1089a14_817d_45de_a8fc_e0d3aaf05f33.slice/crio-140d5f83e8fbc9fc19d82215c9c8b3a5bd450e09500611e60124e687bf7fe2c2 WatchSource:0}: Error finding container 140d5f83e8fbc9fc19d82215c9c8b3a5bd450e09500611e60124e687bf7fe2c2: Status 404 returned error can't find the container with id 140d5f83e8fbc9fc19d82215c9c8b3a5bd450e09500611e60124e687bf7fe2c2
Sep 11 18:01:14.936905 ip-10-0-141-139 kubenswrapper[2577]: I0911 18:01:14.936817    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-tsftd/perf-node-gather-daemonset-pggj8" event={"ID":"f1089a14-817d-45de-a8fc-e0d3aaf05f33","Type":"ContainerStarted","Data":"81820d9bf910f5b0e1030846fb6b73dc907c7611af989bd7219e2f1cd7628bd2"}
Sep 11 18:01:14.936905 ip-10-0-141-139 kubenswrapper[2577]: I0911 18:01:14.936869    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-tsftd/perf-node-gather-daemonset-pggj8" event={"ID":"f1089a14-817d-45de-a8fc-e0d3aaf05f33","Type":"ContainerStarted","Data":"140d5f83e8fbc9fc19d82215c9c8b3a5bd450e09500611e60124e687bf7fe2c2"}
Sep 11 18:01:14.936905 ip-10-0-141-139 kubenswrapper[2577]: I0911 18:01:14.936900    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-tsftd/perf-node-gather-daemonset-pggj8"
Sep 11 18:01:14.955329 ip-10-0-141-139 kubenswrapper[2577]: I0911 18:01:14.955022    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-tsftd/perf-node-gather-daemonset-pggj8" podStartSLOduration=1.955003241 podStartE2EDuration="1.955003241s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 18:01:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:01:14.953765973 +0000 UTC m=+2552.526986085" watchObservedRunningTime="2026-09-11 18:01:14.955003241 +0000 UTC m=+2552.528223354"
Sep 11 18:01:16.600243 ip-10-0-141-139 kubenswrapper[2577]: I0911 18:01:16.600201    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-g6qq7_d6d8f0b3-5098-4746-8e75-aee84eafa17b/dns/0.log"
Sep 11 18:01:16.622294 ip-10-0-141-139 kubenswrapper[2577]: I0911 18:01:16.622266    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-g6qq7_d6d8f0b3-5098-4746-8e75-aee84eafa17b/kube-rbac-proxy/0.log"
Sep 11 18:01:16.696827 ip-10-0-141-139 kubenswrapper[2577]: I0911 18:01:16.696796    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-9v9dt_66d2b8f8-a4f7-4811-8034-2523351499ee/dns-node-resolver/0.log"
Sep 11 18:01:17.350001 ip-10-0-141-139 kubenswrapper[2577]: I0911 18:01:17.349965    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-k5j4d_c5878f27-a359-47d1-b223-08f4bd829380/node-ca/0.log"
Sep 11 18:01:19.153049 ip-10-0-141-139 kubenswrapper[2577]: I0911 18:01:19.153019    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-7d6df6ff46-hkdnw_78758cdb-2c93-44c5-9039-ff181899b15f/kube-auth-proxy/0.log"
Sep 11 18:01:19.252962 ip-10-0-141-139 kubenswrapper[2577]: I0911 18:01:19.252924    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-59fbc47bf8-r7l5l_c779770c-0017-4b67-bf14-415f9ec89cc2/payload-pre-processing/0.log"
Sep 11 18:01:20.316866 ip-10-0-141-139 kubenswrapper[2577]: I0911 18:01:20.316838    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-tt7ph_894d4606-cd1f-45ae-a9fa-a3b9b5e00975/serve-healthcheck-canary/0.log"
Sep 11 18:01:20.952699 ip-10-0-141-139 kubenswrapper[2577]: I0911 18:01:20.952669    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-tsftd/perf-node-gather-daemonset-pggj8"
Sep 11 18:01:20.961489 ip-10-0-141-139 kubenswrapper[2577]: I0911 18:01:20.961464    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-vnvj4_f68e4b95-ae13-4835-976a-a1af09fc70cf/kube-rbac-proxy/0.log"
Sep 11 18:01:20.986889 ip-10-0-141-139 kubenswrapper[2577]: I0911 18:01:20.986862    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-vnvj4_f68e4b95-ae13-4835-976a-a1af09fc70cf/exporter/0.log"
Sep 11 18:01:21.011387 ip-10-0-141-139 kubenswrapper[2577]: I0911 18:01:21.011360    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-vnvj4_f68e4b95-ae13-4835-976a-a1af09fc70cf/extractor/0.log"
Sep 11 18:01:24.384155 ip-10-0-141-139 kubenswrapper[2577]: I0911 18:01:24.384121    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-b887fdd9f-jc4h9_65106181-ae0e-493b-b7d7-b6707f6f1bab/manager/0.log"
Sep 11 18:01:24.410378 ip-10-0-141-139 kubenswrapper[2577]: I0911 18:01:24.410351    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-b887fdd9f-jc4h9_65106181-ae0e-493b-b7d7-b6707f6f1bab/copy-manifests/0.log"
Sep 11 18:01:25.581237 ip-10-0-141-139 kubenswrapper[2577]: I0911 18:01:25.581210    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-7d6df6ff46-hkdnw_78758cdb-2c93-44c5-9039-ff181899b15f/kube-auth-proxy/0.log"
Sep 11 18:01:25.646286 ip-10-0-141-139 kubenswrapper[2577]: I0911 18:01:25.646233    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-59fbc47bf8-r7l5l_c779770c-0017-4b67-bf14-415f9ec89cc2/payload-pre-processing/0.log"
Sep 11 18:01:25.762322 ip-10-0-141-139 kubenswrapper[2577]: I0911 18:01:25.762292    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_lws-controller-manager-845d467d5c-xnvgd_e0102433-2481-47fa-ac24-665e4326f8e8/manager/0.log"
Sep 11 18:01:28.148896 ip-10-0-141-139 kubenswrapper[2577]: I0911 18:01:28.148866    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-7d6df6ff46-hkdnw_78758cdb-2c93-44c5-9039-ff181899b15f/kube-auth-proxy/0.log"
Sep 11 18:01:28.214728 ip-10-0-141-139 kubenswrapper[2577]: I0911 18:01:28.214689    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-59fbc47bf8-r7l5l_c779770c-0017-4b67-bf14-415f9ec89cc2/payload-pre-processing/0.log"