Sep 11 03:40:22.235416 ip-10-0-141-105 systemd[1]: Starting Kubernetes Kubelet...
Sep 11 03:40:22.614883 ip-10-0-141-105 kubenswrapper[2579]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 11 03:40:22.614883 ip-10-0-141-105 kubenswrapper[2579]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Sep 11 03:40:22.614883 ip-10-0-141-105 kubenswrapper[2579]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 11 03:40:22.614883 ip-10-0-141-105 kubenswrapper[2579]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI.
Sep 11 03:40:22.614883 ip-10-0-141-105 kubenswrapper[2579]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 11 03:40:22.615817 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:40:22.615730    2579 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime"
Sep 11 03:40:22.619590 ip-10-0-141-105 kubenswrapper[2579]: W0911 03:40:22.619574    2579 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 11 03:40:22.619590 ip-10-0-141-105 kubenswrapper[2579]: W0911 03:40:22.619590    2579 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 11 03:40:22.619653 ip-10-0-141-105 kubenswrapper[2579]: W0911 03:40:22.619594    2579 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 11 03:40:22.619653 ip-10-0-141-105 kubenswrapper[2579]: W0911 03:40:22.619597    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 11 03:40:22.619653 ip-10-0-141-105 kubenswrapper[2579]: W0911 03:40:22.619599    2579 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 11 03:40:22.619653 ip-10-0-141-105 kubenswrapper[2579]: W0911 03:40:22.619603    2579 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 11 03:40:22.619653 ip-10-0-141-105 kubenswrapper[2579]: W0911 03:40:22.619606    2579 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 11 03:40:22.619653 ip-10-0-141-105 kubenswrapper[2579]: W0911 03:40:22.619609    2579 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 11 03:40:22.619653 ip-10-0-141-105 kubenswrapper[2579]: W0911 03:40:22.619611    2579 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 11 03:40:22.619653 ip-10-0-141-105 kubenswrapper[2579]: W0911 03:40:22.619614    2579 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 11 03:40:22.619653 ip-10-0-141-105 kubenswrapper[2579]: W0911 03:40:22.619617    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 11 03:40:22.619653 ip-10-0-141-105 kubenswrapper[2579]: W0911 03:40:22.619619    2579 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 11 03:40:22.619653 ip-10-0-141-105 kubenswrapper[2579]: W0911 03:40:22.619622    2579 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 11 03:40:22.619653 ip-10-0-141-105 kubenswrapper[2579]: W0911 03:40:22.619624    2579 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 11 03:40:22.619653 ip-10-0-141-105 kubenswrapper[2579]: W0911 03:40:22.619627    2579 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 11 03:40:22.619653 ip-10-0-141-105 kubenswrapper[2579]: W0911 03:40:22.619629    2579 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 11 03:40:22.619653 ip-10-0-141-105 kubenswrapper[2579]: W0911 03:40:22.619632    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 11 03:40:22.619653 ip-10-0-141-105 kubenswrapper[2579]: W0911 03:40:22.619636    2579 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 11 03:40:22.619653 ip-10-0-141-105 kubenswrapper[2579]: W0911 03:40:22.619639    2579 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 11 03:40:22.619653 ip-10-0-141-105 kubenswrapper[2579]: W0911 03:40:22.619641    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 11 03:40:22.619653 ip-10-0-141-105 kubenswrapper[2579]: W0911 03:40:22.619644    2579 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 11 03:40:22.620125 ip-10-0-141-105 kubenswrapper[2579]: W0911 03:40:22.619647    2579 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 11 03:40:22.620125 ip-10-0-141-105 kubenswrapper[2579]: W0911 03:40:22.619650    2579 feature_gate.go:328] unrecognized feature gate: Example2
Sep 11 03:40:22.620125 ip-10-0-141-105 kubenswrapper[2579]: W0911 03:40:22.619652    2579 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 11 03:40:22.620125 ip-10-0-141-105 kubenswrapper[2579]: W0911 03:40:22.619655    2579 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 11 03:40:22.620125 ip-10-0-141-105 kubenswrapper[2579]: W0911 03:40:22.619658    2579 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 11 03:40:22.620125 ip-10-0-141-105 kubenswrapper[2579]: W0911 03:40:22.619661    2579 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 11 03:40:22.620125 ip-10-0-141-105 kubenswrapper[2579]: W0911 03:40:22.619663    2579 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 11 03:40:22.620125 ip-10-0-141-105 kubenswrapper[2579]: W0911 03:40:22.619666    2579 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 11 03:40:22.620125 ip-10-0-141-105 kubenswrapper[2579]: W0911 03:40:22.619669    2579 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 11 03:40:22.620125 ip-10-0-141-105 kubenswrapper[2579]: W0911 03:40:22.619671    2579 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 11 03:40:22.620125 ip-10-0-141-105 kubenswrapper[2579]: W0911 03:40:22.619673    2579 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 11 03:40:22.620125 ip-10-0-141-105 kubenswrapper[2579]: W0911 03:40:22.619676    2579 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 11 03:40:22.620125 ip-10-0-141-105 kubenswrapper[2579]: W0911 03:40:22.619679    2579 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 11 03:40:22.620125 ip-10-0-141-105 kubenswrapper[2579]: W0911 03:40:22.619681    2579 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 11 03:40:22.620125 ip-10-0-141-105 kubenswrapper[2579]: W0911 03:40:22.619683    2579 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 11 03:40:22.620125 ip-10-0-141-105 kubenswrapper[2579]: W0911 03:40:22.619686    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 11 03:40:22.620125 ip-10-0-141-105 kubenswrapper[2579]: W0911 03:40:22.619688    2579 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 11 03:40:22.620125 ip-10-0-141-105 kubenswrapper[2579]: W0911 03:40:22.619690    2579 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 11 03:40:22.620125 ip-10-0-141-105 kubenswrapper[2579]: W0911 03:40:22.619693    2579 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 11 03:40:22.620125 ip-10-0-141-105 kubenswrapper[2579]: W0911 03:40:22.619696    2579 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 11 03:40:22.620601 ip-10-0-141-105 kubenswrapper[2579]: W0911 03:40:22.619698    2579 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 11 03:40:22.620601 ip-10-0-141-105 kubenswrapper[2579]: W0911 03:40:22.619702    2579 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 11 03:40:22.620601 ip-10-0-141-105 kubenswrapper[2579]: W0911 03:40:22.619704    2579 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 11 03:40:22.620601 ip-10-0-141-105 kubenswrapper[2579]: W0911 03:40:22.619707    2579 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 11 03:40:22.620601 ip-10-0-141-105 kubenswrapper[2579]: W0911 03:40:22.619709    2579 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 11 03:40:22.620601 ip-10-0-141-105 kubenswrapper[2579]: W0911 03:40:22.619711    2579 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 11 03:40:22.620601 ip-10-0-141-105 kubenswrapper[2579]: W0911 03:40:22.619714    2579 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 11 03:40:22.620601 ip-10-0-141-105 kubenswrapper[2579]: W0911 03:40:22.619716    2579 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 11 03:40:22.620601 ip-10-0-141-105 kubenswrapper[2579]: W0911 03:40:22.619719    2579 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 11 03:40:22.620601 ip-10-0-141-105 kubenswrapper[2579]: W0911 03:40:22.619721    2579 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 11 03:40:22.620601 ip-10-0-141-105 kubenswrapper[2579]: W0911 03:40:22.619725    2579 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 11 03:40:22.620601 ip-10-0-141-105 kubenswrapper[2579]: W0911 03:40:22.619728    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 11 03:40:22.620601 ip-10-0-141-105 kubenswrapper[2579]: W0911 03:40:22.619733    2579 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 11 03:40:22.620601 ip-10-0-141-105 kubenswrapper[2579]: W0911 03:40:22.619737    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 11 03:40:22.620601 ip-10-0-141-105 kubenswrapper[2579]: W0911 03:40:22.619741    2579 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 11 03:40:22.620601 ip-10-0-141-105 kubenswrapper[2579]: W0911 03:40:22.619744    2579 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 11 03:40:22.620601 ip-10-0-141-105 kubenswrapper[2579]: W0911 03:40:22.619746    2579 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 11 03:40:22.620601 ip-10-0-141-105 kubenswrapper[2579]: W0911 03:40:22.619749    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 11 03:40:22.620601 ip-10-0-141-105 kubenswrapper[2579]: W0911 03:40:22.619752    2579 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 11 03:40:22.621056 ip-10-0-141-105 kubenswrapper[2579]: W0911 03:40:22.619755    2579 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 11 03:40:22.621056 ip-10-0-141-105 kubenswrapper[2579]: W0911 03:40:22.619757    2579 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 11 03:40:22.621056 ip-10-0-141-105 kubenswrapper[2579]: W0911 03:40:22.619760    2579 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 11 03:40:22.621056 ip-10-0-141-105 kubenswrapper[2579]: W0911 03:40:22.619763    2579 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 11 03:40:22.621056 ip-10-0-141-105 kubenswrapper[2579]: W0911 03:40:22.619766    2579 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 11 03:40:22.621056 ip-10-0-141-105 kubenswrapper[2579]: W0911 03:40:22.619769    2579 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 11 03:40:22.621056 ip-10-0-141-105 kubenswrapper[2579]: W0911 03:40:22.619772    2579 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 11 03:40:22.621056 ip-10-0-141-105 kubenswrapper[2579]: W0911 03:40:22.619774    2579 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 11 03:40:22.621056 ip-10-0-141-105 kubenswrapper[2579]: W0911 03:40:22.619777    2579 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 11 03:40:22.621056 ip-10-0-141-105 kubenswrapper[2579]: W0911 03:40:22.619781    2579 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 11 03:40:22.621056 ip-10-0-141-105 kubenswrapper[2579]: W0911 03:40:22.619783    2579 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 11 03:40:22.621056 ip-10-0-141-105 kubenswrapper[2579]: W0911 03:40:22.619787    2579 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 11 03:40:22.621056 ip-10-0-141-105 kubenswrapper[2579]: W0911 03:40:22.619789    2579 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 11 03:40:22.621056 ip-10-0-141-105 kubenswrapper[2579]: W0911 03:40:22.619792    2579 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 11 03:40:22.621056 ip-10-0-141-105 kubenswrapper[2579]: W0911 03:40:22.619796    2579 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 11 03:40:22.621056 ip-10-0-141-105 kubenswrapper[2579]: W0911 03:40:22.619799    2579 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 11 03:40:22.621056 ip-10-0-141-105 kubenswrapper[2579]: W0911 03:40:22.619802    2579 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 11 03:40:22.621056 ip-10-0-141-105 kubenswrapper[2579]: W0911 03:40:22.619805    2579 feature_gate.go:328] unrecognized feature gate: Example
Sep 11 03:40:22.621056 ip-10-0-141-105 kubenswrapper[2579]: W0911 03:40:22.619808    2579 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 11 03:40:22.621526 ip-10-0-141-105 kubenswrapper[2579]: W0911 03:40:22.619812    2579 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 11 03:40:22.621526 ip-10-0-141-105 kubenswrapper[2579]: W0911 03:40:22.619814    2579 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 11 03:40:22.621526 ip-10-0-141-105 kubenswrapper[2579]: W0911 03:40:22.619817    2579 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 11 03:40:22.621526 ip-10-0-141-105 kubenswrapper[2579]: W0911 03:40:22.619820    2579 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 11 03:40:22.621526 ip-10-0-141-105 kubenswrapper[2579]: W0911 03:40:22.619823    2579 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 11 03:40:22.621526 ip-10-0-141-105 kubenswrapper[2579]: W0911 03:40:22.619826    2579 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 11 03:40:22.621526 ip-10-0-141-105 kubenswrapper[2579]: W0911 03:40:22.619829    2579 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 11 03:40:22.621526 ip-10-0-141-105 kubenswrapper[2579]: W0911 03:40:22.619832    2579 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 11 03:40:22.621526 ip-10-0-141-105 kubenswrapper[2579]: W0911 03:40:22.619835    2579 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 11 03:40:22.621526 ip-10-0-141-105 kubenswrapper[2579]: W0911 03:40:22.619837    2579 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 11 03:40:22.621526 ip-10-0-141-105 kubenswrapper[2579]: W0911 03:40:22.619840    2579 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 11 03:40:22.621526 ip-10-0-141-105 kubenswrapper[2579]: W0911 03:40:22.619842    2579 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 11 03:40:22.621526 ip-10-0-141-105 kubenswrapper[2579]: W0911 03:40:22.619845    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 11 03:40:22.621526 ip-10-0-141-105 kubenswrapper[2579]: W0911 03:40:22.619848    2579 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 11 03:40:22.621526 ip-10-0-141-105 kubenswrapper[2579]: W0911 03:40:22.619850    2579 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 11 03:40:22.621526 ip-10-0-141-105 kubenswrapper[2579]: W0911 03:40:22.619853    2579 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 11 03:40:22.621526 ip-10-0-141-105 kubenswrapper[2579]: W0911 03:40:22.619855    2579 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 11 03:40:22.621526 ip-10-0-141-105 kubenswrapper[2579]: W0911 03:40:22.619858    2579 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 11 03:40:22.621526 ip-10-0-141-105 kubenswrapper[2579]: W0911 03:40:22.619860    2579 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 11 03:40:22.621526 ip-10-0-141-105 kubenswrapper[2579]: W0911 03:40:22.619862    2579 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 11 03:40:22.622040 ip-10-0-141-105 kubenswrapper[2579]: W0911 03:40:22.620560    2579 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 11 03:40:22.622040 ip-10-0-141-105 kubenswrapper[2579]: W0911 03:40:22.620567    2579 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 11 03:40:22.622040 ip-10-0-141-105 kubenswrapper[2579]: W0911 03:40:22.620570    2579 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 11 03:40:22.622040 ip-10-0-141-105 kubenswrapper[2579]: W0911 03:40:22.620572    2579 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 11 03:40:22.622040 ip-10-0-141-105 kubenswrapper[2579]: W0911 03:40:22.620575    2579 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 11 03:40:22.622040 ip-10-0-141-105 kubenswrapper[2579]: W0911 03:40:22.620580    2579 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 11 03:40:22.622040 ip-10-0-141-105 kubenswrapper[2579]: W0911 03:40:22.620582    2579 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 11 03:40:22.622040 ip-10-0-141-105 kubenswrapper[2579]: W0911 03:40:22.620585    2579 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 11 03:40:22.622040 ip-10-0-141-105 kubenswrapper[2579]: W0911 03:40:22.620587    2579 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 11 03:40:22.622040 ip-10-0-141-105 kubenswrapper[2579]: W0911 03:40:22.620590    2579 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 11 03:40:22.622040 ip-10-0-141-105 kubenswrapper[2579]: W0911 03:40:22.620593    2579 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 11 03:40:22.622040 ip-10-0-141-105 kubenswrapper[2579]: W0911 03:40:22.620595    2579 feature_gate.go:328] unrecognized feature gate: Example
Sep 11 03:40:22.622040 ip-10-0-141-105 kubenswrapper[2579]: W0911 03:40:22.620600    2579 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 11 03:40:22.622040 ip-10-0-141-105 kubenswrapper[2579]: W0911 03:40:22.620604    2579 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 11 03:40:22.622040 ip-10-0-141-105 kubenswrapper[2579]: W0911 03:40:22.620607    2579 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 11 03:40:22.622040 ip-10-0-141-105 kubenswrapper[2579]: W0911 03:40:22.620610    2579 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 11 03:40:22.622040 ip-10-0-141-105 kubenswrapper[2579]: W0911 03:40:22.620613    2579 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 11 03:40:22.622040 ip-10-0-141-105 kubenswrapper[2579]: W0911 03:40:22.620617    2579 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 11 03:40:22.622040 ip-10-0-141-105 kubenswrapper[2579]: W0911 03:40:22.620620    2579 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 11 03:40:22.622040 ip-10-0-141-105 kubenswrapper[2579]: W0911 03:40:22.620623    2579 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 11 03:40:22.622583 ip-10-0-141-105 kubenswrapper[2579]: W0911 03:40:22.620626    2579 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 11 03:40:22.622583 ip-10-0-141-105 kubenswrapper[2579]: W0911 03:40:22.620629    2579 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 11 03:40:22.622583 ip-10-0-141-105 kubenswrapper[2579]: W0911 03:40:22.620632    2579 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 11 03:40:22.622583 ip-10-0-141-105 kubenswrapper[2579]: W0911 03:40:22.620634    2579 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 11 03:40:22.622583 ip-10-0-141-105 kubenswrapper[2579]: W0911 03:40:22.620638    2579 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 11 03:40:22.622583 ip-10-0-141-105 kubenswrapper[2579]: W0911 03:40:22.620642    2579 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 11 03:40:22.622583 ip-10-0-141-105 kubenswrapper[2579]: W0911 03:40:22.620644    2579 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 11 03:40:22.622583 ip-10-0-141-105 kubenswrapper[2579]: W0911 03:40:22.620647    2579 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 11 03:40:22.622583 ip-10-0-141-105 kubenswrapper[2579]: W0911 03:40:22.620649    2579 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 11 03:40:22.622583 ip-10-0-141-105 kubenswrapper[2579]: W0911 03:40:22.620652    2579 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 11 03:40:22.622583 ip-10-0-141-105 kubenswrapper[2579]: W0911 03:40:22.620655    2579 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 11 03:40:22.622583 ip-10-0-141-105 kubenswrapper[2579]: W0911 03:40:22.620657    2579 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 11 03:40:22.622583 ip-10-0-141-105 kubenswrapper[2579]: W0911 03:40:22.620660    2579 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 11 03:40:22.622583 ip-10-0-141-105 kubenswrapper[2579]: W0911 03:40:22.620662    2579 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 11 03:40:22.622583 ip-10-0-141-105 kubenswrapper[2579]: W0911 03:40:22.620666    2579 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 11 03:40:22.622583 ip-10-0-141-105 kubenswrapper[2579]: W0911 03:40:22.620668    2579 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 11 03:40:22.622583 ip-10-0-141-105 kubenswrapper[2579]: W0911 03:40:22.620671    2579 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 11 03:40:22.622583 ip-10-0-141-105 kubenswrapper[2579]: W0911 03:40:22.620673    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 11 03:40:22.622583 ip-10-0-141-105 kubenswrapper[2579]: W0911 03:40:22.620676    2579 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 11 03:40:22.622583 ip-10-0-141-105 kubenswrapper[2579]: W0911 03:40:22.620679    2579 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 11 03:40:22.623071 ip-10-0-141-105 kubenswrapper[2579]: W0911 03:40:22.620682    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 11 03:40:22.623071 ip-10-0-141-105 kubenswrapper[2579]: W0911 03:40:22.620684    2579 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 11 03:40:22.623071 ip-10-0-141-105 kubenswrapper[2579]: W0911 03:40:22.620687    2579 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 11 03:40:22.623071 ip-10-0-141-105 kubenswrapper[2579]: W0911 03:40:22.620689    2579 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 11 03:40:22.623071 ip-10-0-141-105 kubenswrapper[2579]: W0911 03:40:22.620691    2579 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 11 03:40:22.623071 ip-10-0-141-105 kubenswrapper[2579]: W0911 03:40:22.620694    2579 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 11 03:40:22.623071 ip-10-0-141-105 kubenswrapper[2579]: W0911 03:40:22.620696    2579 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 11 03:40:22.623071 ip-10-0-141-105 kubenswrapper[2579]: W0911 03:40:22.620699    2579 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 11 03:40:22.623071 ip-10-0-141-105 kubenswrapper[2579]: W0911 03:40:22.620701    2579 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 11 03:40:22.623071 ip-10-0-141-105 kubenswrapper[2579]: W0911 03:40:22.620704    2579 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 11 03:40:22.623071 ip-10-0-141-105 kubenswrapper[2579]: W0911 03:40:22.620707    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 11 03:40:22.623071 ip-10-0-141-105 kubenswrapper[2579]: W0911 03:40:22.620710    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 11 03:40:22.623071 ip-10-0-141-105 kubenswrapper[2579]: W0911 03:40:22.620713    2579 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 11 03:40:22.623071 ip-10-0-141-105 kubenswrapper[2579]: W0911 03:40:22.620715    2579 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 11 03:40:22.623071 ip-10-0-141-105 kubenswrapper[2579]: W0911 03:40:22.620718    2579 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 11 03:40:22.623071 ip-10-0-141-105 kubenswrapper[2579]: W0911 03:40:22.620720    2579 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 11 03:40:22.623071 ip-10-0-141-105 kubenswrapper[2579]: W0911 03:40:22.620723    2579 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 11 03:40:22.623071 ip-10-0-141-105 kubenswrapper[2579]: W0911 03:40:22.620725    2579 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 11 03:40:22.623071 ip-10-0-141-105 kubenswrapper[2579]: W0911 03:40:22.620728    2579 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 11 03:40:22.623071 ip-10-0-141-105 kubenswrapper[2579]: W0911 03:40:22.620730    2579 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 11 03:40:22.623662 ip-10-0-141-105 kubenswrapper[2579]: W0911 03:40:22.620732    2579 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 11 03:40:22.623662 ip-10-0-141-105 kubenswrapper[2579]: W0911 03:40:22.620735    2579 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 11 03:40:22.623662 ip-10-0-141-105 kubenswrapper[2579]: W0911 03:40:22.620737    2579 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 11 03:40:22.623662 ip-10-0-141-105 kubenswrapper[2579]: W0911 03:40:22.620739    2579 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 11 03:40:22.623662 ip-10-0-141-105 kubenswrapper[2579]: W0911 03:40:22.620742    2579 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 11 03:40:22.623662 ip-10-0-141-105 kubenswrapper[2579]: W0911 03:40:22.620745    2579 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 11 03:40:22.623662 ip-10-0-141-105 kubenswrapper[2579]: W0911 03:40:22.620747    2579 feature_gate.go:328] unrecognized feature gate: Example2
Sep 11 03:40:22.623662 ip-10-0-141-105 kubenswrapper[2579]: W0911 03:40:22.620750    2579 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 11 03:40:22.623662 ip-10-0-141-105 kubenswrapper[2579]: W0911 03:40:22.620752    2579 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 11 03:40:22.623662 ip-10-0-141-105 kubenswrapper[2579]: W0911 03:40:22.620755    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 11 03:40:22.623662 ip-10-0-141-105 kubenswrapper[2579]: W0911 03:40:22.620757    2579 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 11 03:40:22.623662 ip-10-0-141-105 kubenswrapper[2579]: W0911 03:40:22.620760    2579 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 11 03:40:22.623662 ip-10-0-141-105 kubenswrapper[2579]: W0911 03:40:22.620762    2579 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 11 03:40:22.623662 ip-10-0-141-105 kubenswrapper[2579]: W0911 03:40:22.620766    2579 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 11 03:40:22.623662 ip-10-0-141-105 kubenswrapper[2579]: W0911 03:40:22.620769    2579 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 11 03:40:22.623662 ip-10-0-141-105 kubenswrapper[2579]: W0911 03:40:22.620771    2579 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 11 03:40:22.623662 ip-10-0-141-105 kubenswrapper[2579]: W0911 03:40:22.620774    2579 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 11 03:40:22.623662 ip-10-0-141-105 kubenswrapper[2579]: W0911 03:40:22.620776    2579 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 11 03:40:22.623662 ip-10-0-141-105 kubenswrapper[2579]: W0911 03:40:22.620778    2579 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 11 03:40:22.623662 ip-10-0-141-105 kubenswrapper[2579]: W0911 03:40:22.620781    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 11 03:40:22.624160 ip-10-0-141-105 kubenswrapper[2579]: W0911 03:40:22.620783    2579 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 11 03:40:22.624160 ip-10-0-141-105 kubenswrapper[2579]: W0911 03:40:22.620786    2579 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 11 03:40:22.624160 ip-10-0-141-105 kubenswrapper[2579]: W0911 03:40:22.620789    2579 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 11 03:40:22.624160 ip-10-0-141-105 kubenswrapper[2579]: W0911 03:40:22.620791    2579 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 11 03:40:22.624160 ip-10-0-141-105 kubenswrapper[2579]: W0911 03:40:22.620794    2579 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 11 03:40:22.624160 ip-10-0-141-105 kubenswrapper[2579]: W0911 03:40:22.620796    2579 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 11 03:40:22.624160 ip-10-0-141-105 kubenswrapper[2579]: W0911 03:40:22.620800    2579 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 11 03:40:22.624160 ip-10-0-141-105 kubenswrapper[2579]: W0911 03:40:22.620802    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 11 03:40:22.624160 ip-10-0-141-105 kubenswrapper[2579]: W0911 03:40:22.620805    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 11 03:40:22.624160 ip-10-0-141-105 kubenswrapper[2579]: W0911 03:40:22.620807    2579 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 11 03:40:22.624160 ip-10-0-141-105 kubenswrapper[2579]: W0911 03:40:22.620811    2579 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 11 03:40:22.624160 ip-10-0-141-105 kubenswrapper[2579]: W0911 03:40:22.620814    2579 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 11 03:40:22.624160 ip-10-0-141-105 kubenswrapper[2579]: W0911 03:40:22.620817    2579 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 11 03:40:22.624160 ip-10-0-141-105 kubenswrapper[2579]: W0911 03:40:22.620820    2579 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 11 03:40:22.624160 ip-10-0-141-105 kubenswrapper[2579]: W0911 03:40:22.620822    2579 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 11 03:40:22.624160 ip-10-0-141-105 kubenswrapper[2579]: W0911 03:40:22.620825    2579 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 11 03:40:22.624160 ip-10-0-141-105 kubenswrapper[2579]: W0911 03:40:22.620827    2579 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 11 03:40:22.624160 ip-10-0-141-105 kubenswrapper[2579]: W0911 03:40:22.620830    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 11 03:40:22.624160 ip-10-0-141-105 kubenswrapper[2579]: W0911 03:40:22.620832    2579 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 11 03:40:22.624614 ip-10-0-141-105 kubenswrapper[2579]: W0911 03:40:22.621211    2579 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 11 03:40:22.624614 ip-10-0-141-105 kubenswrapper[2579]: W0911 03:40:22.621237    2579 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 11 03:40:22.624614 ip-10-0-141-105 kubenswrapper[2579]: W0911 03:40:22.621240    2579 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 11 03:40:22.624614 ip-10-0-141-105 kubenswrapper[2579]: W0911 03:40:22.621242    2579 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 11 03:40:22.624614 ip-10-0-141-105 kubenswrapper[2579]: W0911 03:40:22.621245    2579 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 11 03:40:22.624614 ip-10-0-141-105 kubenswrapper[2579]: W0911 03:40:22.621248    2579 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 11 03:40:22.624614 ip-10-0-141-105 kubenswrapper[2579]: W0911 03:40:22.621253    2579 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 11 03:40:22.624614 ip-10-0-141-105 kubenswrapper[2579]: W0911 03:40:22.621255    2579 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 11 03:40:22.624614 ip-10-0-141-105 kubenswrapper[2579]: W0911 03:40:22.621258    2579 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 11 03:40:22.624614 ip-10-0-141-105 kubenswrapper[2579]: W0911 03:40:22.621261    2579 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 11 03:40:22.624614 ip-10-0-141-105 kubenswrapper[2579]: W0911 03:40:22.621263    2579 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 11 03:40:22.624614 ip-10-0-141-105 kubenswrapper[2579]: W0911 03:40:22.621266    2579 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 11 03:40:22.624614 ip-10-0-141-105 kubenswrapper[2579]: W0911 03:40:22.621268    2579 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 11 03:40:22.624614 ip-10-0-141-105 kubenswrapper[2579]: W0911 03:40:22.621271    2579 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 11 03:40:22.624614 ip-10-0-141-105 kubenswrapper[2579]: W0911 03:40:22.621275    2579 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 11 03:40:22.624614 ip-10-0-141-105 kubenswrapper[2579]: W0911 03:40:22.621279    2579 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 11 03:40:22.624614 ip-10-0-141-105 kubenswrapper[2579]: W0911 03:40:22.621282    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 11 03:40:22.624614 ip-10-0-141-105 kubenswrapper[2579]: W0911 03:40:22.621285    2579 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 11 03:40:22.624614 ip-10-0-141-105 kubenswrapper[2579]: W0911 03:40:22.621288    2579 feature_gate.go:328] unrecognized feature gate: Example
Sep 11 03:40:22.625097 ip-10-0-141-105 kubenswrapper[2579]: W0911 03:40:22.621291    2579 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 11 03:40:22.625097 ip-10-0-141-105 kubenswrapper[2579]: W0911 03:40:22.621293    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 11 03:40:22.625097 ip-10-0-141-105 kubenswrapper[2579]: W0911 03:40:22.621296    2579 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 11 03:40:22.625097 ip-10-0-141-105 kubenswrapper[2579]: W0911 03:40:22.621299    2579 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 11 03:40:22.625097 ip-10-0-141-105 kubenswrapper[2579]: W0911 03:40:22.621302    2579 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 11 03:40:22.625097 ip-10-0-141-105 kubenswrapper[2579]: W0911 03:40:22.621304    2579 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 11 03:40:22.625097 ip-10-0-141-105 kubenswrapper[2579]: W0911 03:40:22.621306    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 11 03:40:22.625097 ip-10-0-141-105 kubenswrapper[2579]: W0911 03:40:22.621309    2579 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 11 03:40:22.625097 ip-10-0-141-105 kubenswrapper[2579]: W0911 03:40:22.621311    2579 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 11 03:40:22.625097 ip-10-0-141-105 kubenswrapper[2579]: W0911 03:40:22.621314    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 11 03:40:22.625097 ip-10-0-141-105 kubenswrapper[2579]: W0911 03:40:22.621316    2579 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 11 03:40:22.625097 ip-10-0-141-105 kubenswrapper[2579]: W0911 03:40:22.621319    2579 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 11 03:40:22.625097 ip-10-0-141-105 kubenswrapper[2579]: W0911 03:40:22.621322    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 11 03:40:22.625097 ip-10-0-141-105 kubenswrapper[2579]: W0911 03:40:22.621324    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 11 03:40:22.625097 ip-10-0-141-105 kubenswrapper[2579]: W0911 03:40:22.621326    2579 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 11 03:40:22.625097 ip-10-0-141-105 kubenswrapper[2579]: W0911 03:40:22.621329    2579 feature_gate.go:328] unrecognized feature gate: Example2
Sep 11 03:40:22.625097 ip-10-0-141-105 kubenswrapper[2579]: W0911 03:40:22.621331    2579 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 11 03:40:22.625097 ip-10-0-141-105 kubenswrapper[2579]: W0911 03:40:22.621334    2579 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 11 03:40:22.625097 ip-10-0-141-105 kubenswrapper[2579]: W0911 03:40:22.621337    2579 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 11 03:40:22.625097 ip-10-0-141-105 kubenswrapper[2579]: W0911 03:40:22.621340    2579 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 11 03:40:22.625097 ip-10-0-141-105 kubenswrapper[2579]: W0911 03:40:22.621343    2579 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 11 03:40:22.625595 ip-10-0-141-105 kubenswrapper[2579]: W0911 03:40:22.621345    2579 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 11 03:40:22.625595 ip-10-0-141-105 kubenswrapper[2579]: W0911 03:40:22.621348    2579 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 11 03:40:22.625595 ip-10-0-141-105 kubenswrapper[2579]: W0911 03:40:22.621350    2579 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 11 03:40:22.625595 ip-10-0-141-105 kubenswrapper[2579]: W0911 03:40:22.621353    2579 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 11 03:40:22.625595 ip-10-0-141-105 kubenswrapper[2579]: W0911 03:40:22.621355    2579 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 11 03:40:22.625595 ip-10-0-141-105 kubenswrapper[2579]: W0911 03:40:22.621357    2579 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 11 03:40:22.625595 ip-10-0-141-105 kubenswrapper[2579]: W0911 03:40:22.621363    2579 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 11 03:40:22.625595 ip-10-0-141-105 kubenswrapper[2579]: W0911 03:40:22.621366    2579 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 11 03:40:22.625595 ip-10-0-141-105 kubenswrapper[2579]: W0911 03:40:22.621368    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 11 03:40:22.625595 ip-10-0-141-105 kubenswrapper[2579]: W0911 03:40:22.621371    2579 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 11 03:40:22.625595 ip-10-0-141-105 kubenswrapper[2579]: W0911 03:40:22.621373    2579 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 11 03:40:22.625595 ip-10-0-141-105 kubenswrapper[2579]: W0911 03:40:22.621375    2579 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 11 03:40:22.625595 ip-10-0-141-105 kubenswrapper[2579]: W0911 03:40:22.621378    2579 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 11 03:40:22.625595 ip-10-0-141-105 kubenswrapper[2579]: W0911 03:40:22.621380    2579 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 11 03:40:22.625595 ip-10-0-141-105 kubenswrapper[2579]: W0911 03:40:22.621382    2579 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 11 03:40:22.625595 ip-10-0-141-105 kubenswrapper[2579]: W0911 03:40:22.621385    2579 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 11 03:40:22.625595 ip-10-0-141-105 kubenswrapper[2579]: W0911 03:40:22.621387    2579 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 11 03:40:22.625595 ip-10-0-141-105 kubenswrapper[2579]: W0911 03:40:22.621390    2579 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 11 03:40:22.625595 ip-10-0-141-105 kubenswrapper[2579]: W0911 03:40:22.621392    2579 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 11 03:40:22.626056 ip-10-0-141-105 kubenswrapper[2579]: W0911 03:40:22.621394    2579 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 11 03:40:22.626056 ip-10-0-141-105 kubenswrapper[2579]: W0911 03:40:22.621397    2579 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 11 03:40:22.626056 ip-10-0-141-105 kubenswrapper[2579]: W0911 03:40:22.621399    2579 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 11 03:40:22.626056 ip-10-0-141-105 kubenswrapper[2579]: W0911 03:40:22.621402    2579 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 11 03:40:22.626056 ip-10-0-141-105 kubenswrapper[2579]: W0911 03:40:22.621404    2579 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 11 03:40:22.626056 ip-10-0-141-105 kubenswrapper[2579]: W0911 03:40:22.621407    2579 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 11 03:40:22.626056 ip-10-0-141-105 kubenswrapper[2579]: W0911 03:40:22.621409    2579 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 11 03:40:22.626056 ip-10-0-141-105 kubenswrapper[2579]: W0911 03:40:22.621411    2579 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 11 03:40:22.626056 ip-10-0-141-105 kubenswrapper[2579]: W0911 03:40:22.621414    2579 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 11 03:40:22.626056 ip-10-0-141-105 kubenswrapper[2579]: W0911 03:40:22.621416    2579 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 11 03:40:22.626056 ip-10-0-141-105 kubenswrapper[2579]: W0911 03:40:22.621420    2579 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 11 03:40:22.626056 ip-10-0-141-105 kubenswrapper[2579]: W0911 03:40:22.621425    2579 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 11 03:40:22.626056 ip-10-0-141-105 kubenswrapper[2579]: W0911 03:40:22.621427    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 11 03:40:22.626056 ip-10-0-141-105 kubenswrapper[2579]: W0911 03:40:22.621430    2579 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 11 03:40:22.626056 ip-10-0-141-105 kubenswrapper[2579]: W0911 03:40:22.621432    2579 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 11 03:40:22.626056 ip-10-0-141-105 kubenswrapper[2579]: W0911 03:40:22.621435    2579 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 11 03:40:22.626056 ip-10-0-141-105 kubenswrapper[2579]: W0911 03:40:22.621437    2579 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 11 03:40:22.626056 ip-10-0-141-105 kubenswrapper[2579]: W0911 03:40:22.621440    2579 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 11 03:40:22.626056 ip-10-0-141-105 kubenswrapper[2579]: W0911 03:40:22.621442    2579 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 11 03:40:22.626540 ip-10-0-141-105 kubenswrapper[2579]: W0911 03:40:22.621446    2579 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 11 03:40:22.626540 ip-10-0-141-105 kubenswrapper[2579]: W0911 03:40:22.621448    2579 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 11 03:40:22.626540 ip-10-0-141-105 kubenswrapper[2579]: W0911 03:40:22.621451    2579 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 11 03:40:22.626540 ip-10-0-141-105 kubenswrapper[2579]: W0911 03:40:22.621453    2579 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 11 03:40:22.626540 ip-10-0-141-105 kubenswrapper[2579]: W0911 03:40:22.621456    2579 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 11 03:40:22.626540 ip-10-0-141-105 kubenswrapper[2579]: W0911 03:40:22.621458    2579 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 11 03:40:22.626540 ip-10-0-141-105 kubenswrapper[2579]: W0911 03:40:22.621461    2579 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 11 03:40:22.626540 ip-10-0-141-105 kubenswrapper[2579]: W0911 03:40:22.621463    2579 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 11 03:40:22.626540 ip-10-0-141-105 kubenswrapper[2579]: W0911 03:40:22.621466    2579 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 11 03:40:22.626540 ip-10-0-141-105 kubenswrapper[2579]: W0911 03:40:22.621468    2579 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 11 03:40:22.626540 ip-10-0-141-105 kubenswrapper[2579]: W0911 03:40:22.621470    2579 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 11 03:40:22.626540 ip-10-0-141-105 kubenswrapper[2579]: W0911 03:40:22.621473    2579 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 11 03:40:22.626540 ip-10-0-141-105 kubenswrapper[2579]: W0911 03:40:22.621475    2579 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 11 03:40:22.626540 ip-10-0-141-105 kubenswrapper[2579]: W0911 03:40:22.621478    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 11 03:40:22.626540 ip-10-0-141-105 kubenswrapper[2579]: W0911 03:40:22.621480    2579 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 11 03:40:22.626540 ip-10-0-141-105 kubenswrapper[2579]: W0911 03:40:22.621483    2579 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 11 03:40:22.626540 ip-10-0-141-105 kubenswrapper[2579]: W0911 03:40:22.621485    2579 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 11 03:40:22.626540 ip-10-0-141-105 kubenswrapper[2579]: W0911 03:40:22.621487    2579 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 11 03:40:22.626540 ip-10-0-141-105 kubenswrapper[2579]: W0911 03:40:22.621490    2579 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 11 03:40:22.626540 ip-10-0-141-105 kubenswrapper[2579]: W0911 03:40:22.621493    2579 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 11 03:40:22.627014 ip-10-0-141-105 kubenswrapper[2579]: W0911 03:40:22.621496    2579 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 11 03:40:22.627014 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:40:22.621589    2579 flags.go:64] FLAG: --address="0.0.0.0"
Sep 11 03:40:22.627014 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:40:22.621626    2579 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Sep 11 03:40:22.627014 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:40:22.621632    2579 flags.go:64] FLAG: --anonymous-auth="true"
Sep 11 03:40:22.627014 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:40:22.621638    2579 flags.go:64] FLAG: --application-metrics-count-limit="100"
Sep 11 03:40:22.627014 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:40:22.621642    2579 flags.go:64] FLAG: --authentication-token-webhook="false"
Sep 11 03:40:22.627014 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:40:22.621645    2579 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Sep 11 03:40:22.627014 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:40:22.621649    2579 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Sep 11 03:40:22.627014 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:40:22.621653    2579 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Sep 11 03:40:22.627014 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:40:22.621656    2579 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Sep 11 03:40:22.627014 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:40:22.621659    2579 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Sep 11 03:40:22.627014 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:40:22.621663    2579 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Sep 11 03:40:22.627014 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:40:22.621667    2579 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Sep 11 03:40:22.627014 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:40:22.621670    2579 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Sep 11 03:40:22.627014 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:40:22.621673    2579 flags.go:64] FLAG: --cgroup-root=""
Sep 11 03:40:22.627014 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:40:22.621676    2579 flags.go:64] FLAG: --cgroups-per-qos="true"
Sep 11 03:40:22.627014 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:40:22.621679    2579 flags.go:64] FLAG: --client-ca-file=""
Sep 11 03:40:22.627014 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:40:22.621682    2579 flags.go:64] FLAG: --cloud-provider="external"
Sep 11 03:40:22.627014 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:40:22.621685    2579 flags.go:64] FLAG: --cluster-dns="[]"
Sep 11 03:40:22.627014 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:40:22.621689    2579 flags.go:64] FLAG: --cluster-domain=""
Sep 11 03:40:22.627014 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:40:22.621692    2579 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Sep 11 03:40:22.627014 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:40:22.621695    2579 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Sep 11 03:40:22.627014 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:40:22.621698    2579 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Sep 11 03:40:22.627014 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:40:22.621701    2579 flags.go:64] FLAG: --container-log-max-files="5"
Sep 11 03:40:22.627658 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:40:22.621704    2579 flags.go:64] FLAG: --container-log-max-size="10Mi"
Sep 11 03:40:22.627658 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:40:22.621707    2579 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Sep 11 03:40:22.627658 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:40:22.621710    2579 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Sep 11 03:40:22.627658 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:40:22.621713    2579 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Sep 11 03:40:22.627658 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:40:22.621716    2579 flags.go:64] FLAG: --contention-profiling="false"
Sep 11 03:40:22.627658 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:40:22.621719    2579 flags.go:64] FLAG: --cpu-cfs-quota="true"
Sep 11 03:40:22.627658 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:40:22.621722    2579 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Sep 11 03:40:22.627658 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:40:22.621725    2579 flags.go:64] FLAG: --cpu-manager-policy="none"
Sep 11 03:40:22.627658 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:40:22.621728    2579 flags.go:64] FLAG: --cpu-manager-policy-options=""
Sep 11 03:40:22.627658 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:40:22.621732    2579 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Sep 11 03:40:22.627658 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:40:22.621734    2579 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Sep 11 03:40:22.627658 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:40:22.621737    2579 flags.go:64] FLAG: --enable-debugging-handlers="true"
Sep 11 03:40:22.627658 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:40:22.621741    2579 flags.go:64] FLAG: --enable-load-reader="false"
Sep 11 03:40:22.627658 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:40:22.621744    2579 flags.go:64] FLAG: --enable-server="true"
Sep 11 03:40:22.627658 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:40:22.621748    2579 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Sep 11 03:40:22.627658 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:40:22.621752    2579 flags.go:64] FLAG: --event-burst="100"
Sep 11 03:40:22.627658 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:40:22.621755    2579 flags.go:64] FLAG: --event-qps="50"
Sep 11 03:40:22.627658 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:40:22.621758    2579 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Sep 11 03:40:22.627658 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:40:22.621761    2579 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Sep 11 03:40:22.627658 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:40:22.621763    2579 flags.go:64] FLAG: --eviction-hard=""
Sep 11 03:40:22.627658 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:40:22.621769    2579 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Sep 11 03:40:22.627658 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:40:22.621772    2579 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Sep 11 03:40:22.627658 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:40:22.621775    2579 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Sep 11 03:40:22.627658 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:40:22.621778    2579 flags.go:64] FLAG: --eviction-soft=""
Sep 11 03:40:22.627658 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:40:22.621781    2579 flags.go:64] FLAG: --eviction-soft-grace-period=""
Sep 11 03:40:22.628272 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:40:22.621784    2579 flags.go:64] FLAG: --exit-on-lock-contention="false"
Sep 11 03:40:22.628272 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:40:22.621787    2579 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Sep 11 03:40:22.628272 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:40:22.621790    2579 flags.go:64] FLAG: --experimental-mounter-path=""
Sep 11 03:40:22.628272 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:40:22.621792    2579 flags.go:64] FLAG: --fail-cgroupv1="false"
Sep 11 03:40:22.628272 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:40:22.621795    2579 flags.go:64] FLAG: --fail-swap-on="true"
Sep 11 03:40:22.628272 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:40:22.621798    2579 flags.go:64] FLAG: --feature-gates=""
Sep 11 03:40:22.628272 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:40:22.621802    2579 flags.go:64] FLAG: --file-check-frequency="20s"
Sep 11 03:40:22.628272 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:40:22.621805    2579 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Sep 11 03:40:22.628272 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:40:22.621808    2579 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Sep 11 03:40:22.628272 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:40:22.621811    2579 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Sep 11 03:40:22.628272 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:40:22.621814    2579 flags.go:64] FLAG: --healthz-port="10248"
Sep 11 03:40:22.628272 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:40:22.621817    2579 flags.go:64] FLAG: --help="false"
Sep 11 03:40:22.628272 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:40:22.621820    2579 flags.go:64] FLAG: --hostname-override="ip-10-0-141-105.ec2.internal"
Sep 11 03:40:22.628272 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:40:22.621823    2579 flags.go:64] FLAG: --housekeeping-interval="10s"
Sep 11 03:40:22.628272 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:40:22.621825    2579 flags.go:64] FLAG: --http-check-frequency="20s"
Sep 11 03:40:22.628272 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:40:22.621828    2579 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Sep 11 03:40:22.628272 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:40:22.621831    2579 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Sep 11 03:40:22.628272 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:40:22.621834    2579 flags.go:64] FLAG: --image-gc-high-threshold="85"
Sep 11 03:40:22.628272 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:40:22.621837    2579 flags.go:64] FLAG: --image-gc-low-threshold="80"
Sep 11 03:40:22.628272 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:40:22.621841    2579 flags.go:64] FLAG: --image-service-endpoint=""
Sep 11 03:40:22.628272 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:40:22.621844    2579 flags.go:64] FLAG: --kernel-memcg-notification="false"
Sep 11 03:40:22.628272 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:40:22.621847    2579 flags.go:64] FLAG: --kube-api-burst="100"
Sep 11 03:40:22.628272 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:40:22.621850    2579 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Sep 11 03:40:22.628866 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:40:22.621853    2579 flags.go:64] FLAG: --kube-api-qps="50"
Sep 11 03:40:22.628866 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:40:22.621856    2579 flags.go:64] FLAG: --kube-reserved=""
Sep 11 03:40:22.628866 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:40:22.621859    2579 flags.go:64] FLAG: --kube-reserved-cgroup=""
Sep 11 03:40:22.628866 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:40:22.621862    2579 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Sep 11 03:40:22.628866 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:40:22.621866    2579 flags.go:64] FLAG: --kubelet-cgroups=""
Sep 11 03:40:22.628866 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:40:22.621868    2579 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Sep 11 03:40:22.628866 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:40:22.621871    2579 flags.go:64] FLAG: --lock-file=""
Sep 11 03:40:22.628866 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:40:22.621874    2579 flags.go:64] FLAG: --log-cadvisor-usage="false"
Sep 11 03:40:22.628866 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:40:22.621877    2579 flags.go:64] FLAG: --log-flush-frequency="5s"
Sep 11 03:40:22.628866 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:40:22.621880    2579 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Sep 11 03:40:22.628866 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:40:22.621885    2579 flags.go:64] FLAG: --log-json-split-stream="false"
Sep 11 03:40:22.628866 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:40:22.621888    2579 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Sep 11 03:40:22.628866 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:40:22.621890    2579 flags.go:64] FLAG: --log-text-split-stream="false"
Sep 11 03:40:22.628866 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:40:22.621893    2579 flags.go:64] FLAG: --logging-format="text"
Sep 11 03:40:22.628866 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:40:22.621896    2579 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Sep 11 03:40:22.628866 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:40:22.621899    2579 flags.go:64] FLAG: --make-iptables-util-chains="true"
Sep 11 03:40:22.628866 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:40:22.621902    2579 flags.go:64] FLAG: --manifest-url=""
Sep 11 03:40:22.628866 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:40:22.621904    2579 flags.go:64] FLAG: --manifest-url-header=""
Sep 11 03:40:22.628866 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:40:22.621908    2579 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Sep 11 03:40:22.628866 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:40:22.621911    2579 flags.go:64] FLAG: --max-open-files="1000000"
Sep 11 03:40:22.628866 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:40:22.621915    2579 flags.go:64] FLAG: --max-pods="110"
Sep 11 03:40:22.628866 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:40:22.621918    2579 flags.go:64] FLAG: --maximum-dead-containers="-1"
Sep 11 03:40:22.628866 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:40:22.621921    2579 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Sep 11 03:40:22.628866 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:40:22.621923    2579 flags.go:64] FLAG: --memory-manager-policy="None"
Sep 11 03:40:22.628866 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:40:22.621926    2579 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Sep 11 03:40:22.629520 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:40:22.621929    2579 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Sep 11 03:40:22.629520 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:40:22.621932    2579 flags.go:64] FLAG: --node-ip="0.0.0.0"
Sep 11 03:40:22.629520 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:40:22.621935    2579 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Sep 11 03:40:22.629520 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:40:22.621943    2579 flags.go:64] FLAG: --node-status-max-images="50"
Sep 11 03:40:22.629520 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:40:22.621946    2579 flags.go:64] FLAG: --node-status-update-frequency="10s"
Sep 11 03:40:22.629520 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:40:22.621949    2579 flags.go:64] FLAG: --oom-score-adj="-999"
Sep 11 03:40:22.629520 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:40:22.621952    2579 flags.go:64] FLAG: --pod-cidr=""
Sep 11 03:40:22.629520 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:40:22.621956    2579 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:eee392dbe3ac2ed4a38bca29ae3bd486f5b7780406c62c600154cd1906871b10"
Sep 11 03:40:22.629520 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:40:22.621961    2579 flags.go:64] FLAG: --pod-manifest-path=""
Sep 11 03:40:22.629520 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:40:22.621964    2579 flags.go:64] FLAG: --pod-max-pids="-1"
Sep 11 03:40:22.629520 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:40:22.621967    2579 flags.go:64] FLAG: --pods-per-core="0"
Sep 11 03:40:22.629520 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:40:22.621970    2579 flags.go:64] FLAG: --port="10250"
Sep 11 03:40:22.629520 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:40:22.621974    2579 flags.go:64] FLAG: --protect-kernel-defaults="false"
Sep 11 03:40:22.629520 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:40:22.621977    2579 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-038127e4781a07339"
Sep 11 03:40:22.629520 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:40:22.621980    2579 flags.go:64] FLAG: --qos-reserved=""
Sep 11 03:40:22.629520 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:40:22.621983    2579 flags.go:64] FLAG: --read-only-port="10255"
Sep 11 03:40:22.629520 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:40:22.621986    2579 flags.go:64] FLAG: --register-node="true"
Sep 11 03:40:22.629520 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:40:22.621988    2579 flags.go:64] FLAG: --register-with-taints=""
Sep 11 03:40:22.629520 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:40:22.621992    2579 flags.go:64] FLAG: --registry-burst="10"
Sep 11 03:40:22.629520 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:40:22.621995    2579 flags.go:64] FLAG: --registry-qps="5"
Sep 11 03:40:22.629520 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:40:22.621997    2579 flags.go:64] FLAG: --reserved-cpus=""
Sep 11 03:40:22.629520 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:40:22.622000    2579 flags.go:64] FLAG: --reserved-memory=""
Sep 11 03:40:22.629520 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:40:22.622004    2579 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Sep 11 03:40:22.629520 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:40:22.622007    2579 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Sep 11 03:40:22.629520 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:40:22.622010    2579 flags.go:64] FLAG: --rotate-certificates="false"
Sep 11 03:40:22.630192 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:40:22.622012    2579 flags.go:64] FLAG: --rotate-server-certificates="false"
Sep 11 03:40:22.630192 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:40:22.622015    2579 flags.go:64] FLAG: --runonce="false"
Sep 11 03:40:22.630192 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:40:22.622018    2579 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Sep 11 03:40:22.630192 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:40:22.622021    2579 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Sep 11 03:40:22.630192 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:40:22.622024    2579 flags.go:64] FLAG: --seccomp-default="false"
Sep 11 03:40:22.630192 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:40:22.622026    2579 flags.go:64] FLAG: --serialize-image-pulls="true"
Sep 11 03:40:22.630192 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:40:22.622029    2579 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Sep 11 03:40:22.630192 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:40:22.622032    2579 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Sep 11 03:40:22.630192 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:40:22.622035    2579 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Sep 11 03:40:22.630192 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:40:22.622038    2579 flags.go:64] FLAG: --storage-driver-password="root"
Sep 11 03:40:22.630192 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:40:22.622043    2579 flags.go:64] FLAG: --storage-driver-secure="false"
Sep 11 03:40:22.630192 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:40:22.622045    2579 flags.go:64] FLAG: --storage-driver-table="stats"
Sep 11 03:40:22.630192 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:40:22.622048    2579 flags.go:64] FLAG: --storage-driver-user="root"
Sep 11 03:40:22.630192 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:40:22.622051    2579 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Sep 11 03:40:22.630192 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:40:22.622054    2579 flags.go:64] FLAG: --sync-frequency="1m0s"
Sep 11 03:40:22.630192 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:40:22.622057    2579 flags.go:64] FLAG: --system-cgroups=""
Sep 11 03:40:22.630192 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:40:22.622060    2579 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Sep 11 03:40:22.630192 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:40:22.622065    2579 flags.go:64] FLAG: --system-reserved-cgroup=""
Sep 11 03:40:22.630192 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:40:22.622068    2579 flags.go:64] FLAG: --tls-cert-file=""
Sep 11 03:40:22.630192 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:40:22.622072    2579 flags.go:64] FLAG: --tls-cipher-suites="[]"
Sep 11 03:40:22.630192 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:40:22.622076    2579 flags.go:64] FLAG: --tls-min-version=""
Sep 11 03:40:22.630192 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:40:22.622093    2579 flags.go:64] FLAG: --tls-private-key-file=""
Sep 11 03:40:22.630192 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:40:22.622095    2579 flags.go:64] FLAG: --topology-manager-policy="none"
Sep 11 03:40:22.630192 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:40:22.622098    2579 flags.go:64] FLAG: --topology-manager-policy-options=""
Sep 11 03:40:22.630192 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:40:22.622101    2579 flags.go:64] FLAG: --topology-manager-scope="container"
Sep 11 03:40:22.630785 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:40:22.622104    2579 flags.go:64] FLAG: --v="2"
Sep 11 03:40:22.630785 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:40:22.622108    2579 flags.go:64] FLAG: --version="false"
Sep 11 03:40:22.630785 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:40:22.622112    2579 flags.go:64] FLAG: --vmodule=""
Sep 11 03:40:22.630785 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:40:22.622116    2579 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Sep 11 03:40:22.630785 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:40:22.622119    2579 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Sep 11 03:40:22.630785 ip-10-0-141-105 kubenswrapper[2579]: W0911 03:40:22.622249    2579 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 11 03:40:22.630785 ip-10-0-141-105 kubenswrapper[2579]: W0911 03:40:22.622253    2579 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 11 03:40:22.630785 ip-10-0-141-105 kubenswrapper[2579]: W0911 03:40:22.622256    2579 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 11 03:40:22.630785 ip-10-0-141-105 kubenswrapper[2579]: W0911 03:40:22.622258    2579 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 11 03:40:22.630785 ip-10-0-141-105 kubenswrapper[2579]: W0911 03:40:22.622261    2579 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 11 03:40:22.630785 ip-10-0-141-105 kubenswrapper[2579]: W0911 03:40:22.622264    2579 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 11 03:40:22.630785 ip-10-0-141-105 kubenswrapper[2579]: W0911 03:40:22.622267    2579 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 11 03:40:22.630785 ip-10-0-141-105 kubenswrapper[2579]: W0911 03:40:22.622269    2579 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 11 03:40:22.630785 ip-10-0-141-105 kubenswrapper[2579]: W0911 03:40:22.622272    2579 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 11 03:40:22.630785 ip-10-0-141-105 kubenswrapper[2579]: W0911 03:40:22.622274    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 11 03:40:22.630785 ip-10-0-141-105 kubenswrapper[2579]: W0911 03:40:22.622277    2579 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 11 03:40:22.630785 ip-10-0-141-105 kubenswrapper[2579]: W0911 03:40:22.622280    2579 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 11 03:40:22.630785 ip-10-0-141-105 kubenswrapper[2579]: W0911 03:40:22.622283    2579 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 11 03:40:22.630785 ip-10-0-141-105 kubenswrapper[2579]: W0911 03:40:22.622286    2579 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 11 03:40:22.630785 ip-10-0-141-105 kubenswrapper[2579]: W0911 03:40:22.622288    2579 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 11 03:40:22.630785 ip-10-0-141-105 kubenswrapper[2579]: W0911 03:40:22.622290    2579 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 11 03:40:22.631344 ip-10-0-141-105 kubenswrapper[2579]: W0911 03:40:22.622293    2579 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 11 03:40:22.631344 ip-10-0-141-105 kubenswrapper[2579]: W0911 03:40:22.622295    2579 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 11 03:40:22.631344 ip-10-0-141-105 kubenswrapper[2579]: W0911 03:40:22.622298    2579 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 11 03:40:22.631344 ip-10-0-141-105 kubenswrapper[2579]: W0911 03:40:22.622301    2579 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 11 03:40:22.631344 ip-10-0-141-105 kubenswrapper[2579]: W0911 03:40:22.622303    2579 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 11 03:40:22.631344 ip-10-0-141-105 kubenswrapper[2579]: W0911 03:40:22.622307    2579 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 11 03:40:22.631344 ip-10-0-141-105 kubenswrapper[2579]: W0911 03:40:22.622310    2579 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 11 03:40:22.631344 ip-10-0-141-105 kubenswrapper[2579]: W0911 03:40:22.622312    2579 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 11 03:40:22.631344 ip-10-0-141-105 kubenswrapper[2579]: W0911 03:40:22.622315    2579 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 11 03:40:22.631344 ip-10-0-141-105 kubenswrapper[2579]: W0911 03:40:22.622317    2579 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 11 03:40:22.631344 ip-10-0-141-105 kubenswrapper[2579]: W0911 03:40:22.622320    2579 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 11 03:40:22.631344 ip-10-0-141-105 kubenswrapper[2579]: W0911 03:40:22.622323    2579 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 11 03:40:22.631344 ip-10-0-141-105 kubenswrapper[2579]: W0911 03:40:22.622327    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 11 03:40:22.631344 ip-10-0-141-105 kubenswrapper[2579]: W0911 03:40:22.622331    2579 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 11 03:40:22.631344 ip-10-0-141-105 kubenswrapper[2579]: W0911 03:40:22.622335    2579 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 11 03:40:22.631344 ip-10-0-141-105 kubenswrapper[2579]: W0911 03:40:22.622339    2579 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 11 03:40:22.631344 ip-10-0-141-105 kubenswrapper[2579]: W0911 03:40:22.622342    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 11 03:40:22.631344 ip-10-0-141-105 kubenswrapper[2579]: W0911 03:40:22.622345    2579 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 11 03:40:22.631344 ip-10-0-141-105 kubenswrapper[2579]: W0911 03:40:22.622347    2579 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 11 03:40:22.631821 ip-10-0-141-105 kubenswrapper[2579]: W0911 03:40:22.622350    2579 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 11 03:40:22.631821 ip-10-0-141-105 kubenswrapper[2579]: W0911 03:40:22.622352    2579 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 11 03:40:22.631821 ip-10-0-141-105 kubenswrapper[2579]: W0911 03:40:22.622355    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 11 03:40:22.631821 ip-10-0-141-105 kubenswrapper[2579]: W0911 03:40:22.622357    2579 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 11 03:40:22.631821 ip-10-0-141-105 kubenswrapper[2579]: W0911 03:40:22.622361    2579 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 11 03:40:22.631821 ip-10-0-141-105 kubenswrapper[2579]: W0911 03:40:22.622364    2579 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 11 03:40:22.631821 ip-10-0-141-105 kubenswrapper[2579]: W0911 03:40:22.622367    2579 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 11 03:40:22.631821 ip-10-0-141-105 kubenswrapper[2579]: W0911 03:40:22.622370    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 11 03:40:22.631821 ip-10-0-141-105 kubenswrapper[2579]: W0911 03:40:22.622373    2579 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 11 03:40:22.631821 ip-10-0-141-105 kubenswrapper[2579]: W0911 03:40:22.622376    2579 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 11 03:40:22.631821 ip-10-0-141-105 kubenswrapper[2579]: W0911 03:40:22.622379    2579 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 11 03:40:22.631821 ip-10-0-141-105 kubenswrapper[2579]: W0911 03:40:22.622382    2579 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 11 03:40:22.631821 ip-10-0-141-105 kubenswrapper[2579]: W0911 03:40:22.622385    2579 feature_gate.go:328] unrecognized feature gate: Example
Sep 11 03:40:22.631821 ip-10-0-141-105 kubenswrapper[2579]: W0911 03:40:22.622387    2579 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 11 03:40:22.631821 ip-10-0-141-105 kubenswrapper[2579]: W0911 03:40:22.622390    2579 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 11 03:40:22.631821 ip-10-0-141-105 kubenswrapper[2579]: W0911 03:40:22.622393    2579 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 11 03:40:22.631821 ip-10-0-141-105 kubenswrapper[2579]: W0911 03:40:22.622396    2579 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 11 03:40:22.631821 ip-10-0-141-105 kubenswrapper[2579]: W0911 03:40:22.622398    2579 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 11 03:40:22.631821 ip-10-0-141-105 kubenswrapper[2579]: W0911 03:40:22.622402    2579 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 11 03:40:22.631821 ip-10-0-141-105 kubenswrapper[2579]: W0911 03:40:22.622405    2579 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 11 03:40:22.632326 ip-10-0-141-105 kubenswrapper[2579]: W0911 03:40:22.622407    2579 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 11 03:40:22.632326 ip-10-0-141-105 kubenswrapper[2579]: W0911 03:40:22.622410    2579 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 11 03:40:22.632326 ip-10-0-141-105 kubenswrapper[2579]: W0911 03:40:22.622412    2579 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 11 03:40:22.632326 ip-10-0-141-105 kubenswrapper[2579]: W0911 03:40:22.622415    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 11 03:40:22.632326 ip-10-0-141-105 kubenswrapper[2579]: W0911 03:40:22.622418    2579 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 11 03:40:22.632326 ip-10-0-141-105 kubenswrapper[2579]: W0911 03:40:22.622420    2579 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 11 03:40:22.632326 ip-10-0-141-105 kubenswrapper[2579]: W0911 03:40:22.622424    2579 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 11 03:40:22.632326 ip-10-0-141-105 kubenswrapper[2579]: W0911 03:40:22.622427    2579 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 11 03:40:22.632326 ip-10-0-141-105 kubenswrapper[2579]: W0911 03:40:22.622430    2579 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 11 03:40:22.632326 ip-10-0-141-105 kubenswrapper[2579]: W0911 03:40:22.622433    2579 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 11 03:40:22.632326 ip-10-0-141-105 kubenswrapper[2579]: W0911 03:40:22.622436    2579 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 11 03:40:22.632326 ip-10-0-141-105 kubenswrapper[2579]: W0911 03:40:22.622438    2579 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 11 03:40:22.632326 ip-10-0-141-105 kubenswrapper[2579]: W0911 03:40:22.622441    2579 feature_gate.go:328] unrecognized feature gate: Example2
Sep 11 03:40:22.632326 ip-10-0-141-105 kubenswrapper[2579]: W0911 03:40:22.622443    2579 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 11 03:40:22.632326 ip-10-0-141-105 kubenswrapper[2579]: W0911 03:40:22.622446    2579 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 11 03:40:22.632326 ip-10-0-141-105 kubenswrapper[2579]: W0911 03:40:22.622448    2579 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 11 03:40:22.632326 ip-10-0-141-105 kubenswrapper[2579]: W0911 03:40:22.622451    2579 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 11 03:40:22.632326 ip-10-0-141-105 kubenswrapper[2579]: W0911 03:40:22.622454    2579 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 11 03:40:22.632326 ip-10-0-141-105 kubenswrapper[2579]: W0911 03:40:22.622456    2579 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 11 03:40:22.632326 ip-10-0-141-105 kubenswrapper[2579]: W0911 03:40:22.622459    2579 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 11 03:40:22.632811 ip-10-0-141-105 kubenswrapper[2579]: W0911 03:40:22.622461    2579 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 11 03:40:22.632811 ip-10-0-141-105 kubenswrapper[2579]: W0911 03:40:22.622464    2579 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 11 03:40:22.632811 ip-10-0-141-105 kubenswrapper[2579]: W0911 03:40:22.622467    2579 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 11 03:40:22.632811 ip-10-0-141-105 kubenswrapper[2579]: W0911 03:40:22.622469    2579 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 11 03:40:22.632811 ip-10-0-141-105 kubenswrapper[2579]: W0911 03:40:22.622471    2579 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 11 03:40:22.632811 ip-10-0-141-105 kubenswrapper[2579]: W0911 03:40:22.622474    2579 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 11 03:40:22.632811 ip-10-0-141-105 kubenswrapper[2579]: W0911 03:40:22.622476    2579 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 11 03:40:22.632811 ip-10-0-141-105 kubenswrapper[2579]: W0911 03:40:22.622479    2579 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 11 03:40:22.632811 ip-10-0-141-105 kubenswrapper[2579]: W0911 03:40:22.622481    2579 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 11 03:40:22.632811 ip-10-0-141-105 kubenswrapper[2579]: W0911 03:40:22.622484    2579 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 11 03:40:22.632811 ip-10-0-141-105 kubenswrapper[2579]: W0911 03:40:22.622488    2579 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 11 03:40:22.632811 ip-10-0-141-105 kubenswrapper[2579]: W0911 03:40:22.622490    2579 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 11 03:40:22.632811 ip-10-0-141-105 kubenswrapper[2579]: W0911 03:40:22.622493    2579 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 11 03:40:22.632811 ip-10-0-141-105 kubenswrapper[2579]: W0911 03:40:22.622495    2579 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 11 03:40:22.632811 ip-10-0-141-105 kubenswrapper[2579]: W0911 03:40:22.622498    2579 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 11 03:40:22.632811 ip-10-0-141-105 kubenswrapper[2579]: W0911 03:40:22.622500    2579 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 11 03:40:22.632811 ip-10-0-141-105 kubenswrapper[2579]: W0911 03:40:22.622503    2579 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 11 03:40:22.632811 ip-10-0-141-105 kubenswrapper[2579]: W0911 03:40:22.622505    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 11 03:40:22.632811 ip-10-0-141-105 kubenswrapper[2579]: W0911 03:40:22.622508    2579 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 11 03:40:22.632811 ip-10-0-141-105 kubenswrapper[2579]: W0911 03:40:22.622510    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 11 03:40:22.633317 ip-10-0-141-105 kubenswrapper[2579]: W0911 03:40:22.622512    2579 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 11 03:40:22.633317 ip-10-0-141-105 kubenswrapper[2579]: W0911 03:40:22.622515    2579 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 11 03:40:22.633317 ip-10-0-141-105 kubenswrapper[2579]: W0911 03:40:22.622517    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 11 03:40:22.633317 ip-10-0-141-105 kubenswrapper[2579]: W0911 03:40:22.622521    2579 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 11 03:40:22.633317 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:40:22.623129    2579 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 11 03:40:22.633317 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:40:22.630605    2579 server.go:529] "Kubelet version" kubeletVersion="v1.34.9"
Sep 11 03:40:22.633317 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:40:22.630621    2579 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Sep 11 03:40:22.633317 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:40:22.631525    2579 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Sep 11 03:40:22.633317 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:40:22.631533    2579 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Sep 11 03:40:22.633317 ip-10-0-141-105 kubenswrapper[2579]: W0911 03:40:22.631594    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 11 03:40:22.633317 ip-10-0-141-105 kubenswrapper[2579]: W0911 03:40:22.631601    2579 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 11 03:40:22.633317 ip-10-0-141-105 kubenswrapper[2579]: W0911 03:40:22.631604    2579 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 11 03:40:22.633317 ip-10-0-141-105 kubenswrapper[2579]: W0911 03:40:22.631608    2579 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 11 03:40:22.633317 ip-10-0-141-105 kubenswrapper[2579]: W0911 03:40:22.631611    2579 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 11 03:40:22.633317 ip-10-0-141-105 kubenswrapper[2579]: W0911 03:40:22.631613    2579 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 11 03:40:22.633687 ip-10-0-141-105 kubenswrapper[2579]: W0911 03:40:22.631616    2579 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 11 03:40:22.633687 ip-10-0-141-105 kubenswrapper[2579]: W0911 03:40:22.631619    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 11 03:40:22.633687 ip-10-0-141-105 kubenswrapper[2579]: W0911 03:40:22.631622    2579 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 11 03:40:22.633687 ip-10-0-141-105 kubenswrapper[2579]: W0911 03:40:22.631624    2579 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 11 03:40:22.633687 ip-10-0-141-105 kubenswrapper[2579]: W0911 03:40:22.631627    2579 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 11 03:40:22.633687 ip-10-0-141-105 kubenswrapper[2579]: W0911 03:40:22.631629    2579 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 11 03:40:22.633687 ip-10-0-141-105 kubenswrapper[2579]: W0911 03:40:22.631633    2579 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 11 03:40:22.633687 ip-10-0-141-105 kubenswrapper[2579]: W0911 03:40:22.631636    2579 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 11 03:40:22.633687 ip-10-0-141-105 kubenswrapper[2579]: W0911 03:40:22.631639    2579 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 11 03:40:22.633687 ip-10-0-141-105 kubenswrapper[2579]: W0911 03:40:22.631642    2579 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 11 03:40:22.633687 ip-10-0-141-105 kubenswrapper[2579]: W0911 03:40:22.631645    2579 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 11 03:40:22.633687 ip-10-0-141-105 kubenswrapper[2579]: W0911 03:40:22.631647    2579 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 11 03:40:22.633687 ip-10-0-141-105 kubenswrapper[2579]: W0911 03:40:22.631650    2579 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 11 03:40:22.633687 ip-10-0-141-105 kubenswrapper[2579]: W0911 03:40:22.631653    2579 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 11 03:40:22.633687 ip-10-0-141-105 kubenswrapper[2579]: W0911 03:40:22.631656    2579 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 11 03:40:22.633687 ip-10-0-141-105 kubenswrapper[2579]: W0911 03:40:22.631660    2579 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 11 03:40:22.633687 ip-10-0-141-105 kubenswrapper[2579]: W0911 03:40:22.631663    2579 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 11 03:40:22.633687 ip-10-0-141-105 kubenswrapper[2579]: W0911 03:40:22.631666    2579 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 11 03:40:22.633687 ip-10-0-141-105 kubenswrapper[2579]: W0911 03:40:22.631669    2579 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 11 03:40:22.633687 ip-10-0-141-105 kubenswrapper[2579]: W0911 03:40:22.631672    2579 feature_gate.go:328] unrecognized feature gate: Example2
Sep 11 03:40:22.634305 ip-10-0-141-105 kubenswrapper[2579]: W0911 03:40:22.631674    2579 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 11 03:40:22.634305 ip-10-0-141-105 kubenswrapper[2579]: W0911 03:40:22.631677    2579 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 11 03:40:22.634305 ip-10-0-141-105 kubenswrapper[2579]: W0911 03:40:22.631679    2579 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 11 03:40:22.634305 ip-10-0-141-105 kubenswrapper[2579]: W0911 03:40:22.631683    2579 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 11 03:40:22.634305 ip-10-0-141-105 kubenswrapper[2579]: W0911 03:40:22.631686    2579 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 11 03:40:22.634305 ip-10-0-141-105 kubenswrapper[2579]: W0911 03:40:22.631688    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 11 03:40:22.634305 ip-10-0-141-105 kubenswrapper[2579]: W0911 03:40:22.631691    2579 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 11 03:40:22.634305 ip-10-0-141-105 kubenswrapper[2579]: W0911 03:40:22.631695    2579 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 11 03:40:22.634305 ip-10-0-141-105 kubenswrapper[2579]: W0911 03:40:22.631698    2579 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 11 03:40:22.634305 ip-10-0-141-105 kubenswrapper[2579]: W0911 03:40:22.631701    2579 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 11 03:40:22.634305 ip-10-0-141-105 kubenswrapper[2579]: W0911 03:40:22.631703    2579 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 11 03:40:22.634305 ip-10-0-141-105 kubenswrapper[2579]: W0911 03:40:22.631706    2579 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 11 03:40:22.634305 ip-10-0-141-105 kubenswrapper[2579]: W0911 03:40:22.631709    2579 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 11 03:40:22.634305 ip-10-0-141-105 kubenswrapper[2579]: W0911 03:40:22.631711    2579 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 11 03:40:22.634305 ip-10-0-141-105 kubenswrapper[2579]: W0911 03:40:22.631714    2579 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 11 03:40:22.634305 ip-10-0-141-105 kubenswrapper[2579]: W0911 03:40:22.631716    2579 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 11 03:40:22.634305 ip-10-0-141-105 kubenswrapper[2579]: W0911 03:40:22.631719    2579 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 11 03:40:22.634305 ip-10-0-141-105 kubenswrapper[2579]: W0911 03:40:22.631721    2579 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 11 03:40:22.634305 ip-10-0-141-105 kubenswrapper[2579]: W0911 03:40:22.631724    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 11 03:40:22.634794 ip-10-0-141-105 kubenswrapper[2579]: W0911 03:40:22.631726    2579 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 11 03:40:22.634794 ip-10-0-141-105 kubenswrapper[2579]: W0911 03:40:22.631729    2579 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 11 03:40:22.634794 ip-10-0-141-105 kubenswrapper[2579]: W0911 03:40:22.631731    2579 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 11 03:40:22.634794 ip-10-0-141-105 kubenswrapper[2579]: W0911 03:40:22.631734    2579 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 11 03:40:22.634794 ip-10-0-141-105 kubenswrapper[2579]: W0911 03:40:22.631737    2579 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 11 03:40:22.634794 ip-10-0-141-105 kubenswrapper[2579]: W0911 03:40:22.631739    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 11 03:40:22.634794 ip-10-0-141-105 kubenswrapper[2579]: W0911 03:40:22.631742    2579 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 11 03:40:22.634794 ip-10-0-141-105 kubenswrapper[2579]: W0911 03:40:22.631745    2579 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 11 03:40:22.634794 ip-10-0-141-105 kubenswrapper[2579]: W0911 03:40:22.631747    2579 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 11 03:40:22.634794 ip-10-0-141-105 kubenswrapper[2579]: W0911 03:40:22.631750    2579 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 11 03:40:22.634794 ip-10-0-141-105 kubenswrapper[2579]: W0911 03:40:22.631752    2579 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 11 03:40:22.634794 ip-10-0-141-105 kubenswrapper[2579]: W0911 03:40:22.631754    2579 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 11 03:40:22.634794 ip-10-0-141-105 kubenswrapper[2579]: W0911 03:40:22.631757    2579 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 11 03:40:22.634794 ip-10-0-141-105 kubenswrapper[2579]: W0911 03:40:22.631759    2579 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 11 03:40:22.634794 ip-10-0-141-105 kubenswrapper[2579]: W0911 03:40:22.631762    2579 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 11 03:40:22.634794 ip-10-0-141-105 kubenswrapper[2579]: W0911 03:40:22.631764    2579 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 11 03:40:22.634794 ip-10-0-141-105 kubenswrapper[2579]: W0911 03:40:22.631767    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 11 03:40:22.634794 ip-10-0-141-105 kubenswrapper[2579]: W0911 03:40:22.631770    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 11 03:40:22.634794 ip-10-0-141-105 kubenswrapper[2579]: W0911 03:40:22.631773    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 11 03:40:22.635266 ip-10-0-141-105 kubenswrapper[2579]: W0911 03:40:22.631775    2579 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 11 03:40:22.635266 ip-10-0-141-105 kubenswrapper[2579]: W0911 03:40:22.631778    2579 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 11 03:40:22.635266 ip-10-0-141-105 kubenswrapper[2579]: W0911 03:40:22.631781    2579 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 11 03:40:22.635266 ip-10-0-141-105 kubenswrapper[2579]: W0911 03:40:22.631783    2579 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 11 03:40:22.635266 ip-10-0-141-105 kubenswrapper[2579]: W0911 03:40:22.631786    2579 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 11 03:40:22.635266 ip-10-0-141-105 kubenswrapper[2579]: W0911 03:40:22.631788    2579 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 11 03:40:22.635266 ip-10-0-141-105 kubenswrapper[2579]: W0911 03:40:22.631791    2579 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 11 03:40:22.635266 ip-10-0-141-105 kubenswrapper[2579]: W0911 03:40:22.631793    2579 feature_gate.go:328] unrecognized feature gate: Example
Sep 11 03:40:22.635266 ip-10-0-141-105 kubenswrapper[2579]: W0911 03:40:22.631796    2579 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 11 03:40:22.635266 ip-10-0-141-105 kubenswrapper[2579]: W0911 03:40:22.631798    2579 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 11 03:40:22.635266 ip-10-0-141-105 kubenswrapper[2579]: W0911 03:40:22.631801    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 11 03:40:22.635266 ip-10-0-141-105 kubenswrapper[2579]: W0911 03:40:22.631803    2579 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 11 03:40:22.635266 ip-10-0-141-105 kubenswrapper[2579]: W0911 03:40:22.631807    2579 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 11 03:40:22.635266 ip-10-0-141-105 kubenswrapper[2579]: W0911 03:40:22.631810    2579 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 11 03:40:22.635266 ip-10-0-141-105 kubenswrapper[2579]: W0911 03:40:22.631814    2579 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 11 03:40:22.635266 ip-10-0-141-105 kubenswrapper[2579]: W0911 03:40:22.631816    2579 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 11 03:40:22.635266 ip-10-0-141-105 kubenswrapper[2579]: W0911 03:40:22.631819    2579 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 11 03:40:22.635266 ip-10-0-141-105 kubenswrapper[2579]: W0911 03:40:22.631822    2579 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 11 03:40:22.635266 ip-10-0-141-105 kubenswrapper[2579]: W0911 03:40:22.631824    2579 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 11 03:40:22.635266 ip-10-0-141-105 kubenswrapper[2579]: W0911 03:40:22.631827    2579 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 11 03:40:22.635797 ip-10-0-141-105 kubenswrapper[2579]: W0911 03:40:22.631830    2579 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 11 03:40:22.635797 ip-10-0-141-105 kubenswrapper[2579]: W0911 03:40:22.631832    2579 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 11 03:40:22.635797 ip-10-0-141-105 kubenswrapper[2579]: W0911 03:40:22.631835    2579 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 11 03:40:22.635797 ip-10-0-141-105 kubenswrapper[2579]: W0911 03:40:22.631838    2579 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 11 03:40:22.635797 ip-10-0-141-105 kubenswrapper[2579]: W0911 03:40:22.631840    2579 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 11 03:40:22.635797 ip-10-0-141-105 kubenswrapper[2579]: W0911 03:40:22.631843    2579 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 11 03:40:22.635797 ip-10-0-141-105 kubenswrapper[2579]: W0911 03:40:22.631845    2579 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 11 03:40:22.635797 ip-10-0-141-105 kubenswrapper[2579]: W0911 03:40:22.631848    2579 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 11 03:40:22.635797 ip-10-0-141-105 kubenswrapper[2579]: W0911 03:40:22.631850    2579 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 11 03:40:22.635797 ip-10-0-141-105 kubenswrapper[2579]: W0911 03:40:22.631853    2579 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 11 03:40:22.635797 ip-10-0-141-105 kubenswrapper[2579]: W0911 03:40:22.631856    2579 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 11 03:40:22.635797 ip-10-0-141-105 kubenswrapper[2579]: W0911 03:40:22.631858    2579 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 11 03:40:22.635797 ip-10-0-141-105 kubenswrapper[2579]: W0911 03:40:22.631861    2579 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 11 03:40:22.635797 ip-10-0-141-105 kubenswrapper[2579]: W0911 03:40:22.631863    2579 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 11 03:40:22.635797 ip-10-0-141-105 kubenswrapper[2579]: W0911 03:40:22.631866    2579 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 11 03:40:22.636250 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:40:22.631871    2579 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 11 03:40:22.636250 ip-10-0-141-105 kubenswrapper[2579]: W0911 03:40:22.631982    2579 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 11 03:40:22.636250 ip-10-0-141-105 kubenswrapper[2579]: W0911 03:40:22.631987    2579 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 11 03:40:22.636250 ip-10-0-141-105 kubenswrapper[2579]: W0911 03:40:22.631990    2579 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 11 03:40:22.636250 ip-10-0-141-105 kubenswrapper[2579]: W0911 03:40:22.631993    2579 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 11 03:40:22.636250 ip-10-0-141-105 kubenswrapper[2579]: W0911 03:40:22.631996    2579 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 11 03:40:22.636250 ip-10-0-141-105 kubenswrapper[2579]: W0911 03:40:22.631999    2579 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 11 03:40:22.636250 ip-10-0-141-105 kubenswrapper[2579]: W0911 03:40:22.632001    2579 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 11 03:40:22.636250 ip-10-0-141-105 kubenswrapper[2579]: W0911 03:40:22.632004    2579 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 11 03:40:22.636250 ip-10-0-141-105 kubenswrapper[2579]: W0911 03:40:22.632006    2579 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 11 03:40:22.636250 ip-10-0-141-105 kubenswrapper[2579]: W0911 03:40:22.632009    2579 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 11 03:40:22.636250 ip-10-0-141-105 kubenswrapper[2579]: W0911 03:40:22.632012    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 11 03:40:22.636250 ip-10-0-141-105 kubenswrapper[2579]: W0911 03:40:22.632015    2579 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 11 03:40:22.636250 ip-10-0-141-105 kubenswrapper[2579]: W0911 03:40:22.632017    2579 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 11 03:40:22.636250 ip-10-0-141-105 kubenswrapper[2579]: W0911 03:40:22.632020    2579 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 11 03:40:22.636707 ip-10-0-141-105 kubenswrapper[2579]: W0911 03:40:22.632022    2579 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 11 03:40:22.636707 ip-10-0-141-105 kubenswrapper[2579]: W0911 03:40:22.632025    2579 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 11 03:40:22.636707 ip-10-0-141-105 kubenswrapper[2579]: W0911 03:40:22.632028    2579 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 11 03:40:22.636707 ip-10-0-141-105 kubenswrapper[2579]: W0911 03:40:22.632030    2579 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 11 03:40:22.636707 ip-10-0-141-105 kubenswrapper[2579]: W0911 03:40:22.632033    2579 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 11 03:40:22.636707 ip-10-0-141-105 kubenswrapper[2579]: W0911 03:40:22.632035    2579 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 11 03:40:22.636707 ip-10-0-141-105 kubenswrapper[2579]: W0911 03:40:22.632037    2579 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 11 03:40:22.636707 ip-10-0-141-105 kubenswrapper[2579]: W0911 03:40:22.632040    2579 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 11 03:40:22.636707 ip-10-0-141-105 kubenswrapper[2579]: W0911 03:40:22.632042    2579 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 11 03:40:22.636707 ip-10-0-141-105 kubenswrapper[2579]: W0911 03:40:22.632045    2579 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 11 03:40:22.636707 ip-10-0-141-105 kubenswrapper[2579]: W0911 03:40:22.632047    2579 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 11 03:40:22.636707 ip-10-0-141-105 kubenswrapper[2579]: W0911 03:40:22.632050    2579 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 11 03:40:22.636707 ip-10-0-141-105 kubenswrapper[2579]: W0911 03:40:22.632053    2579 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 11 03:40:22.636707 ip-10-0-141-105 kubenswrapper[2579]: W0911 03:40:22.632055    2579 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 11 03:40:22.636707 ip-10-0-141-105 kubenswrapper[2579]: W0911 03:40:22.632058    2579 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 11 03:40:22.636707 ip-10-0-141-105 kubenswrapper[2579]: W0911 03:40:22.632060    2579 feature_gate.go:328] unrecognized feature gate: Example2
Sep 11 03:40:22.636707 ip-10-0-141-105 kubenswrapper[2579]: W0911 03:40:22.632063    2579 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 11 03:40:22.636707 ip-10-0-141-105 kubenswrapper[2579]: W0911 03:40:22.632065    2579 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 11 03:40:22.636707 ip-10-0-141-105 kubenswrapper[2579]: W0911 03:40:22.632068    2579 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 11 03:40:22.636707 ip-10-0-141-105 kubenswrapper[2579]: W0911 03:40:22.632070    2579 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 11 03:40:22.637307 ip-10-0-141-105 kubenswrapper[2579]: W0911 03:40:22.632072    2579 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 11 03:40:22.637307 ip-10-0-141-105 kubenswrapper[2579]: W0911 03:40:22.632075    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 11 03:40:22.637307 ip-10-0-141-105 kubenswrapper[2579]: W0911 03:40:22.632077    2579 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 11 03:40:22.637307 ip-10-0-141-105 kubenswrapper[2579]: W0911 03:40:22.632096    2579 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 11 03:40:22.637307 ip-10-0-141-105 kubenswrapper[2579]: W0911 03:40:22.632099    2579 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 11 03:40:22.637307 ip-10-0-141-105 kubenswrapper[2579]: W0911 03:40:22.632101    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 11 03:40:22.637307 ip-10-0-141-105 kubenswrapper[2579]: W0911 03:40:22.632104    2579 feature_gate.go:328] unrecognized feature gate: Example
Sep 11 03:40:22.637307 ip-10-0-141-105 kubenswrapper[2579]: W0911 03:40:22.632107    2579 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 11 03:40:22.637307 ip-10-0-141-105 kubenswrapper[2579]: W0911 03:40:22.632109    2579 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 11 03:40:22.637307 ip-10-0-141-105 kubenswrapper[2579]: W0911 03:40:22.632112    2579 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 11 03:40:22.637307 ip-10-0-141-105 kubenswrapper[2579]: W0911 03:40:22.632118    2579 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 11 03:40:22.637307 ip-10-0-141-105 kubenswrapper[2579]: W0911 03:40:22.632121    2579 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 11 03:40:22.637307 ip-10-0-141-105 kubenswrapper[2579]: W0911 03:40:22.632123    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 11 03:40:22.637307 ip-10-0-141-105 kubenswrapper[2579]: W0911 03:40:22.632126    2579 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 11 03:40:22.637307 ip-10-0-141-105 kubenswrapper[2579]: W0911 03:40:22.632129    2579 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 11 03:40:22.637307 ip-10-0-141-105 kubenswrapper[2579]: W0911 03:40:22.632131    2579 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 11 03:40:22.637307 ip-10-0-141-105 kubenswrapper[2579]: W0911 03:40:22.632134    2579 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 11 03:40:22.637307 ip-10-0-141-105 kubenswrapper[2579]: W0911 03:40:22.632136    2579 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 11 03:40:22.637307 ip-10-0-141-105 kubenswrapper[2579]: W0911 03:40:22.632138    2579 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 11 03:40:22.637307 ip-10-0-141-105 kubenswrapper[2579]: W0911 03:40:22.632141    2579 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 11 03:40:22.638033 ip-10-0-141-105 kubenswrapper[2579]: W0911 03:40:22.632145    2579 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 11 03:40:22.638033 ip-10-0-141-105 kubenswrapper[2579]: W0911 03:40:22.632148    2579 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 11 03:40:22.638033 ip-10-0-141-105 kubenswrapper[2579]: W0911 03:40:22.632152    2579 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 11 03:40:22.638033 ip-10-0-141-105 kubenswrapper[2579]: W0911 03:40:22.632156    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 11 03:40:22.638033 ip-10-0-141-105 kubenswrapper[2579]: W0911 03:40:22.632159    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 11 03:40:22.638033 ip-10-0-141-105 kubenswrapper[2579]: W0911 03:40:22.632161    2579 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 11 03:40:22.638033 ip-10-0-141-105 kubenswrapper[2579]: W0911 03:40:22.632164    2579 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 11 03:40:22.638033 ip-10-0-141-105 kubenswrapper[2579]: W0911 03:40:22.632166    2579 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 11 03:40:22.638033 ip-10-0-141-105 kubenswrapper[2579]: W0911 03:40:22.632169    2579 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 11 03:40:22.638033 ip-10-0-141-105 kubenswrapper[2579]: W0911 03:40:22.632172    2579 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 11 03:40:22.638033 ip-10-0-141-105 kubenswrapper[2579]: W0911 03:40:22.632175    2579 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 11 03:40:22.638033 ip-10-0-141-105 kubenswrapper[2579]: W0911 03:40:22.632178    2579 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 11 03:40:22.638033 ip-10-0-141-105 kubenswrapper[2579]: W0911 03:40:22.632182    2579 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 11 03:40:22.638033 ip-10-0-141-105 kubenswrapper[2579]: W0911 03:40:22.632185    2579 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 11 03:40:22.638033 ip-10-0-141-105 kubenswrapper[2579]: W0911 03:40:22.632188    2579 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 11 03:40:22.638033 ip-10-0-141-105 kubenswrapper[2579]: W0911 03:40:22.632190    2579 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 11 03:40:22.638033 ip-10-0-141-105 kubenswrapper[2579]: W0911 03:40:22.632193    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 11 03:40:22.638033 ip-10-0-141-105 kubenswrapper[2579]: W0911 03:40:22.632196    2579 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 11 03:40:22.638759 ip-10-0-141-105 kubenswrapper[2579]: W0911 03:40:22.632198    2579 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 11 03:40:22.638759 ip-10-0-141-105 kubenswrapper[2579]: W0911 03:40:22.632201    2579 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 11 03:40:22.638759 ip-10-0-141-105 kubenswrapper[2579]: W0911 03:40:22.632203    2579 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 11 03:40:22.638759 ip-10-0-141-105 kubenswrapper[2579]: W0911 03:40:22.632206    2579 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 11 03:40:22.638759 ip-10-0-141-105 kubenswrapper[2579]: W0911 03:40:22.632208    2579 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 11 03:40:22.638759 ip-10-0-141-105 kubenswrapper[2579]: W0911 03:40:22.632211    2579 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 11 03:40:22.638759 ip-10-0-141-105 kubenswrapper[2579]: W0911 03:40:22.632214    2579 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 11 03:40:22.638759 ip-10-0-141-105 kubenswrapper[2579]: W0911 03:40:22.632216    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 11 03:40:22.638759 ip-10-0-141-105 kubenswrapper[2579]: W0911 03:40:22.632219    2579 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 11 03:40:22.638759 ip-10-0-141-105 kubenswrapper[2579]: W0911 03:40:22.632221    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 11 03:40:22.638759 ip-10-0-141-105 kubenswrapper[2579]: W0911 03:40:22.632224    2579 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 11 03:40:22.638759 ip-10-0-141-105 kubenswrapper[2579]: W0911 03:40:22.632226    2579 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 11 03:40:22.638759 ip-10-0-141-105 kubenswrapper[2579]: W0911 03:40:22.632229    2579 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 11 03:40:22.638759 ip-10-0-141-105 kubenswrapper[2579]: W0911 03:40:22.632231    2579 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 11 03:40:22.638759 ip-10-0-141-105 kubenswrapper[2579]: W0911 03:40:22.632234    2579 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 11 03:40:22.638759 ip-10-0-141-105 kubenswrapper[2579]: W0911 03:40:22.632236    2579 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 11 03:40:22.638759 ip-10-0-141-105 kubenswrapper[2579]: W0911 03:40:22.632239    2579 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 11 03:40:22.638759 ip-10-0-141-105 kubenswrapper[2579]: W0911 03:40:22.632242    2579 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 11 03:40:22.638759 ip-10-0-141-105 kubenswrapper[2579]: W0911 03:40:22.632244    2579 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 11 03:40:22.638759 ip-10-0-141-105 kubenswrapper[2579]: W0911 03:40:22.632246    2579 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 11 03:40:22.639579 ip-10-0-141-105 kubenswrapper[2579]: W0911 03:40:22.632249    2579 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 11 03:40:22.639579 ip-10-0-141-105 kubenswrapper[2579]: W0911 03:40:22.632252    2579 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 11 03:40:22.639579 ip-10-0-141-105 kubenswrapper[2579]: W0911 03:40:22.632254    2579 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 11 03:40:22.639579 ip-10-0-141-105 kubenswrapper[2579]: W0911 03:40:22.632257    2579 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 11 03:40:22.639579 ip-10-0-141-105 kubenswrapper[2579]: W0911 03:40:22.632259    2579 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 11 03:40:22.639579 ip-10-0-141-105 kubenswrapper[2579]: W0911 03:40:22.632261    2579 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 11 03:40:22.639579 ip-10-0-141-105 kubenswrapper[2579]: W0911 03:40:22.632264    2579 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 11 03:40:22.639579 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:40:22.632269    2579 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 11 03:40:22.639579 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:40:22.632370    2579 server.go:962] "Client rotation is on, will bootstrap in background"
Sep 11 03:40:22.639579 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:40:22.635186    2579 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Sep 11 03:40:22.639579 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:40:22.636008    2579 server.go:1019] "Starting client certificate rotation"
Sep 11 03:40:22.639579 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:40:22.636125    2579 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 11 03:40:22.639579 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:40:22.636175    2579 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 11 03:40:22.639984 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:40:22.639323    2579 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Sep 11 03:40:22.644459 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:40:22.644435    2579 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Sep 11 03:40:22.654211 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:40:22.654192    2579 log.go:25] "Validated CRI v1 runtime API"
Sep 11 03:40:22.657925 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:40:22.657910    2579 log.go:25] "Validated CRI v1 image API"
Sep 11 03:40:22.658792 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:40:22.658775    2579 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Sep 11 03:40:22.662404 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:40:22.662383    2579 fs.go:135] Filesystem UUIDs: map[7B77-95E7:/dev/nvme0n1p2 a8f19d23-e5a6-4be9-8d4e-b52f8329a6f4:/dev/nvme0n1p3 e6c0153c-9401-4440-9e56-7efb9e1dca75:/dev/nvme0n1p4]
Sep 11 03:40:22.662491 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:40:22.662403    2579 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}]
Sep 11 03:40:22.666704 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:40:22.666678    2579 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Sep 11 03:40:22.668951 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:40:22.668823    2579 manager.go:216] Machine: {Timestamp:2026-09-11 03:40:22.66757388 +0000 UTC m=+0.322779439 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3102606 MemoryCapacity:32812154880 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec249297bab71799c8218c2c5b3475f0 SystemUUID:ec249297-bab7-1799-c821-8c2c5b3475f0 BootID:78730f39-03a6-476e-8d25-1e8a730297ef Filesystems:[{Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16406077440 Type:vfs Inodes:4005390 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6562430976 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16406077440 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:8d:3f:f0:38:0d Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:8d:3f:f0:38:0d Speed:0 Mtu:9001} {Name:ovs-system MacAddress:16:26:53:ad:a4:26 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:32812154880 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:34603008 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Sep 11 03:40:22.668951 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:40:22.668933    2579 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Sep 11 03:40:22.669077 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:40:22.669023    2579 manager.go:232] Version: {KernelVersion:5.14.0-570.140.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260908-1 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Sep 11 03:40:22.671955 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:40:22.671931    2579 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Sep 11 03:40:22.672118 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:40:22.671959    2579 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-141-105.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Sep 11 03:40:22.673287 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:40:22.673276    2579 topology_manager.go:138] "Creating topology manager with none policy"
Sep 11 03:40:22.673329 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:40:22.673290    2579 container_manager_linux.go:309] "Creating device plugin manager"
Sep 11 03:40:22.673329 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:40:22.673297    2579 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 11 03:40:22.674104 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:40:22.674075    2579 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 11 03:40:22.675265 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:40:22.675254    2579 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager"
Sep 11 03:40:22.679159 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:40:22.679147    2579 state_mem.go:36] "Initialized new in-memory state store"
Sep 11 03:40:22.679278 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:40:22.679269    2579 server.go:1240] "Using root directory" path="/var/lib/kubelet"
Sep 11 03:40:22.679321 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:40:22.679300    2579 kubelet.go:486] "Attempting to sync node with API server"
Sep 11 03:40:22.679321 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:40:22.679311    2579 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests"
Sep 11 03:40:22.679377 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:40:22.679325    2579 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Sep 11 03:40:22.679377 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:40:22.679335    2579 kubelet.go:398] "Adding apiserver pod source"
Sep 11 03:40:22.679377 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:40:22.679345    2579 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Sep 11 03:40:22.679726 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:40:22.679709    2579 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Sep 11 03:40:22.679806 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:40:22.679736    2579 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Sep 11 03:40:22.681860 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:40:22.681843    2579 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.12-5.rhaos4.21.gitd1ec63c.el9" apiVersion="v1"
Sep 11 03:40:22.685363 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:40:22.685346    2579 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Sep 11 03:40:22.685448 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:40:22.685377    2579 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Sep 11 03:40:22.687160 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:40:22.687147    2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Sep 11 03:40:22.687160 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:40:22.687163    2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Sep 11 03:40:22.687321 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:40:22.687169    2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Sep 11 03:40:22.687321 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:40:22.687175    2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Sep 11 03:40:22.687321 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:40:22.687184    2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Sep 11 03:40:22.687321 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:40:22.687189    2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Sep 11 03:40:22.687321 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:40:22.687203    2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Sep 11 03:40:22.687321 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:40:22.687208    2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Sep 11 03:40:22.687321 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:40:22.687215    2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Sep 11 03:40:22.687321 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:40:22.687220    2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Sep 11 03:40:22.687321 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:40:22.687228    2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Sep 11 03:40:22.687321 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:40:22.687233    2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Sep 11 03:40:22.688118 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:40:22.688108    2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Sep 11 03:40:22.688118 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:40:22.688117    2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Sep 11 03:40:22.689530 ip-10-0-141-105 kubenswrapper[2579]: E0911 03:40:22.689505    2579 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
Sep 11 03:40:22.689608 ip-10-0-141-105 kubenswrapper[2579]: E0911 03:40:22.689507    2579 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-141-105.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
Sep 11 03:40:22.691872 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:40:22.691844    2579 server.go:1268] "Started kubelet"
Sep 11 03:40:22.691969 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:40:22.691912    2579 server.go:180] "Starting to listen" address="0.0.0.0" port=10250
Sep 11 03:40:22.692016 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:40:22.691959    2579 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Sep 11 03:40:22.692055 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:40:22.692032    2579 server_v1.go:49] "podresources" method="list" useActivePods=true
Sep 11 03:40:22.692148 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:40:22.692130    2579 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-rhxp4"
Sep 11 03:40:22.692784 ip-10-0-141-105 systemd[1]: Started Kubernetes Kubelet.
Sep 11 03:40:22.693605 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:40:22.693435    2579 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Sep 11 03:40:22.694745 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:40:22.694732    2579 server.go:310] "Adding debug handlers to kubelet server"
Sep 11 03:40:22.699763 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:40:22.699744    2579 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-141-105.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 11 03:40:22.699915 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:40:22.699889    2579 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-rhxp4"
Sep 11 03:40:22.699996 ip-10-0-141-105 kubenswrapper[2579]: E0911 03:40:22.699919    2579 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Sep 11 03:40:22.700884 ip-10-0-141-105 kubenswrapper[2579]: E0911 03:40:22.699817    2579 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-141-105.ec2.internal.18d4272c2ee64705  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-141-105.ec2.internal,UID:ip-10-0-141-105.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-141-105.ec2.internal,},FirstTimestamp:2026-09-11 03:40:22.691817221 +0000 UTC m=+0.347022778,LastTimestamp:2026-09-11 03:40:22.691817221 +0000 UTC m=+0.347022778,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-141-105.ec2.internal,}"
Sep 11 03:40:22.701736 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:40:22.701721    2579 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Sep 11 03:40:22.702461 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:40:22.702444    2579 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
Sep 11 03:40:22.702643 ip-10-0-141-105 kubenswrapper[2579]: E0911 03:40:22.702591    2579 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-105.ec2.internal\" not found"
Sep 11 03:40:22.703333 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:40:22.703314    2579 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Sep 11 03:40:22.703333 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:40:22.703331    2579 volume_manager.go:311] "The desired_state_of_world populator starts"
Sep 11 03:40:22.703498 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:40:22.703344    2579 volume_manager.go:313] "Starting Kubelet Volume Manager"
Sep 11 03:40:22.703498 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:40:22.703419    2579 factory.go:153] Registering CRI-O factory
Sep 11 03:40:22.703498 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:40:22.703454    2579 factory.go:223] Registration of the crio container factory successfully
Sep 11 03:40:22.703498 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:40:22.703472    2579 reconstruct.go:97] "Volume reconstruction finished"
Sep 11 03:40:22.703498 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:40:22.703479    2579 reconciler.go:29] "Reconciler: start to sync state"
Sep 11 03:40:22.703721 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:40:22.703517    2579 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Sep 11 03:40:22.703721 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:40:22.703527    2579 factory.go:55] Registering systemd factory
Sep 11 03:40:22.703721 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:40:22.703536    2579 factory.go:223] Registration of the systemd container factory successfully
Sep 11 03:40:22.703721 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:40:22.703553    2579 factory.go:103] Registering Raw factory
Sep 11 03:40:22.703721 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:40:22.703566    2579 manager.go:1196] Started watching for new ooms in manager
Sep 11 03:40:22.704053 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:40:22.704039    2579 manager.go:349] Starting recovery of all containers
Sep 11 03:40:22.705655 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:40:22.705631    2579 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 11 03:40:22.708396 ip-10-0-141-105 kubenswrapper[2579]: E0911 03:40:22.708366    2579 nodelease.go:49] "Failed to get node when trying to set owner ref to the node lease" err="nodes \"ip-10-0-141-105.ec2.internal\" not found" node="ip-10-0-141-105.ec2.internal"
Sep 11 03:40:22.716283 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:40:22.716265    2579 manager.go:354] Recovery completed
Sep 11 03:40:22.720284 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:40:22.720272    2579 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 11 03:40:22.722177 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:40:22.722159    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-105.ec2.internal" event="NodeHasSufficientMemory"
Sep 11 03:40:22.722256 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:40:22.722189    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-105.ec2.internal" event="NodeHasNoDiskPressure"
Sep 11 03:40:22.722256 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:40:22.722199    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-105.ec2.internal" event="NodeHasSufficientPID"
Sep 11 03:40:22.722808 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:40:22.722794    2579 cpu_manager.go:222] "Starting CPU manager" policy="none"
Sep 11 03:40:22.722851 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:40:22.722809    2579 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s"
Sep 11 03:40:22.722851 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:40:22.722813    2579 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 11 03:40:22.722851 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:40:22.722828    2579 state_mem.go:36] "Initialized new in-memory state store"
Sep 11 03:40:22.723903 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:40:22.723881    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-105.ec2.internal" event="NodeHasSufficientMemory"
Sep 11 03:40:22.723976 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:40:22.723917    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-105.ec2.internal" event="NodeHasNoDiskPressure"
Sep 11 03:40:22.723976 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:40:22.723921    2579 policy_none.go:49] "None policy: Start"
Sep 11 03:40:22.723976 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:40:22.723933    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-105.ec2.internal" event="NodeHasSufficientPID"
Sep 11 03:40:22.723976 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:40:22.723934    2579 memory_manager.go:187] "Starting memorymanager" policy="None"
Sep 11 03:40:22.723976 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:40:22.723965    2579 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Sep 11 03:40:22.724893 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:40:22.724881    2579 policy_none.go:47] "Start"
Sep 11 03:40:22.762695 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:40:22.762677    2579 manager.go:337] "Starting Device Plugin manager"
Sep 11 03:40:22.775955 ip-10-0-141-105 kubenswrapper[2579]: E0911 03:40:22.762722    2579 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Sep 11 03:40:22.775955 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:40:22.762736    2579 server.go:87] "Starting device plugin registration server"
Sep 11 03:40:22.775955 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:40:22.763037    2579 eviction_manager.go:189] "Eviction manager: starting control loop"
Sep 11 03:40:22.775955 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:40:22.763051    2579 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Sep 11 03:40:22.775955 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:40:22.763161    2579 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Sep 11 03:40:22.775955 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:40:22.763262    2579 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts"
Sep 11 03:40:22.775955 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:40:22.763271    2579 plugin_manager.go:118] "Starting Kubelet Plugin Manager"
Sep 11 03:40:22.775955 ip-10-0-141-105 kubenswrapper[2579]: E0911 03:40:22.763759    2579 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Sep 11 03:40:22.775955 ip-10-0-141-105 kubenswrapper[2579]: E0911 03:40:22.763804    2579 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-141-105.ec2.internal\" not found"
Sep 11 03:40:22.843804 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:40:22.843764    2579 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Sep 11 03:40:22.845220 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:40:22.845199    2579 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Sep 11 03:40:22.845314 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:40:22.845268    2579 status_manager.go:244] "Starting to sync pod status with apiserver"
Sep 11 03:40:22.845314 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:40:22.845299    2579 kubelet.go:2448] "Starting kubelet main sync loop"
Sep 11 03:40:22.845411 ip-10-0-141-105 kubenswrapper[2579]: E0911 03:40:22.845348    2579 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Sep 11 03:40:22.847809 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:40:22.847789    2579 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 11 03:40:22.863835 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:40:22.863797    2579 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 11 03:40:22.864753 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:40:22.864728    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-105.ec2.internal" event="NodeHasSufficientMemory"
Sep 11 03:40:22.864861 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:40:22.864770    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-105.ec2.internal" event="NodeHasNoDiskPressure"
Sep 11 03:40:22.864861 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:40:22.864786    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-105.ec2.internal" event="NodeHasSufficientPID"
Sep 11 03:40:22.864861 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:40:22.864817    2579 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-141-105.ec2.internal"
Sep 11 03:40:22.875849 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:40:22.875786    2579 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-141-105.ec2.internal"
Sep 11 03:40:22.875849 ip-10-0-141-105 kubenswrapper[2579]: E0911 03:40:22.875813    2579 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-141-105.ec2.internal\": node \"ip-10-0-141-105.ec2.internal\" not found"
Sep 11 03:40:22.899618 ip-10-0-141-105 kubenswrapper[2579]: E0911 03:40:22.899593    2579 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-105.ec2.internal\" not found"
Sep 11 03:40:22.945893 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:40:22.945863    2579 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-105.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-141-105.ec2.internal"]
Sep 11 03:40:22.946008 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:40:22.945947    2579 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 11 03:40:22.946868 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:40:22.946852    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-105.ec2.internal" event="NodeHasSufficientMemory"
Sep 11 03:40:22.946955 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:40:22.946887    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-105.ec2.internal" event="NodeHasNoDiskPressure"
Sep 11 03:40:22.946955 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:40:22.946897    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-105.ec2.internal" event="NodeHasSufficientPID"
Sep 11 03:40:22.948146 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:40:22.948134    2579 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 11 03:40:22.948309 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:40:22.948295    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-105.ec2.internal"
Sep 11 03:40:22.948357 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:40:22.948349    2579 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 11 03:40:22.948887 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:40:22.948857    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-105.ec2.internal" event="NodeHasSufficientMemory"
Sep 11 03:40:22.948959 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:40:22.948890    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-105.ec2.internal" event="NodeHasNoDiskPressure"
Sep 11 03:40:22.948959 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:40:22.948903    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-105.ec2.internal" event="NodeHasSufficientPID"
Sep 11 03:40:22.949397 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:40:22.949384    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-105.ec2.internal" event="NodeHasSufficientMemory"
Sep 11 03:40:22.949486 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:40:22.949413    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-105.ec2.internal" event="NodeHasNoDiskPressure"
Sep 11 03:40:22.949486 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:40:22.949428    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-105.ec2.internal" event="NodeHasSufficientPID"
Sep 11 03:40:22.950674 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:40:22.950659    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-105.ec2.internal"
Sep 11 03:40:22.950772 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:40:22.950691    2579 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 11 03:40:22.951388 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:40:22.951374    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-105.ec2.internal" event="NodeHasSufficientMemory"
Sep 11 03:40:22.951457 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:40:22.951401    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-105.ec2.internal" event="NodeHasNoDiskPressure"
Sep 11 03:40:22.951457 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:40:22.951416    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-105.ec2.internal" event="NodeHasSufficientPID"
Sep 11 03:40:22.975707 ip-10-0-141-105 kubenswrapper[2579]: E0911 03:40:22.975689    2579 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-141-105.ec2.internal\" not found" node="ip-10-0-141-105.ec2.internal"
Sep 11 03:40:22.980559 ip-10-0-141-105 kubenswrapper[2579]: E0911 03:40:22.980542    2579 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-141-105.ec2.internal\" not found" node="ip-10-0-141-105.ec2.internal"
Sep 11 03:40:23.000174 ip-10-0-141-105 kubenswrapper[2579]: E0911 03:40:23.000152    2579 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-105.ec2.internal\" not found"
Sep 11 03:40:23.100270 ip-10-0-141-105 kubenswrapper[2579]: E0911 03:40:23.100231    2579 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-105.ec2.internal\" not found"
Sep 11 03:40:23.104504 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:40:23.104489    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/c9ea5fe55135c7b1e5541fdd12fac4be-config\") pod \"kube-apiserver-proxy-ip-10-0-141-105.ec2.internal\" (UID: \"c9ea5fe55135c7b1e5541fdd12fac4be\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-141-105.ec2.internal"
Sep 11 03:40:23.104566 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:40:23.104511    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/466709fffb065514ddf1d3dfe2c8d5d2-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-141-105.ec2.internal\" (UID: \"466709fffb065514ddf1d3dfe2c8d5d2\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-105.ec2.internal"
Sep 11 03:40:23.104566 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:40:23.104530    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/466709fffb065514ddf1d3dfe2c8d5d2-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-141-105.ec2.internal\" (UID: \"466709fffb065514ddf1d3dfe2c8d5d2\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-105.ec2.internal"
Sep 11 03:40:23.200992 ip-10-0-141-105 kubenswrapper[2579]: E0911 03:40:23.200917    2579 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-105.ec2.internal\" not found"
Sep 11 03:40:23.205250 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:40:23.205229    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/c9ea5fe55135c7b1e5541fdd12fac4be-config\") pod \"kube-apiserver-proxy-ip-10-0-141-105.ec2.internal\" (UID: \"c9ea5fe55135c7b1e5541fdd12fac4be\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-141-105.ec2.internal"
Sep 11 03:40:23.205356 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:40:23.205258    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/466709fffb065514ddf1d3dfe2c8d5d2-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-141-105.ec2.internal\" (UID: \"466709fffb065514ddf1d3dfe2c8d5d2\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-105.ec2.internal"
Sep 11 03:40:23.205356 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:40:23.205280    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/466709fffb065514ddf1d3dfe2c8d5d2-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-141-105.ec2.internal\" (UID: \"466709fffb065514ddf1d3dfe2c8d5d2\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-105.ec2.internal"
Sep 11 03:40:23.205356 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:40:23.205339    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/466709fffb065514ddf1d3dfe2c8d5d2-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-141-105.ec2.internal\" (UID: \"466709fffb065514ddf1d3dfe2c8d5d2\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-105.ec2.internal"
Sep 11 03:40:23.205477 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:40:23.205341    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/c9ea5fe55135c7b1e5541fdd12fac4be-config\") pod \"kube-apiserver-proxy-ip-10-0-141-105.ec2.internal\" (UID: \"c9ea5fe55135c7b1e5541fdd12fac4be\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-141-105.ec2.internal"
Sep 11 03:40:23.205477 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:40:23.205385    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/466709fffb065514ddf1d3dfe2c8d5d2-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-141-105.ec2.internal\" (UID: \"466709fffb065514ddf1d3dfe2c8d5d2\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-105.ec2.internal"
Sep 11 03:40:23.278358 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:40:23.278330    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-105.ec2.internal"
Sep 11 03:40:23.284653 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:40:23.284528    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-105.ec2.internal"
Sep 11 03:40:23.301763 ip-10-0-141-105 kubenswrapper[2579]: E0911 03:40:23.301734    2579 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-105.ec2.internal\" not found"
Sep 11 03:40:23.402255 ip-10-0-141-105 kubenswrapper[2579]: E0911 03:40:23.402211    2579 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-105.ec2.internal\" not found"
Sep 11 03:40:23.502789 ip-10-0-141-105 kubenswrapper[2579]: E0911 03:40:23.502717    2579 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-105.ec2.internal\" not found"
Sep 11 03:40:23.603191 ip-10-0-141-105 kubenswrapper[2579]: E0911 03:40:23.603152    2579 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-105.ec2.internal\" not found"
Sep 11 03:40:23.635515 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:40:23.635486    2579 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Sep 11 03:40:23.635929 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:40:23.635620    2579 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Sep 11 03:40:23.635929 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:40:23.635634    2579 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Sep 11 03:40:23.693421 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:40:23.693392    2579 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 11 03:40:23.702200 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:40:23.702161    2579 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-09-10 03:35:23 +0000 UTC" deadline="2028-06-18 16:45:59.028643492 +0000 UTC"
Sep 11 03:40:23.702306 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:40:23.702194    2579 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Sep 11 03:40:23.702357 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:40:23.702195    2579 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="15517h5m35.326452344s"
Sep 11 03:40:23.702627 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:40:23.702614    2579 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-105.ec2.internal"
Sep 11 03:40:23.712901 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:40:23.712882    2579 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Sep 11 03:40:23.714427 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:40:23.714413    2579 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-105.ec2.internal"
Sep 11 03:40:23.715197 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:40:23.715182    2579 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Sep 11 03:40:23.723989 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:40:23.723972    2579 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Sep 11 03:40:23.735159 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:40:23.735142    2579 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-p5cs8"
Sep 11 03:40:23.742581 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:40:23.742566    2579 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-p5cs8"
Sep 11 03:40:23.931741 ip-10-0-141-105 kubenswrapper[2579]: W0911 03:40:23.931699    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod466709fffb065514ddf1d3dfe2c8d5d2.slice/crio-0667a90be31d1d4db838854212dacfd25bc6af25bf1c617058b29080d75f9b0f WatchSource:0}: Error finding container 0667a90be31d1d4db838854212dacfd25bc6af25bf1c617058b29080d75f9b0f: Status 404 returned error can't find the container with id 0667a90be31d1d4db838854212dacfd25bc6af25bf1c617058b29080d75f9b0f
Sep 11 03:40:23.932375 ip-10-0-141-105 kubenswrapper[2579]: W0911 03:40:23.932007    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc9ea5fe55135c7b1e5541fdd12fac4be.slice/crio-38086cc8ad9d216d887b9175a690d73f82255d78a246536895637c77d52d88ff WatchSource:0}: Error finding container 38086cc8ad9d216d887b9175a690d73f82255d78a246536895637c77d52d88ff: Status 404 returned error can't find the container with id 38086cc8ad9d216d887b9175a690d73f82255d78a246536895637c77d52d88ff
Sep 11 03:40:23.936847 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:40:23.936828    2579 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 11 03:40:24.126781 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:40:24.126686    2579 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 11 03:40:24.659316 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:40:24.659283    2579 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 11 03:40:24.680694 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:40:24.680666    2579 apiserver.go:52] "Watching apiserver"
Sep 11 03:40:24.690531 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:40:24.690500    2579 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Sep 11 03:40:24.691002 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:40:24.690964    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kube-system/global-pull-secret-syncer-jwm97","openshift-cluster-node-tuning-operator/tuned-nv6wz","openshift-dns/node-resolver-mhjgr","openshift-image-registry/node-ca-zg5qq","openshift-multus/multus-48m57","openshift-multus/network-metrics-daemon-9cmdj","openshift-network-diagnostics/network-check-target-p54z5","openshift-network-operator/iptables-alerter-2lk5x","kube-system/konnectivity-agent-s9xqv","kube-system/kube-apiserver-proxy-ip-10-0-141-105.ec2.internal","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5lt7m","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-105.ec2.internal","openshift-multus/multus-additional-cni-plugins-cgtgv","openshift-ovn-kubernetes/ovnkube-node-j72hj"]
Sep 11 03:40:24.692709 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:40:24.692685    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-p54z5"
Sep 11 03:40:24.692811 ip-10-0-141-105 kubenswrapper[2579]: E0911 03:40:24.692776    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-p54z5" podUID="801557d7-c47c-43dd-977e-58a9766436c1"
Sep 11 03:40:24.693781 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:40:24.693765    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-cgtgv"
Sep 11 03:40:24.695069 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:40:24.695048    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-mhjgr"
Sep 11 03:40:24.697139 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:40:24.697122    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-zg5qq"
Sep 11 03:40:24.699217 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:40:24.698245    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Sep 11 03:40:24.699217 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:40:24.698308    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-m74qq\""
Sep 11 03:40:24.699217 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:40:24.698472    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Sep 11 03:40:24.699217 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:40:24.698614    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-4qfc6\""
Sep 11 03:40:24.699217 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:40:24.698668    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Sep 11 03:40:24.699217 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:40:24.698770    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Sep 11 03:40:24.699217 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:40:24.698896    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Sep 11 03:40:24.699217 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:40:24.698901    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Sep 11 03:40:24.699217 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:40:24.699066    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Sep 11 03:40:24.699730 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:40:24.699461    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Sep 11 03:40:24.704261 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:40:24.703256    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-48m57"
Sep 11 03:40:24.704261 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:40:24.703407    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9cmdj"
Sep 11 03:40:24.704261 ip-10-0-141-105 kubenswrapper[2579]: E0911 03:40:24.703503    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9cmdj" podUID="3517687a-5b4a-4804-9239-e0a5d1c415e0"
Sep 11 03:40:24.704261 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:40:24.703853    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Sep 11 03:40:24.704261 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:40:24.703922    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Sep 11 03:40:24.704261 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:40:24.704026    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-5c4gv\""
Sep 11 03:40:24.705557 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:40:24.705531    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-s9xqv"
Sep 11 03:40:24.705694 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:40:24.705671    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Sep 11 03:40:24.706038 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:40:24.706019    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-vx8ml\""
Sep 11 03:40:24.707009 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:40:24.706989    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-j72hj"
Sep 11 03:40:24.708517 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:40:24.707927    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Sep 11 03:40:24.708517 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:40:24.708232    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Sep 11 03:40:24.708517 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:40:24.708270    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-5vd9x\""
Sep 11 03:40:24.708778 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:40:24.708762    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-2lk5x"
Sep 11 03:40:24.709716 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:40:24.709695    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Sep 11 03:40:24.709815 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:40:24.709722    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Sep 11 03:40:24.709815 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:40:24.709783    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Sep 11 03:40:24.709815 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:40:24.709789    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Sep 11 03:40:24.709973 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:40:24.709727    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-fmqrc\""
Sep 11 03:40:24.710671 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:40:24.710279    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5lt7m"
Sep 11 03:40:24.711558 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:40:24.711538    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Sep 11 03:40:24.711803 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:40:24.711780    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Sep 11 03:40:24.712056 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:40:24.712039    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Sep 11 03:40:24.712943 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:40:24.712921    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Sep 11 03:40:24.713038 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:40:24.712977    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Sep 11 03:40:24.713038 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:40:24.713003    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-jshf8\""
Sep 11 03:40:24.713216 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:40:24.713200    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Sep 11 03:40:24.713341 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:40:24.713320    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Sep 11 03:40:24.713408 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:40:24.713273    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Sep 11 03:40:24.713584 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:40:24.713321    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-mxbvh\""
Sep 11 03:40:24.713666 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:40:24.713258    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-nv6wz"
Sep 11 03:40:24.713970 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:40:24.713945    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/08caa83a-ff39-4f1a-955c-a3b164eb57f7-cni-binary-copy\") pod \"multus-additional-cni-plugins-cgtgv\" (UID: \"08caa83a-ff39-4f1a-955c-a3b164eb57f7\") " pod="openshift-multus/multus-additional-cni-plugins-cgtgv"
Sep 11 03:40:24.714059 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:40:24.713986    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/9fc36641-9051-4f34-a897-bb4c92977307-host\") pod \"node-ca-zg5qq\" (UID: \"9fc36641-9051-4f34-a897-bb4c92977307\") " pod="openshift-image-registry/node-ca-zg5qq"
Sep 11 03:40:24.715291 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:40:24.715267    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/975d3d70-a13c-46bb-afc2-05c3609050f4-system-cni-dir\") pod \"multus-48m57\" (UID: \"975d3d70-a13c-46bb-afc2-05c3609050f4\") " pod="openshift-multus/multus-48m57"
Sep 11 03:40:24.715376 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:40:24.715307    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/975d3d70-a13c-46bb-afc2-05c3609050f4-host-var-lib-cni-bin\") pod \"multus-48m57\" (UID: \"975d3d70-a13c-46bb-afc2-05c3609050f4\") " pod="openshift-multus/multus-48m57"
Sep 11 03:40:24.715376 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:40:24.715342    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/975d3d70-a13c-46bb-afc2-05c3609050f4-host-var-lib-cni-multus\") pod \"multus-48m57\" (UID: \"975d3d70-a13c-46bb-afc2-05c3609050f4\") " pod="openshift-multus/multus-48m57"
Sep 11 03:40:24.715376 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:40:24.715371    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b8wlz\" (UniqueName: \"kubernetes.io/projected/801557d7-c47c-43dd-977e-58a9766436c1-kube-api-access-b8wlz\") pod \"network-check-target-p54z5\" (UID: \"801557d7-c47c-43dd-977e-58a9766436c1\") " pod="openshift-network-diagnostics/network-check-target-p54z5"
Sep 11 03:40:24.715545 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:40:24.715400    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/975d3d70-a13c-46bb-afc2-05c3609050f4-multus-daemon-config\") pod \"multus-48m57\" (UID: \"975d3d70-a13c-46bb-afc2-05c3609050f4\") " pod="openshift-multus/multus-48m57"
Sep 11 03:40:24.715545 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:40:24.715427    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/08caa83a-ff39-4f1a-955c-a3b164eb57f7-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-cgtgv\" (UID: \"08caa83a-ff39-4f1a-955c-a3b164eb57f7\") " pod="openshift-multus/multus-additional-cni-plugins-cgtgv"
Sep 11 03:40:24.715545 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:40:24.715480    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h565q\" (UniqueName: \"kubernetes.io/projected/08caa83a-ff39-4f1a-955c-a3b164eb57f7-kube-api-access-h565q\") pod \"multus-additional-cni-plugins-cgtgv\" (UID: \"08caa83a-ff39-4f1a-955c-a3b164eb57f7\") " pod="openshift-multus/multus-additional-cni-plugins-cgtgv"
Sep 11 03:40:24.715545 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:40:24.715509    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-597tt\" (UniqueName: \"kubernetes.io/projected/3517687a-5b4a-4804-9239-e0a5d1c415e0-kube-api-access-597tt\") pod \"network-metrics-daemon-9cmdj\" (UID: \"3517687a-5b4a-4804-9239-e0a5d1c415e0\") " pod="openshift-multus/network-metrics-daemon-9cmdj"
Sep 11 03:40:24.715545 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:40:24.715539    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/975d3d70-a13c-46bb-afc2-05c3609050f4-etc-kubernetes\") pod \"multus-48m57\" (UID: \"975d3d70-a13c-46bb-afc2-05c3609050f4\") " pod="openshift-multus/multus-48m57"
Sep 11 03:40:24.715814 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:40:24.715584    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/9fc36641-9051-4f34-a897-bb4c92977307-serviceca\") pod \"node-ca-zg5qq\" (UID: \"9fc36641-9051-4f34-a897-bb4c92977307\") " pod="openshift-image-registry/node-ca-zg5qq"
Sep 11 03:40:24.715814 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:40:24.715611    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f4lsw\" (UniqueName: \"kubernetes.io/projected/9fc36641-9051-4f34-a897-bb4c92977307-kube-api-access-f4lsw\") pod \"node-ca-zg5qq\" (UID: \"9fc36641-9051-4f34-a897-bb4c92977307\") " pod="openshift-image-registry/node-ca-zg5qq"
Sep 11 03:40:24.715814 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:40:24.715651    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/975d3d70-a13c-46bb-afc2-05c3609050f4-host-run-k8s-cni-cncf-io\") pod \"multus-48m57\" (UID: \"975d3d70-a13c-46bb-afc2-05c3609050f4\") " pod="openshift-multus/multus-48m57"
Sep 11 03:40:24.715814 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:40:24.715675    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/975d3d70-a13c-46bb-afc2-05c3609050f4-host-run-netns\") pod \"multus-48m57\" (UID: \"975d3d70-a13c-46bb-afc2-05c3609050f4\") " pod="openshift-multus/multus-48m57"
Sep 11 03:40:24.715814 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:40:24.715701    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/975d3d70-a13c-46bb-afc2-05c3609050f4-host-run-multus-certs\") pod \"multus-48m57\" (UID: \"975d3d70-a13c-46bb-afc2-05c3609050f4\") " pod="openshift-multus/multus-48m57"
Sep 11 03:40:24.715814 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:40:24.715754    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/975d3d70-a13c-46bb-afc2-05c3609050f4-hostroot\") pod \"multus-48m57\" (UID: \"975d3d70-a13c-46bb-afc2-05c3609050f4\") " pod="openshift-multus/multus-48m57"
Sep 11 03:40:24.715814 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:40:24.715778    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/08caa83a-ff39-4f1a-955c-a3b164eb57f7-cnibin\") pod \"multus-additional-cni-plugins-cgtgv\" (UID: \"08caa83a-ff39-4f1a-955c-a3b164eb57f7\") " pod="openshift-multus/multus-additional-cni-plugins-cgtgv"
Sep 11 03:40:24.716146 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:40:24.715968    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/08caa83a-ff39-4f1a-955c-a3b164eb57f7-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-cgtgv\" (UID: \"08caa83a-ff39-4f1a-955c-a3b164eb57f7\") " pod="openshift-multus/multus-additional-cni-plugins-cgtgv"
Sep 11 03:40:24.716146 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:40:24.716047    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/975d3d70-a13c-46bb-afc2-05c3609050f4-cnibin\") pod \"multus-48m57\" (UID: \"975d3d70-a13c-46bb-afc2-05c3609050f4\") " pod="openshift-multus/multus-48m57"
Sep 11 03:40:24.716146 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:40:24.716073    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/ff572c4a-7a16-4c2b-b0a8-160b3b274834-agent-certs\") pod \"konnectivity-agent-s9xqv\" (UID: \"ff572c4a-7a16-4c2b-b0a8-160b3b274834\") " pod="kube-system/konnectivity-agent-s9xqv"
Sep 11 03:40:24.716146 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:40:24.716112    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3517687a-5b4a-4804-9239-e0a5d1c415e0-metrics-certs\") pod \"network-metrics-daemon-9cmdj\" (UID: \"3517687a-5b4a-4804-9239-e0a5d1c415e0\") " pod="openshift-multus/network-metrics-daemon-9cmdj"
Sep 11 03:40:24.716314 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:40:24.716160    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/08caa83a-ff39-4f1a-955c-a3b164eb57f7-system-cni-dir\") pod \"multus-additional-cni-plugins-cgtgv\" (UID: \"08caa83a-ff39-4f1a-955c-a3b164eb57f7\") " pod="openshift-multus/multus-additional-cni-plugins-cgtgv"
Sep 11 03:40:24.716314 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:40:24.716246    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Sep 11 03:40:24.716402 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:40:24.716377    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Sep 11 03:40:24.716603 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:40:24.716581    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-mqgmz\""
Sep 11 03:40:24.716667 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:40:24.716593    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jwm97"
Sep 11 03:40:24.716667 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:40:24.716621    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/975d3d70-a13c-46bb-afc2-05c3609050f4-multus-cni-dir\") pod \"multus-48m57\" (UID: \"975d3d70-a13c-46bb-afc2-05c3609050f4\") " pod="openshift-multus/multus-48m57"
Sep 11 03:40:24.716667 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:40:24.716652    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fkffr\" (UniqueName: \"kubernetes.io/projected/975d3d70-a13c-46bb-afc2-05c3609050f4-kube-api-access-fkffr\") pod \"multus-48m57\" (UID: \"975d3d70-a13c-46bb-afc2-05c3609050f4\") " pod="openshift-multus/multus-48m57"
Sep 11 03:40:24.716815 ip-10-0-141-105 kubenswrapper[2579]: E0911 03:40:24.716679    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jwm97" podUID="1a7a4b58-e08a-45ef-ab7e-692d3a4f5314"
Sep 11 03:40:24.716815 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:40:24.716679    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c35abd6b-ad57-4b78-b072-56e6cd032381-tmp-dir\") pod \"node-resolver-mhjgr\" (UID: \"c35abd6b-ad57-4b78-b072-56e6cd032381\") " pod="openshift-dns/node-resolver-mhjgr"
Sep 11 03:40:24.716815 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:40:24.716750    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/975d3d70-a13c-46bb-afc2-05c3609050f4-multus-socket-dir-parent\") pod \"multus-48m57\" (UID: \"975d3d70-a13c-46bb-afc2-05c3609050f4\") " pod="openshift-multus/multus-48m57"
Sep 11 03:40:24.716815 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:40:24.716779    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/975d3d70-a13c-46bb-afc2-05c3609050f4-multus-conf-dir\") pod \"multus-48m57\" (UID: \"975d3d70-a13c-46bb-afc2-05c3609050f4\") " pod="openshift-multus/multus-48m57"
Sep 11 03:40:24.717008 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:40:24.716824    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qhxfd\" (UniqueName: \"kubernetes.io/projected/c35abd6b-ad57-4b78-b072-56e6cd032381-kube-api-access-qhxfd\") pod \"node-resolver-mhjgr\" (UID: \"c35abd6b-ad57-4b78-b072-56e6cd032381\") " pod="openshift-dns/node-resolver-mhjgr"
Sep 11 03:40:24.717008 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:40:24.716852    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/08caa83a-ff39-4f1a-955c-a3b164eb57f7-tuning-conf-dir\") pod \"multus-additional-cni-plugins-cgtgv\" (UID: \"08caa83a-ff39-4f1a-955c-a3b164eb57f7\") " pod="openshift-multus/multus-additional-cni-plugins-cgtgv"
Sep 11 03:40:24.717008 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:40:24.716890    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/975d3d70-a13c-46bb-afc2-05c3609050f4-cni-binary-copy\") pod \"multus-48m57\" (UID: \"975d3d70-a13c-46bb-afc2-05c3609050f4\") " pod="openshift-multus/multus-48m57"
Sep 11 03:40:24.717008 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:40:24.716937    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/975d3d70-a13c-46bb-afc2-05c3609050f4-host-var-lib-kubelet\") pod \"multus-48m57\" (UID: \"975d3d70-a13c-46bb-afc2-05c3609050f4\") " pod="openshift-multus/multus-48m57"
Sep 11 03:40:24.717008 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:40:24.716961    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/ff572c4a-7a16-4c2b-b0a8-160b3b274834-konnectivity-ca\") pod \"konnectivity-agent-s9xqv\" (UID: \"ff572c4a-7a16-4c2b-b0a8-160b3b274834\") " pod="kube-system/konnectivity-agent-s9xqv"
Sep 11 03:40:24.717008 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:40:24.716994    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/c35abd6b-ad57-4b78-b072-56e6cd032381-hosts-file\") pod \"node-resolver-mhjgr\" (UID: \"c35abd6b-ad57-4b78-b072-56e6cd032381\") " pod="openshift-dns/node-resolver-mhjgr"
Sep 11 03:40:24.744366 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:40:24.744311    2579 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-10 03:35:24 +0000 UTC" deadline="2028-05-05 13:50:58.370768105 +0000 UTC"
Sep 11 03:40:24.744456 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:40:24.744367    2579 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14458h10m33.626404771s"
Sep 11 03:40:24.804911 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:40:24.804881    2579 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Sep 11 03:40:24.818271 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:40:24.818243    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/202d85de-7969-4e66-a77e-14029cd0597c-device-dir\") pod \"aws-ebs-csi-driver-node-5lt7m\" (UID: \"202d85de-7969-4e66-a77e-14029cd0597c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5lt7m"
Sep 11 03:40:24.818415 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:40:24.818278    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/5c976635-b437-40c3-9721-43754ae3a058-systemd-units\") pod \"ovnkube-node-j72hj\" (UID: \"5c976635-b437-40c3-9721-43754ae3a058\") " pod="openshift-ovn-kubernetes/ovnkube-node-j72hj"
Sep 11 03:40:24.818415 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:40:24.818295    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/5c976635-b437-40c3-9721-43754ae3a058-node-log\") pod \"ovnkube-node-j72hj\" (UID: \"5c976635-b437-40c3-9721-43754ae3a058\") " pod="openshift-ovn-kubernetes/ovnkube-node-j72hj"
Sep 11 03:40:24.818415 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:40:24.818316    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/08caa83a-ff39-4f1a-955c-a3b164eb57f7-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-cgtgv\" (UID: \"08caa83a-ff39-4f1a-955c-a3b164eb57f7\") " pod="openshift-multus/multus-additional-cni-plugins-cgtgv"
Sep 11 03:40:24.818415 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:40:24.818334    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h565q\" (UniqueName: \"kubernetes.io/projected/08caa83a-ff39-4f1a-955c-a3b164eb57f7-kube-api-access-h565q\") pod \"multus-additional-cni-plugins-cgtgv\" (UID: \"08caa83a-ff39-4f1a-955c-a3b164eb57f7\") " pod="openshift-multus/multus-additional-cni-plugins-cgtgv"
Sep 11 03:40:24.818415 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:40:24.818351    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-597tt\" (UniqueName: \"kubernetes.io/projected/3517687a-5b4a-4804-9239-e0a5d1c415e0-kube-api-access-597tt\") pod \"network-metrics-daemon-9cmdj\" (UID: \"3517687a-5b4a-4804-9239-e0a5d1c415e0\") " pod="openshift-multus/network-metrics-daemon-9cmdj"
Sep 11 03:40:24.818415 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:40:24.818366    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/5bf35a4f-d6d3-4c6d-906f-5f88bc6767b6-etc-tuned\") pod \"tuned-nv6wz\" (UID: \"5bf35a4f-d6d3-4c6d-906f-5f88bc6767b6\") " pod="openshift-cluster-node-tuning-operator/tuned-nv6wz"
Sep 11 03:40:24.818415 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:40:24.818380    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/5bf35a4f-d6d3-4c6d-906f-5f88bc6767b6-var-lib-kubelet\") pod \"tuned-nv6wz\" (UID: \"5bf35a4f-d6d3-4c6d-906f-5f88bc6767b6\") " pod="openshift-cluster-node-tuning-operator/tuned-nv6wz"
Sep 11 03:40:24.818415 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:40:24.818395    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/9fc36641-9051-4f34-a897-bb4c92977307-serviceca\") pod \"node-ca-zg5qq\" (UID: \"9fc36641-9051-4f34-a897-bb4c92977307\") " pod="openshift-image-registry/node-ca-zg5qq"
Sep 11 03:40:24.818415 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:40:24.818410    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/5bf35a4f-d6d3-4c6d-906f-5f88bc6767b6-etc-modprobe-d\") pod \"tuned-nv6wz\" (UID: \"5bf35a4f-d6d3-4c6d-906f-5f88bc6767b6\") " pod="openshift-cluster-node-tuning-operator/tuned-nv6wz"
Sep 11 03:40:24.818854 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:40:24.818425    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/5c976635-b437-40c3-9721-43754ae3a058-ovn-node-metrics-cert\") pod \"ovnkube-node-j72hj\" (UID: \"5c976635-b437-40c3-9721-43754ae3a058\") " pod="openshift-ovn-kubernetes/ovnkube-node-j72hj"
Sep 11 03:40:24.818854 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:40:24.818439    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/5bf35a4f-d6d3-4c6d-906f-5f88bc6767b6-etc-sysctl-d\") pod \"tuned-nv6wz\" (UID: \"5bf35a4f-d6d3-4c6d-906f-5f88bc6767b6\") " pod="openshift-cluster-node-tuning-operator/tuned-nv6wz"
Sep 11 03:40:24.818854 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:40:24.818456    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/975d3d70-a13c-46bb-afc2-05c3609050f4-hostroot\") pod \"multus-48m57\" (UID: \"975d3d70-a13c-46bb-afc2-05c3609050f4\") " pod="openshift-multus/multus-48m57"
Sep 11 03:40:24.818854 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:40:24.818472    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/08caa83a-ff39-4f1a-955c-a3b164eb57f7-cnibin\") pod \"multus-additional-cni-plugins-cgtgv\" (UID: \"08caa83a-ff39-4f1a-955c-a3b164eb57f7\") " pod="openshift-multus/multus-additional-cni-plugins-cgtgv"
Sep 11 03:40:24.818854 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:40:24.818487    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/08caa83a-ff39-4f1a-955c-a3b164eb57f7-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-cgtgv\" (UID: \"08caa83a-ff39-4f1a-955c-a3b164eb57f7\") " pod="openshift-multus/multus-additional-cni-plugins-cgtgv"
Sep 11 03:40:24.818854 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:40:24.818503    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3517687a-5b4a-4804-9239-e0a5d1c415e0-metrics-certs\") pod \"network-metrics-daemon-9cmdj\" (UID: \"3517687a-5b4a-4804-9239-e0a5d1c415e0\") " pod="openshift-multus/network-metrics-daemon-9cmdj"
Sep 11 03:40:24.818854 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:40:24.818518    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/202d85de-7969-4e66-a77e-14029cd0597c-socket-dir\") pod \"aws-ebs-csi-driver-node-5lt7m\" (UID: \"202d85de-7969-4e66-a77e-14029cd0597c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5lt7m"
Sep 11 03:40:24.818854 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:40:24.818536    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/1a7a4b58-e08a-45ef-ab7e-692d3a4f5314-kubelet-config\") pod \"global-pull-secret-syncer-jwm97\" (UID: \"1a7a4b58-e08a-45ef-ab7e-692d3a4f5314\") " pod="kube-system/global-pull-secret-syncer-jwm97"
Sep 11 03:40:24.818854 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:40:24.818551    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/20c994e9-80cd-4378-9d56-60044e385640-host-slash\") pod \"iptables-alerter-2lk5x\" (UID: \"20c994e9-80cd-4378-9d56-60044e385640\") " pod="openshift-network-operator/iptables-alerter-2lk5x"
Sep 11 03:40:24.818854 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:40:24.818566    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/08caa83a-ff39-4f1a-955c-a3b164eb57f7-system-cni-dir\") pod \"multus-additional-cni-plugins-cgtgv\" (UID: \"08caa83a-ff39-4f1a-955c-a3b164eb57f7\") " pod="openshift-multus/multus-additional-cni-plugins-cgtgv"
Sep 11 03:40:24.818854 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:40:24.818582    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fkffr\" (UniqueName: \"kubernetes.io/projected/975d3d70-a13c-46bb-afc2-05c3609050f4-kube-api-access-fkffr\") pod \"multus-48m57\" (UID: \"975d3d70-a13c-46bb-afc2-05c3609050f4\") " pod="openshift-multus/multus-48m57"
Sep 11 03:40:24.818854 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:40:24.818596    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/5bf35a4f-d6d3-4c6d-906f-5f88bc6767b6-sys\") pod \"tuned-nv6wz\" (UID: \"5bf35a4f-d6d3-4c6d-906f-5f88bc6767b6\") " pod="openshift-cluster-node-tuning-operator/tuned-nv6wz"
Sep 11 03:40:24.818854 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:40:24.818611    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/5c976635-b437-40c3-9721-43754ae3a058-host-cni-netd\") pod \"ovnkube-node-j72hj\" (UID: \"5c976635-b437-40c3-9721-43754ae3a058\") " pod="openshift-ovn-kubernetes/ovnkube-node-j72hj"
Sep 11 03:40:24.818854 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:40:24.818633    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/5c976635-b437-40c3-9721-43754ae3a058-ovnkube-config\") pod \"ovnkube-node-j72hj\" (UID: \"5c976635-b437-40c3-9721-43754ae3a058\") " pod="openshift-ovn-kubernetes/ovnkube-node-j72hj"
Sep 11 03:40:24.818854 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:40:24.818648    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6d7c6\" (UniqueName: \"kubernetes.io/projected/5c976635-b437-40c3-9721-43754ae3a058-kube-api-access-6d7c6\") pod \"ovnkube-node-j72hj\" (UID: \"5c976635-b437-40c3-9721-43754ae3a058\") " pod="openshift-ovn-kubernetes/ovnkube-node-j72hj"
Sep 11 03:40:24.818854 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:40:24.818664    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/975d3d70-a13c-46bb-afc2-05c3609050f4-multus-conf-dir\") pod \"multus-48m57\" (UID: \"975d3d70-a13c-46bb-afc2-05c3609050f4\") " pod="openshift-multus/multus-48m57"
Sep 11 03:40:24.818854 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:40:24.818684    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/5c976635-b437-40c3-9721-43754ae3a058-var-lib-openvswitch\") pod \"ovnkube-node-j72hj\" (UID: \"5c976635-b437-40c3-9721-43754ae3a058\") " pod="openshift-ovn-kubernetes/ovnkube-node-j72hj"
Sep 11 03:40:24.819383 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:40:24.818699    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/5c976635-b437-40c3-9721-43754ae3a058-run-openvswitch\") pod \"ovnkube-node-j72hj\" (UID: \"5c976635-b437-40c3-9721-43754ae3a058\") " pod="openshift-ovn-kubernetes/ovnkube-node-j72hj"
Sep 11 03:40:24.819383 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:40:24.818714    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/5c976635-b437-40c3-9721-43754ae3a058-log-socket\") pod \"ovnkube-node-j72hj\" (UID: \"5c976635-b437-40c3-9721-43754ae3a058\") " pod="openshift-ovn-kubernetes/ovnkube-node-j72hj"
Sep 11 03:40:24.819383 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:40:24.818729    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/20c994e9-80cd-4378-9d56-60044e385640-iptables-alerter-script\") pod \"iptables-alerter-2lk5x\" (UID: \"20c994e9-80cd-4378-9d56-60044e385640\") " pod="openshift-network-operator/iptables-alerter-2lk5x"
Sep 11 03:40:24.819383 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:40:24.818751    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/975d3d70-a13c-46bb-afc2-05c3609050f4-host-var-lib-kubelet\") pod \"multus-48m57\" (UID: \"975d3d70-a13c-46bb-afc2-05c3609050f4\") " pod="openshift-multus/multus-48m57"
Sep 11 03:40:24.819383 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:40:24.818766    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/202d85de-7969-4e66-a77e-14029cd0597c-registration-dir\") pod \"aws-ebs-csi-driver-node-5lt7m\" (UID: \"202d85de-7969-4e66-a77e-14029cd0597c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5lt7m"
Sep 11 03:40:24.819383 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:40:24.818780    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/5bf35a4f-d6d3-4c6d-906f-5f88bc6767b6-host\") pod \"tuned-nv6wz\" (UID: \"5bf35a4f-d6d3-4c6d-906f-5f88bc6767b6\") " pod="openshift-cluster-node-tuning-operator/tuned-nv6wz"
Sep 11 03:40:24.819383 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:40:24.818794    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/1a7a4b58-e08a-45ef-ab7e-692d3a4f5314-dbus\") pod \"global-pull-secret-syncer-jwm97\" (UID: \"1a7a4b58-e08a-45ef-ab7e-692d3a4f5314\") " pod="kube-system/global-pull-secret-syncer-jwm97"
Sep 11 03:40:24.819383 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:40:24.818815    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/5c976635-b437-40c3-9721-43754ae3a058-env-overrides\") pod \"ovnkube-node-j72hj\" (UID: \"5c976635-b437-40c3-9721-43754ae3a058\") " pod="openshift-ovn-kubernetes/ovnkube-node-j72hj"
Sep 11 03:40:24.819383 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:40:24.818831    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/5c976635-b437-40c3-9721-43754ae3a058-ovnkube-script-lib\") pod \"ovnkube-node-j72hj\" (UID: \"5c976635-b437-40c3-9721-43754ae3a058\") " pod="openshift-ovn-kubernetes/ovnkube-node-j72hj"
Sep 11 03:40:24.819383 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:40:24.818848    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/08caa83a-ff39-4f1a-955c-a3b164eb57f7-cni-binary-copy\") pod \"multus-additional-cni-plugins-cgtgv\" (UID: \"08caa83a-ff39-4f1a-955c-a3b164eb57f7\") " pod="openshift-multus/multus-additional-cni-plugins-cgtgv"
Sep 11 03:40:24.819383 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:40:24.818865    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/9fc36641-9051-4f34-a897-bb4c92977307-host\") pod \"node-ca-zg5qq\" (UID: \"9fc36641-9051-4f34-a897-bb4c92977307\") " pod="openshift-image-registry/node-ca-zg5qq"
Sep 11 03:40:24.819383 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:40:24.818888    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/975d3d70-a13c-46bb-afc2-05c3609050f4-host-var-lib-cni-bin\") pod \"multus-48m57\" (UID: \"975d3d70-a13c-46bb-afc2-05c3609050f4\") " pod="openshift-multus/multus-48m57"
Sep 11 03:40:24.819383 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:40:24.818913    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/975d3d70-a13c-46bb-afc2-05c3609050f4-host-var-lib-cni-multus\") pod \"multus-48m57\" (UID: \"975d3d70-a13c-46bb-afc2-05c3609050f4\") " pod="openshift-multus/multus-48m57"
Sep 11 03:40:24.819383 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:40:24.818932    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/5bf35a4f-d6d3-4c6d-906f-5f88bc6767b6-etc-sysctl-conf\") pod \"tuned-nv6wz\" (UID: \"5bf35a4f-d6d3-4c6d-906f-5f88bc6767b6\") " pod="openshift-cluster-node-tuning-operator/tuned-nv6wz"
Sep 11 03:40:24.819383 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:40:24.818948    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5c976635-b437-40c3-9721-43754ae3a058-host-run-ovn-kubernetes\") pod \"ovnkube-node-j72hj\" (UID: \"5c976635-b437-40c3-9721-43754ae3a058\") " pod="openshift-ovn-kubernetes/ovnkube-node-j72hj"
Sep 11 03:40:24.819383 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:40:24.818965    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/975d3d70-a13c-46bb-afc2-05c3609050f4-multus-daemon-config\") pod \"multus-48m57\" (UID: \"975d3d70-a13c-46bb-afc2-05c3609050f4\") " pod="openshift-multus/multus-48m57"
Sep 11 03:40:24.819383 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:40:24.818980    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/202d85de-7969-4e66-a77e-14029cd0597c-etc-selinux\") pod \"aws-ebs-csi-driver-node-5lt7m\" (UID: \"202d85de-7969-4e66-a77e-14029cd0597c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5lt7m"
Sep 11 03:40:24.820005 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:40:24.818995    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/5bf35a4f-d6d3-4c6d-906f-5f88bc6767b6-etc-systemd\") pod \"tuned-nv6wz\" (UID: \"5bf35a4f-d6d3-4c6d-906f-5f88bc6767b6\") " pod="openshift-cluster-node-tuning-operator/tuned-nv6wz"
Sep 11 03:40:24.820005 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:40:24.819012    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j7jvz\" (UniqueName: \"kubernetes.io/projected/20c994e9-80cd-4378-9d56-60044e385640-kube-api-access-j7jvz\") pod \"iptables-alerter-2lk5x\" (UID: \"20c994e9-80cd-4378-9d56-60044e385640\") " pod="openshift-network-operator/iptables-alerter-2lk5x"
Sep 11 03:40:24.820005 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:40:24.819028    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/5c976635-b437-40c3-9721-43754ae3a058-etc-openvswitch\") pod \"ovnkube-node-j72hj\" (UID: \"5c976635-b437-40c3-9721-43754ae3a058\") " pod="openshift-ovn-kubernetes/ovnkube-node-j72hj"
Sep 11 03:40:24.820005 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:40:24.819044    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/975d3d70-a13c-46bb-afc2-05c3609050f4-etc-kubernetes\") pod \"multus-48m57\" (UID: \"975d3d70-a13c-46bb-afc2-05c3609050f4\") " pod="openshift-multus/multus-48m57"
Sep 11 03:40:24.820005 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:40:24.819060    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f4lsw\" (UniqueName: \"kubernetes.io/projected/9fc36641-9051-4f34-a897-bb4c92977307-kube-api-access-f4lsw\") pod \"node-ca-zg5qq\" (UID: \"9fc36641-9051-4f34-a897-bb4c92977307\") " pod="openshift-image-registry/node-ca-zg5qq"
Sep 11 03:40:24.820005 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:40:24.819077    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/975d3d70-a13c-46bb-afc2-05c3609050f4-host-run-k8s-cni-cncf-io\") pod \"multus-48m57\" (UID: \"975d3d70-a13c-46bb-afc2-05c3609050f4\") " pod="openshift-multus/multus-48m57"
Sep 11 03:40:24.820005 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:40:24.819094    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/975d3d70-a13c-46bb-afc2-05c3609050f4-host-run-netns\") pod \"multus-48m57\" (UID: \"975d3d70-a13c-46bb-afc2-05c3609050f4\") " pod="openshift-multus/multus-48m57"
Sep 11 03:40:24.820005 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:40:24.819110    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/975d3d70-a13c-46bb-afc2-05c3609050f4-host-run-multus-certs\") pod \"multus-48m57\" (UID: \"975d3d70-a13c-46bb-afc2-05c3609050f4\") " pod="openshift-multus/multus-48m57"
Sep 11 03:40:24.820005 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:40:24.819125    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/202d85de-7969-4e66-a77e-14029cd0597c-sys-fs\") pod \"aws-ebs-csi-driver-node-5lt7m\" (UID: \"202d85de-7969-4e66-a77e-14029cd0597c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5lt7m"
Sep 11 03:40:24.820005 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:40:24.819143    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/1a7a4b58-e08a-45ef-ab7e-692d3a4f5314-original-pull-secret\") pod \"global-pull-secret-syncer-jwm97\" (UID: \"1a7a4b58-e08a-45ef-ab7e-692d3a4f5314\") " pod="kube-system/global-pull-secret-syncer-jwm97"
Sep 11 03:40:24.820005 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:40:24.819174    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/5c976635-b437-40c3-9721-43754ae3a058-host-cni-bin\") pod \"ovnkube-node-j72hj\" (UID: \"5c976635-b437-40c3-9721-43754ae3a058\") " pod="openshift-ovn-kubernetes/ovnkube-node-j72hj"
Sep 11 03:40:24.820005 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:40:24.819191    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/975d3d70-a13c-46bb-afc2-05c3609050f4-cnibin\") pod \"multus-48m57\" (UID: \"975d3d70-a13c-46bb-afc2-05c3609050f4\") " pod="openshift-multus/multus-48m57"
Sep 11 03:40:24.820005 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:40:24.819215    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/ff572c4a-7a16-4c2b-b0a8-160b3b274834-agent-certs\") pod \"konnectivity-agent-s9xqv\" (UID: \"ff572c4a-7a16-4c2b-b0a8-160b3b274834\") " pod="kube-system/konnectivity-agent-s9xqv"
Sep 11 03:40:24.820005 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:40:24.819231    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/5bf35a4f-d6d3-4c6d-906f-5f88bc6767b6-etc-sysconfig\") pod \"tuned-nv6wz\" (UID: \"5bf35a4f-d6d3-4c6d-906f-5f88bc6767b6\") " pod="openshift-cluster-node-tuning-operator/tuned-nv6wz"
Sep 11 03:40:24.820005 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:40:24.819247    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/5c976635-b437-40c3-9721-43754ae3a058-host-kubelet\") pod \"ovnkube-node-j72hj\" (UID: \"5c976635-b437-40c3-9721-43754ae3a058\") " pod="openshift-ovn-kubernetes/ovnkube-node-j72hj"
Sep 11 03:40:24.820005 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:40:24.819264    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5c976635-b437-40c3-9721-43754ae3a058-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-j72hj\" (UID: \"5c976635-b437-40c3-9721-43754ae3a058\") " pod="openshift-ovn-kubernetes/ovnkube-node-j72hj"
Sep 11 03:40:24.820005 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:40:24.819283    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/975d3d70-a13c-46bb-afc2-05c3609050f4-multus-cni-dir\") pod \"multus-48m57\" (UID: \"975d3d70-a13c-46bb-afc2-05c3609050f4\") " pod="openshift-multus/multus-48m57"
Sep 11 03:40:24.820742 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:40:24.819300    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c35abd6b-ad57-4b78-b072-56e6cd032381-tmp-dir\") pod \"node-resolver-mhjgr\" (UID: \"c35abd6b-ad57-4b78-b072-56e6cd032381\") " pod="openshift-dns/node-resolver-mhjgr"
Sep 11 03:40:24.820742 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:40:24.819315    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/5bf35a4f-d6d3-4c6d-906f-5f88bc6767b6-lib-modules\") pod \"tuned-nv6wz\" (UID: \"5bf35a4f-d6d3-4c6d-906f-5f88bc6767b6\") " pod="openshift-cluster-node-tuning-operator/tuned-nv6wz"
Sep 11 03:40:24.820742 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:40:24.819330    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/5c976635-b437-40c3-9721-43754ae3a058-host-slash\") pod \"ovnkube-node-j72hj\" (UID: \"5c976635-b437-40c3-9721-43754ae3a058\") " pod="openshift-ovn-kubernetes/ovnkube-node-j72hj"
Sep 11 03:40:24.820742 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:40:24.819347    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/975d3d70-a13c-46bb-afc2-05c3609050f4-multus-socket-dir-parent\") pod \"multus-48m57\" (UID: \"975d3d70-a13c-46bb-afc2-05c3609050f4\") " pod="openshift-multus/multus-48m57"
Sep 11 03:40:24.820742 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:40:24.819364    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qhxfd\" (UniqueName: \"kubernetes.io/projected/c35abd6b-ad57-4b78-b072-56e6cd032381-kube-api-access-qhxfd\") pod \"node-resolver-mhjgr\" (UID: \"c35abd6b-ad57-4b78-b072-56e6cd032381\") " pod="openshift-dns/node-resolver-mhjgr"
Sep 11 03:40:24.820742 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:40:24.819383    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5bf35a4f-d6d3-4c6d-906f-5f88bc6767b6-etc-kubernetes\") pod \"tuned-nv6wz\" (UID: \"5bf35a4f-d6d3-4c6d-906f-5f88bc6767b6\") " pod="openshift-cluster-node-tuning-operator/tuned-nv6wz"
Sep 11 03:40:24.820742 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:40:24.819398    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/5bf35a4f-d6d3-4c6d-906f-5f88bc6767b6-run\") pod \"tuned-nv6wz\" (UID: \"5bf35a4f-d6d3-4c6d-906f-5f88bc6767b6\") " pod="openshift-cluster-node-tuning-operator/tuned-nv6wz"
Sep 11 03:40:24.820742 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:40:24.819417    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f94wq\" (UniqueName: \"kubernetes.io/projected/5bf35a4f-d6d3-4c6d-906f-5f88bc6767b6-kube-api-access-f94wq\") pod \"tuned-nv6wz\" (UID: \"5bf35a4f-d6d3-4c6d-906f-5f88bc6767b6\") " pod="openshift-cluster-node-tuning-operator/tuned-nv6wz"
Sep 11 03:40:24.820742 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:40:24.819433    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/5c976635-b437-40c3-9721-43754ae3a058-host-run-netns\") pod \"ovnkube-node-j72hj\" (UID: \"5c976635-b437-40c3-9721-43754ae3a058\") " pod="openshift-ovn-kubernetes/ovnkube-node-j72hj"
Sep 11 03:40:24.820742 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:40:24.819449    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/5c976635-b437-40c3-9721-43754ae3a058-run-systemd\") pod \"ovnkube-node-j72hj\" (UID: \"5c976635-b437-40c3-9721-43754ae3a058\") " pod="openshift-ovn-kubernetes/ovnkube-node-j72hj"
Sep 11 03:40:24.820742 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:40:24.819465    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/08caa83a-ff39-4f1a-955c-a3b164eb57f7-tuning-conf-dir\") pod \"multus-additional-cni-plugins-cgtgv\" (UID: \"08caa83a-ff39-4f1a-955c-a3b164eb57f7\") " pod="openshift-multus/multus-additional-cni-plugins-cgtgv"
Sep 11 03:40:24.820742 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:40:24.819482    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/975d3d70-a13c-46bb-afc2-05c3609050f4-cni-binary-copy\") pod \"multus-48m57\" (UID: \"975d3d70-a13c-46bb-afc2-05c3609050f4\") " pod="openshift-multus/multus-48m57"
Sep 11 03:40:24.820742 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:40:24.819500    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/ff572c4a-7a16-4c2b-b0a8-160b3b274834-konnectivity-ca\") pod \"konnectivity-agent-s9xqv\" (UID: \"ff572c4a-7a16-4c2b-b0a8-160b3b274834\") " pod="kube-system/konnectivity-agent-s9xqv"
Sep 11 03:40:24.820742 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:40:24.819520    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/c35abd6b-ad57-4b78-b072-56e6cd032381-hosts-file\") pod \"node-resolver-mhjgr\" (UID: \"c35abd6b-ad57-4b78-b072-56e6cd032381\") " pod="openshift-dns/node-resolver-mhjgr"
Sep 11 03:40:24.820742 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:40:24.819535    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/202d85de-7969-4e66-a77e-14029cd0597c-kubelet-dir\") pod \"aws-ebs-csi-driver-node-5lt7m\" (UID: \"202d85de-7969-4e66-a77e-14029cd0597c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5lt7m"
Sep 11 03:40:24.820742 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:40:24.819551    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/5bf35a4f-d6d3-4c6d-906f-5f88bc6767b6-tmp\") pod \"tuned-nv6wz\" (UID: \"5bf35a4f-d6d3-4c6d-906f-5f88bc6767b6\") " pod="openshift-cluster-node-tuning-operator/tuned-nv6wz"
Sep 11 03:40:24.820742 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:40:24.819569    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/975d3d70-a13c-46bb-afc2-05c3609050f4-system-cni-dir\") pod \"multus-48m57\" (UID: \"975d3d70-a13c-46bb-afc2-05c3609050f4\") " pod="openshift-multus/multus-48m57"
Sep 11 03:40:24.821300 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:40:24.819586    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b8wlz\" (UniqueName: \"kubernetes.io/projected/801557d7-c47c-43dd-977e-58a9766436c1-kube-api-access-b8wlz\") pod \"network-check-target-p54z5\" (UID: \"801557d7-c47c-43dd-977e-58a9766436c1\") " pod="openshift-network-diagnostics/network-check-target-p54z5"
Sep 11 03:40:24.821300 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:40:24.819603    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2tqw9\" (UniqueName: \"kubernetes.io/projected/202d85de-7969-4e66-a77e-14029cd0597c-kube-api-access-2tqw9\") pod \"aws-ebs-csi-driver-node-5lt7m\" (UID: \"202d85de-7969-4e66-a77e-14029cd0597c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5lt7m"
Sep 11 03:40:24.821300 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:40:24.819619    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/5c976635-b437-40c3-9721-43754ae3a058-run-ovn\") pod \"ovnkube-node-j72hj\" (UID: \"5c976635-b437-40c3-9721-43754ae3a058\") " pod="openshift-ovn-kubernetes/ovnkube-node-j72hj"
Sep 11 03:40:24.821300 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:40:24.820099    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/08caa83a-ff39-4f1a-955c-a3b164eb57f7-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-cgtgv\" (UID: \"08caa83a-ff39-4f1a-955c-a3b164eb57f7\") " pod="openshift-multus/multus-additional-cni-plugins-cgtgv"
Sep 11 03:40:24.821300 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:40:24.820775    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/975d3d70-a13c-46bb-afc2-05c3609050f4-host-var-lib-kubelet\") pod \"multus-48m57\" (UID: \"975d3d70-a13c-46bb-afc2-05c3609050f4\") " pod="openshift-multus/multus-48m57"
Sep 11 03:40:24.821300 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:40:24.820952    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/08caa83a-ff39-4f1a-955c-a3b164eb57f7-tuning-conf-dir\") pod \"multus-additional-cni-plugins-cgtgv\" (UID: \"08caa83a-ff39-4f1a-955c-a3b164eb57f7\") " pod="openshift-multus/multus-additional-cni-plugins-cgtgv"
Sep 11 03:40:24.821300 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:40:24.820990    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/975d3d70-a13c-46bb-afc2-05c3609050f4-etc-kubernetes\") pod \"multus-48m57\" (UID: \"975d3d70-a13c-46bb-afc2-05c3609050f4\") " pod="openshift-multus/multus-48m57"
Sep 11 03:40:24.821300 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:40:24.821183    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/975d3d70-a13c-46bb-afc2-05c3609050f4-multus-daemon-config\") pod \"multus-48m57\" (UID: \"975d3d70-a13c-46bb-afc2-05c3609050f4\") " pod="openshift-multus/multus-48m57"
Sep 11 03:40:24.821300 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:40:24.821243    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/ff572c4a-7a16-4c2b-b0a8-160b3b274834-konnectivity-ca\") pod \"konnectivity-agent-s9xqv\" (UID: \"ff572c4a-7a16-4c2b-b0a8-160b3b274834\") " pod="kube-system/konnectivity-agent-s9xqv"
Sep 11 03:40:24.821300 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:40:24.821243    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/975d3d70-a13c-46bb-afc2-05c3609050f4-cnibin\") pod \"multus-48m57\" (UID: \"975d3d70-a13c-46bb-afc2-05c3609050f4\") " pod="openshift-multus/multus-48m57"
Sep 11 03:40:24.821300 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:40:24.821295    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/c35abd6b-ad57-4b78-b072-56e6cd032381-hosts-file\") pod \"node-resolver-mhjgr\" (UID: \"c35abd6b-ad57-4b78-b072-56e6cd032381\") " pod="openshift-dns/node-resolver-mhjgr"
Sep 11 03:40:24.821662 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:40:24.821419    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/08caa83a-ff39-4f1a-955c-a3b164eb57f7-cni-binary-copy\") pod \"multus-additional-cni-plugins-cgtgv\" (UID: \"08caa83a-ff39-4f1a-955c-a3b164eb57f7\") " pod="openshift-multus/multus-additional-cni-plugins-cgtgv"
Sep 11 03:40:24.821662 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:40:24.821464    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/975d3d70-a13c-46bb-afc2-05c3609050f4-host-var-lib-cni-bin\") pod \"multus-48m57\" (UID: \"975d3d70-a13c-46bb-afc2-05c3609050f4\") " pod="openshift-multus/multus-48m57"
Sep 11 03:40:24.821662 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:40:24.821479    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c35abd6b-ad57-4b78-b072-56e6cd032381-tmp-dir\") pod \"node-resolver-mhjgr\" (UID: \"c35abd6b-ad57-4b78-b072-56e6cd032381\") " pod="openshift-dns/node-resolver-mhjgr"
Sep 11 03:40:24.821662 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:40:24.821513    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/9fc36641-9051-4f34-a897-bb4c92977307-host\") pod \"node-ca-zg5qq\" (UID: \"9fc36641-9051-4f34-a897-bb4c92977307\") " pod="openshift-image-registry/node-ca-zg5qq"
Sep 11 03:40:24.821662 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:40:24.821531    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/975d3d70-a13c-46bb-afc2-05c3609050f4-multus-socket-dir-parent\") pod \"multus-48m57\" (UID: \"975d3d70-a13c-46bb-afc2-05c3609050f4\") " pod="openshift-multus/multus-48m57"
Sep 11 03:40:24.821662 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:40:24.821554    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/975d3d70-a13c-46bb-afc2-05c3609050f4-host-var-lib-cni-multus\") pod \"multus-48m57\" (UID: \"975d3d70-a13c-46bb-afc2-05c3609050f4\") " pod="openshift-multus/multus-48m57"
Sep 11 03:40:24.821662 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:40:24.821587    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/975d3d70-a13c-46bb-afc2-05c3609050f4-cni-binary-copy\") pod \"multus-48m57\" (UID: \"975d3d70-a13c-46bb-afc2-05c3609050f4\") " pod="openshift-multus/multus-48m57"
Sep 11 03:40:24.821662 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:40:24.821607    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/975d3d70-a13c-46bb-afc2-05c3609050f4-host-run-multus-certs\") pod \"multus-48m57\" (UID: \"975d3d70-a13c-46bb-afc2-05c3609050f4\") " pod="openshift-multus/multus-48m57"
Sep 11 03:40:24.821662 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:40:24.821633    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/975d3d70-a13c-46bb-afc2-05c3609050f4-host-run-k8s-cni-cncf-io\") pod \"multus-48m57\" (UID: \"975d3d70-a13c-46bb-afc2-05c3609050f4\") " pod="openshift-multus/multus-48m57"
Sep 11 03:40:24.821662 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:40:24.821634    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/975d3d70-a13c-46bb-afc2-05c3609050f4-system-cni-dir\") pod \"multus-48m57\" (UID: \"975d3d70-a13c-46bb-afc2-05c3609050f4\") " pod="openshift-multus/multus-48m57"
Sep 11 03:40:24.821979 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:40:24.821668    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/975d3d70-a13c-46bb-afc2-05c3609050f4-host-run-netns\") pod \"multus-48m57\" (UID: \"975d3d70-a13c-46bb-afc2-05c3609050f4\") " pod="openshift-multus/multus-48m57"
Sep 11 03:40:24.821979 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:40:24.821730    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/975d3d70-a13c-46bb-afc2-05c3609050f4-multus-cni-dir\") pod \"multus-48m57\" (UID: \"975d3d70-a13c-46bb-afc2-05c3609050f4\") " pod="openshift-multus/multus-48m57"
Sep 11 03:40:24.821979 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:40:24.821766    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/08caa83a-ff39-4f1a-955c-a3b164eb57f7-cnibin\") pod \"multus-additional-cni-plugins-cgtgv\" (UID: \"08caa83a-ff39-4f1a-955c-a3b164eb57f7\") " pod="openshift-multus/multus-additional-cni-plugins-cgtgv"
Sep 11 03:40:24.821979 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:40:24.821791    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/975d3d70-a13c-46bb-afc2-05c3609050f4-hostroot\") pod \"multus-48m57\" (UID: \"975d3d70-a13c-46bb-afc2-05c3609050f4\") " pod="openshift-multus/multus-48m57"
Sep 11 03:40:24.821979 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:40:24.821837    2579 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Sep 11 03:40:24.821979 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:40:24.820778    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/975d3d70-a13c-46bb-afc2-05c3609050f4-multus-conf-dir\") pod \"multus-48m57\" (UID: \"975d3d70-a13c-46bb-afc2-05c3609050f4\") " pod="openshift-multus/multus-48m57"
Sep 11 03:40:24.821979 ip-10-0-141-105 kubenswrapper[2579]: E0911 03:40:24.821951    2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 11 03:40:24.822236 ip-10-0-141-105 kubenswrapper[2579]: E0911 03:40:24.822027    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3517687a-5b4a-4804-9239-e0a5d1c415e0-metrics-certs podName:3517687a-5b4a-4804-9239-e0a5d1c415e0 nodeName:}" failed. No retries permitted until 2026-09-11 03:40:25.321993068 +0000 UTC m=+2.977198615 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/3517687a-5b4a-4804-9239-e0a5d1c415e0-metrics-certs") pod "network-metrics-daemon-9cmdj" (UID: "3517687a-5b4a-4804-9239-e0a5d1c415e0") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 11 03:40:24.822236 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:40:24.822133    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/9fc36641-9051-4f34-a897-bb4c92977307-serviceca\") pod \"node-ca-zg5qq\" (UID: \"9fc36641-9051-4f34-a897-bb4c92977307\") " pod="openshift-image-registry/node-ca-zg5qq"
Sep 11 03:40:24.822236 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:40:24.822203    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/08caa83a-ff39-4f1a-955c-a3b164eb57f7-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-cgtgv\" (UID: \"08caa83a-ff39-4f1a-955c-a3b164eb57f7\") " pod="openshift-multus/multus-additional-cni-plugins-cgtgv"
Sep 11 03:40:24.822236 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:40:24.822214    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/08caa83a-ff39-4f1a-955c-a3b164eb57f7-system-cni-dir\") pod \"multus-additional-cni-plugins-cgtgv\" (UID: \"08caa83a-ff39-4f1a-955c-a3b164eb57f7\") " pod="openshift-multus/multus-additional-cni-plugins-cgtgv"
Sep 11 03:40:24.826824 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:40:24.826796    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/ff572c4a-7a16-4c2b-b0a8-160b3b274834-agent-certs\") pod \"konnectivity-agent-s9xqv\" (UID: \"ff572c4a-7a16-4c2b-b0a8-160b3b274834\") " pod="kube-system/konnectivity-agent-s9xqv"
Sep 11 03:40:24.828072 ip-10-0-141-105 kubenswrapper[2579]: E0911 03:40:24.828053    2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 11 03:40:24.828212 ip-10-0-141-105 kubenswrapper[2579]: E0911 03:40:24.828080    2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 11 03:40:24.828212 ip-10-0-141-105 kubenswrapper[2579]: E0911 03:40:24.828094    2579 projected.go:196] Error preparing data for projected volume kube-api-access-b8wlz for pod openshift-network-diagnostics/network-check-target-p54z5: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 11 03:40:24.828212 ip-10-0-141-105 kubenswrapper[2579]: E0911 03:40:24.828170    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/801557d7-c47c-43dd-977e-58a9766436c1-kube-api-access-b8wlz podName:801557d7-c47c-43dd-977e-58a9766436c1 nodeName:}" failed. No retries permitted until 2026-09-11 03:40:25.328151992 +0000 UTC m=+2.983357540 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-b8wlz" (UniqueName: "kubernetes.io/projected/801557d7-c47c-43dd-977e-58a9766436c1-kube-api-access-b8wlz") pod "network-check-target-p54z5" (UID: "801557d7-c47c-43dd-977e-58a9766436c1") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 11 03:40:24.829352 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:40:24.829315    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-h565q\" (UniqueName: \"kubernetes.io/projected/08caa83a-ff39-4f1a-955c-a3b164eb57f7-kube-api-access-h565q\") pod \"multus-additional-cni-plugins-cgtgv\" (UID: \"08caa83a-ff39-4f1a-955c-a3b164eb57f7\") " pod="openshift-multus/multus-additional-cni-plugins-cgtgv"
Sep 11 03:40:24.830895 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:40:24.830864    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-597tt\" (UniqueName: \"kubernetes.io/projected/3517687a-5b4a-4804-9239-e0a5d1c415e0-kube-api-access-597tt\") pod \"network-metrics-daemon-9cmdj\" (UID: \"3517687a-5b4a-4804-9239-e0a5d1c415e0\") " pod="openshift-multus/network-metrics-daemon-9cmdj"
Sep 11 03:40:24.832579 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:40:24.832556    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-f4lsw\" (UniqueName: \"kubernetes.io/projected/9fc36641-9051-4f34-a897-bb4c92977307-kube-api-access-f4lsw\") pod \"node-ca-zg5qq\" (UID: \"9fc36641-9051-4f34-a897-bb4c92977307\") " pod="openshift-image-registry/node-ca-zg5qq"
Sep 11 03:40:24.833353 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:40:24.833307    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qhxfd\" (UniqueName: \"kubernetes.io/projected/c35abd6b-ad57-4b78-b072-56e6cd032381-kube-api-access-qhxfd\") pod \"node-resolver-mhjgr\" (UID: \"c35abd6b-ad57-4b78-b072-56e6cd032381\") " pod="openshift-dns/node-resolver-mhjgr"
Sep 11 03:40:24.833966 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:40:24.833940    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fkffr\" (UniqueName: \"kubernetes.io/projected/975d3d70-a13c-46bb-afc2-05c3609050f4-kube-api-access-fkffr\") pod \"multus-48m57\" (UID: \"975d3d70-a13c-46bb-afc2-05c3609050f4\") " pod="openshift-multus/multus-48m57"
Sep 11 03:40:24.851804 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:40:24.851755    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-105.ec2.internal" event={"ID":"c9ea5fe55135c7b1e5541fdd12fac4be","Type":"ContainerStarted","Data":"38086cc8ad9d216d887b9175a690d73f82255d78a246536895637c77d52d88ff"}
Sep 11 03:40:24.853423 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:40:24.853393    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-105.ec2.internal" event={"ID":"466709fffb065514ddf1d3dfe2c8d5d2","Type":"ContainerStarted","Data":"0667a90be31d1d4db838854212dacfd25bc6af25bf1c617058b29080d75f9b0f"}
Sep 11 03:40:24.921243 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:40:24.920083    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/5c976635-b437-40c3-9721-43754ae3a058-env-overrides\") pod \"ovnkube-node-j72hj\" (UID: \"5c976635-b437-40c3-9721-43754ae3a058\") " pod="openshift-ovn-kubernetes/ovnkube-node-j72hj"
Sep 11 03:40:24.921243 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:40:24.920140    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/5c976635-b437-40c3-9721-43754ae3a058-ovnkube-script-lib\") pod \"ovnkube-node-j72hj\" (UID: \"5c976635-b437-40c3-9721-43754ae3a058\") " pod="openshift-ovn-kubernetes/ovnkube-node-j72hj"
Sep 11 03:40:24.921243 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:40:24.920162    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/5bf35a4f-d6d3-4c6d-906f-5f88bc6767b6-etc-sysctl-conf\") pod \"tuned-nv6wz\" (UID: \"5bf35a4f-d6d3-4c6d-906f-5f88bc6767b6\") " pod="openshift-cluster-node-tuning-operator/tuned-nv6wz"
Sep 11 03:40:24.921243 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:40:24.920180    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5c976635-b437-40c3-9721-43754ae3a058-host-run-ovn-kubernetes\") pod \"ovnkube-node-j72hj\" (UID: \"5c976635-b437-40c3-9721-43754ae3a058\") " pod="openshift-ovn-kubernetes/ovnkube-node-j72hj"
Sep 11 03:40:24.921243 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:40:24.920209    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/202d85de-7969-4e66-a77e-14029cd0597c-etc-selinux\") pod \"aws-ebs-csi-driver-node-5lt7m\" (UID: \"202d85de-7969-4e66-a77e-14029cd0597c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5lt7m"
Sep 11 03:40:24.921243 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:40:24.920251    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/5bf35a4f-d6d3-4c6d-906f-5f88bc6767b6-etc-systemd\") pod \"tuned-nv6wz\" (UID: \"5bf35a4f-d6d3-4c6d-906f-5f88bc6767b6\") " pod="openshift-cluster-node-tuning-operator/tuned-nv6wz"
Sep 11 03:40:24.921243 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:40:24.920268    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j7jvz\" (UniqueName: \"kubernetes.io/projected/20c994e9-80cd-4378-9d56-60044e385640-kube-api-access-j7jvz\") pod \"iptables-alerter-2lk5x\" (UID: \"20c994e9-80cd-4378-9d56-60044e385640\") " pod="openshift-network-operator/iptables-alerter-2lk5x"
Sep 11 03:40:24.921243 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:40:24.920301    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/5c976635-b437-40c3-9721-43754ae3a058-etc-openvswitch\") pod \"ovnkube-node-j72hj\" (UID: \"5c976635-b437-40c3-9721-43754ae3a058\") " pod="openshift-ovn-kubernetes/ovnkube-node-j72hj"
Sep 11 03:40:24.921243 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:40:24.920329    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/202d85de-7969-4e66-a77e-14029cd0597c-sys-fs\") pod \"aws-ebs-csi-driver-node-5lt7m\" (UID: \"202d85de-7969-4e66-a77e-14029cd0597c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5lt7m"
Sep 11 03:40:24.921243 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:40:24.920346    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/1a7a4b58-e08a-45ef-ab7e-692d3a4f5314-original-pull-secret\") pod \"global-pull-secret-syncer-jwm97\" (UID: \"1a7a4b58-e08a-45ef-ab7e-692d3a4f5314\") " pod="kube-system/global-pull-secret-syncer-jwm97"
Sep 11 03:40:24.921243 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:40:24.920362    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/5c976635-b437-40c3-9721-43754ae3a058-host-cni-bin\") pod \"ovnkube-node-j72hj\" (UID: \"5c976635-b437-40c3-9721-43754ae3a058\") " pod="openshift-ovn-kubernetes/ovnkube-node-j72hj"
Sep 11 03:40:24.921243 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:40:24.920384    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/5bf35a4f-d6d3-4c6d-906f-5f88bc6767b6-etc-sysconfig\") pod \"tuned-nv6wz\" (UID: \"5bf35a4f-d6d3-4c6d-906f-5f88bc6767b6\") " pod="openshift-cluster-node-tuning-operator/tuned-nv6wz"
Sep 11 03:40:24.921243 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:40:24.920406    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/5c976635-b437-40c3-9721-43754ae3a058-host-kubelet\") pod \"ovnkube-node-j72hj\" (UID: \"5c976635-b437-40c3-9721-43754ae3a058\") " pod="openshift-ovn-kubernetes/ovnkube-node-j72hj"
Sep 11 03:40:24.921243 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:40:24.920423    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5c976635-b437-40c3-9721-43754ae3a058-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-j72hj\" (UID: \"5c976635-b437-40c3-9721-43754ae3a058\") " pod="openshift-ovn-kubernetes/ovnkube-node-j72hj"
Sep 11 03:40:24.921243 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:40:24.920445    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/5bf35a4f-d6d3-4c6d-906f-5f88bc6767b6-lib-modules\") pod \"tuned-nv6wz\" (UID: \"5bf35a4f-d6d3-4c6d-906f-5f88bc6767b6\") " pod="openshift-cluster-node-tuning-operator/tuned-nv6wz"
Sep 11 03:40:24.921243 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:40:24.920461    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/5c976635-b437-40c3-9721-43754ae3a058-host-slash\") pod \"ovnkube-node-j72hj\" (UID: \"5c976635-b437-40c3-9721-43754ae3a058\") " pod="openshift-ovn-kubernetes/ovnkube-node-j72hj"
Sep 11 03:40:24.921243 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:40:24.920477    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5bf35a4f-d6d3-4c6d-906f-5f88bc6767b6-etc-kubernetes\") pod \"tuned-nv6wz\" (UID: \"5bf35a4f-d6d3-4c6d-906f-5f88bc6767b6\") " pod="openshift-cluster-node-tuning-operator/tuned-nv6wz"
Sep 11 03:40:24.922602 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:40:24.920492    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/5bf35a4f-d6d3-4c6d-906f-5f88bc6767b6-run\") pod \"tuned-nv6wz\" (UID: \"5bf35a4f-d6d3-4c6d-906f-5f88bc6767b6\") " pod="openshift-cluster-node-tuning-operator/tuned-nv6wz"
Sep 11 03:40:24.922602 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:40:24.920508    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f94wq\" (UniqueName: \"kubernetes.io/projected/5bf35a4f-d6d3-4c6d-906f-5f88bc6767b6-kube-api-access-f94wq\") pod \"tuned-nv6wz\" (UID: \"5bf35a4f-d6d3-4c6d-906f-5f88bc6767b6\") " pod="openshift-cluster-node-tuning-operator/tuned-nv6wz"
Sep 11 03:40:24.922602 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:40:24.920523    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/5c976635-b437-40c3-9721-43754ae3a058-host-run-netns\") pod \"ovnkube-node-j72hj\" (UID: \"5c976635-b437-40c3-9721-43754ae3a058\") " pod="openshift-ovn-kubernetes/ovnkube-node-j72hj"
Sep 11 03:40:24.922602 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:40:24.920538    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/5c976635-b437-40c3-9721-43754ae3a058-run-systemd\") pod \"ovnkube-node-j72hj\" (UID: \"5c976635-b437-40c3-9721-43754ae3a058\") " pod="openshift-ovn-kubernetes/ovnkube-node-j72hj"
Sep 11 03:40:24.922602 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:40:24.920567    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/202d85de-7969-4e66-a77e-14029cd0597c-kubelet-dir\") pod \"aws-ebs-csi-driver-node-5lt7m\" (UID: \"202d85de-7969-4e66-a77e-14029cd0597c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5lt7m"
Sep 11 03:40:24.922602 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:40:24.920593    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/5bf35a4f-d6d3-4c6d-906f-5f88bc6767b6-tmp\") pod \"tuned-nv6wz\" (UID: \"5bf35a4f-d6d3-4c6d-906f-5f88bc6767b6\") " pod="openshift-cluster-node-tuning-operator/tuned-nv6wz"
Sep 11 03:40:24.922602 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:40:24.920632    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2tqw9\" (UniqueName: \"kubernetes.io/projected/202d85de-7969-4e66-a77e-14029cd0597c-kube-api-access-2tqw9\") pod \"aws-ebs-csi-driver-node-5lt7m\" (UID: \"202d85de-7969-4e66-a77e-14029cd0597c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5lt7m"
Sep 11 03:40:24.922602 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:40:24.920656    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/5c976635-b437-40c3-9721-43754ae3a058-run-ovn\") pod \"ovnkube-node-j72hj\" (UID: \"5c976635-b437-40c3-9721-43754ae3a058\") " pod="openshift-ovn-kubernetes/ovnkube-node-j72hj"
Sep 11 03:40:24.922602 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:40:24.920684    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/202d85de-7969-4e66-a77e-14029cd0597c-device-dir\") pod \"aws-ebs-csi-driver-node-5lt7m\" (UID: \"202d85de-7969-4e66-a77e-14029cd0597c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5lt7m"
Sep 11 03:40:24.922602 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:40:24.920707    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/5c976635-b437-40c3-9721-43754ae3a058-systemd-units\") pod \"ovnkube-node-j72hj\" (UID: \"5c976635-b437-40c3-9721-43754ae3a058\") " pod="openshift-ovn-kubernetes/ovnkube-node-j72hj"
Sep 11 03:40:24.922602 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:40:24.920722    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/5c976635-b437-40c3-9721-43754ae3a058-node-log\") pod \"ovnkube-node-j72hj\" (UID: \"5c976635-b437-40c3-9721-43754ae3a058\") " pod="openshift-ovn-kubernetes/ovnkube-node-j72hj"
Sep 11 03:40:24.922602 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:40:24.920740    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/5bf35a4f-d6d3-4c6d-906f-5f88bc6767b6-etc-tuned\") pod \"tuned-nv6wz\" (UID: \"5bf35a4f-d6d3-4c6d-906f-5f88bc6767b6\") " pod="openshift-cluster-node-tuning-operator/tuned-nv6wz"
Sep 11 03:40:24.922602 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:40:24.920756    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/5bf35a4f-d6d3-4c6d-906f-5f88bc6767b6-var-lib-kubelet\") pod \"tuned-nv6wz\" (UID: \"5bf35a4f-d6d3-4c6d-906f-5f88bc6767b6\") " pod="openshift-cluster-node-tuning-operator/tuned-nv6wz"
Sep 11 03:40:24.922602 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:40:24.920778    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/5bf35a4f-d6d3-4c6d-906f-5f88bc6767b6-etc-modprobe-d\") pod \"tuned-nv6wz\" (UID: \"5bf35a4f-d6d3-4c6d-906f-5f88bc6767b6\") " pod="openshift-cluster-node-tuning-operator/tuned-nv6wz"
Sep 11 03:40:24.922602 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:40:24.920827    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/5c976635-b437-40c3-9721-43754ae3a058-ovn-node-metrics-cert\") pod \"ovnkube-node-j72hj\" (UID: \"5c976635-b437-40c3-9721-43754ae3a058\") " pod="openshift-ovn-kubernetes/ovnkube-node-j72hj"
Sep 11 03:40:24.922602 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:40:24.920855    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/5bf35a4f-d6d3-4c6d-906f-5f88bc6767b6-etc-sysctl-d\") pod \"tuned-nv6wz\" (UID: \"5bf35a4f-d6d3-4c6d-906f-5f88bc6767b6\") " pod="openshift-cluster-node-tuning-operator/tuned-nv6wz"
Sep 11 03:40:24.922602 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:40:24.920858    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/5c976635-b437-40c3-9721-43754ae3a058-env-overrides\") pod \"ovnkube-node-j72hj\" (UID: \"5c976635-b437-40c3-9721-43754ae3a058\") " pod="openshift-ovn-kubernetes/ovnkube-node-j72hj"
Sep 11 03:40:24.923555 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:40:24.920890    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/202d85de-7969-4e66-a77e-14029cd0597c-socket-dir\") pod \"aws-ebs-csi-driver-node-5lt7m\" (UID: \"202d85de-7969-4e66-a77e-14029cd0597c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5lt7m"
Sep 11 03:40:24.923555 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:40:24.920918    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/1a7a4b58-e08a-45ef-ab7e-692d3a4f5314-kubelet-config\") pod \"global-pull-secret-syncer-jwm97\" (UID: \"1a7a4b58-e08a-45ef-ab7e-692d3a4f5314\") " pod="kube-system/global-pull-secret-syncer-jwm97"
Sep 11 03:40:24.923555 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:40:24.920984    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/20c994e9-80cd-4378-9d56-60044e385640-host-slash\") pod \"iptables-alerter-2lk5x\" (UID: \"20c994e9-80cd-4378-9d56-60044e385640\") " pod="openshift-network-operator/iptables-alerter-2lk5x"
Sep 11 03:40:24.923555 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:40:24.921004    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/5bf35a4f-d6d3-4c6d-906f-5f88bc6767b6-sys\") pod \"tuned-nv6wz\" (UID: \"5bf35a4f-d6d3-4c6d-906f-5f88bc6767b6\") " pod="openshift-cluster-node-tuning-operator/tuned-nv6wz"
Sep 11 03:40:24.923555 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:40:24.921022    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/5c976635-b437-40c3-9721-43754ae3a058-host-cni-netd\") pod \"ovnkube-node-j72hj\" (UID: \"5c976635-b437-40c3-9721-43754ae3a058\") " pod="openshift-ovn-kubernetes/ovnkube-node-j72hj"
Sep 11 03:40:24.923555 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:40:24.921067    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/5c976635-b437-40c3-9721-43754ae3a058-ovnkube-config\") pod \"ovnkube-node-j72hj\" (UID: \"5c976635-b437-40c3-9721-43754ae3a058\") " pod="openshift-ovn-kubernetes/ovnkube-node-j72hj"
Sep 11 03:40:24.923555 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:40:24.921097    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6d7c6\" (UniqueName: \"kubernetes.io/projected/5c976635-b437-40c3-9721-43754ae3a058-kube-api-access-6d7c6\") pod \"ovnkube-node-j72hj\" (UID: \"5c976635-b437-40c3-9721-43754ae3a058\") " pod="openshift-ovn-kubernetes/ovnkube-node-j72hj"
Sep 11 03:40:24.923555 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:40:24.921123    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/5c976635-b437-40c3-9721-43754ae3a058-var-lib-openvswitch\") pod \"ovnkube-node-j72hj\" (UID: \"5c976635-b437-40c3-9721-43754ae3a058\") " pod="openshift-ovn-kubernetes/ovnkube-node-j72hj"
Sep 11 03:40:24.923555 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:40:24.921141    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/5c976635-b437-40c3-9721-43754ae3a058-run-openvswitch\") pod \"ovnkube-node-j72hj\" (UID: \"5c976635-b437-40c3-9721-43754ae3a058\") " pod="openshift-ovn-kubernetes/ovnkube-node-j72hj"
Sep 11 03:40:24.923555 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:40:24.921187    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/5c976635-b437-40c3-9721-43754ae3a058-log-socket\") pod \"ovnkube-node-j72hj\" (UID: \"5c976635-b437-40c3-9721-43754ae3a058\") " pod="openshift-ovn-kubernetes/ovnkube-node-j72hj"
Sep 11 03:40:24.923555 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:40:24.921212    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/20c994e9-80cd-4378-9d56-60044e385640-iptables-alerter-script\") pod \"iptables-alerter-2lk5x\" (UID: \"20c994e9-80cd-4378-9d56-60044e385640\") " pod="openshift-network-operator/iptables-alerter-2lk5x"
Sep 11 03:40:24.923555 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:40:24.921240    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/202d85de-7969-4e66-a77e-14029cd0597c-registration-dir\") pod \"aws-ebs-csi-driver-node-5lt7m\" (UID: \"202d85de-7969-4e66-a77e-14029cd0597c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5lt7m"
Sep 11 03:40:24.923555 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:40:24.921264    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/5bf35a4f-d6d3-4c6d-906f-5f88bc6767b6-host\") pod \"tuned-nv6wz\" (UID: \"5bf35a4f-d6d3-4c6d-906f-5f88bc6767b6\") " pod="openshift-cluster-node-tuning-operator/tuned-nv6wz"
Sep 11 03:40:24.923555 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:40:24.921286    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/1a7a4b58-e08a-45ef-ab7e-692d3a4f5314-dbus\") pod \"global-pull-secret-syncer-jwm97\" (UID: \"1a7a4b58-e08a-45ef-ab7e-692d3a4f5314\") " pod="kube-system/global-pull-secret-syncer-jwm97"
Sep 11 03:40:24.923555 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:40:24.921336    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/5bf35a4f-d6d3-4c6d-906f-5f88bc6767b6-lib-modules\") pod \"tuned-nv6wz\" (UID: \"5bf35a4f-d6d3-4c6d-906f-5f88bc6767b6\") " pod="openshift-cluster-node-tuning-operator/tuned-nv6wz"
Sep 11 03:40:24.923555 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:40:24.921364    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/5c976635-b437-40c3-9721-43754ae3a058-ovnkube-script-lib\") pod \"ovnkube-node-j72hj\" (UID: \"5c976635-b437-40c3-9721-43754ae3a058\") " pod="openshift-ovn-kubernetes/ovnkube-node-j72hj"
Sep 11 03:40:24.923555 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:40:24.921386    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/5c976635-b437-40c3-9721-43754ae3a058-etc-openvswitch\") pod \"ovnkube-node-j72hj\" (UID: \"5c976635-b437-40c3-9721-43754ae3a058\") " pod="openshift-ovn-kubernetes/ovnkube-node-j72hj"
Sep 11 03:40:24.924546 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:40:24.921430    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/202d85de-7969-4e66-a77e-14029cd0597c-sys-fs\") pod \"aws-ebs-csi-driver-node-5lt7m\" (UID: \"202d85de-7969-4e66-a77e-14029cd0597c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5lt7m"
Sep 11 03:40:24.924546 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:40:24.921471    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/5c976635-b437-40c3-9721-43754ae3a058-node-log\") pod \"ovnkube-node-j72hj\" (UID: \"5c976635-b437-40c3-9721-43754ae3a058\") " pod="openshift-ovn-kubernetes/ovnkube-node-j72hj"
Sep 11 03:40:24.924546 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:40:24.921486    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/5c976635-b437-40c3-9721-43754ae3a058-run-systemd\") pod \"ovnkube-node-j72hj\" (UID: \"5c976635-b437-40c3-9721-43754ae3a058\") " pod="openshift-ovn-kubernetes/ovnkube-node-j72hj"
Sep 11 03:40:24.924546 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:40:24.921488    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/5c976635-b437-40c3-9721-43754ae3a058-host-slash\") pod \"ovnkube-node-j72hj\" (UID: \"5c976635-b437-40c3-9721-43754ae3a058\") " pod="openshift-ovn-kubernetes/ovnkube-node-j72hj"
Sep 11 03:40:24.924546 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:40:24.921411    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/1a7a4b58-e08a-45ef-ab7e-692d3a4f5314-dbus\") pod \"global-pull-secret-syncer-jwm97\" (UID: \"1a7a4b58-e08a-45ef-ab7e-692d3a4f5314\") " pod="kube-system/global-pull-secret-syncer-jwm97"
Sep 11 03:40:24.924546 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:40:24.921519    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5bf35a4f-d6d3-4c6d-906f-5f88bc6767b6-etc-kubernetes\") pod \"tuned-nv6wz\" (UID: \"5bf35a4f-d6d3-4c6d-906f-5f88bc6767b6\") " pod="openshift-cluster-node-tuning-operator/tuned-nv6wz"
Sep 11 03:40:24.924546 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:40:24.921560    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/5c976635-b437-40c3-9721-43754ae3a058-host-cni-bin\") pod \"ovnkube-node-j72hj\" (UID: \"5c976635-b437-40c3-9721-43754ae3a058\") " pod="openshift-ovn-kubernetes/ovnkube-node-j72hj"
Sep 11 03:40:24.924546 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:40:24.921696    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/202d85de-7969-4e66-a77e-14029cd0597c-device-dir\") pod \"aws-ebs-csi-driver-node-5lt7m\" (UID: \"202d85de-7969-4e66-a77e-14029cd0597c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5lt7m"
Sep 11 03:40:24.924546 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:40:24.921738    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/5c976635-b437-40c3-9721-43754ae3a058-host-run-netns\") pod \"ovnkube-node-j72hj\" (UID: \"5c976635-b437-40c3-9721-43754ae3a058\") " pod="openshift-ovn-kubernetes/ovnkube-node-j72hj"
Sep 11 03:40:24.924546 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:40:24.921738    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/5c976635-b437-40c3-9721-43754ae3a058-run-ovn\") pod \"ovnkube-node-j72hj\" (UID: \"5c976635-b437-40c3-9721-43754ae3a058\") " pod="openshift-ovn-kubernetes/ovnkube-node-j72hj"
Sep 11 03:40:24.924546 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:40:24.921781    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/5c976635-b437-40c3-9721-43754ae3a058-host-kubelet\") pod \"ovnkube-node-j72hj\" (UID: \"5c976635-b437-40c3-9721-43754ae3a058\") " pod="openshift-ovn-kubernetes/ovnkube-node-j72hj"
Sep 11 03:40:24.924546 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:40:24.921787    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/5bf35a4f-d6d3-4c6d-906f-5f88bc6767b6-var-lib-kubelet\") pod \"tuned-nv6wz\" (UID: \"5bf35a4f-d6d3-4c6d-906f-5f88bc6767b6\") " pod="openshift-cluster-node-tuning-operator/tuned-nv6wz"
Sep 11 03:40:24.924546 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:40:24.921819    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/202d85de-7969-4e66-a77e-14029cd0597c-kubelet-dir\") pod \"aws-ebs-csi-driver-node-5lt7m\" (UID: \"202d85de-7969-4e66-a77e-14029cd0597c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5lt7m"
Sep 11 03:40:24.924546 ip-10-0-141-105 kubenswrapper[2579]: E0911 03:40:24.921903    2579 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 11 03:40:24.924546 ip-10-0-141-105 kubenswrapper[2579]: E0911 03:40:24.921958    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1a7a4b58-e08a-45ef-ab7e-692d3a4f5314-original-pull-secret podName:1a7a4b58-e08a-45ef-ab7e-692d3a4f5314 nodeName:}" failed. No retries permitted until 2026-09-11 03:40:25.421940472 +0000 UTC m=+3.077146037 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/1a7a4b58-e08a-45ef-ab7e-692d3a4f5314-original-pull-secret") pod "global-pull-secret-syncer-jwm97" (UID: "1a7a4b58-e08a-45ef-ab7e-692d3a4f5314") : object "kube-system"/"original-pull-secret" not registered
Sep 11 03:40:24.924546 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:40:24.922012    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/5bf35a4f-d6d3-4c6d-906f-5f88bc6767b6-etc-modprobe-d\") pod \"tuned-nv6wz\" (UID: \"5bf35a4f-d6d3-4c6d-906f-5f88bc6767b6\") " pod="openshift-cluster-node-tuning-operator/tuned-nv6wz"
Sep 11 03:40:24.924546 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:40:24.922101    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/5bf35a4f-d6d3-4c6d-906f-5f88bc6767b6-etc-sysctl-d\") pod \"tuned-nv6wz\" (UID: \"5bf35a4f-d6d3-4c6d-906f-5f88bc6767b6\") " pod="openshift-cluster-node-tuning-operator/tuned-nv6wz"
Sep 11 03:40:24.925258 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:40:24.921821    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/5c976635-b437-40c3-9721-43754ae3a058-host-cni-netd\") pod \"ovnkube-node-j72hj\" (UID: \"5c976635-b437-40c3-9721-43754ae3a058\") " pod="openshift-ovn-kubernetes/ovnkube-node-j72hj"
Sep 11 03:40:24.925258 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:40:24.922141    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/20c994e9-80cd-4378-9d56-60044e385640-host-slash\") pod \"iptables-alerter-2lk5x\" (UID: \"20c994e9-80cd-4378-9d56-60044e385640\") " pod="openshift-network-operator/iptables-alerter-2lk5x"
Sep 11 03:40:24.925258 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:40:24.922193    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/1a7a4b58-e08a-45ef-ab7e-692d3a4f5314-kubelet-config\") pod \"global-pull-secret-syncer-jwm97\" (UID: \"1a7a4b58-e08a-45ef-ab7e-692d3a4f5314\") " pod="kube-system/global-pull-secret-syncer-jwm97"
Sep 11 03:40:24.925258 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:40:24.922206    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/5bf35a4f-d6d3-4c6d-906f-5f88bc6767b6-host\") pod \"tuned-nv6wz\" (UID: \"5bf35a4f-d6d3-4c6d-906f-5f88bc6767b6\") " pod="openshift-cluster-node-tuning-operator/tuned-nv6wz"
Sep 11 03:40:24.925258 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:40:24.922394    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/5c976635-b437-40c3-9721-43754ae3a058-ovnkube-config\") pod \"ovnkube-node-j72hj\" (UID: \"5c976635-b437-40c3-9721-43754ae3a058\") " pod="openshift-ovn-kubernetes/ovnkube-node-j72hj"
Sep 11 03:40:24.925258 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:40:24.922463    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/5c976635-b437-40c3-9721-43754ae3a058-run-openvswitch\") pod \"ovnkube-node-j72hj\" (UID: \"5c976635-b437-40c3-9721-43754ae3a058\") " pod="openshift-ovn-kubernetes/ovnkube-node-j72hj"
Sep 11 03:40:24.925258 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:40:24.922502    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/5c976635-b437-40c3-9721-43754ae3a058-var-lib-openvswitch\") pod \"ovnkube-node-j72hj\" (UID: \"5c976635-b437-40c3-9721-43754ae3a058\") " pod="openshift-ovn-kubernetes/ovnkube-node-j72hj"
Sep 11 03:40:24.925258 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:40:24.922557    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/202d85de-7969-4e66-a77e-14029cd0597c-registration-dir\") pod \"aws-ebs-csi-driver-node-5lt7m\" (UID: \"202d85de-7969-4e66-a77e-14029cd0597c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5lt7m"
Sep 11 03:40:24.925258 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:40:24.922603    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/5c976635-b437-40c3-9721-43754ae3a058-systemd-units\") pod \"ovnkube-node-j72hj\" (UID: \"5c976635-b437-40c3-9721-43754ae3a058\") " pod="openshift-ovn-kubernetes/ovnkube-node-j72hj"
Sep 11 03:40:24.925258 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:40:24.922604    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5c976635-b437-40c3-9721-43754ae3a058-host-run-ovn-kubernetes\") pod \"ovnkube-node-j72hj\" (UID: \"5c976635-b437-40c3-9721-43754ae3a058\") " pod="openshift-ovn-kubernetes/ovnkube-node-j72hj"
Sep 11 03:40:24.925258 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:40:24.922652    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/20c994e9-80cd-4378-9d56-60044e385640-iptables-alerter-script\") pod \"iptables-alerter-2lk5x\" (UID: \"20c994e9-80cd-4378-9d56-60044e385640\") " pod="openshift-network-operator/iptables-alerter-2lk5x"
Sep 11 03:40:24.925258 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:40:24.922685    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/202d85de-7969-4e66-a77e-14029cd0597c-socket-dir\") pod \"aws-ebs-csi-driver-node-5lt7m\" (UID: \"202d85de-7969-4e66-a77e-14029cd0597c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5lt7m"
Sep 11 03:40:24.925258 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:40:24.922715    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/5bf35a4f-d6d3-4c6d-906f-5f88bc6767b6-etc-sysconfig\") pod \"tuned-nv6wz\" (UID: \"5bf35a4f-d6d3-4c6d-906f-5f88bc6767b6\") " pod="openshift-cluster-node-tuning-operator/tuned-nv6wz"
Sep 11 03:40:24.925258 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:40:24.922740    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/5bf35a4f-d6d3-4c6d-906f-5f88bc6767b6-etc-systemd\") pod \"tuned-nv6wz\" (UID: \"5bf35a4f-d6d3-4c6d-906f-5f88bc6767b6\") " pod="openshift-cluster-node-tuning-operator/tuned-nv6wz"
Sep 11 03:40:24.925258 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:40:24.922751    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/5c976635-b437-40c3-9721-43754ae3a058-log-socket\") pod \"ovnkube-node-j72hj\" (UID: \"5c976635-b437-40c3-9721-43754ae3a058\") " pod="openshift-ovn-kubernetes/ovnkube-node-j72hj"
Sep 11 03:40:24.925258 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:40:24.922762    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/5bf35a4f-d6d3-4c6d-906f-5f88bc6767b6-etc-sysctl-conf\") pod \"tuned-nv6wz\" (UID: \"5bf35a4f-d6d3-4c6d-906f-5f88bc6767b6\") " pod="openshift-cluster-node-tuning-operator/tuned-nv6wz"
Sep 11 03:40:24.925258 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:40:24.922796    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/5bf35a4f-d6d3-4c6d-906f-5f88bc6767b6-sys\") pod \"tuned-nv6wz\" (UID: \"5bf35a4f-d6d3-4c6d-906f-5f88bc6767b6\") " pod="openshift-cluster-node-tuning-operator/tuned-nv6wz"
Sep 11 03:40:24.925872 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:40:24.922839    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/202d85de-7969-4e66-a77e-14029cd0597c-etc-selinux\") pod \"aws-ebs-csi-driver-node-5lt7m\" (UID: \"202d85de-7969-4e66-a77e-14029cd0597c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5lt7m"
Sep 11 03:40:24.925872 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:40:24.922815    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5c976635-b437-40c3-9721-43754ae3a058-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-j72hj\" (UID: \"5c976635-b437-40c3-9721-43754ae3a058\") " pod="openshift-ovn-kubernetes/ovnkube-node-j72hj"
Sep 11 03:40:24.925872 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:40:24.922970    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/5bf35a4f-d6d3-4c6d-906f-5f88bc6767b6-run\") pod \"tuned-nv6wz\" (UID: \"5bf35a4f-d6d3-4c6d-906f-5f88bc6767b6\") " pod="openshift-cluster-node-tuning-operator/tuned-nv6wz"
Sep 11 03:40:24.925872 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:40:24.924272    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/5bf35a4f-d6d3-4c6d-906f-5f88bc6767b6-tmp\") pod \"tuned-nv6wz\" (UID: \"5bf35a4f-d6d3-4c6d-906f-5f88bc6767b6\") " pod="openshift-cluster-node-tuning-operator/tuned-nv6wz"
Sep 11 03:40:24.925872 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:40:24.924630    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/5bf35a4f-d6d3-4c6d-906f-5f88bc6767b6-etc-tuned\") pod \"tuned-nv6wz\" (UID: \"5bf35a4f-d6d3-4c6d-906f-5f88bc6767b6\") " pod="openshift-cluster-node-tuning-operator/tuned-nv6wz"
Sep 11 03:40:24.926069 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:40:24.925985    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/5c976635-b437-40c3-9721-43754ae3a058-ovn-node-metrics-cert\") pod \"ovnkube-node-j72hj\" (UID: \"5c976635-b437-40c3-9721-43754ae3a058\") " pod="openshift-ovn-kubernetes/ovnkube-node-j72hj"
Sep 11 03:40:24.929529 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:40:24.929507    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2tqw9\" (UniqueName: \"kubernetes.io/projected/202d85de-7969-4e66-a77e-14029cd0597c-kube-api-access-2tqw9\") pod \"aws-ebs-csi-driver-node-5lt7m\" (UID: \"202d85de-7969-4e66-a77e-14029cd0597c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5lt7m"
Sep 11 03:40:24.936283 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:40:24.936256    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-f94wq\" (UniqueName: \"kubernetes.io/projected/5bf35a4f-d6d3-4c6d-906f-5f88bc6767b6-kube-api-access-f94wq\") pod \"tuned-nv6wz\" (UID: \"5bf35a4f-d6d3-4c6d-906f-5f88bc6767b6\") " pod="openshift-cluster-node-tuning-operator/tuned-nv6wz"
Sep 11 03:40:24.936596 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:40:24.936570    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6d7c6\" (UniqueName: \"kubernetes.io/projected/5c976635-b437-40c3-9721-43754ae3a058-kube-api-access-6d7c6\") pod \"ovnkube-node-j72hj\" (UID: \"5c976635-b437-40c3-9721-43754ae3a058\") " pod="openshift-ovn-kubernetes/ovnkube-node-j72hj"
Sep 11 03:40:24.937079 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:40:24.937058    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-j7jvz\" (UniqueName: \"kubernetes.io/projected/20c994e9-80cd-4378-9d56-60044e385640-kube-api-access-j7jvz\") pod \"iptables-alerter-2lk5x\" (UID: \"20c994e9-80cd-4378-9d56-60044e385640\") " pod="openshift-network-operator/iptables-alerter-2lk5x"
Sep 11 03:40:24.945384 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:40:24.945356    2579 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 11 03:40:25.013038 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:40:25.012988    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-cgtgv"
Sep 11 03:40:25.022055 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:40:25.022025    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-mhjgr"
Sep 11 03:40:25.032129 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:40:25.032090    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-zg5qq"
Sep 11 03:40:25.039043 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:40:25.039008    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-48m57"
Sep 11 03:40:25.047901 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:40:25.047868    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-s9xqv"
Sep 11 03:40:25.065664 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:40:25.065640    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-j72hj"
Sep 11 03:40:25.076732 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:40:25.076708    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-2lk5x"
Sep 11 03:40:25.084575 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:40:25.084552    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5lt7m"
Sep 11 03:40:25.093370 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:40:25.093350    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-nv6wz"
Sep 11 03:40:25.325354 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:40:25.325251    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3517687a-5b4a-4804-9239-e0a5d1c415e0-metrics-certs\") pod \"network-metrics-daemon-9cmdj\" (UID: \"3517687a-5b4a-4804-9239-e0a5d1c415e0\") " pod="openshift-multus/network-metrics-daemon-9cmdj"
Sep 11 03:40:25.325568 ip-10-0-141-105 kubenswrapper[2579]: E0911 03:40:25.325406    2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 11 03:40:25.325568 ip-10-0-141-105 kubenswrapper[2579]: E0911 03:40:25.325477    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3517687a-5b4a-4804-9239-e0a5d1c415e0-metrics-certs podName:3517687a-5b4a-4804-9239-e0a5d1c415e0 nodeName:}" failed. No retries permitted until 2026-09-11 03:40:26.325457827 +0000 UTC m=+3.980663378 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/3517687a-5b4a-4804-9239-e0a5d1c415e0-metrics-certs") pod "network-metrics-daemon-9cmdj" (UID: "3517687a-5b4a-4804-9239-e0a5d1c415e0") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 11 03:40:25.425957 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:40:25.425919    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/1a7a4b58-e08a-45ef-ab7e-692d3a4f5314-original-pull-secret\") pod \"global-pull-secret-syncer-jwm97\" (UID: \"1a7a4b58-e08a-45ef-ab7e-692d3a4f5314\") " pod="kube-system/global-pull-secret-syncer-jwm97"
Sep 11 03:40:25.426182 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:40:25.425968    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b8wlz\" (UniqueName: \"kubernetes.io/projected/801557d7-c47c-43dd-977e-58a9766436c1-kube-api-access-b8wlz\") pod \"network-check-target-p54z5\" (UID: \"801557d7-c47c-43dd-977e-58a9766436c1\") " pod="openshift-network-diagnostics/network-check-target-p54z5"
Sep 11 03:40:25.426182 ip-10-0-141-105 kubenswrapper[2579]: E0911 03:40:25.426081    2579 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 11 03:40:25.426182 ip-10-0-141-105 kubenswrapper[2579]: E0911 03:40:25.426177    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1a7a4b58-e08a-45ef-ab7e-692d3a4f5314-original-pull-secret podName:1a7a4b58-e08a-45ef-ab7e-692d3a4f5314 nodeName:}" failed. No retries permitted until 2026-09-11 03:40:26.426160787 +0000 UTC m=+4.081366333 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/1a7a4b58-e08a-45ef-ab7e-692d3a4f5314-original-pull-secret") pod "global-pull-secret-syncer-jwm97" (UID: "1a7a4b58-e08a-45ef-ab7e-692d3a4f5314") : object "kube-system"/"original-pull-secret" not registered
Sep 11 03:40:25.426389 ip-10-0-141-105 kubenswrapper[2579]: E0911 03:40:25.426101    2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 11 03:40:25.426389 ip-10-0-141-105 kubenswrapper[2579]: E0911 03:40:25.426254    2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 11 03:40:25.426389 ip-10-0-141-105 kubenswrapper[2579]: E0911 03:40:25.426273    2579 projected.go:196] Error preparing data for projected volume kube-api-access-b8wlz for pod openshift-network-diagnostics/network-check-target-p54z5: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 11 03:40:25.426389 ip-10-0-141-105 kubenswrapper[2579]: E0911 03:40:25.426333    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/801557d7-c47c-43dd-977e-58a9766436c1-kube-api-access-b8wlz podName:801557d7-c47c-43dd-977e-58a9766436c1 nodeName:}" failed. No retries permitted until 2026-09-11 03:40:26.42631632 +0000 UTC m=+4.081521882 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-b8wlz" (UniqueName: "kubernetes.io/projected/801557d7-c47c-43dd-977e-58a9766436c1-kube-api-access-b8wlz") pod "network-check-target-p54z5" (UID: "801557d7-c47c-43dd-977e-58a9766436c1") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 11 03:40:25.744807 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:40:25.744714    2579 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-10 03:35:24 +0000 UTC" deadline="2028-05-26 01:56:28.903140408 +0000 UTC"
Sep 11 03:40:25.744807 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:40:25.744760    2579 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14950h16m3.158383465s"
Sep 11 03:40:25.996721 ip-10-0-141-105 kubenswrapper[2579]: W0911 03:40:25.996686    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podff572c4a_7a16_4c2b_b0a8_160b3b274834.slice/crio-08e76c0ca76ad02cf302f5f0fd59a8b91213939c22f229f181c48b173cde5c61 WatchSource:0}: Error finding container 08e76c0ca76ad02cf302f5f0fd59a8b91213939c22f229f181c48b173cde5c61: Status 404 returned error can't find the container with id 08e76c0ca76ad02cf302f5f0fd59a8b91213939c22f229f181c48b173cde5c61
Sep 11 03:40:25.997457 ip-10-0-141-105 kubenswrapper[2579]: W0911 03:40:25.997385    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5c976635_b437_40c3_9721_43754ae3a058.slice/crio-535378d6a076e5ecfd2efeaa7f448432a065a175c1d9fb9a053e201e9520ec6c WatchSource:0}: Error finding container 535378d6a076e5ecfd2efeaa7f448432a065a175c1d9fb9a053e201e9520ec6c: Status 404 returned error can't find the container with id 535378d6a076e5ecfd2efeaa7f448432a065a175c1d9fb9a053e201e9520ec6c
Sep 11 03:40:25.998249 ip-10-0-141-105 kubenswrapper[2579]: W0911 03:40:25.998219    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc35abd6b_ad57_4b78_b072_56e6cd032381.slice/crio-299f7542ceca7155f3447dfa423d65e543ff3597eb9f9d85799ec7e565d6183f WatchSource:0}: Error finding container 299f7542ceca7155f3447dfa423d65e543ff3597eb9f9d85799ec7e565d6183f: Status 404 returned error can't find the container with id 299f7542ceca7155f3447dfa423d65e543ff3597eb9f9d85799ec7e565d6183f
Sep 11 03:40:26.002020 ip-10-0-141-105 kubenswrapper[2579]: W0911 03:40:26.001990    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod975d3d70_a13c_46bb_afc2_05c3609050f4.slice/crio-831888fcef21e1621bf9b6ffa9034a543d70ecaae26e3956ceb4835c3235a771 WatchSource:0}: Error finding container 831888fcef21e1621bf9b6ffa9034a543d70ecaae26e3956ceb4835c3235a771: Status 404 returned error can't find the container with id 831888fcef21e1621bf9b6ffa9034a543d70ecaae26e3956ceb4835c3235a771
Sep 11 03:40:26.003694 ip-10-0-141-105 kubenswrapper[2579]: W0911 03:40:26.003665    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9fc36641_9051_4f34_a897_bb4c92977307.slice/crio-429aac580a8f69c5abb7bc362c7537f9f5cb089fe8921c0e731857b8bd22532f WatchSource:0}: Error finding container 429aac580a8f69c5abb7bc362c7537f9f5cb089fe8921c0e731857b8bd22532f: Status 404 returned error can't find the container with id 429aac580a8f69c5abb7bc362c7537f9f5cb089fe8921c0e731857b8bd22532f
Sep 11 03:40:26.011891 ip-10-0-141-105 kubenswrapper[2579]: W0911 03:40:26.010110    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod08caa83a_ff39_4f1a_955c_a3b164eb57f7.slice/crio-0e6f41ec623b3460dd43f82659c6e772fd89e3e33845cbf86f6d58a5cafff8fb WatchSource:0}: Error finding container 0e6f41ec623b3460dd43f82659c6e772fd89e3e33845cbf86f6d58a5cafff8fb: Status 404 returned error can't find the container with id 0e6f41ec623b3460dd43f82659c6e772fd89e3e33845cbf86f6d58a5cafff8fb
Sep 11 03:40:26.333708 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:40:26.333456    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3517687a-5b4a-4804-9239-e0a5d1c415e0-metrics-certs\") pod \"network-metrics-daemon-9cmdj\" (UID: \"3517687a-5b4a-4804-9239-e0a5d1c415e0\") " pod="openshift-multus/network-metrics-daemon-9cmdj"
Sep 11 03:40:26.333844 ip-10-0-141-105 kubenswrapper[2579]: E0911 03:40:26.333666    2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 11 03:40:26.333844 ip-10-0-141-105 kubenswrapper[2579]: E0911 03:40:26.333833    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3517687a-5b4a-4804-9239-e0a5d1c415e0-metrics-certs podName:3517687a-5b4a-4804-9239-e0a5d1c415e0 nodeName:}" failed. No retries permitted until 2026-09-11 03:40:28.333816233 +0000 UTC m=+5.989021780 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/3517687a-5b4a-4804-9239-e0a5d1c415e0-metrics-certs") pod "network-metrics-daemon-9cmdj" (UID: "3517687a-5b4a-4804-9239-e0a5d1c415e0") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 11 03:40:26.434921 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:40:26.434182    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/1a7a4b58-e08a-45ef-ab7e-692d3a4f5314-original-pull-secret\") pod \"global-pull-secret-syncer-jwm97\" (UID: \"1a7a4b58-e08a-45ef-ab7e-692d3a4f5314\") " pod="kube-system/global-pull-secret-syncer-jwm97"
Sep 11 03:40:26.434921 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:40:26.434254    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b8wlz\" (UniqueName: \"kubernetes.io/projected/801557d7-c47c-43dd-977e-58a9766436c1-kube-api-access-b8wlz\") pod \"network-check-target-p54z5\" (UID: \"801557d7-c47c-43dd-977e-58a9766436c1\") " pod="openshift-network-diagnostics/network-check-target-p54z5"
Sep 11 03:40:26.434921 ip-10-0-141-105 kubenswrapper[2579]: E0911 03:40:26.434386    2579 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 11 03:40:26.434921 ip-10-0-141-105 kubenswrapper[2579]: E0911 03:40:26.434449    2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 11 03:40:26.434921 ip-10-0-141-105 kubenswrapper[2579]: E0911 03:40:26.434487    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1a7a4b58-e08a-45ef-ab7e-692d3a4f5314-original-pull-secret podName:1a7a4b58-e08a-45ef-ab7e-692d3a4f5314 nodeName:}" failed. No retries permitted until 2026-09-11 03:40:28.434460615 +0000 UTC m=+6.089666182 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/1a7a4b58-e08a-45ef-ab7e-692d3a4f5314-original-pull-secret") pod "global-pull-secret-syncer-jwm97" (UID: "1a7a4b58-e08a-45ef-ab7e-692d3a4f5314") : object "kube-system"/"original-pull-secret" not registered
Sep 11 03:40:26.434921 ip-10-0-141-105 kubenswrapper[2579]: E0911 03:40:26.434487    2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 11 03:40:26.434921 ip-10-0-141-105 kubenswrapper[2579]: E0911 03:40:26.434508    2579 projected.go:196] Error preparing data for projected volume kube-api-access-b8wlz for pod openshift-network-diagnostics/network-check-target-p54z5: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 11 03:40:26.434921 ip-10-0-141-105 kubenswrapper[2579]: E0911 03:40:26.434572    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/801557d7-c47c-43dd-977e-58a9766436c1-kube-api-access-b8wlz podName:801557d7-c47c-43dd-977e-58a9766436c1 nodeName:}" failed. No retries permitted until 2026-09-11 03:40:28.434543274 +0000 UTC m=+6.089748830 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-b8wlz" (UniqueName: "kubernetes.io/projected/801557d7-c47c-43dd-977e-58a9766436c1-kube-api-access-b8wlz") pod "network-check-target-p54z5" (UID: "801557d7-c47c-43dd-977e-58a9766436c1") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 11 03:40:26.856183 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:40:26.853541    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9cmdj"
Sep 11 03:40:26.856183 ip-10-0-141-105 kubenswrapper[2579]: E0911 03:40:26.853807    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9cmdj" podUID="3517687a-5b4a-4804-9239-e0a5d1c415e0"
Sep 11 03:40:26.856183 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:40:26.854441    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jwm97"
Sep 11 03:40:26.856183 ip-10-0-141-105 kubenswrapper[2579]: E0911 03:40:26.854622    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jwm97" podUID="1a7a4b58-e08a-45ef-ab7e-692d3a4f5314"
Sep 11 03:40:26.856183 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:40:26.854722    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-p54z5"
Sep 11 03:40:26.856183 ip-10-0-141-105 kubenswrapper[2579]: E0911 03:40:26.854853    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-p54z5" podUID="801557d7-c47c-43dd-977e-58a9766436c1"
Sep 11 03:40:26.864939 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:40:26.864908    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5lt7m" event={"ID":"202d85de-7969-4e66-a77e-14029cd0597c","Type":"ContainerStarted","Data":"bcec64b46f5a62a4792ff6752515c86d2d45639bc19f32e0eddab5b9c3ebad0a"}
Sep 11 03:40:26.876211 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:40:26.872772    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-zg5qq" event={"ID":"9fc36641-9051-4f34-a897-bb4c92977307","Type":"ContainerStarted","Data":"429aac580a8f69c5abb7bc362c7537f9f5cb089fe8921c0e731857b8bd22532f"}
Sep 11 03:40:26.883475 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:40:26.883424    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-s9xqv" event={"ID":"ff572c4a-7a16-4c2b-b0a8-160b3b274834","Type":"ContainerStarted","Data":"08e76c0ca76ad02cf302f5f0fd59a8b91213939c22f229f181c48b173cde5c61"}
Sep 11 03:40:26.899848 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:40:26.899811    2579 generic.go:356] "Generic (PLEG): container finished" podID="466709fffb065514ddf1d3dfe2c8d5d2" containerID="55fa1783422b552bfdfb2216fb2b04fa7bbf7bff98eaddac917f25165f2b65fe" exitCode=0
Sep 11 03:40:26.900007 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:40:26.899905    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-105.ec2.internal" event={"ID":"466709fffb065514ddf1d3dfe2c8d5d2","Type":"ContainerDied","Data":"55fa1783422b552bfdfb2216fb2b04fa7bbf7bff98eaddac917f25165f2b65fe"}
Sep 11 03:40:26.914366 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:40:26.914336    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-nv6wz" event={"ID":"5bf35a4f-d6d3-4c6d-906f-5f88bc6767b6","Type":"ContainerStarted","Data":"cca5cb994555f49c5c74f3c08b6b5f734dce307cb3e04a132e166beada96199c"}
Sep 11 03:40:26.921626 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:40:26.921596    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-2lk5x" event={"ID":"20c994e9-80cd-4378-9d56-60044e385640","Type":"ContainerStarted","Data":"a3b8500d72c47051182d8f392bf440b0510658f32f40bbf749a2dadb6192dc5d"}
Sep 11 03:40:26.927611 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:40:26.927583    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-48m57" event={"ID":"975d3d70-a13c-46bb-afc2-05c3609050f4","Type":"ContainerStarted","Data":"831888fcef21e1621bf9b6ffa9034a543d70ecaae26e3956ceb4835c3235a771"}
Sep 11 03:40:26.945712 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:40:26.945671    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-j72hj" event={"ID":"5c976635-b437-40c3-9721-43754ae3a058","Type":"ContainerStarted","Data":"535378d6a076e5ecfd2efeaa7f448432a065a175c1d9fb9a053e201e9520ec6c"}
Sep 11 03:40:26.957924 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:40:26.957889    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-mhjgr" event={"ID":"c35abd6b-ad57-4b78-b072-56e6cd032381","Type":"ContainerStarted","Data":"299f7542ceca7155f3447dfa423d65e543ff3597eb9f9d85799ec7e565d6183f"}
Sep 11 03:40:26.971029 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:40:26.970994    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-105.ec2.internal" event={"ID":"c9ea5fe55135c7b1e5541fdd12fac4be","Type":"ContainerStarted","Data":"f29c4472245281f23ed886dddef5759147061684095979252973a296aa72df62"}
Sep 11 03:40:26.980532 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:40:26.980500    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-cgtgv" event={"ID":"08caa83a-ff39-4f1a-955c-a3b164eb57f7","Type":"ContainerStarted","Data":"0e6f41ec623b3460dd43f82659c6e772fd89e3e33845cbf86f6d58a5cafff8fb"}
Sep 11 03:40:28.002461 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:40:28.002422    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-105.ec2.internal" event={"ID":"466709fffb065514ddf1d3dfe2c8d5d2","Type":"ContainerStarted","Data":"020108afa2e906c837e253613ccda069552bc32509a46f064b8161a68d775277"}
Sep 11 03:40:28.019287 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:40:28.019221    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-105.ec2.internal" podStartSLOduration=4.019196831 podStartE2EDuration="4.019196831s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 03:40:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 03:40:26.986775844 +0000 UTC m=+4.641981416" watchObservedRunningTime="2026-09-11 03:40:28.019196831 +0000 UTC m=+5.674402404"
Sep 11 03:40:28.353294 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:40:28.353196    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3517687a-5b4a-4804-9239-e0a5d1c415e0-metrics-certs\") pod \"network-metrics-daemon-9cmdj\" (UID: \"3517687a-5b4a-4804-9239-e0a5d1c415e0\") " pod="openshift-multus/network-metrics-daemon-9cmdj"
Sep 11 03:40:28.353475 ip-10-0-141-105 kubenswrapper[2579]: E0911 03:40:28.353365    2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 11 03:40:28.353475 ip-10-0-141-105 kubenswrapper[2579]: E0911 03:40:28.353434    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3517687a-5b4a-4804-9239-e0a5d1c415e0-metrics-certs podName:3517687a-5b4a-4804-9239-e0a5d1c415e0 nodeName:}" failed. No retries permitted until 2026-09-11 03:40:32.353413788 +0000 UTC m=+10.008619359 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/3517687a-5b4a-4804-9239-e0a5d1c415e0-metrics-certs") pod "network-metrics-daemon-9cmdj" (UID: "3517687a-5b4a-4804-9239-e0a5d1c415e0") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 11 03:40:28.453808 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:40:28.453743    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/1a7a4b58-e08a-45ef-ab7e-692d3a4f5314-original-pull-secret\") pod \"global-pull-secret-syncer-jwm97\" (UID: \"1a7a4b58-e08a-45ef-ab7e-692d3a4f5314\") " pod="kube-system/global-pull-secret-syncer-jwm97"
Sep 11 03:40:28.453989 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:40:28.453817    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b8wlz\" (UniqueName: \"kubernetes.io/projected/801557d7-c47c-43dd-977e-58a9766436c1-kube-api-access-b8wlz\") pod \"network-check-target-p54z5\" (UID: \"801557d7-c47c-43dd-977e-58a9766436c1\") " pod="openshift-network-diagnostics/network-check-target-p54z5"
Sep 11 03:40:28.454106 ip-10-0-141-105 kubenswrapper[2579]: E0911 03:40:28.454011    2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 11 03:40:28.454106 ip-10-0-141-105 kubenswrapper[2579]: E0911 03:40:28.454033    2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 11 03:40:28.454106 ip-10-0-141-105 kubenswrapper[2579]: E0911 03:40:28.454047    2579 projected.go:196] Error preparing data for projected volume kube-api-access-b8wlz for pod openshift-network-diagnostics/network-check-target-p54z5: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 11 03:40:28.454299 ip-10-0-141-105 kubenswrapper[2579]: E0911 03:40:28.454108    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/801557d7-c47c-43dd-977e-58a9766436c1-kube-api-access-b8wlz podName:801557d7-c47c-43dd-977e-58a9766436c1 nodeName:}" failed. No retries permitted until 2026-09-11 03:40:32.454089834 +0000 UTC m=+10.109295388 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-b8wlz" (UniqueName: "kubernetes.io/projected/801557d7-c47c-43dd-977e-58a9766436c1-kube-api-access-b8wlz") pod "network-check-target-p54z5" (UID: "801557d7-c47c-43dd-977e-58a9766436c1") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 11 03:40:28.454590 ip-10-0-141-105 kubenswrapper[2579]: E0911 03:40:28.454568    2579 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 11 03:40:28.454662 ip-10-0-141-105 kubenswrapper[2579]: E0911 03:40:28.454631    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1a7a4b58-e08a-45ef-ab7e-692d3a4f5314-original-pull-secret podName:1a7a4b58-e08a-45ef-ab7e-692d3a4f5314 nodeName:}" failed. No retries permitted until 2026-09-11 03:40:32.454614075 +0000 UTC m=+10.109819652 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/1a7a4b58-e08a-45ef-ab7e-692d3a4f5314-original-pull-secret") pod "global-pull-secret-syncer-jwm97" (UID: "1a7a4b58-e08a-45ef-ab7e-692d3a4f5314") : object "kube-system"/"original-pull-secret" not registered
Sep 11 03:40:28.848705 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:40:28.848608    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jwm97"
Sep 11 03:40:28.848938 ip-10-0-141-105 kubenswrapper[2579]: E0911 03:40:28.848767    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jwm97" podUID="1a7a4b58-e08a-45ef-ab7e-692d3a4f5314"
Sep 11 03:40:28.849194 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:40:28.849174    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-p54z5"
Sep 11 03:40:28.849302 ip-10-0-141-105 kubenswrapper[2579]: E0911 03:40:28.849280    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-p54z5" podUID="801557d7-c47c-43dd-977e-58a9766436c1"
Sep 11 03:40:28.849362 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:40:28.849342    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9cmdj"
Sep 11 03:40:28.849480 ip-10-0-141-105 kubenswrapper[2579]: E0911 03:40:28.849460    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9cmdj" podUID="3517687a-5b4a-4804-9239-e0a5d1c415e0"
Sep 11 03:40:30.847230 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:40:30.846266    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9cmdj"
Sep 11 03:40:30.847230 ip-10-0-141-105 kubenswrapper[2579]: E0911 03:40:30.846439    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9cmdj" podUID="3517687a-5b4a-4804-9239-e0a5d1c415e0"
Sep 11 03:40:30.847230 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:40:30.846921    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jwm97"
Sep 11 03:40:30.847230 ip-10-0-141-105 kubenswrapper[2579]: E0911 03:40:30.847024    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jwm97" podUID="1a7a4b58-e08a-45ef-ab7e-692d3a4f5314"
Sep 11 03:40:30.847230 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:40:30.847086    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-p54z5"
Sep 11 03:40:30.847230 ip-10-0-141-105 kubenswrapper[2579]: E0911 03:40:30.847192    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-p54z5" podUID="801557d7-c47c-43dd-977e-58a9766436c1"
Sep 11 03:40:32.388966 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:40:32.388922    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3517687a-5b4a-4804-9239-e0a5d1c415e0-metrics-certs\") pod \"network-metrics-daemon-9cmdj\" (UID: \"3517687a-5b4a-4804-9239-e0a5d1c415e0\") " pod="openshift-multus/network-metrics-daemon-9cmdj"
Sep 11 03:40:32.389471 ip-10-0-141-105 kubenswrapper[2579]: E0911 03:40:32.389093    2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 11 03:40:32.389471 ip-10-0-141-105 kubenswrapper[2579]: E0911 03:40:32.389196    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3517687a-5b4a-4804-9239-e0a5d1c415e0-metrics-certs podName:3517687a-5b4a-4804-9239-e0a5d1c415e0 nodeName:}" failed. No retries permitted until 2026-09-11 03:40:40.389171688 +0000 UTC m=+18.044377239 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/3517687a-5b4a-4804-9239-e0a5d1c415e0-metrics-certs") pod "network-metrics-daemon-9cmdj" (UID: "3517687a-5b4a-4804-9239-e0a5d1c415e0") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 11 03:40:32.489302 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:40:32.489260    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/1a7a4b58-e08a-45ef-ab7e-692d3a4f5314-original-pull-secret\") pod \"global-pull-secret-syncer-jwm97\" (UID: \"1a7a4b58-e08a-45ef-ab7e-692d3a4f5314\") " pod="kube-system/global-pull-secret-syncer-jwm97"
Sep 11 03:40:32.489501 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:40:32.489322    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b8wlz\" (UniqueName: \"kubernetes.io/projected/801557d7-c47c-43dd-977e-58a9766436c1-kube-api-access-b8wlz\") pod \"network-check-target-p54z5\" (UID: \"801557d7-c47c-43dd-977e-58a9766436c1\") " pod="openshift-network-diagnostics/network-check-target-p54z5"
Sep 11 03:40:32.489501 ip-10-0-141-105 kubenswrapper[2579]: E0911 03:40:32.489452    2579 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 11 03:40:32.489501 ip-10-0-141-105 kubenswrapper[2579]: E0911 03:40:32.489497    2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 11 03:40:32.489675 ip-10-0-141-105 kubenswrapper[2579]: E0911 03:40:32.489520    2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 11 03:40:32.489675 ip-10-0-141-105 kubenswrapper[2579]: E0911 03:40:32.489533    2579 projected.go:196] Error preparing data for projected volume kube-api-access-b8wlz for pod openshift-network-diagnostics/network-check-target-p54z5: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 11 03:40:32.489675 ip-10-0-141-105 kubenswrapper[2579]: E0911 03:40:32.489538    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1a7a4b58-e08a-45ef-ab7e-692d3a4f5314-original-pull-secret podName:1a7a4b58-e08a-45ef-ab7e-692d3a4f5314 nodeName:}" failed. No retries permitted until 2026-09-11 03:40:40.489514338 +0000 UTC m=+18.144719899 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/1a7a4b58-e08a-45ef-ab7e-692d3a4f5314-original-pull-secret") pod "global-pull-secret-syncer-jwm97" (UID: "1a7a4b58-e08a-45ef-ab7e-692d3a4f5314") : object "kube-system"/"original-pull-secret" not registered
Sep 11 03:40:32.489675 ip-10-0-141-105 kubenswrapper[2579]: E0911 03:40:32.489585    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/801557d7-c47c-43dd-977e-58a9766436c1-kube-api-access-b8wlz podName:801557d7-c47c-43dd-977e-58a9766436c1 nodeName:}" failed. No retries permitted until 2026-09-11 03:40:40.489568823 +0000 UTC m=+18.144774373 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-b8wlz" (UniqueName: "kubernetes.io/projected/801557d7-c47c-43dd-977e-58a9766436c1-kube-api-access-b8wlz") pod "network-check-target-p54z5" (UID: "801557d7-c47c-43dd-977e-58a9766436c1") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 11 03:40:32.847095 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:40:32.847003    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jwm97"
Sep 11 03:40:32.847286 ip-10-0-141-105 kubenswrapper[2579]: E0911 03:40:32.847121    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jwm97" podUID="1a7a4b58-e08a-45ef-ab7e-692d3a4f5314"
Sep 11 03:40:32.847286 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:40:32.847186    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9cmdj"
Sep 11 03:40:32.847286 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:40:32.847267    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-p54z5"
Sep 11 03:40:32.847472 ip-10-0-141-105 kubenswrapper[2579]: E0911 03:40:32.847385    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-p54z5" podUID="801557d7-c47c-43dd-977e-58a9766436c1"
Sep 11 03:40:32.847545 ip-10-0-141-105 kubenswrapper[2579]: E0911 03:40:32.847503    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9cmdj" podUID="3517687a-5b4a-4804-9239-e0a5d1c415e0"
Sep 11 03:40:34.847085 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:40:34.846518    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-p54z5"
Sep 11 03:40:34.847085 ip-10-0-141-105 kubenswrapper[2579]: E0911 03:40:34.846669    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-p54z5" podUID="801557d7-c47c-43dd-977e-58a9766436c1"
Sep 11 03:40:34.847085 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:40:34.847012    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9cmdj"
Sep 11 03:40:34.847748 ip-10-0-141-105 kubenswrapper[2579]: E0911 03:40:34.847124    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9cmdj" podUID="3517687a-5b4a-4804-9239-e0a5d1c415e0"
Sep 11 03:40:34.847748 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:40:34.847160    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jwm97"
Sep 11 03:40:34.847748 ip-10-0-141-105 kubenswrapper[2579]: E0911 03:40:34.847250    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jwm97" podUID="1a7a4b58-e08a-45ef-ab7e-692d3a4f5314"
Sep 11 03:40:36.845880 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:40:36.845845    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-p54z5"
Sep 11 03:40:36.846349 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:40:36.845921    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jwm97"
Sep 11 03:40:36.846349 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:40:36.845925    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9cmdj"
Sep 11 03:40:36.846349 ip-10-0-141-105 kubenswrapper[2579]: E0911 03:40:36.846002    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jwm97" podUID="1a7a4b58-e08a-45ef-ab7e-692d3a4f5314"
Sep 11 03:40:36.846349 ip-10-0-141-105 kubenswrapper[2579]: E0911 03:40:36.846096    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-p54z5" podUID="801557d7-c47c-43dd-977e-58a9766436c1"
Sep 11 03:40:36.846349 ip-10-0-141-105 kubenswrapper[2579]: E0911 03:40:36.846184    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9cmdj" podUID="3517687a-5b4a-4804-9239-e0a5d1c415e0"
Sep 11 03:40:38.845526 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:40:38.845489    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-p54z5"
Sep 11 03:40:38.845987 ip-10-0-141-105 kubenswrapper[2579]: E0911 03:40:38.845616    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-p54z5" podUID="801557d7-c47c-43dd-977e-58a9766436c1"
Sep 11 03:40:38.845987 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:40:38.845490    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jwm97"
Sep 11 03:40:38.845987 ip-10-0-141-105 kubenswrapper[2579]: E0911 03:40:38.845680    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jwm97" podUID="1a7a4b58-e08a-45ef-ab7e-692d3a4f5314"
Sep 11 03:40:38.845987 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:40:38.845489    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9cmdj"
Sep 11 03:40:38.845987 ip-10-0-141-105 kubenswrapper[2579]: E0911 03:40:38.845748    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9cmdj" podUID="3517687a-5b4a-4804-9239-e0a5d1c415e0"
Sep 11 03:40:40.453141 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:40:40.453102    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3517687a-5b4a-4804-9239-e0a5d1c415e0-metrics-certs\") pod \"network-metrics-daemon-9cmdj\" (UID: \"3517687a-5b4a-4804-9239-e0a5d1c415e0\") " pod="openshift-multus/network-metrics-daemon-9cmdj"
Sep 11 03:40:40.453641 ip-10-0-141-105 kubenswrapper[2579]: E0911 03:40:40.453249    2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 11 03:40:40.453641 ip-10-0-141-105 kubenswrapper[2579]: E0911 03:40:40.453310    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3517687a-5b4a-4804-9239-e0a5d1c415e0-metrics-certs podName:3517687a-5b4a-4804-9239-e0a5d1c415e0 nodeName:}" failed. No retries permitted until 2026-09-11 03:40:56.453295327 +0000 UTC m=+34.108500877 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/3517687a-5b4a-4804-9239-e0a5d1c415e0-metrics-certs") pod "network-metrics-daemon-9cmdj" (UID: "3517687a-5b4a-4804-9239-e0a5d1c415e0") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 11 03:40:40.554002 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:40:40.553960    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/1a7a4b58-e08a-45ef-ab7e-692d3a4f5314-original-pull-secret\") pod \"global-pull-secret-syncer-jwm97\" (UID: \"1a7a4b58-e08a-45ef-ab7e-692d3a4f5314\") " pod="kube-system/global-pull-secret-syncer-jwm97"
Sep 11 03:40:40.554216 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:40:40.554024    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b8wlz\" (UniqueName: \"kubernetes.io/projected/801557d7-c47c-43dd-977e-58a9766436c1-kube-api-access-b8wlz\") pod \"network-check-target-p54z5\" (UID: \"801557d7-c47c-43dd-977e-58a9766436c1\") " pod="openshift-network-diagnostics/network-check-target-p54z5"
Sep 11 03:40:40.554282 ip-10-0-141-105 kubenswrapper[2579]: E0911 03:40:40.554242    2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 11 03:40:40.554282 ip-10-0-141-105 kubenswrapper[2579]: E0911 03:40:40.554266    2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 11 03:40:40.554282 ip-10-0-141-105 kubenswrapper[2579]: E0911 03:40:40.554281    2579 projected.go:196] Error preparing data for projected volume kube-api-access-b8wlz for pod openshift-network-diagnostics/network-check-target-p54z5: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 11 03:40:40.554412 ip-10-0-141-105 kubenswrapper[2579]: E0911 03:40:40.554344    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/801557d7-c47c-43dd-977e-58a9766436c1-kube-api-access-b8wlz podName:801557d7-c47c-43dd-977e-58a9766436c1 nodeName:}" failed. No retries permitted until 2026-09-11 03:40:56.554324815 +0000 UTC m=+34.209530383 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-b8wlz" (UniqueName: "kubernetes.io/projected/801557d7-c47c-43dd-977e-58a9766436c1-kube-api-access-b8wlz") pod "network-check-target-p54z5" (UID: "801557d7-c47c-43dd-977e-58a9766436c1") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 11 03:40:40.554624 ip-10-0-141-105 kubenswrapper[2579]: E0911 03:40:40.554593    2579 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 11 03:40:40.554769 ip-10-0-141-105 kubenswrapper[2579]: E0911 03:40:40.554674    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1a7a4b58-e08a-45ef-ab7e-692d3a4f5314-original-pull-secret podName:1a7a4b58-e08a-45ef-ab7e-692d3a4f5314 nodeName:}" failed. No retries permitted until 2026-09-11 03:40:56.554655185 +0000 UTC m=+34.209860730 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/1a7a4b58-e08a-45ef-ab7e-692d3a4f5314-original-pull-secret") pod "global-pull-secret-syncer-jwm97" (UID: "1a7a4b58-e08a-45ef-ab7e-692d3a4f5314") : object "kube-system"/"original-pull-secret" not registered
Sep 11 03:40:40.846689 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:40:40.846612    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-p54z5"
Sep 11 03:40:40.846689 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:40:40.846642    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9cmdj"
Sep 11 03:40:40.846689 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:40:40.846677    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jwm97"
Sep 11 03:40:40.846960 ip-10-0-141-105 kubenswrapper[2579]: E0911 03:40:40.846779    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9cmdj" podUID="3517687a-5b4a-4804-9239-e0a5d1c415e0"
Sep 11 03:40:40.846960 ip-10-0-141-105 kubenswrapper[2579]: E0911 03:40:40.846861    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-p54z5" podUID="801557d7-c47c-43dd-977e-58a9766436c1"
Sep 11 03:40:40.846960 ip-10-0-141-105 kubenswrapper[2579]: E0911 03:40:40.846903    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jwm97" podUID="1a7a4b58-e08a-45ef-ab7e-692d3a4f5314"
Sep 11 03:40:42.846669 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:40:42.846623    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9cmdj"
Sep 11 03:40:42.847158 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:40:42.846718    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jwm97"
Sep 11 03:40:42.847158 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:40:42.846828    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-p54z5"
Sep 11 03:40:42.847158 ip-10-0-141-105 kubenswrapper[2579]: E0911 03:40:42.846849    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9cmdj" podUID="3517687a-5b4a-4804-9239-e0a5d1c415e0"
Sep 11 03:40:42.847158 ip-10-0-141-105 kubenswrapper[2579]: E0911 03:40:42.846925    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-p54z5" podUID="801557d7-c47c-43dd-977e-58a9766436c1"
Sep 11 03:40:42.847158 ip-10-0-141-105 kubenswrapper[2579]: E0911 03:40:42.847023    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jwm97" podUID="1a7a4b58-e08a-45ef-ab7e-692d3a4f5314"
Sep 11 03:40:44.845602 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:40:44.845573    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9cmdj"
Sep 11 03:40:44.845602 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:40:44.845588    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jwm97"
Sep 11 03:40:44.846190 ip-10-0-141-105 kubenswrapper[2579]: E0911 03:40:44.845703    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9cmdj" podUID="3517687a-5b4a-4804-9239-e0a5d1c415e0"
Sep 11 03:40:44.846190 ip-10-0-141-105 kubenswrapper[2579]: E0911 03:40:44.845776    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jwm97" podUID="1a7a4b58-e08a-45ef-ab7e-692d3a4f5314"
Sep 11 03:40:44.846190 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:40:44.845826    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-p54z5"
Sep 11 03:40:44.846190 ip-10-0-141-105 kubenswrapper[2579]: E0911 03:40:44.845894    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-p54z5" podUID="801557d7-c47c-43dd-977e-58a9766436c1"
Sep 11 03:40:46.044242 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:40:46.043983    2579 generic.go:356] "Generic (PLEG): container finished" podID="08caa83a-ff39-4f1a-955c-a3b164eb57f7" containerID="4cd7a75a4973b97c3e6a8a282d81a25970a720a175345bdb364e38982d12ee27" exitCode=0
Sep 11 03:40:46.044242 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:40:46.044067    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-cgtgv" event={"ID":"08caa83a-ff39-4f1a-955c-a3b164eb57f7","Type":"ContainerDied","Data":"4cd7a75a4973b97c3e6a8a282d81a25970a720a175345bdb364e38982d12ee27"}
Sep 11 03:40:46.046043 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:40:46.045834    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5lt7m" event={"ID":"202d85de-7969-4e66-a77e-14029cd0597c","Type":"ContainerStarted","Data":"9eca215f2c1863fb2c9b22db2c8d824ad110aa16df6e7410ad33eb2384679c30"}
Sep 11 03:40:46.047515 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:40:46.047481    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-zg5qq" event={"ID":"9fc36641-9051-4f34-a897-bb4c92977307","Type":"ContainerStarted","Data":"8cae2643f0f7b7c86c99d7936def7b6c1a608c3d491c8c0c02bb0dae872ef8fd"}
Sep 11 03:40:46.051038 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:40:46.051008    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-s9xqv" event={"ID":"ff572c4a-7a16-4c2b-b0a8-160b3b274834","Type":"ContainerStarted","Data":"4d6d0a24b4cdfc382bb99ee711e968b5f3a8edc62e440acdb5e67dc48c5b625f"}
Sep 11 03:40:46.055451 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:40:46.055427    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-nv6wz" event={"ID":"5bf35a4f-d6d3-4c6d-906f-5f88bc6767b6","Type":"ContainerStarted","Data":"75a76c473f8ca0c2a728317de7b16e0054f92e0cfcf72b156955880039ce0bee"}
Sep 11 03:40:46.057099 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:40:46.057058    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-48m57" event={"ID":"975d3d70-a13c-46bb-afc2-05c3609050f4","Type":"ContainerStarted","Data":"88670cdac288ee04396b68a4e1ff3b9b794543eb6a21d3002758670df82c6d5f"}
Sep 11 03:40:46.060646 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:40:46.060626    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-j72hj" event={"ID":"5c976635-b437-40c3-9721-43754ae3a058","Type":"ContainerStarted","Data":"4273587e88589ae9ac0894da8921a46b526b7aeef30a03d803046e0dfc45df7f"}
Sep 11 03:40:46.060748 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:40:46.060694    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-j72hj" event={"ID":"5c976635-b437-40c3-9721-43754ae3a058","Type":"ContainerStarted","Data":"f0cfdedc00f9ad8bad77261e76171817237bb3702162b4b9a3a422aaf250fd13"}
Sep 11 03:40:46.060748 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:40:46.060743    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-j72hj" event={"ID":"5c976635-b437-40c3-9721-43754ae3a058","Type":"ContainerStarted","Data":"2b0134a2d5cad5030c643d08f7472c611362b35b079f6ce5f64b67e4c3d3c9f6"}
Sep 11 03:40:46.060862 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:40:46.060789    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-j72hj" event={"ID":"5c976635-b437-40c3-9721-43754ae3a058","Type":"ContainerStarted","Data":"e9d1d8e8008b5ef3cba6b8dbbaff84daa52a0e675b41d38dd9f7eff3aacd2392"}
Sep 11 03:40:46.060862 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:40:46.060836    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-j72hj" event={"ID":"5c976635-b437-40c3-9721-43754ae3a058","Type":"ContainerStarted","Data":"bdf113f6d81dd943d1d7d30928eae31cb30c0865774fb6278095f101d2f034fe"}
Sep 11 03:40:46.060951 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:40:46.060883    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-j72hj" event={"ID":"5c976635-b437-40c3-9721-43754ae3a058","Type":"ContainerStarted","Data":"f2a5ec4328e24f8e98c547d09ab206556485f7891dbb7863852431951b12a7bf"}
Sep 11 03:40:46.062245 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:40:46.062223    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-mhjgr" event={"ID":"c35abd6b-ad57-4b78-b072-56e6cd032381","Type":"ContainerStarted","Data":"a13691047f89986990835a915af9d85a8fde9ec8b0f0e93265e679845bd33038"}
Sep 11 03:40:46.065935 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:40:46.065899    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-105.ec2.internal" podStartSLOduration=22.065887932 podStartE2EDuration="22.065887932s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 03:40:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 03:40:28.01971229 +0000 UTC m=+5.674917868" watchObservedRunningTime="2026-09-11 03:40:46.065887932 +0000 UTC m=+23.721093499"
Sep 11 03:40:46.079056 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:40:46.079006    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-zg5qq" podStartSLOduration=3.933263364 podStartE2EDuration="23.078991786s" totalImagesPullingTime="19.145728422s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 03:40:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 03:40:46.078917677 +0000 UTC m=+23.734123246" watchObservedRunningTime="2026-09-11 03:40:46.078991786 +0000 UTC m=+23.734197370"
Sep 11 03:40:46.094397 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:40:46.094348    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-48m57" podStartSLOduration=3.890005804 podStartE2EDuration="23.094329917s" totalImagesPullingTime="19.204324113s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 03:40:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 03:40:46.094257138 +0000 UTC m=+23.749462705" watchObservedRunningTime="2026-09-11 03:40:46.094329917 +0000 UTC m=+23.749535484"
Sep 11 03:40:46.109196 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:40:46.109154    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-s9xqv" podStartSLOduration=11.220273393 podStartE2EDuration="23.109136464s" totalImagesPullingTime="11.888863071s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 03:40:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 03:40:46.108815254 +0000 UTC m=+23.764020821" watchObservedRunningTime="2026-09-11 03:40:46.109136464 +0000 UTC m=+23.764342030"
Sep 11 03:40:46.134349 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:40:46.134295    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-mhjgr" podStartSLOduration=3.982566806 podStartE2EDuration="23.134275799s" totalImagesPullingTime="19.151708993s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 03:40:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 03:40:46.133759942 +0000 UTC m=+23.788965503" watchObservedRunningTime="2026-09-11 03:40:46.134275799 +0000 UTC m=+23.789481366"
Sep 11 03:40:46.173382 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:40:46.173324    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-nv6wz" podStartSLOduration=3.027924424 podStartE2EDuration="22.173303779s" totalImagesPullingTime="19.145379355s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 03:40:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 03:40:46.173000438 +0000 UTC m=+23.828206006" watchObservedRunningTime="2026-09-11 03:40:46.173303779 +0000 UTC m=+23.828509347"
Sep 11 03:40:46.523007 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:40:46.522977    2579 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Sep 11 03:40:46.777797 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:40:46.777647    2579 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-09-11T03:40:46.523006446Z","UUID":"3736dd2d-f739-483b-b10a-ebc5db9ec881","Handler":null,"Name":"","Endpoint":""}
Sep 11 03:40:46.779155 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:40:46.779137    2579 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Sep 11 03:40:46.779273 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:40:46.779167    2579 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Sep 11 03:40:46.846449 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:40:46.846420    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-p54z5"
Sep 11 03:40:46.846615 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:40:46.846420    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jwm97"
Sep 11 03:40:46.846615 ip-10-0-141-105 kubenswrapper[2579]: E0911 03:40:46.846536    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-p54z5" podUID="801557d7-c47c-43dd-977e-58a9766436c1"
Sep 11 03:40:46.846615 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:40:46.846422    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9cmdj"
Sep 11 03:40:46.846773 ip-10-0-141-105 kubenswrapper[2579]: E0911 03:40:46.846614    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jwm97" podUID="1a7a4b58-e08a-45ef-ab7e-692d3a4f5314"
Sep 11 03:40:46.846773 ip-10-0-141-105 kubenswrapper[2579]: E0911 03:40:46.846678    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9cmdj" podUID="3517687a-5b4a-4804-9239-e0a5d1c415e0"
Sep 11 03:40:47.066814 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:40:47.066731    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5lt7m" event={"ID":"202d85de-7969-4e66-a77e-14029cd0597c","Type":"ContainerStarted","Data":"9701a07a9c909ded0355f283d346702c253951efa3c94d4006f831467e6cdca1"}
Sep 11 03:40:47.068341 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:40:47.068272    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-2lk5x" event={"ID":"20c994e9-80cd-4378-9d56-60044e385640","Type":"ContainerStarted","Data":"712cfa302e53889d16d69803ab612214d0aceef558d2c8d4a3c5bfab63cfae10"}
Sep 11 03:40:47.085435 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:40:47.085381    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-2lk5x" podStartSLOduration=4.938999393 podStartE2EDuration="24.085359762s" totalImagesPullingTime="19.146360369s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 03:40:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 03:40:47.08500786 +0000 UTC m=+24.740213428" watchObservedRunningTime="2026-09-11 03:40:47.085359762 +0000 UTC m=+24.740565329"
Sep 11 03:40:48.074815 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:40:48.074719    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-j72hj" event={"ID":"5c976635-b437-40c3-9721-43754ae3a058","Type":"ContainerStarted","Data":"2e133a1ff015354a87f3693d7d9ada309e186bc1fe755b21d8373d533fd19963"}
Sep 11 03:40:48.077135 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:40:48.077103    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5lt7m" event={"ID":"202d85de-7969-4e66-a77e-14029cd0597c","Type":"ContainerStarted","Data":"dfd867ebffaae37b7aa598c1aa623f75c2546afaf8176a6b9d8f83f29a768cf7"}
Sep 11 03:40:48.095015 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:40:48.094964    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5lt7m" podStartSLOduration=3.7864558759999998 podStartE2EDuration="25.094948705s" totalImagesPullingTime="21.308492829s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 03:40:23 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 03:40:48.094558987 +0000 UTC m=+25.749764556" watchObservedRunningTime="2026-09-11 03:40:48.094948705 +0000 UTC m=+25.750154277"
Sep 11 03:40:48.846294 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:40:48.846261    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9cmdj"
Sep 11 03:40:48.846500 ip-10-0-141-105 kubenswrapper[2579]: E0911 03:40:48.846406    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9cmdj" podUID="3517687a-5b4a-4804-9239-e0a5d1c415e0"
Sep 11 03:40:48.846805 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:40:48.846780    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jwm97"
Sep 11 03:40:48.846892 ip-10-0-141-105 kubenswrapper[2579]: E0911 03:40:48.846874    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jwm97" podUID="1a7a4b58-e08a-45ef-ab7e-692d3a4f5314"
Sep 11 03:40:48.847100 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:40:48.847061    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-p54z5"
Sep 11 03:40:48.847215 ip-10-0-141-105 kubenswrapper[2579]: E0911 03:40:48.847195    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-p54z5" podUID="801557d7-c47c-43dd-977e-58a9766436c1"
Sep 11 03:40:49.976265 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:40:49.976211    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-s9xqv"
Sep 11 03:40:49.976678 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:40:49.976602    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-s9xqv"
Sep 11 03:40:50.083230 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:40:50.083199    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-s9xqv"
Sep 11 03:40:50.083916 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:40:50.083897    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-s9xqv"
Sep 11 03:40:50.846265 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:40:50.846044    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-p54z5"
Sep 11 03:40:50.846433 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:40:50.846067    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9cmdj"
Sep 11 03:40:50.846433 ip-10-0-141-105 kubenswrapper[2579]: E0911 03:40:50.846361    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-p54z5" podUID="801557d7-c47c-43dd-977e-58a9766436c1"
Sep 11 03:40:50.846541 ip-10-0-141-105 kubenswrapper[2579]: E0911 03:40:50.846461    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9cmdj" podUID="3517687a-5b4a-4804-9239-e0a5d1c415e0"
Sep 11 03:40:50.846541 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:40:50.846128    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jwm97"
Sep 11 03:40:50.846634 ip-10-0-141-105 kubenswrapper[2579]: E0911 03:40:50.846612    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jwm97" podUID="1a7a4b58-e08a-45ef-ab7e-692d3a4f5314"
Sep 11 03:40:51.089586 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:40:51.089508    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-j72hj" event={"ID":"5c976635-b437-40c3-9721-43754ae3a058","Type":"ContainerStarted","Data":"b2b50be05ac6e79f9a695ae70c701dab4bc3baf78388e291bf638d53977a7f31"}
Sep 11 03:40:51.090323 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:40:51.089912    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-j72hj"
Sep 11 03:40:51.091821 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:40:51.091789    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-cgtgv" event={"ID":"08caa83a-ff39-4f1a-955c-a3b164eb57f7","Type":"ContainerStarted","Data":"8b996bdad363e27c8b6cfde280dbc1943da9c6d6149903d5855d8490c8b6194d"}
Sep 11 03:40:51.105262 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:40:51.105235    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-j72hj"
Sep 11 03:40:51.116433 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:40:51.116392    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-j72hj" podStartSLOduration=8.904020685999999 podStartE2EDuration="28.116376476s" totalImagesPullingTime="19.21235579s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 03:40:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 03:40:51.115840473 +0000 UTC m=+28.771046039" watchObservedRunningTime="2026-09-11 03:40:51.116376476 +0000 UTC m=+28.771582040"
Sep 11 03:40:52.097173 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:40:52.097123    2579 generic.go:356] "Generic (PLEG): container finished" podID="08caa83a-ff39-4f1a-955c-a3b164eb57f7" containerID="8b996bdad363e27c8b6cfde280dbc1943da9c6d6149903d5855d8490c8b6194d" exitCode=0
Sep 11 03:40:52.097864 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:40:52.097223    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-cgtgv" event={"ID":"08caa83a-ff39-4f1a-955c-a3b164eb57f7","Type":"ContainerDied","Data":"8b996bdad363e27c8b6cfde280dbc1943da9c6d6149903d5855d8490c8b6194d"}
Sep 11 03:40:52.097934 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:40:52.097872    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-j72hj"
Sep 11 03:40:52.097934 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:40:52.097931    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-j72hj"
Sep 11 03:40:52.116244 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:40:52.116212    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-j72hj"
Sep 11 03:40:52.757617 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:40:52.757405    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-9cmdj"]
Sep 11 03:40:52.757809 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:40:52.757760    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9cmdj"
Sep 11 03:40:52.757912 ip-10-0-141-105 kubenswrapper[2579]: E0911 03:40:52.757861    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9cmdj" podUID="3517687a-5b4a-4804-9239-e0a5d1c415e0"
Sep 11 03:40:52.760525 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:40:52.760501    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-p54z5"]
Sep 11 03:40:52.760638 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:40:52.760614    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-p54z5"
Sep 11 03:40:52.760735 ip-10-0-141-105 kubenswrapper[2579]: E0911 03:40:52.760695    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-p54z5" podUID="801557d7-c47c-43dd-977e-58a9766436c1"
Sep 11 03:40:52.761277 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:40:52.761253    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-jwm97"]
Sep 11 03:40:52.761369 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:40:52.761348    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jwm97"
Sep 11 03:40:52.761448 ip-10-0-141-105 kubenswrapper[2579]: E0911 03:40:52.761424    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jwm97" podUID="1a7a4b58-e08a-45ef-ab7e-692d3a4f5314"
Sep 11 03:40:53.101391 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:40:53.101358    2579 generic.go:356] "Generic (PLEG): container finished" podID="08caa83a-ff39-4f1a-955c-a3b164eb57f7" containerID="a8a7b52d99f8dfae7ff13bfcad4c0bbe8ec11ff57138d3e3b769e995fdea0e8c" exitCode=0
Sep 11 03:40:53.101756 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:40:53.101444    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-cgtgv" event={"ID":"08caa83a-ff39-4f1a-955c-a3b164eb57f7","Type":"ContainerDied","Data":"a8a7b52d99f8dfae7ff13bfcad4c0bbe8ec11ff57138d3e3b769e995fdea0e8c"}
Sep 11 03:40:53.846460 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:40:53.846434    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9cmdj"
Sep 11 03:40:53.846571 ip-10-0-141-105 kubenswrapper[2579]: E0911 03:40:53.846551    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9cmdj" podUID="3517687a-5b4a-4804-9239-e0a5d1c415e0"
Sep 11 03:40:54.105706 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:40:54.105674    2579 generic.go:356] "Generic (PLEG): container finished" podID="08caa83a-ff39-4f1a-955c-a3b164eb57f7" containerID="edeb81ed874eb1611a39923445d87ac74602391a58861c48bebfe384fa4ce906" exitCode=0
Sep 11 03:40:54.106032 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:40:54.105765    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-cgtgv" event={"ID":"08caa83a-ff39-4f1a-955c-a3b164eb57f7","Type":"ContainerDied","Data":"edeb81ed874eb1611a39923445d87ac74602391a58861c48bebfe384fa4ce906"}
Sep 11 03:40:54.846617 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:40:54.846509    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-p54z5"
Sep 11 03:40:54.846799 ip-10-0-141-105 kubenswrapper[2579]: E0911 03:40:54.846675    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-p54z5" podUID="801557d7-c47c-43dd-977e-58a9766436c1"
Sep 11 03:40:54.846799 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:40:54.846698    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jwm97"
Sep 11 03:40:54.846909 ip-10-0-141-105 kubenswrapper[2579]: E0911 03:40:54.846812    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jwm97" podUID="1a7a4b58-e08a-45ef-ab7e-692d3a4f5314"
Sep 11 03:40:55.845823 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:40:55.845788    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9cmdj"
Sep 11 03:40:55.846250 ip-10-0-141-105 kubenswrapper[2579]: E0911 03:40:55.845936    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9cmdj" podUID="3517687a-5b4a-4804-9239-e0a5d1c415e0"
Sep 11 03:40:56.461986 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:40:56.461954    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3517687a-5b4a-4804-9239-e0a5d1c415e0-metrics-certs\") pod \"network-metrics-daemon-9cmdj\" (UID: \"3517687a-5b4a-4804-9239-e0a5d1c415e0\") " pod="openshift-multus/network-metrics-daemon-9cmdj"
Sep 11 03:40:56.462233 ip-10-0-141-105 kubenswrapper[2579]: E0911 03:40:56.462143    2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 11 03:40:56.462233 ip-10-0-141-105 kubenswrapper[2579]: E0911 03:40:56.462216    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3517687a-5b4a-4804-9239-e0a5d1c415e0-metrics-certs podName:3517687a-5b4a-4804-9239-e0a5d1c415e0 nodeName:}" failed. No retries permitted until 2026-09-11 03:41:28.462196586 +0000 UTC m=+66.117402144 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/3517687a-5b4a-4804-9239-e0a5d1c415e0-metrics-certs") pod "network-metrics-daemon-9cmdj" (UID: "3517687a-5b4a-4804-9239-e0a5d1c415e0") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 11 03:40:56.562415 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:40:56.562383    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/1a7a4b58-e08a-45ef-ab7e-692d3a4f5314-original-pull-secret\") pod \"global-pull-secret-syncer-jwm97\" (UID: \"1a7a4b58-e08a-45ef-ab7e-692d3a4f5314\") " pod="kube-system/global-pull-secret-syncer-jwm97"
Sep 11 03:40:56.562559 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:40:56.562427    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b8wlz\" (UniqueName: \"kubernetes.io/projected/801557d7-c47c-43dd-977e-58a9766436c1-kube-api-access-b8wlz\") pod \"network-check-target-p54z5\" (UID: \"801557d7-c47c-43dd-977e-58a9766436c1\") " pod="openshift-network-diagnostics/network-check-target-p54z5"
Sep 11 03:40:56.562559 ip-10-0-141-105 kubenswrapper[2579]: E0911 03:40:56.562523    2579 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 11 03:40:56.562559 ip-10-0-141-105 kubenswrapper[2579]: E0911 03:40:56.562538    2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 11 03:40:56.562559 ip-10-0-141-105 kubenswrapper[2579]: E0911 03:40:56.562555    2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 11 03:40:56.562726 ip-10-0-141-105 kubenswrapper[2579]: E0911 03:40:56.562567    2579 projected.go:196] Error preparing data for projected volume kube-api-access-b8wlz for pod openshift-network-diagnostics/network-check-target-p54z5: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 11 03:40:56.562726 ip-10-0-141-105 kubenswrapper[2579]: E0911 03:40:56.562598    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1a7a4b58-e08a-45ef-ab7e-692d3a4f5314-original-pull-secret podName:1a7a4b58-e08a-45ef-ab7e-692d3a4f5314 nodeName:}" failed. No retries permitted until 2026-09-11 03:41:28.562580382 +0000 UTC m=+66.217785941 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/1a7a4b58-e08a-45ef-ab7e-692d3a4f5314-original-pull-secret") pod "global-pull-secret-syncer-jwm97" (UID: "1a7a4b58-e08a-45ef-ab7e-692d3a4f5314") : object "kube-system"/"original-pull-secret" not registered
Sep 11 03:40:56.562726 ip-10-0-141-105 kubenswrapper[2579]: E0911 03:40:56.562614    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/801557d7-c47c-43dd-977e-58a9766436c1-kube-api-access-b8wlz podName:801557d7-c47c-43dd-977e-58a9766436c1 nodeName:}" failed. No retries permitted until 2026-09-11 03:41:28.562606162 +0000 UTC m=+66.217811706 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-b8wlz" (UniqueName: "kubernetes.io/projected/801557d7-c47c-43dd-977e-58a9766436c1-kube-api-access-b8wlz") pod "network-check-target-p54z5" (UID: "801557d7-c47c-43dd-977e-58a9766436c1") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 11 03:40:56.850168 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:40:56.850075    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jwm97"
Sep 11 03:40:56.850168 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:40:56.850114    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-p54z5"
Sep 11 03:40:56.850700 ip-10-0-141-105 kubenswrapper[2579]: E0911 03:40:56.850217    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jwm97" podUID="1a7a4b58-e08a-45ef-ab7e-692d3a4f5314"
Sep 11 03:40:56.850700 ip-10-0-141-105 kubenswrapper[2579]: E0911 03:40:56.850340    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-p54z5" podUID="801557d7-c47c-43dd-977e-58a9766436c1"
Sep 11 03:40:57.846448 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:40:57.846243    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9cmdj"
Sep 11 03:40:57.846611 ip-10-0-141-105 kubenswrapper[2579]: E0911 03:40:57.846556    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9cmdj" podUID="3517687a-5b4a-4804-9239-e0a5d1c415e0"
Sep 11 03:40:58.063464 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:40:58.063432    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-105.ec2.internal" event="NodeReady"
Sep 11 03:40:58.063924 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:40:58.063586    2579 kubelet_node_status.go:489] "Fast updating node status as it just became ready"
Sep 11 03:40:58.103614 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:40:58.103563    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-rkpx7"]
Sep 11 03:40:58.138096 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:40:58.138040    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-wttz6"]
Sep 11 03:40:58.138262 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:40:58.138190    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-rkpx7"
Sep 11 03:40:58.141001 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:40:58.140740    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Sep 11 03:40:58.141001 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:40:58.140756    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-qlqn4\""
Sep 11 03:40:58.141001 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:40:58.140774    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Sep 11 03:40:58.152000 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:40:58.151979    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-rkpx7"]
Sep 11 03:40:58.152130 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:40:58.152018    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-wttz6"]
Sep 11 03:40:58.152130 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:40:58.152121    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-wttz6"
Sep 11 03:40:58.154699 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:40:58.154671    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Sep 11 03:40:58.154819 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:40:58.154748    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Sep 11 03:40:58.154819 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:40:58.154774    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-qd2rt\""
Sep 11 03:40:58.154819 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:40:58.154752    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Sep 11 03:40:58.276872 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:40:58.276786    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/7eae32af-5f15-4e41-993e-ae4397cdd634-cert\") pod \"ingress-canary-wttz6\" (UID: \"7eae32af-5f15-4e41-993e-ae4397cdd634\") " pod="openshift-ingress-canary/ingress-canary-wttz6"
Sep 11 03:40:58.276872 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:40:58.276851    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/82475b03-f46c-4157-a864-0e92015820af-tmp-dir\") pod \"dns-default-rkpx7\" (UID: \"82475b03-f46c-4157-a864-0e92015820af\") " pod="openshift-dns/dns-default-rkpx7"
Sep 11 03:40:58.277071 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:40:58.276915    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/82475b03-f46c-4157-a864-0e92015820af-metrics-tls\") pod \"dns-default-rkpx7\" (UID: \"82475b03-f46c-4157-a864-0e92015820af\") " pod="openshift-dns/dns-default-rkpx7"
Sep 11 03:40:58.277071 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:40:58.276970    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-59cbf\" (UniqueName: \"kubernetes.io/projected/82475b03-f46c-4157-a864-0e92015820af-kube-api-access-59cbf\") pod \"dns-default-rkpx7\" (UID: \"82475b03-f46c-4157-a864-0e92015820af\") " pod="openshift-dns/dns-default-rkpx7"
Sep 11 03:40:58.277071 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:40:58.276999    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9gqg6\" (UniqueName: \"kubernetes.io/projected/7eae32af-5f15-4e41-993e-ae4397cdd634-kube-api-access-9gqg6\") pod \"ingress-canary-wttz6\" (UID: \"7eae32af-5f15-4e41-993e-ae4397cdd634\") " pod="openshift-ingress-canary/ingress-canary-wttz6"
Sep 11 03:40:58.277071 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:40:58.277032    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/82475b03-f46c-4157-a864-0e92015820af-config-volume\") pod \"dns-default-rkpx7\" (UID: \"82475b03-f46c-4157-a864-0e92015820af\") " pod="openshift-dns/dns-default-rkpx7"
Sep 11 03:40:58.378137 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:40:58.378069    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/7eae32af-5f15-4e41-993e-ae4397cdd634-cert\") pod \"ingress-canary-wttz6\" (UID: \"7eae32af-5f15-4e41-993e-ae4397cdd634\") " pod="openshift-ingress-canary/ingress-canary-wttz6"
Sep 11 03:40:58.378300 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:40:58.378151    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/82475b03-f46c-4157-a864-0e92015820af-tmp-dir\") pod \"dns-default-rkpx7\" (UID: \"82475b03-f46c-4157-a864-0e92015820af\") " pod="openshift-dns/dns-default-rkpx7"
Sep 11 03:40:58.378300 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:40:58.378177    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/82475b03-f46c-4157-a864-0e92015820af-metrics-tls\") pod \"dns-default-rkpx7\" (UID: \"82475b03-f46c-4157-a864-0e92015820af\") " pod="openshift-dns/dns-default-rkpx7"
Sep 11 03:40:58.378300 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:40:58.378199    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-59cbf\" (UniqueName: \"kubernetes.io/projected/82475b03-f46c-4157-a864-0e92015820af-kube-api-access-59cbf\") pod \"dns-default-rkpx7\" (UID: \"82475b03-f46c-4157-a864-0e92015820af\") " pod="openshift-dns/dns-default-rkpx7"
Sep 11 03:40:58.378300 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:40:58.378218    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9gqg6\" (UniqueName: \"kubernetes.io/projected/7eae32af-5f15-4e41-993e-ae4397cdd634-kube-api-access-9gqg6\") pod \"ingress-canary-wttz6\" (UID: \"7eae32af-5f15-4e41-993e-ae4397cdd634\") " pod="openshift-ingress-canary/ingress-canary-wttz6"
Sep 11 03:40:58.378485 ip-10-0-141-105 kubenswrapper[2579]: E0911 03:40:58.378317    2579 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 11 03:40:58.378485 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:40:58.378358    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/82475b03-f46c-4157-a864-0e92015820af-config-volume\") pod \"dns-default-rkpx7\" (UID: \"82475b03-f46c-4157-a864-0e92015820af\") " pod="openshift-dns/dns-default-rkpx7"
Sep 11 03:40:58.378485 ip-10-0-141-105 kubenswrapper[2579]: E0911 03:40:58.378383    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/82475b03-f46c-4157-a864-0e92015820af-metrics-tls podName:82475b03-f46c-4157-a864-0e92015820af nodeName:}" failed. No retries permitted until 2026-09-11 03:40:58.878362521 +0000 UTC m=+36.533568083 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/82475b03-f46c-4157-a864-0e92015820af-metrics-tls") pod "dns-default-rkpx7" (UID: "82475b03-f46c-4157-a864-0e92015820af") : secret "dns-default-metrics-tls" not found
Sep 11 03:40:58.378584 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:40:58.378511    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/82475b03-f46c-4157-a864-0e92015820af-tmp-dir\") pod \"dns-default-rkpx7\" (UID: \"82475b03-f46c-4157-a864-0e92015820af\") " pod="openshift-dns/dns-default-rkpx7"
Sep 11 03:40:58.378835 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:40:58.378817    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/82475b03-f46c-4157-a864-0e92015820af-config-volume\") pod \"dns-default-rkpx7\" (UID: \"82475b03-f46c-4157-a864-0e92015820af\") " pod="openshift-dns/dns-default-rkpx7"
Sep 11 03:40:58.382611 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:40:58.382596    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/7eae32af-5f15-4e41-993e-ae4397cdd634-cert\") pod \"ingress-canary-wttz6\" (UID: \"7eae32af-5f15-4e41-993e-ae4397cdd634\") " pod="openshift-ingress-canary/ingress-canary-wttz6"
Sep 11 03:40:58.386696 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:40:58.386667    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-59cbf\" (UniqueName: \"kubernetes.io/projected/82475b03-f46c-4157-a864-0e92015820af-kube-api-access-59cbf\") pod \"dns-default-rkpx7\" (UID: \"82475b03-f46c-4157-a864-0e92015820af\") " pod="openshift-dns/dns-default-rkpx7"
Sep 11 03:40:58.386831 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:40:58.386810    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9gqg6\" (UniqueName: \"kubernetes.io/projected/7eae32af-5f15-4e41-993e-ae4397cdd634-kube-api-access-9gqg6\") pod \"ingress-canary-wttz6\" (UID: \"7eae32af-5f15-4e41-993e-ae4397cdd634\") " pod="openshift-ingress-canary/ingress-canary-wttz6"
Sep 11 03:40:58.461845 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:40:58.461815    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-wttz6"
Sep 11 03:40:58.849336 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:40:58.849303    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jwm97"
Sep 11 03:40:58.849523 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:40:58.849311    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-p54z5"
Sep 11 03:40:58.852283 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:40:58.852257    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Sep 11 03:40:58.852409 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:40:58.852304    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-hqw27\""
Sep 11 03:40:58.853020 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:40:58.852995    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Sep 11 03:40:58.853166 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:40:58.853023    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Sep 11 03:40:58.882684 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:40:58.882624    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/82475b03-f46c-4157-a864-0e92015820af-metrics-tls\") pod \"dns-default-rkpx7\" (UID: \"82475b03-f46c-4157-a864-0e92015820af\") " pod="openshift-dns/dns-default-rkpx7"
Sep 11 03:40:58.882791 ip-10-0-141-105 kubenswrapper[2579]: E0911 03:40:58.882760    2579 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 11 03:40:58.882831 ip-10-0-141-105 kubenswrapper[2579]: E0911 03:40:58.882821    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/82475b03-f46c-4157-a864-0e92015820af-metrics-tls podName:82475b03-f46c-4157-a864-0e92015820af nodeName:}" failed. No retries permitted until 2026-09-11 03:40:59.882805457 +0000 UTC m=+37.538011005 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/82475b03-f46c-4157-a864-0e92015820af-metrics-tls") pod "dns-default-rkpx7" (UID: "82475b03-f46c-4157-a864-0e92015820af") : secret "dns-default-metrics-tls" not found
Sep 11 03:40:58.915105 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:40:58.915060    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-wttz6"]
Sep 11 03:40:58.947979 ip-10-0-141-105 kubenswrapper[2579]: W0911 03:40:58.947947    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7eae32af_5f15_4e41_993e_ae4397cdd634.slice/crio-e5728bacb0ce461ed2c3f62c9aa3194da8e6cb9ddf069dc3aac2aae18de8b8dd WatchSource:0}: Error finding container e5728bacb0ce461ed2c3f62c9aa3194da8e6cb9ddf069dc3aac2aae18de8b8dd: Status 404 returned error can't find the container with id e5728bacb0ce461ed2c3f62c9aa3194da8e6cb9ddf069dc3aac2aae18de8b8dd
Sep 11 03:40:59.120444 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:40:59.120394    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-wttz6" event={"ID":"7eae32af-5f15-4e41-993e-ae4397cdd634","Type":"ContainerStarted","Data":"e5728bacb0ce461ed2c3f62c9aa3194da8e6cb9ddf069dc3aac2aae18de8b8dd"}
Sep 11 03:40:59.123197 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:40:59.123167    2579 generic.go:356] "Generic (PLEG): container finished" podID="08caa83a-ff39-4f1a-955c-a3b164eb57f7" containerID="21d14151f7a2fb6f3fd77a1570d51a7a17b483402573f44be7d83e8d06ba8810" exitCode=0
Sep 11 03:40:59.123307 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:40:59.123213    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-cgtgv" event={"ID":"08caa83a-ff39-4f1a-955c-a3b164eb57f7","Type":"ContainerDied","Data":"21d14151f7a2fb6f3fd77a1570d51a7a17b483402573f44be7d83e8d06ba8810"}
Sep 11 03:40:59.845683 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:40:59.845500    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9cmdj"
Sep 11 03:40:59.848587 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:40:59.848559    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Sep 11 03:40:59.848717 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:40:59.848571    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-sfchv\""
Sep 11 03:40:59.890357 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:40:59.890332    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/82475b03-f46c-4157-a864-0e92015820af-metrics-tls\") pod \"dns-default-rkpx7\" (UID: \"82475b03-f46c-4157-a864-0e92015820af\") " pod="openshift-dns/dns-default-rkpx7"
Sep 11 03:40:59.890468 ip-10-0-141-105 kubenswrapper[2579]: E0911 03:40:59.890446    2579 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 11 03:40:59.890514 ip-10-0-141-105 kubenswrapper[2579]: E0911 03:40:59.890499    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/82475b03-f46c-4157-a864-0e92015820af-metrics-tls podName:82475b03-f46c-4157-a864-0e92015820af nodeName:}" failed. No retries permitted until 2026-09-11 03:41:01.890486281 +0000 UTC m=+39.545691827 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/82475b03-f46c-4157-a864-0e92015820af-metrics-tls") pod "dns-default-rkpx7" (UID: "82475b03-f46c-4157-a864-0e92015820af") : secret "dns-default-metrics-tls" not found
Sep 11 03:41:00.130519 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:41:00.130441    2579 generic.go:356] "Generic (PLEG): container finished" podID="08caa83a-ff39-4f1a-955c-a3b164eb57f7" containerID="42c2e8c2aa248eba7bc7a33f050bbaa9c7b938edf5221d40a5425c0677a21aa6" exitCode=0
Sep 11 03:41:00.130519 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:41:00.130491    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-cgtgv" event={"ID":"08caa83a-ff39-4f1a-955c-a3b164eb57f7","Type":"ContainerDied","Data":"42c2e8c2aa248eba7bc7a33f050bbaa9c7b938edf5221d40a5425c0677a21aa6"}
Sep 11 03:41:01.137487 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:41:01.137450    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-cgtgv" event={"ID":"08caa83a-ff39-4f1a-955c-a3b164eb57f7","Type":"ContainerStarted","Data":"5d033d0d18888d302f62bf1ceba13d0a8e2a0da173f64a70dead835305692b9b"}
Sep 11 03:41:01.159626 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:41:01.159568    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-cgtgv" podStartSLOduration=8.209969109 podStartE2EDuration="38.159551069s" totalImagesPullingTime="29.94958196s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 03:40:23 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 03:41:01.158624745 +0000 UTC m=+38.813830313" watchObservedRunningTime="2026-09-11 03:41:01.159551069 +0000 UTC m=+38.814756630"
Sep 11 03:41:01.907911 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:41:01.907830    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/82475b03-f46c-4157-a864-0e92015820af-metrics-tls\") pod \"dns-default-rkpx7\" (UID: \"82475b03-f46c-4157-a864-0e92015820af\") " pod="openshift-dns/dns-default-rkpx7"
Sep 11 03:41:01.908047 ip-10-0-141-105 kubenswrapper[2579]: E0911 03:41:01.907940    2579 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 11 03:41:01.908047 ip-10-0-141-105 kubenswrapper[2579]: E0911 03:41:01.908013    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/82475b03-f46c-4157-a864-0e92015820af-metrics-tls podName:82475b03-f46c-4157-a864-0e92015820af nodeName:}" failed. No retries permitted until 2026-09-11 03:41:05.907995771 +0000 UTC m=+43.563201336 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/82475b03-f46c-4157-a864-0e92015820af-metrics-tls") pod "dns-default-rkpx7" (UID: "82475b03-f46c-4157-a864-0e92015820af") : secret "dns-default-metrics-tls" not found
Sep 11 03:41:02.142023 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:41:02.141986    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-wttz6" event={"ID":"7eae32af-5f15-4e41-993e-ae4397cdd634","Type":"ContainerStarted","Data":"fa4d07f50566d11c8fa6a5e62b5e9c7cb8e528f5d03570924348477b1a8b5338"}
Sep 11 03:41:02.156515 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:41:02.156468    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-wttz6" podStartSLOduration=1.5415230979999999 podStartE2EDuration="4.156449705s" totalImagesPullingTime="2.614926607s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 03:40:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 03:41:02.155671059 +0000 UTC m=+39.810876641" watchObservedRunningTime="2026-09-11 03:41:02.156449705 +0000 UTC m=+39.811655270"
Sep 11 03:41:04.578855 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:41:04.578819    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-82sjt"]
Sep 11 03:41:04.612350 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:41:04.612318    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-82sjt"]
Sep 11 03:41:04.612482 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:41:04.612416    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-82sjt"
Sep 11 03:41:04.615791 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:41:04.615768    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"openshift-service-ca.crt\""
Sep 11 03:41:04.615881 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:41:04.615791    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"kube-root-ca.crt\""
Sep 11 03:41:04.615959 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:41:04.615942    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-storage-operator\"/\"volume-data-source-validator-dockercfg-7cf5l\""
Sep 11 03:41:04.697410 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:41:04.697376    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-6hjz5"]
Sep 11 03:41:04.720294 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:41:04.720261    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/router-default-7ff5fb48d-7h989"]
Sep 11 03:41:04.720420 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:41:04.720397    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-6hjz5"
Sep 11 03:41:04.724349 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:41:04.724328    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"openshift-service-ca.crt\""
Sep 11 03:41:04.724470 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:41:04.724328    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"serving-cert\""
Sep 11 03:41:04.724470 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:41:04.724333    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-root-ca.crt\""
Sep 11 03:41:04.724870 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:41:04.724851    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"config\""
Sep 11 03:41:04.724961 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:41:04.724887    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-storage-version-migrator-operator-dockercfg-2f6n7\""
Sep 11 03:41:04.728551 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:41:04.728527    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bmvf2\" (UniqueName: \"kubernetes.io/projected/5a0a9ca4-faf7-4b53-8f35-7f5138ab0895-kube-api-access-bmvf2\") pod \"volume-data-source-validator-86695b7cc7-82sjt\" (UID: \"5a0a9ca4-faf7-4b53-8f35-7f5138ab0895\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-82sjt"
Sep 11 03:41:04.740284 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:41:04.740264    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-6hjz5"]
Sep 11 03:41:04.740380 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:41:04.740291    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-7ff5fb48d-7h989"]
Sep 11 03:41:04.740380 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:41:04.740354    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-7ff5fb48d-7h989"
Sep 11 03:41:04.743163 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:41:04.743144    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
Sep 11 03:41:04.743340 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:41:04.743190    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-stats-default\""
Sep 11 03:41:04.743340 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:41:04.743305    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"default-ingress-cert\""
Sep 11 03:41:04.743520 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:41:04.743448    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
Sep 11 03:41:04.743658 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:41:04.743642    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"service-ca-bundle\""
Sep 11 03:41:04.744209 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:41:04.744193    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-hlpk7\""
Sep 11 03:41:04.744735 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:41:04.744719    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-metrics-certs-default\""
Sep 11 03:41:04.829590 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:41:04.829508    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2695af89-0e52-4bd4-9bc5-da707f50efc9-metrics-certs\") pod \"router-default-7ff5fb48d-7h989\" (UID: \"2695af89-0e52-4bd4-9bc5-da707f50efc9\") " pod="openshift-ingress/router-default-7ff5fb48d-7h989"
Sep 11 03:41:04.829590 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:41:04.829543    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vczmk\" (UniqueName: \"kubernetes.io/projected/2695af89-0e52-4bd4-9bc5-da707f50efc9-kube-api-access-vczmk\") pod \"router-default-7ff5fb48d-7h989\" (UID: \"2695af89-0e52-4bd4-9bc5-da707f50efc9\") " pod="openshift-ingress/router-default-7ff5fb48d-7h989"
Sep 11 03:41:04.829590 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:41:04.829577    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p888s\" (UniqueName: \"kubernetes.io/projected/9dbdbcd2-cfbf-447b-8064-35a1852b1a9d-kube-api-access-p888s\") pod \"kube-storage-version-migrator-operator-6c8f84f897-6hjz5\" (UID: \"9dbdbcd2-cfbf-447b-8064-35a1852b1a9d\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-6hjz5"
Sep 11 03:41:04.829590 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:41:04.829594    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/2695af89-0e52-4bd4-9bc5-da707f50efc9-stats-auth\") pod \"router-default-7ff5fb48d-7h989\" (UID: \"2695af89-0e52-4bd4-9bc5-da707f50efc9\") " pod="openshift-ingress/router-default-7ff5fb48d-7h989"
Sep 11 03:41:04.829912 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:41:04.829660    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bmvf2\" (UniqueName: \"kubernetes.io/projected/5a0a9ca4-faf7-4b53-8f35-7f5138ab0895-kube-api-access-bmvf2\") pod \"volume-data-source-validator-86695b7cc7-82sjt\" (UID: \"5a0a9ca4-faf7-4b53-8f35-7f5138ab0895\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-82sjt"
Sep 11 03:41:04.829912 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:41:04.829697    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/9dbdbcd2-cfbf-447b-8064-35a1852b1a9d-serving-cert\") pod \"kube-storage-version-migrator-operator-6c8f84f897-6hjz5\" (UID: \"9dbdbcd2-cfbf-447b-8064-35a1852b1a9d\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-6hjz5"
Sep 11 03:41:04.829912 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:41:04.829759    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/9dbdbcd2-cfbf-447b-8064-35a1852b1a9d-config\") pod \"kube-storage-version-migrator-operator-6c8f84f897-6hjz5\" (UID: \"9dbdbcd2-cfbf-447b-8064-35a1852b1a9d\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-6hjz5"
Sep 11 03:41:04.829912 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:41:04.829792    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2695af89-0e52-4bd4-9bc5-da707f50efc9-service-ca-bundle\") pod \"router-default-7ff5fb48d-7h989\" (UID: \"2695af89-0e52-4bd4-9bc5-da707f50efc9\") " pod="openshift-ingress/router-default-7ff5fb48d-7h989"
Sep 11 03:41:04.829912 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:41:04.829886    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/2695af89-0e52-4bd4-9bc5-da707f50efc9-default-certificate\") pod \"router-default-7ff5fb48d-7h989\" (UID: \"2695af89-0e52-4bd4-9bc5-da707f50efc9\") " pod="openshift-ingress/router-default-7ff5fb48d-7h989"
Sep 11 03:41:04.843802 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:41:04.843771    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bmvf2\" (UniqueName: \"kubernetes.io/projected/5a0a9ca4-faf7-4b53-8f35-7f5138ab0895-kube-api-access-bmvf2\") pod \"volume-data-source-validator-86695b7cc7-82sjt\" (UID: \"5a0a9ca4-faf7-4b53-8f35-7f5138ab0895\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-82sjt"
Sep 11 03:41:04.922856 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:41:04.922822    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-82sjt"
Sep 11 03:41:04.931196 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:41:04.931164    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2695af89-0e52-4bd4-9bc5-da707f50efc9-metrics-certs\") pod \"router-default-7ff5fb48d-7h989\" (UID: \"2695af89-0e52-4bd4-9bc5-da707f50efc9\") " pod="openshift-ingress/router-default-7ff5fb48d-7h989"
Sep 11 03:41:04.931196 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:41:04.931198    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vczmk\" (UniqueName: \"kubernetes.io/projected/2695af89-0e52-4bd4-9bc5-da707f50efc9-kube-api-access-vczmk\") pod \"router-default-7ff5fb48d-7h989\" (UID: \"2695af89-0e52-4bd4-9bc5-da707f50efc9\") " pod="openshift-ingress/router-default-7ff5fb48d-7h989"
Sep 11 03:41:04.931351 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:41:04.931231    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p888s\" (UniqueName: \"kubernetes.io/projected/9dbdbcd2-cfbf-447b-8064-35a1852b1a9d-kube-api-access-p888s\") pod \"kube-storage-version-migrator-operator-6c8f84f897-6hjz5\" (UID: \"9dbdbcd2-cfbf-447b-8064-35a1852b1a9d\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-6hjz5"
Sep 11 03:41:04.931351 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:41:04.931254    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/2695af89-0e52-4bd4-9bc5-da707f50efc9-stats-auth\") pod \"router-default-7ff5fb48d-7h989\" (UID: \"2695af89-0e52-4bd4-9bc5-da707f50efc9\") " pod="openshift-ingress/router-default-7ff5fb48d-7h989"
Sep 11 03:41:04.931351 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:41:04.931282    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/9dbdbcd2-cfbf-447b-8064-35a1852b1a9d-serving-cert\") pod \"kube-storage-version-migrator-operator-6c8f84f897-6hjz5\" (UID: \"9dbdbcd2-cfbf-447b-8064-35a1852b1a9d\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-6hjz5"
Sep 11 03:41:04.931351 ip-10-0-141-105 kubenswrapper[2579]: E0911 03:41:04.931321    2579 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Sep 11 03:41:04.931351 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:41:04.931332    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/9dbdbcd2-cfbf-447b-8064-35a1852b1a9d-config\") pod \"kube-storage-version-migrator-operator-6c8f84f897-6hjz5\" (UID: \"9dbdbcd2-cfbf-447b-8064-35a1852b1a9d\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-6hjz5"
Sep 11 03:41:04.931582 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:41:04.931368    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2695af89-0e52-4bd4-9bc5-da707f50efc9-service-ca-bundle\") pod \"router-default-7ff5fb48d-7h989\" (UID: \"2695af89-0e52-4bd4-9bc5-da707f50efc9\") " pod="openshift-ingress/router-default-7ff5fb48d-7h989"
Sep 11 03:41:04.931582 ip-10-0-141-105 kubenswrapper[2579]: E0911 03:41:04.931407    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2695af89-0e52-4bd4-9bc5-da707f50efc9-metrics-certs podName:2695af89-0e52-4bd4-9bc5-da707f50efc9 nodeName:}" failed. No retries permitted until 2026-09-11 03:41:05.431382741 +0000 UTC m=+43.086588312 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/2695af89-0e52-4bd4-9bc5-da707f50efc9-metrics-certs") pod "router-default-7ff5fb48d-7h989" (UID: "2695af89-0e52-4bd4-9bc5-da707f50efc9") : secret "router-metrics-certs-default" not found
Sep 11 03:41:04.931582 ip-10-0-141-105 kubenswrapper[2579]: E0911 03:41:04.931482    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/2695af89-0e52-4bd4-9bc5-da707f50efc9-service-ca-bundle podName:2695af89-0e52-4bd4-9bc5-da707f50efc9 nodeName:}" failed. No retries permitted until 2026-09-11 03:41:05.431464016 +0000 UTC m=+43.086669581 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/2695af89-0e52-4bd4-9bc5-da707f50efc9-service-ca-bundle") pod "router-default-7ff5fb48d-7h989" (UID: "2695af89-0e52-4bd4-9bc5-da707f50efc9") : configmap references non-existent config key: service-ca.crt
Sep 11 03:41:04.931582 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:41:04.931555    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/2695af89-0e52-4bd4-9bc5-da707f50efc9-default-certificate\") pod \"router-default-7ff5fb48d-7h989\" (UID: \"2695af89-0e52-4bd4-9bc5-da707f50efc9\") " pod="openshift-ingress/router-default-7ff5fb48d-7h989"
Sep 11 03:41:04.932329 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:41:04.932306    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/9dbdbcd2-cfbf-447b-8064-35a1852b1a9d-config\") pod \"kube-storage-version-migrator-operator-6c8f84f897-6hjz5\" (UID: \"9dbdbcd2-cfbf-447b-8064-35a1852b1a9d\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-6hjz5"
Sep 11 03:41:04.933759 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:41:04.933737    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/9dbdbcd2-cfbf-447b-8064-35a1852b1a9d-serving-cert\") pod \"kube-storage-version-migrator-operator-6c8f84f897-6hjz5\" (UID: \"9dbdbcd2-cfbf-447b-8064-35a1852b1a9d\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-6hjz5"
Sep 11 03:41:04.934028 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:41:04.934012    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/2695af89-0e52-4bd4-9bc5-da707f50efc9-default-certificate\") pod \"router-default-7ff5fb48d-7h989\" (UID: \"2695af89-0e52-4bd4-9bc5-da707f50efc9\") " pod="openshift-ingress/router-default-7ff5fb48d-7h989"
Sep 11 03:41:04.934133 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:41:04.934108    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/2695af89-0e52-4bd4-9bc5-da707f50efc9-stats-auth\") pod \"router-default-7ff5fb48d-7h989\" (UID: \"2695af89-0e52-4bd4-9bc5-da707f50efc9\") " pod="openshift-ingress/router-default-7ff5fb48d-7h989"
Sep 11 03:41:04.945266 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:41:04.945243    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vczmk\" (UniqueName: \"kubernetes.io/projected/2695af89-0e52-4bd4-9bc5-da707f50efc9-kube-api-access-vczmk\") pod \"router-default-7ff5fb48d-7h989\" (UID: \"2695af89-0e52-4bd4-9bc5-da707f50efc9\") " pod="openshift-ingress/router-default-7ff5fb48d-7h989"
Sep 11 03:41:04.945705 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:41:04.945684    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-p888s\" (UniqueName: \"kubernetes.io/projected/9dbdbcd2-cfbf-447b-8064-35a1852b1a9d-kube-api-access-p888s\") pod \"kube-storage-version-migrator-operator-6c8f84f897-6hjz5\" (UID: \"9dbdbcd2-cfbf-447b-8064-35a1852b1a9d\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-6hjz5"
Sep 11 03:41:05.029499 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:41:05.029449    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-6hjz5"
Sep 11 03:41:05.209310 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:41:05.209278    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-82sjt"]
Sep 11 03:41:05.212577 ip-10-0-141-105 kubenswrapper[2579]: W0911 03:41:05.212544    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5a0a9ca4_faf7_4b53_8f35_7f5138ab0895.slice/crio-b59652d4b76e4774d41fde42189614eab5cfb3765b7aeda1d8622ebd67ac05d7 WatchSource:0}: Error finding container b59652d4b76e4774d41fde42189614eab5cfb3765b7aeda1d8622ebd67ac05d7: Status 404 returned error can't find the container with id b59652d4b76e4774d41fde42189614eab5cfb3765b7aeda1d8622ebd67ac05d7
Sep 11 03:41:05.328610 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:41:05.328582    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-6hjz5"]
Sep 11 03:41:05.331729 ip-10-0-141-105 kubenswrapper[2579]: W0911 03:41:05.331703    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9dbdbcd2_cfbf_447b_8064_35a1852b1a9d.slice/crio-c158c85c5e4e6cc28bb24254ef4c474b77bbc49969061b991758512b8591ec35 WatchSource:0}: Error finding container c158c85c5e4e6cc28bb24254ef4c474b77bbc49969061b991758512b8591ec35: Status 404 returned error can't find the container with id c158c85c5e4e6cc28bb24254ef4c474b77bbc49969061b991758512b8591ec35
Sep 11 03:41:05.435703 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:41:05.435609    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2695af89-0e52-4bd4-9bc5-da707f50efc9-service-ca-bundle\") pod \"router-default-7ff5fb48d-7h989\" (UID: \"2695af89-0e52-4bd4-9bc5-da707f50efc9\") " pod="openshift-ingress/router-default-7ff5fb48d-7h989"
Sep 11 03:41:05.435703 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:41:05.435674    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2695af89-0e52-4bd4-9bc5-da707f50efc9-metrics-certs\") pod \"router-default-7ff5fb48d-7h989\" (UID: \"2695af89-0e52-4bd4-9bc5-da707f50efc9\") " pod="openshift-ingress/router-default-7ff5fb48d-7h989"
Sep 11 03:41:05.435910 ip-10-0-141-105 kubenswrapper[2579]: E0911 03:41:05.435796    2579 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Sep 11 03:41:05.435910 ip-10-0-141-105 kubenswrapper[2579]: E0911 03:41:05.435826    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/2695af89-0e52-4bd4-9bc5-da707f50efc9-service-ca-bundle podName:2695af89-0e52-4bd4-9bc5-da707f50efc9 nodeName:}" failed. No retries permitted until 2026-09-11 03:41:06.435801341 +0000 UTC m=+44.091006899 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/2695af89-0e52-4bd4-9bc5-da707f50efc9-service-ca-bundle") pod "router-default-7ff5fb48d-7h989" (UID: "2695af89-0e52-4bd4-9bc5-da707f50efc9") : configmap references non-existent config key: service-ca.crt
Sep 11 03:41:05.435910 ip-10-0-141-105 kubenswrapper[2579]: E0911 03:41:05.435856    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2695af89-0e52-4bd4-9bc5-da707f50efc9-metrics-certs podName:2695af89-0e52-4bd4-9bc5-da707f50efc9 nodeName:}" failed. No retries permitted until 2026-09-11 03:41:06.435846061 +0000 UTC m=+44.091051613 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/2695af89-0e52-4bd4-9bc5-da707f50efc9-metrics-certs") pod "router-default-7ff5fb48d-7h989" (UID: "2695af89-0e52-4bd4-9bc5-da707f50efc9") : secret "router-metrics-certs-default" not found
Sep 11 03:41:05.939470 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:41:05.939438    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/82475b03-f46c-4157-a864-0e92015820af-metrics-tls\") pod \"dns-default-rkpx7\" (UID: \"82475b03-f46c-4157-a864-0e92015820af\") " pod="openshift-dns/dns-default-rkpx7"
Sep 11 03:41:05.939871 ip-10-0-141-105 kubenswrapper[2579]: E0911 03:41:05.939600    2579 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 11 03:41:05.939871 ip-10-0-141-105 kubenswrapper[2579]: E0911 03:41:05.939673    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/82475b03-f46c-4157-a864-0e92015820af-metrics-tls podName:82475b03-f46c-4157-a864-0e92015820af nodeName:}" failed. No retries permitted until 2026-09-11 03:41:13.939654376 +0000 UTC m=+51.594859936 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/82475b03-f46c-4157-a864-0e92015820af-metrics-tls") pod "dns-default-rkpx7" (UID: "82475b03-f46c-4157-a864-0e92015820af") : secret "dns-default-metrics-tls" not found
Sep 11 03:41:06.153252 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:41:06.153205    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-82sjt" event={"ID":"5a0a9ca4-faf7-4b53-8f35-7f5138ab0895","Type":"ContainerStarted","Data":"b59652d4b76e4774d41fde42189614eab5cfb3765b7aeda1d8622ebd67ac05d7"}
Sep 11 03:41:06.155055 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:41:06.155021    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-6hjz5" event={"ID":"9dbdbcd2-cfbf-447b-8064-35a1852b1a9d","Type":"ContainerStarted","Data":"c158c85c5e4e6cc28bb24254ef4c474b77bbc49969061b991758512b8591ec35"}
Sep 11 03:41:06.442729 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:41:06.442684    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2695af89-0e52-4bd4-9bc5-da707f50efc9-service-ca-bundle\") pod \"router-default-7ff5fb48d-7h989\" (UID: \"2695af89-0e52-4bd4-9bc5-da707f50efc9\") " pod="openshift-ingress/router-default-7ff5fb48d-7h989"
Sep 11 03:41:06.442945 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:41:06.442784    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2695af89-0e52-4bd4-9bc5-da707f50efc9-metrics-certs\") pod \"router-default-7ff5fb48d-7h989\" (UID: \"2695af89-0e52-4bd4-9bc5-da707f50efc9\") " pod="openshift-ingress/router-default-7ff5fb48d-7h989"
Sep 11 03:41:06.442945 ip-10-0-141-105 kubenswrapper[2579]: E0911 03:41:06.442921    2579 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Sep 11 03:41:06.443123 ip-10-0-141-105 kubenswrapper[2579]: E0911 03:41:06.442978    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2695af89-0e52-4bd4-9bc5-da707f50efc9-metrics-certs podName:2695af89-0e52-4bd4-9bc5-da707f50efc9 nodeName:}" failed. No retries permitted until 2026-09-11 03:41:08.442962287 +0000 UTC m=+46.098167847 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/2695af89-0e52-4bd4-9bc5-da707f50efc9-metrics-certs") pod "router-default-7ff5fb48d-7h989" (UID: "2695af89-0e52-4bd4-9bc5-da707f50efc9") : secret "router-metrics-certs-default" not found
Sep 11 03:41:06.443413 ip-10-0-141-105 kubenswrapper[2579]: E0911 03:41:06.443390    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/2695af89-0e52-4bd4-9bc5-da707f50efc9-service-ca-bundle podName:2695af89-0e52-4bd4-9bc5-da707f50efc9 nodeName:}" failed. No retries permitted until 2026-09-11 03:41:08.443373225 +0000 UTC m=+46.098578775 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/2695af89-0e52-4bd4-9bc5-da707f50efc9-service-ca-bundle") pod "router-default-7ff5fb48d-7h989" (UID: "2695af89-0e52-4bd4-9bc5-da707f50efc9") : configmap references non-existent config key: service-ca.crt
Sep 11 03:41:08.161403 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:41:08.161364    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-82sjt" event={"ID":"5a0a9ca4-faf7-4b53-8f35-7f5138ab0895","Type":"ContainerStarted","Data":"a08693fae875802e686e2b0ac99cd1134f539e2f38e29ad1eb3aaadbb7bbd1a4"}
Sep 11 03:41:08.162887 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:41:08.162858    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-6hjz5" event={"ID":"9dbdbcd2-cfbf-447b-8064-35a1852b1a9d","Type":"ContainerStarted","Data":"b6d5215edd4b3b5889732d43c15828f38bb479cef9d10d18841a268cf27e9ed1"}
Sep 11 03:41:08.176741 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:41:08.176703    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-82sjt" podStartSLOduration=1.7728882750000001 podStartE2EDuration="4.176687476s" totalImagesPullingTime="2.403799201s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 03:41:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 03:41:08.176275466 +0000 UTC m=+45.831481037" watchObservedRunningTime="2026-09-11 03:41:08.176687476 +0000 UTC m=+45.831893043"
Sep 11 03:41:08.193898 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:41:08.193847    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-6hjz5" podStartSLOduration=1.56168949 podStartE2EDuration="4.193827753s" totalImagesPullingTime="2.632138263s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 03:41:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 03:41:08.192673996 +0000 UTC m=+45.847879563" watchObservedRunningTime="2026-09-11 03:41:08.193827753 +0000 UTC m=+45.849033325"
Sep 11 03:41:08.459136 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:41:08.459106    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2695af89-0e52-4bd4-9bc5-da707f50efc9-metrics-certs\") pod \"router-default-7ff5fb48d-7h989\" (UID: \"2695af89-0e52-4bd4-9bc5-da707f50efc9\") " pod="openshift-ingress/router-default-7ff5fb48d-7h989"
Sep 11 03:41:08.459337 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:41:08.459200    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2695af89-0e52-4bd4-9bc5-da707f50efc9-service-ca-bundle\") pod \"router-default-7ff5fb48d-7h989\" (UID: \"2695af89-0e52-4bd4-9bc5-da707f50efc9\") " pod="openshift-ingress/router-default-7ff5fb48d-7h989"
Sep 11 03:41:08.459337 ip-10-0-141-105 kubenswrapper[2579]: E0911 03:41:08.459255    2579 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Sep 11 03:41:08.459337 ip-10-0-141-105 kubenswrapper[2579]: E0911 03:41:08.459326    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/2695af89-0e52-4bd4-9bc5-da707f50efc9-service-ca-bundle podName:2695af89-0e52-4bd4-9bc5-da707f50efc9 nodeName:}" failed. No retries permitted until 2026-09-11 03:41:12.459307832 +0000 UTC m=+50.114513391 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/2695af89-0e52-4bd4-9bc5-da707f50efc9-service-ca-bundle") pod "router-default-7ff5fb48d-7h989" (UID: "2695af89-0e52-4bd4-9bc5-da707f50efc9") : configmap references non-existent config key: service-ca.crt
Sep 11 03:41:08.459467 ip-10-0-141-105 kubenswrapper[2579]: E0911 03:41:08.459348    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2695af89-0e52-4bd4-9bc5-da707f50efc9-metrics-certs podName:2695af89-0e52-4bd4-9bc5-da707f50efc9 nodeName:}" failed. No retries permitted until 2026-09-11 03:41:12.459338341 +0000 UTC m=+50.114543886 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/2695af89-0e52-4bd4-9bc5-da707f50efc9-metrics-certs") pod "router-default-7ff5fb48d-7h989" (UID: "2695af89-0e52-4bd4-9bc5-da707f50efc9") : secret "router-metrics-certs-default" not found
Sep 11 03:41:08.872214 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:41:08.872184    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-8zk4s"]
Sep 11 03:41:08.903114 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:41:08.903067    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-8zk4s"]
Sep 11 03:41:08.903263 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:41:08.903200    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-8zk4s"
Sep 11 03:41:08.906298 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:41:08.906270    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Sep 11 03:41:08.906444 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:41:08.906270    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Sep 11 03:41:08.906444 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:41:08.906426    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-8h9gh\""
Sep 11 03:41:08.906659 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:41:08.906644    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Sep 11 03:41:08.906745 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:41:08.906700    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Sep 11 03:41:09.062749 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:41:09.062714    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/6bc2f317-5df1-4b84-9cc6-496fd598ee89-crio-socket\") pod \"insights-runtime-extractor-8zk4s\" (UID: \"6bc2f317-5df1-4b84-9cc6-496fd598ee89\") " pod="openshift-insights/insights-runtime-extractor-8zk4s"
Sep 11 03:41:09.062950 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:41:09.062779    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/6bc2f317-5df1-4b84-9cc6-496fd598ee89-data-volume\") pod \"insights-runtime-extractor-8zk4s\" (UID: \"6bc2f317-5df1-4b84-9cc6-496fd598ee89\") " pod="openshift-insights/insights-runtime-extractor-8zk4s"
Sep 11 03:41:09.062950 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:41:09.062850    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/6bc2f317-5df1-4b84-9cc6-496fd598ee89-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-8zk4s\" (UID: \"6bc2f317-5df1-4b84-9cc6-496fd598ee89\") " pod="openshift-insights/insights-runtime-extractor-8zk4s"
Sep 11 03:41:09.062950 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:41:09.062899    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x56qn\" (UniqueName: \"kubernetes.io/projected/6bc2f317-5df1-4b84-9cc6-496fd598ee89-kube-api-access-x56qn\") pod \"insights-runtime-extractor-8zk4s\" (UID: \"6bc2f317-5df1-4b84-9cc6-496fd598ee89\") " pod="openshift-insights/insights-runtime-extractor-8zk4s"
Sep 11 03:41:09.063264 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:41:09.063012    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/6bc2f317-5df1-4b84-9cc6-496fd598ee89-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-8zk4s\" (UID: \"6bc2f317-5df1-4b84-9cc6-496fd598ee89\") " pod="openshift-insights/insights-runtime-extractor-8zk4s"
Sep 11 03:41:09.163991 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:41:09.163908    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/6bc2f317-5df1-4b84-9cc6-496fd598ee89-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-8zk4s\" (UID: \"6bc2f317-5df1-4b84-9cc6-496fd598ee89\") " pod="openshift-insights/insights-runtime-extractor-8zk4s"
Sep 11 03:41:09.163991 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:41:09.163977    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/6bc2f317-5df1-4b84-9cc6-496fd598ee89-crio-socket\") pod \"insights-runtime-extractor-8zk4s\" (UID: \"6bc2f317-5df1-4b84-9cc6-496fd598ee89\") " pod="openshift-insights/insights-runtime-extractor-8zk4s"
Sep 11 03:41:09.164479 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:41:09.164018    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/6bc2f317-5df1-4b84-9cc6-496fd598ee89-data-volume\") pod \"insights-runtime-extractor-8zk4s\" (UID: \"6bc2f317-5df1-4b84-9cc6-496fd598ee89\") " pod="openshift-insights/insights-runtime-extractor-8zk4s"
Sep 11 03:41:09.164479 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:41:09.164054    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/6bc2f317-5df1-4b84-9cc6-496fd598ee89-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-8zk4s\" (UID: \"6bc2f317-5df1-4b84-9cc6-496fd598ee89\") " pod="openshift-insights/insights-runtime-extractor-8zk4s"
Sep 11 03:41:09.164479 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:41:09.164076    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x56qn\" (UniqueName: \"kubernetes.io/projected/6bc2f317-5df1-4b84-9cc6-496fd598ee89-kube-api-access-x56qn\") pod \"insights-runtime-extractor-8zk4s\" (UID: \"6bc2f317-5df1-4b84-9cc6-496fd598ee89\") " pod="openshift-insights/insights-runtime-extractor-8zk4s"
Sep 11 03:41:09.164479 ip-10-0-141-105 kubenswrapper[2579]: E0911 03:41:09.164125    2579 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 11 03:41:09.164479 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:41:09.164234    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/6bc2f317-5df1-4b84-9cc6-496fd598ee89-crio-socket\") pod \"insights-runtime-extractor-8zk4s\" (UID: \"6bc2f317-5df1-4b84-9cc6-496fd598ee89\") " pod="openshift-insights/insights-runtime-extractor-8zk4s"
Sep 11 03:41:09.164479 ip-10-0-141-105 kubenswrapper[2579]: E0911 03:41:09.164305    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6bc2f317-5df1-4b84-9cc6-496fd598ee89-insights-runtime-extractor-tls podName:6bc2f317-5df1-4b84-9cc6-496fd598ee89 nodeName:}" failed. No retries permitted until 2026-09-11 03:41:09.66428607 +0000 UTC m=+47.319491619 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/6bc2f317-5df1-4b84-9cc6-496fd598ee89-insights-runtime-extractor-tls") pod "insights-runtime-extractor-8zk4s" (UID: "6bc2f317-5df1-4b84-9cc6-496fd598ee89") : secret "insights-runtime-extractor-tls" not found
Sep 11 03:41:09.164479 ip-10-0-141-105 kubenswrapper[2579]: E0911 03:41:09.164324    2579 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 11 03:41:09.164479 ip-10-0-141-105 kubenswrapper[2579]: E0911 03:41:09.164404    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/6bc2f317-5df1-4b84-9cc6-496fd598ee89-kube-rbac-proxy-cm podName:6bc2f317-5df1-4b84-9cc6-496fd598ee89 nodeName:}" failed. No retries permitted until 2026-09-11 03:41:09.66438675 +0000 UTC m=+47.319592295 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/6bc2f317-5df1-4b84-9cc6-496fd598ee89-kube-rbac-proxy-cm") pod "insights-runtime-extractor-8zk4s" (UID: "6bc2f317-5df1-4b84-9cc6-496fd598ee89") : configmap "kube-rbac-proxy" not found
Sep 11 03:41:09.164479 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:41:09.164447    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/6bc2f317-5df1-4b84-9cc6-496fd598ee89-data-volume\") pod \"insights-runtime-extractor-8zk4s\" (UID: \"6bc2f317-5df1-4b84-9cc6-496fd598ee89\") " pod="openshift-insights/insights-runtime-extractor-8zk4s"
Sep 11 03:41:09.174587 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:41:09.174563    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-x56qn\" (UniqueName: \"kubernetes.io/projected/6bc2f317-5df1-4b84-9cc6-496fd598ee89-kube-api-access-x56qn\") pod \"insights-runtime-extractor-8zk4s\" (UID: \"6bc2f317-5df1-4b84-9cc6-496fd598ee89\") " pod="openshift-insights/insights-runtime-extractor-8zk4s"
Sep 11 03:41:09.667374 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:41:09.667331    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/6bc2f317-5df1-4b84-9cc6-496fd598ee89-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-8zk4s\" (UID: \"6bc2f317-5df1-4b84-9cc6-496fd598ee89\") " pod="openshift-insights/insights-runtime-extractor-8zk4s"
Sep 11 03:41:09.667567 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:41:09.667425    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/6bc2f317-5df1-4b84-9cc6-496fd598ee89-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-8zk4s\" (UID: \"6bc2f317-5df1-4b84-9cc6-496fd598ee89\") " pod="openshift-insights/insights-runtime-extractor-8zk4s"
Sep 11 03:41:09.667567 ip-10-0-141-105 kubenswrapper[2579]: E0911 03:41:09.667482    2579 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 11 03:41:09.667567 ip-10-0-141-105 kubenswrapper[2579]: E0911 03:41:09.667554    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6bc2f317-5df1-4b84-9cc6-496fd598ee89-insights-runtime-extractor-tls podName:6bc2f317-5df1-4b84-9cc6-496fd598ee89 nodeName:}" failed. No retries permitted until 2026-09-11 03:41:10.667538517 +0000 UTC m=+48.322744063 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/6bc2f317-5df1-4b84-9cc6-496fd598ee89-insights-runtime-extractor-tls") pod "insights-runtime-extractor-8zk4s" (UID: "6bc2f317-5df1-4b84-9cc6-496fd598ee89") : secret "insights-runtime-extractor-tls" not found
Sep 11 03:41:09.667567 ip-10-0-141-105 kubenswrapper[2579]: E0911 03:41:09.667553    2579 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 11 03:41:09.667716 ip-10-0-141-105 kubenswrapper[2579]: E0911 03:41:09.667606    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/6bc2f317-5df1-4b84-9cc6-496fd598ee89-kube-rbac-proxy-cm podName:6bc2f317-5df1-4b84-9cc6-496fd598ee89 nodeName:}" failed. No retries permitted until 2026-09-11 03:41:10.667592758 +0000 UTC m=+48.322798307 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/6bc2f317-5df1-4b84-9cc6-496fd598ee89-kube-rbac-proxy-cm") pod "insights-runtime-extractor-8zk4s" (UID: "6bc2f317-5df1-4b84-9cc6-496fd598ee89") : configmap "kube-rbac-proxy" not found
Sep 11 03:41:10.676290 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:41:10.676248    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/6bc2f317-5df1-4b84-9cc6-496fd598ee89-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-8zk4s\" (UID: \"6bc2f317-5df1-4b84-9cc6-496fd598ee89\") " pod="openshift-insights/insights-runtime-extractor-8zk4s"
Sep 11 03:41:10.676694 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:41:10.676338    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/6bc2f317-5df1-4b84-9cc6-496fd598ee89-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-8zk4s\" (UID: \"6bc2f317-5df1-4b84-9cc6-496fd598ee89\") " pod="openshift-insights/insights-runtime-extractor-8zk4s"
Sep 11 03:41:10.676694 ip-10-0-141-105 kubenswrapper[2579]: E0911 03:41:10.676410    2579 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 11 03:41:10.676694 ip-10-0-141-105 kubenswrapper[2579]: E0911 03:41:10.676453    2579 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 11 03:41:10.676694 ip-10-0-141-105 kubenswrapper[2579]: E0911 03:41:10.676493    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6bc2f317-5df1-4b84-9cc6-496fd598ee89-insights-runtime-extractor-tls podName:6bc2f317-5df1-4b84-9cc6-496fd598ee89 nodeName:}" failed. No retries permitted until 2026-09-11 03:41:12.676473262 +0000 UTC m=+50.331678825 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/6bc2f317-5df1-4b84-9cc6-496fd598ee89-insights-runtime-extractor-tls") pod "insights-runtime-extractor-8zk4s" (UID: "6bc2f317-5df1-4b84-9cc6-496fd598ee89") : secret "insights-runtime-extractor-tls" not found
Sep 11 03:41:10.676694 ip-10-0-141-105 kubenswrapper[2579]: E0911 03:41:10.676512    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/6bc2f317-5df1-4b84-9cc6-496fd598ee89-kube-rbac-proxy-cm podName:6bc2f317-5df1-4b84-9cc6-496fd598ee89 nodeName:}" failed. No retries permitted until 2026-09-11 03:41:12.676503391 +0000 UTC m=+50.331708940 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/6bc2f317-5df1-4b84-9cc6-496fd598ee89-kube-rbac-proxy-cm") pod "insights-runtime-extractor-8zk4s" (UID: "6bc2f317-5df1-4b84-9cc6-496fd598ee89") : configmap "kube-rbac-proxy" not found
Sep 11 03:41:10.889815 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:41:10.889785    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-mhjgr_c35abd6b-ad57-4b78-b072-56e6cd032381/dns-node-resolver/0.log"
Sep 11 03:41:11.351909 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:41:11.351875    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-service-ca/service-ca-568869fdc5-bw58m"]
Sep 11 03:41:11.374021 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:41:11.373992    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-568869fdc5-bw58m"]
Sep 11 03:41:11.374169 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:41:11.374128    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-568869fdc5-bw58m"
Sep 11 03:41:11.376907 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:41:11.376881    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"openshift-service-ca.crt\""
Sep 11 03:41:11.377050 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:41:11.376908    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"signing-key\""
Sep 11 03:41:11.378026 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:41:11.378007    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"service-ca-dockercfg-xxl7j\""
Sep 11 03:41:11.378193 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:41:11.378011    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"signing-cabundle\""
Sep 11 03:41:11.378304 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:41:11.378282    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"kube-root-ca.crt\""
Sep 11 03:41:11.483239 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:41:11.483208    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/fcd3e8ff-f3fc-4498-a1ba-f178811ea89a-signing-key\") pod \"service-ca-568869fdc5-bw58m\" (UID: \"fcd3e8ff-f3fc-4498-a1ba-f178811ea89a\") " pod="openshift-service-ca/service-ca-568869fdc5-bw58m"
Sep 11 03:41:11.483399 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:41:11.483258    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q6msr\" (UniqueName: \"kubernetes.io/projected/fcd3e8ff-f3fc-4498-a1ba-f178811ea89a-kube-api-access-q6msr\") pod \"service-ca-568869fdc5-bw58m\" (UID: \"fcd3e8ff-f3fc-4498-a1ba-f178811ea89a\") " pod="openshift-service-ca/service-ca-568869fdc5-bw58m"
Sep 11 03:41:11.483399 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:41:11.483322    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/fcd3e8ff-f3fc-4498-a1ba-f178811ea89a-signing-cabundle\") pod \"service-ca-568869fdc5-bw58m\" (UID: \"fcd3e8ff-f3fc-4498-a1ba-f178811ea89a\") " pod="openshift-service-ca/service-ca-568869fdc5-bw58m"
Sep 11 03:41:11.584324 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:41:11.584282    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q6msr\" (UniqueName: \"kubernetes.io/projected/fcd3e8ff-f3fc-4498-a1ba-f178811ea89a-kube-api-access-q6msr\") pod \"service-ca-568869fdc5-bw58m\" (UID: \"fcd3e8ff-f3fc-4498-a1ba-f178811ea89a\") " pod="openshift-service-ca/service-ca-568869fdc5-bw58m"
Sep 11 03:41:11.584506 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:41:11.584394    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/fcd3e8ff-f3fc-4498-a1ba-f178811ea89a-signing-cabundle\") pod \"service-ca-568869fdc5-bw58m\" (UID: \"fcd3e8ff-f3fc-4498-a1ba-f178811ea89a\") " pod="openshift-service-ca/service-ca-568869fdc5-bw58m"
Sep 11 03:41:11.584506 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:41:11.584462    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/fcd3e8ff-f3fc-4498-a1ba-f178811ea89a-signing-key\") pod \"service-ca-568869fdc5-bw58m\" (UID: \"fcd3e8ff-f3fc-4498-a1ba-f178811ea89a\") " pod="openshift-service-ca/service-ca-568869fdc5-bw58m"
Sep 11 03:41:11.585046 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:41:11.585024    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/fcd3e8ff-f3fc-4498-a1ba-f178811ea89a-signing-cabundle\") pod \"service-ca-568869fdc5-bw58m\" (UID: \"fcd3e8ff-f3fc-4498-a1ba-f178811ea89a\") " pod="openshift-service-ca/service-ca-568869fdc5-bw58m"
Sep 11 03:41:11.587020 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:41:11.586997    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/fcd3e8ff-f3fc-4498-a1ba-f178811ea89a-signing-key\") pod \"service-ca-568869fdc5-bw58m\" (UID: \"fcd3e8ff-f3fc-4498-a1ba-f178811ea89a\") " pod="openshift-service-ca/service-ca-568869fdc5-bw58m"
Sep 11 03:41:11.593099 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:41:11.593056    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-q6msr\" (UniqueName: \"kubernetes.io/projected/fcd3e8ff-f3fc-4498-a1ba-f178811ea89a-kube-api-access-q6msr\") pod \"service-ca-568869fdc5-bw58m\" (UID: \"fcd3e8ff-f3fc-4498-a1ba-f178811ea89a\") " pod="openshift-service-ca/service-ca-568869fdc5-bw58m"
Sep 11 03:41:11.683772 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:41:11.683740    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-568869fdc5-bw58m"
Sep 11 03:41:11.779557 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:41:11.779526    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator/migrator-8b4d697c5-k7vtl"]
Sep 11 03:41:11.794191 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:41:11.794161    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-8b4d697c5-k7vtl"]
Sep 11 03:41:11.794317 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:41:11.794273    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-8b4d697c5-k7vtl"
Sep 11 03:41:11.797514 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:41:11.797493    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\""
Sep 11 03:41:11.797514 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:41:11.797504    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-crwlb\""
Sep 11 03:41:11.797689 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:41:11.797564    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\""
Sep 11 03:41:11.886680 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:41:11.886647    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-brzxk\" (UniqueName: \"kubernetes.io/projected/7d444dfd-481f-4684-a469-94371b9331fc-kube-api-access-brzxk\") pod \"migrator-8b4d697c5-k7vtl\" (UID: \"7d444dfd-481f-4684-a469-94371b9331fc\") " pod="openshift-kube-storage-version-migrator/migrator-8b4d697c5-k7vtl"
Sep 11 03:41:11.968296 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:41:11.968194    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-568869fdc5-bw58m"]
Sep 11 03:41:11.970887 ip-10-0-141-105 kubenswrapper[2579]: W0911 03:41:11.970853    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfcd3e8ff_f3fc_4498_a1ba_f178811ea89a.slice/crio-d5fe8ece06c0d4fb9471f38e160b6ff529f352accde066b9cae0b9e549408f2c WatchSource:0}: Error finding container d5fe8ece06c0d4fb9471f38e160b6ff529f352accde066b9cae0b9e549408f2c: Status 404 returned error can't find the container with id d5fe8ece06c0d4fb9471f38e160b6ff529f352accde066b9cae0b9e549408f2c
Sep 11 03:41:11.987801 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:41:11.987781    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-brzxk\" (UniqueName: \"kubernetes.io/projected/7d444dfd-481f-4684-a469-94371b9331fc-kube-api-access-brzxk\") pod \"migrator-8b4d697c5-k7vtl\" (UID: \"7d444dfd-481f-4684-a469-94371b9331fc\") " pod="openshift-kube-storage-version-migrator/migrator-8b4d697c5-k7vtl"
Sep 11 03:41:11.996418 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:41:11.996398    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-brzxk\" (UniqueName: \"kubernetes.io/projected/7d444dfd-481f-4684-a469-94371b9331fc-kube-api-access-brzxk\") pod \"migrator-8b4d697c5-k7vtl\" (UID: \"7d444dfd-481f-4684-a469-94371b9331fc\") " pod="openshift-kube-storage-version-migrator/migrator-8b4d697c5-k7vtl"
Sep 11 03:41:12.093343 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:41:12.093316    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-zg5qq_9fc36641-9051-4f34-a897-bb4c92977307/node-ca/0.log"
Sep 11 03:41:12.105194 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:41:12.105169    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-8b4d697c5-k7vtl"
Sep 11 03:41:12.175668 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:41:12.175626    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-568869fdc5-bw58m" event={"ID":"fcd3e8ff-f3fc-4498-a1ba-f178811ea89a","Type":"ContainerStarted","Data":"d5fe8ece06c0d4fb9471f38e160b6ff529f352accde066b9cae0b9e549408f2c"}
Sep 11 03:41:12.385021 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:41:12.384871    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-8b4d697c5-k7vtl"]
Sep 11 03:41:12.387334 ip-10-0-141-105 kubenswrapper[2579]: W0911 03:41:12.387308    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7d444dfd_481f_4684_a469_94371b9331fc.slice/crio-8f529e5e6edf40e80a9dad780723f349eb476f9a115395e251405566e5a47c4f WatchSource:0}: Error finding container 8f529e5e6edf40e80a9dad780723f349eb476f9a115395e251405566e5a47c4f: Status 404 returned error can't find the container with id 8f529e5e6edf40e80a9dad780723f349eb476f9a115395e251405566e5a47c4f
Sep 11 03:41:12.492349 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:41:12.492318    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2695af89-0e52-4bd4-9bc5-da707f50efc9-service-ca-bundle\") pod \"router-default-7ff5fb48d-7h989\" (UID: \"2695af89-0e52-4bd4-9bc5-da707f50efc9\") " pod="openshift-ingress/router-default-7ff5fb48d-7h989"
Sep 11 03:41:12.492528 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:41:12.492372    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2695af89-0e52-4bd4-9bc5-da707f50efc9-metrics-certs\") pod \"router-default-7ff5fb48d-7h989\" (UID: \"2695af89-0e52-4bd4-9bc5-da707f50efc9\") " pod="openshift-ingress/router-default-7ff5fb48d-7h989"
Sep 11 03:41:12.492528 ip-10-0-141-105 kubenswrapper[2579]: E0911 03:41:12.492499    2579 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Sep 11 03:41:12.492634 ip-10-0-141-105 kubenswrapper[2579]: E0911 03:41:12.492543    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/2695af89-0e52-4bd4-9bc5-da707f50efc9-service-ca-bundle podName:2695af89-0e52-4bd4-9bc5-da707f50efc9 nodeName:}" failed. No retries permitted until 2026-09-11 03:41:20.492519729 +0000 UTC m=+58.147725301 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/2695af89-0e52-4bd4-9bc5-da707f50efc9-service-ca-bundle") pod "router-default-7ff5fb48d-7h989" (UID: "2695af89-0e52-4bd4-9bc5-da707f50efc9") : configmap references non-existent config key: service-ca.crt
Sep 11 03:41:12.492634 ip-10-0-141-105 kubenswrapper[2579]: E0911 03:41:12.492572    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2695af89-0e52-4bd4-9bc5-da707f50efc9-metrics-certs podName:2695af89-0e52-4bd4-9bc5-da707f50efc9 nodeName:}" failed. No retries permitted until 2026-09-11 03:41:20.4925612 +0000 UTC m=+58.147766751 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/2695af89-0e52-4bd4-9bc5-da707f50efc9-metrics-certs") pod "router-default-7ff5fb48d-7h989" (UID: "2695af89-0e52-4bd4-9bc5-da707f50efc9") : secret "router-metrics-certs-default" not found
Sep 11 03:41:12.694037 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:41:12.693955    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/6bc2f317-5df1-4b84-9cc6-496fd598ee89-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-8zk4s\" (UID: \"6bc2f317-5df1-4b84-9cc6-496fd598ee89\") " pod="openshift-insights/insights-runtime-extractor-8zk4s"
Sep 11 03:41:12.694492 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:41:12.694049    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/6bc2f317-5df1-4b84-9cc6-496fd598ee89-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-8zk4s\" (UID: \"6bc2f317-5df1-4b84-9cc6-496fd598ee89\") " pod="openshift-insights/insights-runtime-extractor-8zk4s"
Sep 11 03:41:12.694492 ip-10-0-141-105 kubenswrapper[2579]: E0911 03:41:12.694163    2579 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 11 03:41:12.694492 ip-10-0-141-105 kubenswrapper[2579]: E0911 03:41:12.694162    2579 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 11 03:41:12.694492 ip-10-0-141-105 kubenswrapper[2579]: E0911 03:41:12.694236    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/6bc2f317-5df1-4b84-9cc6-496fd598ee89-kube-rbac-proxy-cm podName:6bc2f317-5df1-4b84-9cc6-496fd598ee89 nodeName:}" failed. No retries permitted until 2026-09-11 03:41:16.694217822 +0000 UTC m=+54.349423386 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/6bc2f317-5df1-4b84-9cc6-496fd598ee89-kube-rbac-proxy-cm") pod "insights-runtime-extractor-8zk4s" (UID: "6bc2f317-5df1-4b84-9cc6-496fd598ee89") : configmap "kube-rbac-proxy" not found
Sep 11 03:41:12.694492 ip-10-0-141-105 kubenswrapper[2579]: E0911 03:41:12.694256    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6bc2f317-5df1-4b84-9cc6-496fd598ee89-insights-runtime-extractor-tls podName:6bc2f317-5df1-4b84-9cc6-496fd598ee89 nodeName:}" failed. No retries permitted until 2026-09-11 03:41:16.694246073 +0000 UTC m=+54.349451620 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/6bc2f317-5df1-4b84-9cc6-496fd598ee89-insights-runtime-extractor-tls") pod "insights-runtime-extractor-8zk4s" (UID: "6bc2f317-5df1-4b84-9cc6-496fd598ee89") : secret "insights-runtime-extractor-tls" not found
Sep 11 03:41:12.891092 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:41:12.891059    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-wttz6_7eae32af-5f15-4e41-993e-ae4397cdd634/serve-healthcheck-canary/0.log"
Sep 11 03:41:13.180818 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:41:13.180777    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-8b4d697c5-k7vtl" event={"ID":"7d444dfd-481f-4684-a469-94371b9331fc","Type":"ContainerStarted","Data":"8f529e5e6edf40e80a9dad780723f349eb476f9a115395e251405566e5a47c4f"}
Sep 11 03:41:14.006457 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:41:14.006412    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/82475b03-f46c-4157-a864-0e92015820af-metrics-tls\") pod \"dns-default-rkpx7\" (UID: \"82475b03-f46c-4157-a864-0e92015820af\") " pod="openshift-dns/dns-default-rkpx7"
Sep 11 03:41:14.006975 ip-10-0-141-105 kubenswrapper[2579]: E0911 03:41:14.006593    2579 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 11 03:41:14.006975 ip-10-0-141-105 kubenswrapper[2579]: E0911 03:41:14.006672    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/82475b03-f46c-4157-a864-0e92015820af-metrics-tls podName:82475b03-f46c-4157-a864-0e92015820af nodeName:}" failed. No retries permitted until 2026-09-11 03:41:30.00665008 +0000 UTC m=+67.661855629 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/82475b03-f46c-4157-a864-0e92015820af-metrics-tls") pod "dns-default-rkpx7" (UID: "82475b03-f46c-4157-a864-0e92015820af") : secret "dns-default-metrics-tls" not found
Sep 11 03:41:15.189352 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:41:15.189245    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-568869fdc5-bw58m" event={"ID":"fcd3e8ff-f3fc-4498-a1ba-f178811ea89a","Type":"ContainerStarted","Data":"10103d9abff6d5e98350f443d9f61cc81c086cf439f509c19c228c1f2cb0b0cf"}
Sep 11 03:41:15.191230 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:41:15.191204    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-8b4d697c5-k7vtl" event={"ID":"7d444dfd-481f-4684-a469-94371b9331fc","Type":"ContainerStarted","Data":"416d43fe23cccd44232b81642e698316a571708089531fc1144c3e425e3994a5"}
Sep 11 03:41:15.191361 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:41:15.191252    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-8b4d697c5-k7vtl" event={"ID":"7d444dfd-481f-4684-a469-94371b9331fc","Type":"ContainerStarted","Data":"89e5a3a5dd079ea559f3973ca650a628a48052c475430d6713d42cf2405dbaa7"}
Sep 11 03:41:15.207028 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:41:15.206987    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-service-ca/service-ca-568869fdc5-bw58m" podStartSLOduration=1.5084040220000001 podStartE2EDuration="4.206973729s" totalImagesPullingTime="2.698569707s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 03:41:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 03:41:15.205996441 +0000 UTC m=+52.861202008" watchObservedRunningTime="2026-09-11 03:41:15.206973729 +0000 UTC m=+52.862179351"
Sep 11 03:41:15.221674 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:41:15.221633    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator/migrator-8b4d697c5-k7vtl" podStartSLOduration=1.9452297440000001 podStartE2EDuration="4.221619079s" totalImagesPullingTime="2.276389335s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 03:41:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 03:41:15.220675883 +0000 UTC m=+52.875881449" watchObservedRunningTime="2026-09-11 03:41:15.221619079 +0000 UTC m=+52.876824645"
Sep 11 03:41:16.728604 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:41:16.728564    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/6bc2f317-5df1-4b84-9cc6-496fd598ee89-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-8zk4s\" (UID: \"6bc2f317-5df1-4b84-9cc6-496fd598ee89\") " pod="openshift-insights/insights-runtime-extractor-8zk4s"
Sep 11 03:41:16.729059 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:41:16.728638    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/6bc2f317-5df1-4b84-9cc6-496fd598ee89-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-8zk4s\" (UID: \"6bc2f317-5df1-4b84-9cc6-496fd598ee89\") " pod="openshift-insights/insights-runtime-extractor-8zk4s"
Sep 11 03:41:16.729059 ip-10-0-141-105 kubenswrapper[2579]: E0911 03:41:16.728708    2579 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 11 03:41:16.729059 ip-10-0-141-105 kubenswrapper[2579]: E0911 03:41:16.728752    2579 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 11 03:41:16.729059 ip-10-0-141-105 kubenswrapper[2579]: E0911 03:41:16.728775    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6bc2f317-5df1-4b84-9cc6-496fd598ee89-insights-runtime-extractor-tls podName:6bc2f317-5df1-4b84-9cc6-496fd598ee89 nodeName:}" failed. No retries permitted until 2026-09-11 03:41:24.728759741 +0000 UTC m=+62.383965285 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/6bc2f317-5df1-4b84-9cc6-496fd598ee89-insights-runtime-extractor-tls") pod "insights-runtime-extractor-8zk4s" (UID: "6bc2f317-5df1-4b84-9cc6-496fd598ee89") : secret "insights-runtime-extractor-tls" not found
Sep 11 03:41:16.729059 ip-10-0-141-105 kubenswrapper[2579]: E0911 03:41:16.728805    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/6bc2f317-5df1-4b84-9cc6-496fd598ee89-kube-rbac-proxy-cm podName:6bc2f317-5df1-4b84-9cc6-496fd598ee89 nodeName:}" failed. No retries permitted until 2026-09-11 03:41:24.72878244 +0000 UTC m=+62.383987989 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/6bc2f317-5df1-4b84-9cc6-496fd598ee89-kube-rbac-proxy-cm") pod "insights-runtime-extractor-8zk4s" (UID: "6bc2f317-5df1-4b84-9cc6-496fd598ee89") : configmap "kube-rbac-proxy" not found
Sep 11 03:41:20.560875 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:41:20.560832    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2695af89-0e52-4bd4-9bc5-da707f50efc9-service-ca-bundle\") pod \"router-default-7ff5fb48d-7h989\" (UID: \"2695af89-0e52-4bd4-9bc5-da707f50efc9\") " pod="openshift-ingress/router-default-7ff5fb48d-7h989"
Sep 11 03:41:20.561380 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:41:20.560903    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2695af89-0e52-4bd4-9bc5-da707f50efc9-metrics-certs\") pod \"router-default-7ff5fb48d-7h989\" (UID: \"2695af89-0e52-4bd4-9bc5-da707f50efc9\") " pod="openshift-ingress/router-default-7ff5fb48d-7h989"
Sep 11 03:41:20.561380 ip-10-0-141-105 kubenswrapper[2579]: E0911 03:41:20.561030    2579 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Sep 11 03:41:20.561380 ip-10-0-141-105 kubenswrapper[2579]: E0911 03:41:20.561076    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/2695af89-0e52-4bd4-9bc5-da707f50efc9-service-ca-bundle podName:2695af89-0e52-4bd4-9bc5-da707f50efc9 nodeName:}" failed. No retries permitted until 2026-09-11 03:41:36.561050421 +0000 UTC m=+74.216255978 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/2695af89-0e52-4bd4-9bc5-da707f50efc9-service-ca-bundle") pod "router-default-7ff5fb48d-7h989" (UID: "2695af89-0e52-4bd4-9bc5-da707f50efc9") : configmap references non-existent config key: service-ca.crt
Sep 11 03:41:20.561380 ip-10-0-141-105 kubenswrapper[2579]: E0911 03:41:20.561130    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2695af89-0e52-4bd4-9bc5-da707f50efc9-metrics-certs podName:2695af89-0e52-4bd4-9bc5-da707f50efc9 nodeName:}" failed. No retries permitted until 2026-09-11 03:41:36.561117879 +0000 UTC m=+74.216323427 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/2695af89-0e52-4bd4-9bc5-da707f50efc9-metrics-certs") pod "router-default-7ff5fb48d-7h989" (UID: "2695af89-0e52-4bd4-9bc5-da707f50efc9") : secret "router-metrics-certs-default" not found
Sep 11 03:41:24.117278 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:41:24.117250    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-j72hj"
Sep 11 03:41:24.797149 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:41:24.797113    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/6bc2f317-5df1-4b84-9cc6-496fd598ee89-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-8zk4s\" (UID: \"6bc2f317-5df1-4b84-9cc6-496fd598ee89\") " pod="openshift-insights/insights-runtime-extractor-8zk4s"
Sep 11 03:41:24.797363 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:41:24.797180    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/6bc2f317-5df1-4b84-9cc6-496fd598ee89-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-8zk4s\" (UID: \"6bc2f317-5df1-4b84-9cc6-496fd598ee89\") " pod="openshift-insights/insights-runtime-extractor-8zk4s"
Sep 11 03:41:24.797363 ip-10-0-141-105 kubenswrapper[2579]: E0911 03:41:24.797290    2579 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 11 03:41:24.797480 ip-10-0-141-105 kubenswrapper[2579]: E0911 03:41:24.797373    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6bc2f317-5df1-4b84-9cc6-496fd598ee89-insights-runtime-extractor-tls podName:6bc2f317-5df1-4b84-9cc6-496fd598ee89 nodeName:}" failed. No retries permitted until 2026-09-11 03:41:40.797351311 +0000 UTC m=+78.452556872 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/6bc2f317-5df1-4b84-9cc6-496fd598ee89-insights-runtime-extractor-tls") pod "insights-runtime-extractor-8zk4s" (UID: "6bc2f317-5df1-4b84-9cc6-496fd598ee89") : secret "insights-runtime-extractor-tls" not found
Sep 11 03:41:24.797653 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:41:24.797633    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/6bc2f317-5df1-4b84-9cc6-496fd598ee89-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-8zk4s\" (UID: \"6bc2f317-5df1-4b84-9cc6-496fd598ee89\") " pod="openshift-insights/insights-runtime-extractor-8zk4s"
Sep 11 03:41:28.524538 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:41:28.524496    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3517687a-5b4a-4804-9239-e0a5d1c415e0-metrics-certs\") pod \"network-metrics-daemon-9cmdj\" (UID: \"3517687a-5b4a-4804-9239-e0a5d1c415e0\") " pod="openshift-multus/network-metrics-daemon-9cmdj"
Sep 11 03:41:28.527123 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:41:28.527106    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Sep 11 03:41:28.537575 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:41:28.537556    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3517687a-5b4a-4804-9239-e0a5d1c415e0-metrics-certs\") pod \"network-metrics-daemon-9cmdj\" (UID: \"3517687a-5b4a-4804-9239-e0a5d1c415e0\") " pod="openshift-multus/network-metrics-daemon-9cmdj"
Sep 11 03:41:28.625415 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:41:28.625385    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/1a7a4b58-e08a-45ef-ab7e-692d3a4f5314-original-pull-secret\") pod \"global-pull-secret-syncer-jwm97\" (UID: \"1a7a4b58-e08a-45ef-ab7e-692d3a4f5314\") " pod="kube-system/global-pull-secret-syncer-jwm97"
Sep 11 03:41:28.625583 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:41:28.625428    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b8wlz\" (UniqueName: \"kubernetes.io/projected/801557d7-c47c-43dd-977e-58a9766436c1-kube-api-access-b8wlz\") pod \"network-check-target-p54z5\" (UID: \"801557d7-c47c-43dd-977e-58a9766436c1\") " pod="openshift-network-diagnostics/network-check-target-p54z5"
Sep 11 03:41:28.628464 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:41:28.628441    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Sep 11 03:41:28.628573 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:41:28.628440    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Sep 11 03:41:28.637939 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:41:28.637914    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Sep 11 03:41:28.638181 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:41:28.638152    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/1a7a4b58-e08a-45ef-ab7e-692d3a4f5314-original-pull-secret\") pod \"global-pull-secret-syncer-jwm97\" (UID: \"1a7a4b58-e08a-45ef-ab7e-692d3a4f5314\") " pod="kube-system/global-pull-secret-syncer-jwm97"
Sep 11 03:41:28.648162 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:41:28.648135    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-b8wlz\" (UniqueName: \"kubernetes.io/projected/801557d7-c47c-43dd-977e-58a9766436c1-kube-api-access-b8wlz\") pod \"network-check-target-p54z5\" (UID: \"801557d7-c47c-43dd-977e-58a9766436c1\") " pod="openshift-network-diagnostics/network-check-target-p54z5"
Sep 11 03:41:28.659558 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:41:28.659542    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-sfchv\""
Sep 11 03:41:28.667658 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:41:28.667646    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9cmdj"
Sep 11 03:41:28.861909 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:41:28.861881    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jwm97"
Sep 11 03:41:28.868631 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:41:28.868613    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-hqw27\""
Sep 11 03:41:28.876709 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:41:28.876693    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-p54z5"
Sep 11 03:41:28.973311 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:41:28.973281    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-9cmdj"]
Sep 11 03:41:28.976947 ip-10-0-141-105 kubenswrapper[2579]: W0911 03:41:28.976918    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3517687a_5b4a_4804_9239_e0a5d1c415e0.slice/crio-1586d029898c51bfc6062afad81adf6ef70ea266005e2ef0964342e095922615 WatchSource:0}: Error finding container 1586d029898c51bfc6062afad81adf6ef70ea266005e2ef0964342e095922615: Status 404 returned error can't find the container with id 1586d029898c51bfc6062afad81adf6ef70ea266005e2ef0964342e095922615
Sep 11 03:41:29.152291 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:41:29.152267    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-jwm97"]
Sep 11 03:41:29.154712 ip-10-0-141-105 kubenswrapper[2579]: W0911 03:41:29.154685    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1a7a4b58_e08a_45ef_ab7e_692d3a4f5314.slice/crio-e88c711851075d339b4661f5a91273f6876aede65cdf4f9def76ebde54e761e5 WatchSource:0}: Error finding container e88c711851075d339b4661f5a91273f6876aede65cdf4f9def76ebde54e761e5: Status 404 returned error can't find the container with id e88c711851075d339b4661f5a91273f6876aede65cdf4f9def76ebde54e761e5
Sep 11 03:41:29.176409 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:41:29.176380    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-p54z5"]
Sep 11 03:41:29.179225 ip-10-0-141-105 kubenswrapper[2579]: W0911 03:41:29.179199    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod801557d7_c47c_43dd_977e_58a9766436c1.slice/crio-971f38c65e1e7192faa3da90219afa8ad835b41fcb083c35cfcdaec4e1a5dc27 WatchSource:0}: Error finding container 971f38c65e1e7192faa3da90219afa8ad835b41fcb083c35cfcdaec4e1a5dc27: Status 404 returned error can't find the container with id 971f38c65e1e7192faa3da90219afa8ad835b41fcb083c35cfcdaec4e1a5dc27
Sep 11 03:41:29.236880 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:41:29.236844    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-9cmdj" event={"ID":"3517687a-5b4a-4804-9239-e0a5d1c415e0","Type":"ContainerStarted","Data":"1586d029898c51bfc6062afad81adf6ef70ea266005e2ef0964342e095922615"}
Sep 11 03:41:29.238035 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:41:29.238009    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-jwm97" event={"ID":"1a7a4b58-e08a-45ef-ab7e-692d3a4f5314","Type":"ContainerStarted","Data":"e88c711851075d339b4661f5a91273f6876aede65cdf4f9def76ebde54e761e5"}
Sep 11 03:41:29.239184 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:41:29.239160    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-p54z5" event={"ID":"801557d7-c47c-43dd-977e-58a9766436c1","Type":"ContainerStarted","Data":"971f38c65e1e7192faa3da90219afa8ad835b41fcb083c35cfcdaec4e1a5dc27"}
Sep 11 03:41:30.035870 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:41:30.035835    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/82475b03-f46c-4157-a864-0e92015820af-metrics-tls\") pod \"dns-default-rkpx7\" (UID: \"82475b03-f46c-4157-a864-0e92015820af\") " pod="openshift-dns/dns-default-rkpx7"
Sep 11 03:41:30.039575 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:41:30.039545    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/82475b03-f46c-4157-a864-0e92015820af-metrics-tls\") pod \"dns-default-rkpx7\" (UID: \"82475b03-f46c-4157-a864-0e92015820af\") " pod="openshift-dns/dns-default-rkpx7"
Sep 11 03:41:30.262376 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:41:30.262341    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-qlqn4\""
Sep 11 03:41:30.269616 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:41:30.269577    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-rkpx7"
Sep 11 03:41:30.576133 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:41:30.575475    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-rkpx7"]
Sep 11 03:41:30.580048 ip-10-0-141-105 kubenswrapper[2579]: W0911 03:41:30.580017    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod82475b03_f46c_4157_a864_0e92015820af.slice/crio-7fa0585b4ba7a16a37a5454a47a18fbda3cc96760c6ee7c928b1490c55e193b7 WatchSource:0}: Error finding container 7fa0585b4ba7a16a37a5454a47a18fbda3cc96760c6ee7c928b1490c55e193b7: Status 404 returned error can't find the container with id 7fa0585b4ba7a16a37a5454a47a18fbda3cc96760c6ee7c928b1490c55e193b7
Sep 11 03:41:31.249290 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:41:31.249176    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-rkpx7" event={"ID":"82475b03-f46c-4157-a864-0e92015820af","Type":"ContainerStarted","Data":"7fa0585b4ba7a16a37a5454a47a18fbda3cc96760c6ee7c928b1490c55e193b7"}
Sep 11 03:41:31.252598 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:41:31.252341    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-9cmdj" event={"ID":"3517687a-5b4a-4804-9239-e0a5d1c415e0","Type":"ContainerStarted","Data":"5e56ab42c54c6316b8bb09f8b6651bf530bf77db45dc458b2a619e711a96fe44"}
Sep 11 03:41:31.252598 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:41:31.252394    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-9cmdj" event={"ID":"3517687a-5b4a-4804-9239-e0a5d1c415e0","Type":"ContainerStarted","Data":"786624b69c1fe57a7d6110baa6eb3b6390039239de14f19e6512d65fe86726ee"}
Sep 11 03:41:31.271295 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:41:31.271240    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-9cmdj" podStartSLOduration=66.921590745 podStartE2EDuration="1m8.271218772s" totalImagesPullingTime="1.349628027s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 03:40:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 03:41:31.269963043 +0000 UTC m=+68.925168624" watchObservedRunningTime="2026-09-11 03:41:31.271218772 +0000 UTC m=+68.926424341"
Sep 11 03:41:32.118686 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:41:32.118646    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-5df84ddc6-cmdtl"]
Sep 11 03:41:32.120856 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:41:32.120834    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-5df84ddc6-cmdtl"
Sep 11 03:41:32.126867 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:41:32.126837    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\""
Sep 11 03:41:32.127038 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:41:32.126837    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-hub-kubeconfig\""
Sep 11 03:41:32.127512 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:41:32.127491    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\""
Sep 11 03:41:32.127722 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:41:32.127706    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\""
Sep 11 03:41:32.127917 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:41:32.127904    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-dockercfg-wpb8w\""
Sep 11 03:41:32.140386 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:41:32.140359    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-5df84ddc6-cmdtl"]
Sep 11 03:41:32.154072 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:41:32.154041    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/2104734e-8efc-43b0-ad48-f3292b001e87-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-5df84ddc6-cmdtl\" (UID: \"2104734e-8efc-43b0-ad48-f3292b001e87\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-5df84ddc6-cmdtl"
Sep 11 03:41:32.154360 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:41:32.154332    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5kc22\" (UniqueName: \"kubernetes.io/projected/2104734e-8efc-43b0-ad48-f3292b001e87-kube-api-access-5kc22\") pod \"managed-serviceaccount-addon-agent-5df84ddc6-cmdtl\" (UID: \"2104734e-8efc-43b0-ad48-f3292b001e87\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-5df84ddc6-cmdtl"
Sep 11 03:41:32.254689 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:41:32.254659    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5kc22\" (UniqueName: \"kubernetes.io/projected/2104734e-8efc-43b0-ad48-f3292b001e87-kube-api-access-5kc22\") pod \"managed-serviceaccount-addon-agent-5df84ddc6-cmdtl\" (UID: \"2104734e-8efc-43b0-ad48-f3292b001e87\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-5df84ddc6-cmdtl"
Sep 11 03:41:32.255142 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:41:32.254758    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/2104734e-8efc-43b0-ad48-f3292b001e87-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-5df84ddc6-cmdtl\" (UID: \"2104734e-8efc-43b0-ad48-f3292b001e87\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-5df84ddc6-cmdtl"
Sep 11 03:41:32.259686 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:41:32.259651    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/2104734e-8efc-43b0-ad48-f3292b001e87-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-5df84ddc6-cmdtl\" (UID: \"2104734e-8efc-43b0-ad48-f3292b001e87\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-5df84ddc6-cmdtl"
Sep 11 03:41:32.271880 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:41:32.271855    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5kc22\" (UniqueName: \"kubernetes.io/projected/2104734e-8efc-43b0-ad48-f3292b001e87-kube-api-access-5kc22\") pod \"managed-serviceaccount-addon-agent-5df84ddc6-cmdtl\" (UID: \"2104734e-8efc-43b0-ad48-f3292b001e87\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-5df84ddc6-cmdtl"
Sep 11 03:41:32.441382 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:41:32.441295    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-5df84ddc6-cmdtl"
Sep 11 03:41:35.074537 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:41:35.074455    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-5df84ddc6-cmdtl"]
Sep 11 03:41:35.077928 ip-10-0-141-105 kubenswrapper[2579]: W0911 03:41:35.077907    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod2104734e_8efc_43b0_ad48_f3292b001e87.slice/crio-bb6dc4418136ac7a779c74a7bf4dff2b925ea050f1ca9b58ed86711b4c519e9e WatchSource:0}: Error finding container bb6dc4418136ac7a779c74a7bf4dff2b925ea050f1ca9b58ed86711b4c519e9e: Status 404 returned error can't find the container with id bb6dc4418136ac7a779c74a7bf4dff2b925ea050f1ca9b58ed86711b4c519e9e
Sep 11 03:41:35.268871 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:41:35.268841    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-p54z5" event={"ID":"801557d7-c47c-43dd-977e-58a9766436c1","Type":"ContainerStarted","Data":"184128399a862e1254c05797a2813fc6581090d1a0e530f453629d8ef18438f1"}
Sep 11 03:41:35.269062 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:41:35.268976    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-p54z5"
Sep 11 03:41:35.270124 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:41:35.270103    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-5df84ddc6-cmdtl" event={"ID":"2104734e-8efc-43b0-ad48-f3292b001e87","Type":"ContainerStarted","Data":"bb6dc4418136ac7a779c74a7bf4dff2b925ea050f1ca9b58ed86711b4c519e9e"}
Sep 11 03:41:35.271547 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:41:35.271528    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-jwm97" event={"ID":"1a7a4b58-e08a-45ef-ab7e-692d3a4f5314","Type":"ContainerStarted","Data":"292d2e11c5c486fcef5d7cb8dae268d45e940fbbc83d370bb61daa28a59cedf9"}
Sep 11 03:41:35.273254 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:41:35.273232    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-rkpx7" event={"ID":"82475b03-f46c-4157-a864-0e92015820af","Type":"ContainerStarted","Data":"17be8cec6cdec068f4c929e63d42240fd66113dfbdfa546fc89ba2bdade8f0ed"}
Sep 11 03:41:35.273343 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:41:35.273294    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-rkpx7" event={"ID":"82475b03-f46c-4157-a864-0e92015820af","Type":"ContainerStarted","Data":"90a886f66e64dcffb7edbfda3ee10496cb1a85bd7d30563c412ebcc6fa8189fe"}
Sep 11 03:41:35.273399 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:41:35.273385    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-rkpx7"
Sep 11 03:41:35.284777 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:41:35.284736    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-p54z5" podStartSLOduration=66.654487226 podStartE2EDuration="1m12.284721579s" totalImagesPullingTime="5.630234353s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 03:40:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 03:41:35.284329203 +0000 UTC m=+72.939534771" watchObservedRunningTime="2026-09-11 03:41:35.284721579 +0000 UTC m=+72.939927124"
Sep 11 03:41:35.301663 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:41:35.301624    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-rkpx7" podStartSLOduration=33.083780508 podStartE2EDuration="37.301610009s" totalImagesPullingTime="4.217829501s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 03:40:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 03:41:35.301076073 +0000 UTC m=+72.956281640" watchObservedRunningTime="2026-09-11 03:41:35.301610009 +0000 UTC m=+72.956815575"
Sep 11 03:41:36.590722 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:41:36.590681    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2695af89-0e52-4bd4-9bc5-da707f50efc9-service-ca-bundle\") pod \"router-default-7ff5fb48d-7h989\" (UID: \"2695af89-0e52-4bd4-9bc5-da707f50efc9\") " pod="openshift-ingress/router-default-7ff5fb48d-7h989"
Sep 11 03:41:36.591217 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:41:36.590742    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2695af89-0e52-4bd4-9bc5-da707f50efc9-metrics-certs\") pod \"router-default-7ff5fb48d-7h989\" (UID: \"2695af89-0e52-4bd4-9bc5-da707f50efc9\") " pod="openshift-ingress/router-default-7ff5fb48d-7h989"
Sep 11 03:41:36.591465 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:41:36.591434    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2695af89-0e52-4bd4-9bc5-da707f50efc9-service-ca-bundle\") pod \"router-default-7ff5fb48d-7h989\" (UID: \"2695af89-0e52-4bd4-9bc5-da707f50efc9\") " pod="openshift-ingress/router-default-7ff5fb48d-7h989"
Sep 11 03:41:36.595163 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:41:36.595137    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2695af89-0e52-4bd4-9bc5-da707f50efc9-metrics-certs\") pod \"router-default-7ff5fb48d-7h989\" (UID: \"2695af89-0e52-4bd4-9bc5-da707f50efc9\") " pod="openshift-ingress/router-default-7ff5fb48d-7h989"
Sep 11 03:41:36.852169 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:41:36.852056    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-hlpk7\""
Sep 11 03:41:36.859477 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:41:36.859455    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-7ff5fb48d-7h989"
Sep 11 03:41:37.624859 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:41:37.624818    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-jwm97" podStartSLOduration=67.968520125 podStartE2EDuration="1m13.624799342s" totalImagesPullingTime="5.656279217s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 03:40:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 03:41:35.316099678 +0000 UTC m=+72.971305236" watchObservedRunningTime="2026-09-11 03:41:37.624799342 +0000 UTC m=+75.280004906"
Sep 11 03:41:37.625233 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:41:37.625049    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-7ff5fb48d-7h989"]
Sep 11 03:41:37.627152 ip-10-0-141-105 kubenswrapper[2579]: W0911 03:41:37.627124    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2695af89_0e52_4bd4_9bc5_da707f50efc9.slice/crio-3ade5974342c788db7f8bc2a01321d023c644394b3777e5c5d6464f54714af50 WatchSource:0}: Error finding container 3ade5974342c788db7f8bc2a01321d023c644394b3777e5c5d6464f54714af50: Status 404 returned error can't find the container with id 3ade5974342c788db7f8bc2a01321d023c644394b3777e5c5d6464f54714af50
Sep 11 03:41:38.286695 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:41:38.286663    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-7ff5fb48d-7h989" event={"ID":"2695af89-0e52-4bd4-9bc5-da707f50efc9","Type":"ContainerStarted","Data":"7678e4a123fdaf5c29f5293e3ca482d03f11c88a4d9693d79e41b7b9e27d4a51"}
Sep 11 03:41:38.286891 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:41:38.286716    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-7ff5fb48d-7h989" event={"ID":"2695af89-0e52-4bd4-9bc5-da707f50efc9","Type":"ContainerStarted","Data":"3ade5974342c788db7f8bc2a01321d023c644394b3777e5c5d6464f54714af50"}
Sep 11 03:41:38.288129 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:41:38.288108    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-5df84ddc6-cmdtl" event={"ID":"2104734e-8efc-43b0-ad48-f3292b001e87","Type":"ContainerStarted","Data":"be97f275dd3cc26080fbfc2efc6e9e51e6cab4098a6e177349b8eb8ac0a5f1e7"}
Sep 11 03:41:38.310481 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:41:38.310432    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/router-default-7ff5fb48d-7h989" podStartSLOduration=34.310416577 podStartE2EDuration="34.310416577s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 03:41:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 03:41:38.308489698 +0000 UTC m=+75.963695267" watchObservedRunningTime="2026-09-11 03:41:38.310416577 +0000 UTC m=+75.965622144"
Sep 11 03:41:38.326740 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:41:38.326700    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-5df84ddc6-cmdtl" podStartSLOduration=4.01772669 podStartE2EDuration="6.32668123s" totalImagesPullingTime="2.30895454s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 03:41:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 03:41:38.325676789 +0000 UTC m=+75.980882357" watchObservedRunningTime="2026-09-11 03:41:38.32668123 +0000 UTC m=+75.981886798"
Sep 11 03:41:38.860592 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:41:38.860563    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/router-default-7ff5fb48d-7h989"
Sep 11 03:41:38.863003 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:41:38.862982    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/router-default-7ff5fb48d-7h989"
Sep 11 03:41:39.292137 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:41:39.292024    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/router-default-7ff5fb48d-7h989"
Sep 11 03:41:39.293283 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:41:39.293264    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/router-default-7ff5fb48d-7h989"
Sep 11 03:41:40.819300 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:41:40.819271    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/6bc2f317-5df1-4b84-9cc6-496fd598ee89-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-8zk4s\" (UID: \"6bc2f317-5df1-4b84-9cc6-496fd598ee89\") " pod="openshift-insights/insights-runtime-extractor-8zk4s"
Sep 11 03:41:40.821740 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:41:40.821718    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/6bc2f317-5df1-4b84-9cc6-496fd598ee89-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-8zk4s\" (UID: \"6bc2f317-5df1-4b84-9cc6-496fd598ee89\") " pod="openshift-insights/insights-runtime-extractor-8zk4s"
Sep 11 03:41:41.015225 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:41:41.015197    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-8h9gh\""
Sep 11 03:41:41.022485 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:41:41.022463    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-8zk4s"
Sep 11 03:41:41.325592 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:41:41.325524    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-8zk4s"]
Sep 11 03:41:41.329030 ip-10-0-141-105 kubenswrapper[2579]: W0911 03:41:41.329002    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6bc2f317_5df1_4b84_9cc6_496fd598ee89.slice/crio-f40fa3ead5c5ce80137a6e17dbf0f068c12b57b1f0a69cfd251991e387b684e5 WatchSource:0}: Error finding container f40fa3ead5c5ce80137a6e17dbf0f068c12b57b1f0a69cfd251991e387b684e5: Status 404 returned error can't find the container with id f40fa3ead5c5ce80137a6e17dbf0f068c12b57b1f0a69cfd251991e387b684e5
Sep 11 03:41:42.304543 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:41:42.304462    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-8zk4s" event={"ID":"6bc2f317-5df1-4b84-9cc6-496fd598ee89","Type":"ContainerStarted","Data":"8e5afc78be4ef26ad86e7775c548f5ae18b4b4dd696550d53df69a604263a6f8"}
Sep 11 03:41:42.304543 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:41:42.304535    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-8zk4s" event={"ID":"6bc2f317-5df1-4b84-9cc6-496fd598ee89","Type":"ContainerStarted","Data":"ab9a3dff6174b67f63faff82393949481f5ca173e525bfc1c047afbf0e09257e"}
Sep 11 03:41:42.304945 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:41:42.304555    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-8zk4s" event={"ID":"6bc2f317-5df1-4b84-9cc6-496fd598ee89","Type":"ContainerStarted","Data":"f40fa3ead5c5ce80137a6e17dbf0f068c12b57b1f0a69cfd251991e387b684e5"}
Sep 11 03:41:44.313617 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:41:44.313580    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-8zk4s" event={"ID":"6bc2f317-5df1-4b84-9cc6-496fd598ee89","Type":"ContainerStarted","Data":"4cc520365af128d07c819fb6f6dc5ff2dbe48f33560ea4a0fe859b6c42291ee6"}
Sep 11 03:41:44.333072 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:41:44.333026    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-8zk4s" podStartSLOduration=34.478158517 podStartE2EDuration="36.333010903s" totalImagesPullingTime="1.854852386s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 03:41:08 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 03:41:44.331574881 +0000 UTC m=+81.986780449" watchObservedRunningTime="2026-09-11 03:41:44.333010903 +0000 UTC m=+81.988216470"
Sep 11 03:41:45.279464 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:41:45.279437    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-rkpx7"
Sep 11 03:41:49.496068 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:41:49.495906    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/kube-state-metrics-59fbdd77d-b5qx4"]
Sep 11 03:41:49.498977 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:41:49.498958    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-59fbdd77d-b5qx4"
Sep 11 03:41:49.502974 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:41:49.502939    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-dockercfg-jnxk6\""
Sep 11 03:41:49.503167 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:41:49.503149    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-custom-resource-state-configmap\""
Sep 11 03:41:49.503253 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:41:49.502949    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-tls\""
Sep 11 03:41:49.503253 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:41:49.503042    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Sep 11 03:41:49.503253 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:41:49.503042    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Sep 11 03:41:49.503411 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:41:49.503155    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Sep 11 03:41:49.503411 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:41:49.503044    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-kube-rbac-proxy-config\""
Sep 11 03:41:49.512880 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:41:49.512857    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-59fbdd77d-b5qx4"]
Sep 11 03:41:49.513782 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:41:49.513756    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-ph5tp"]
Sep 11 03:41:49.516284 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:41:49.516263    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-ph5tp"
Sep 11 03:41:49.519676 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:41:49.519651    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Sep 11 03:41:49.519878 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:41:49.519676    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-cv25j\""
Sep 11 03:41:49.519984 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:41:49.519705    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Sep 11 03:41:49.519984 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:41:49.519734    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Sep 11 03:41:49.576138 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:41:49.576111    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/a3b10037-14c1-484f-9733-d0b6913c73a9-sys\") pod \"node-exporter-ph5tp\" (UID: \"a3b10037-14c1-484f-9733-d0b6913c73a9\") " pod="openshift-monitoring/node-exporter-ph5tp"
Sep 11 03:41:49.576138 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:41:49.576148    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/a3b10037-14c1-484f-9733-d0b6913c73a9-node-exporter-wtmp\") pod \"node-exporter-ph5tp\" (UID: \"a3b10037-14c1-484f-9733-d0b6913c73a9\") " pod="openshift-monitoring/node-exporter-ph5tp"
Sep 11 03:41:49.576377 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:41:49.576170    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/addfafec-32a6-4f08-95a8-80909d0e6d73-kube-state-metrics-tls\") pod \"kube-state-metrics-59fbdd77d-b5qx4\" (UID: \"addfafec-32a6-4f08-95a8-80909d0e6d73\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-b5qx4"
Sep 11 03:41:49.576377 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:41:49.576203    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/a3b10037-14c1-484f-9733-d0b6913c73a9-node-exporter-tls\") pod \"node-exporter-ph5tp\" (UID: \"a3b10037-14c1-484f-9733-d0b6913c73a9\") " pod="openshift-monitoring/node-exporter-ph5tp"
Sep 11 03:41:49.576377 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:41:49.576234    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/a3b10037-14c1-484f-9733-d0b6913c73a9-node-exporter-textfile\") pod \"node-exporter-ph5tp\" (UID: \"a3b10037-14c1-484f-9733-d0b6913c73a9\") " pod="openshift-monitoring/node-exporter-ph5tp"
Sep 11 03:41:49.576377 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:41:49.576260    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/a3b10037-14c1-484f-9733-d0b6913c73a9-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-ph5tp\" (UID: \"a3b10037-14c1-484f-9733-d0b6913c73a9\") " pod="openshift-monitoring/node-exporter-ph5tp"
Sep 11 03:41:49.576377 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:41:49.576313    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-htpxp\" (UniqueName: \"kubernetes.io/projected/a3b10037-14c1-484f-9733-d0b6913c73a9-kube-api-access-htpxp\") pod \"node-exporter-ph5tp\" (UID: \"a3b10037-14c1-484f-9733-d0b6913c73a9\") " pod="openshift-monitoring/node-exporter-ph5tp"
Sep 11 03:41:49.576377 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:41:49.576353    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/addfafec-32a6-4f08-95a8-80909d0e6d73-volume-directive-shadow\") pod \"kube-state-metrics-59fbdd77d-b5qx4\" (UID: \"addfafec-32a6-4f08-95a8-80909d0e6d73\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-b5qx4"
Sep 11 03:41:49.576655 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:41:49.576380    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/addfafec-32a6-4f08-95a8-80909d0e6d73-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-59fbdd77d-b5qx4\" (UID: \"addfafec-32a6-4f08-95a8-80909d0e6d73\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-b5qx4"
Sep 11 03:41:49.576655 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:41:49.576415    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/addfafec-32a6-4f08-95a8-80909d0e6d73-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-59fbdd77d-b5qx4\" (UID: \"addfafec-32a6-4f08-95a8-80909d0e6d73\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-b5qx4"
Sep 11 03:41:49.576655 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:41:49.576446    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hjflz\" (UniqueName: \"kubernetes.io/projected/addfafec-32a6-4f08-95a8-80909d0e6d73-kube-api-access-hjflz\") pod \"kube-state-metrics-59fbdd77d-b5qx4\" (UID: \"addfafec-32a6-4f08-95a8-80909d0e6d73\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-b5qx4"
Sep 11 03:41:49.576655 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:41:49.576473    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a3b10037-14c1-484f-9733-d0b6913c73a9-metrics-client-ca\") pod \"node-exporter-ph5tp\" (UID: \"a3b10037-14c1-484f-9733-d0b6913c73a9\") " pod="openshift-monitoring/node-exporter-ph5tp"
Sep 11 03:41:49.576655 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:41:49.576496    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/a3b10037-14c1-484f-9733-d0b6913c73a9-root\") pod \"node-exporter-ph5tp\" (UID: \"a3b10037-14c1-484f-9733-d0b6913c73a9\") " pod="openshift-monitoring/node-exporter-ph5tp"
Sep 11 03:41:49.576655 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:41:49.576575    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/a3b10037-14c1-484f-9733-d0b6913c73a9-node-exporter-accelerators-collector-config\") pod \"node-exporter-ph5tp\" (UID: \"a3b10037-14c1-484f-9733-d0b6913c73a9\") " pod="openshift-monitoring/node-exporter-ph5tp"
Sep 11 03:41:49.576655 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:41:49.576613    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/addfafec-32a6-4f08-95a8-80909d0e6d73-metrics-client-ca\") pod \"kube-state-metrics-59fbdd77d-b5qx4\" (UID: \"addfafec-32a6-4f08-95a8-80909d0e6d73\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-b5qx4"
Sep 11 03:41:49.677668 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:41:49.677633    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/a3b10037-14c1-484f-9733-d0b6913c73a9-node-exporter-tls\") pod \"node-exporter-ph5tp\" (UID: \"a3b10037-14c1-484f-9733-d0b6913c73a9\") " pod="openshift-monitoring/node-exporter-ph5tp"
Sep 11 03:41:49.677848 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:41:49.677677    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/a3b10037-14c1-484f-9733-d0b6913c73a9-node-exporter-textfile\") pod \"node-exporter-ph5tp\" (UID: \"a3b10037-14c1-484f-9733-d0b6913c73a9\") " pod="openshift-monitoring/node-exporter-ph5tp"
Sep 11 03:41:49.677848 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:41:49.677818    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/a3b10037-14c1-484f-9733-d0b6913c73a9-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-ph5tp\" (UID: \"a3b10037-14c1-484f-9733-d0b6913c73a9\") " pod="openshift-monitoring/node-exporter-ph5tp"
Sep 11 03:41:49.677953 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:41:49.677862    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-htpxp\" (UniqueName: \"kubernetes.io/projected/a3b10037-14c1-484f-9733-d0b6913c73a9-kube-api-access-htpxp\") pod \"node-exporter-ph5tp\" (UID: \"a3b10037-14c1-484f-9733-d0b6913c73a9\") " pod="openshift-monitoring/node-exporter-ph5tp"
Sep 11 03:41:49.677953 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:41:49.677889    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/addfafec-32a6-4f08-95a8-80909d0e6d73-volume-directive-shadow\") pod \"kube-state-metrics-59fbdd77d-b5qx4\" (UID: \"addfafec-32a6-4f08-95a8-80909d0e6d73\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-b5qx4"
Sep 11 03:41:49.677953 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:41:49.677942    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/addfafec-32a6-4f08-95a8-80909d0e6d73-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-59fbdd77d-b5qx4\" (UID: \"addfafec-32a6-4f08-95a8-80909d0e6d73\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-b5qx4"
Sep 11 03:41:49.678114 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:41:49.677980    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/addfafec-32a6-4f08-95a8-80909d0e6d73-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-59fbdd77d-b5qx4\" (UID: \"addfafec-32a6-4f08-95a8-80909d0e6d73\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-b5qx4"
Sep 11 03:41:49.678114 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:41:49.678013    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hjflz\" (UniqueName: \"kubernetes.io/projected/addfafec-32a6-4f08-95a8-80909d0e6d73-kube-api-access-hjflz\") pod \"kube-state-metrics-59fbdd77d-b5qx4\" (UID: \"addfafec-32a6-4f08-95a8-80909d0e6d73\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-b5qx4"
Sep 11 03:41:49.678114 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:41:49.678040    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/a3b10037-14c1-484f-9733-d0b6913c73a9-node-exporter-textfile\") pod \"node-exporter-ph5tp\" (UID: \"a3b10037-14c1-484f-9733-d0b6913c73a9\") " pod="openshift-monitoring/node-exporter-ph5tp"
Sep 11 03:41:49.678114 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:41:49.678042    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a3b10037-14c1-484f-9733-d0b6913c73a9-metrics-client-ca\") pod \"node-exporter-ph5tp\" (UID: \"a3b10037-14c1-484f-9733-d0b6913c73a9\") " pod="openshift-monitoring/node-exporter-ph5tp"
Sep 11 03:41:49.678308 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:41:49.678118    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/a3b10037-14c1-484f-9733-d0b6913c73a9-root\") pod \"node-exporter-ph5tp\" (UID: \"a3b10037-14c1-484f-9733-d0b6913c73a9\") " pod="openshift-monitoring/node-exporter-ph5tp"
Sep 11 03:41:49.678308 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:41:49.678157    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/a3b10037-14c1-484f-9733-d0b6913c73a9-node-exporter-accelerators-collector-config\") pod \"node-exporter-ph5tp\" (UID: \"a3b10037-14c1-484f-9733-d0b6913c73a9\") " pod="openshift-monitoring/node-exporter-ph5tp"
Sep 11 03:41:49.678308 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:41:49.678185    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/addfafec-32a6-4f08-95a8-80909d0e6d73-metrics-client-ca\") pod \"kube-state-metrics-59fbdd77d-b5qx4\" (UID: \"addfafec-32a6-4f08-95a8-80909d0e6d73\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-b5qx4"
Sep 11 03:41:49.678308 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:41:49.678241    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/a3b10037-14c1-484f-9733-d0b6913c73a9-sys\") pod \"node-exporter-ph5tp\" (UID: \"a3b10037-14c1-484f-9733-d0b6913c73a9\") " pod="openshift-monitoring/node-exporter-ph5tp"
Sep 11 03:41:49.678308 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:41:49.678265    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/a3b10037-14c1-484f-9733-d0b6913c73a9-node-exporter-wtmp\") pod \"node-exporter-ph5tp\" (UID: \"a3b10037-14c1-484f-9733-d0b6913c73a9\") " pod="openshift-monitoring/node-exporter-ph5tp"
Sep 11 03:41:49.678308 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:41:49.678288    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/addfafec-32a6-4f08-95a8-80909d0e6d73-kube-state-metrics-tls\") pod \"kube-state-metrics-59fbdd77d-b5qx4\" (UID: \"addfafec-32a6-4f08-95a8-80909d0e6d73\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-b5qx4"
Sep 11 03:41:49.678845 ip-10-0-141-105 kubenswrapper[2579]: E0911 03:41:49.678396    2579 secret.go:189] Couldn't get secret openshift-monitoring/kube-state-metrics-tls: secret "kube-state-metrics-tls" not found
Sep 11 03:41:49.678845 ip-10-0-141-105 kubenswrapper[2579]: E0911 03:41:49.678452    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/addfafec-32a6-4f08-95a8-80909d0e6d73-kube-state-metrics-tls podName:addfafec-32a6-4f08-95a8-80909d0e6d73 nodeName:}" failed. No retries permitted until 2026-09-11 03:41:50.178433028 +0000 UTC m=+87.833638579 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-state-metrics-tls" (UniqueName: "kubernetes.io/secret/addfafec-32a6-4f08-95a8-80909d0e6d73-kube-state-metrics-tls") pod "kube-state-metrics-59fbdd77d-b5qx4" (UID: "addfafec-32a6-4f08-95a8-80909d0e6d73") : secret "kube-state-metrics-tls" not found
Sep 11 03:41:49.678845 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:41:49.678685    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/a3b10037-14c1-484f-9733-d0b6913c73a9-sys\") pod \"node-exporter-ph5tp\" (UID: \"a3b10037-14c1-484f-9733-d0b6913c73a9\") " pod="openshift-monitoring/node-exporter-ph5tp"
Sep 11 03:41:49.678845 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:41:49.678801    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/a3b10037-14c1-484f-9733-d0b6913c73a9-node-exporter-wtmp\") pod \"node-exporter-ph5tp\" (UID: \"a3b10037-14c1-484f-9733-d0b6913c73a9\") " pod="openshift-monitoring/node-exporter-ph5tp"
Sep 11 03:41:49.678845 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:41:49.678801    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/addfafec-32a6-4f08-95a8-80909d0e6d73-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-59fbdd77d-b5qx4\" (UID: \"addfafec-32a6-4f08-95a8-80909d0e6d73\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-b5qx4"
Sep 11 03:41:49.678845 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:41:49.678842    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/a3b10037-14c1-484f-9733-d0b6913c73a9-root\") pod \"node-exporter-ph5tp\" (UID: \"a3b10037-14c1-484f-9733-d0b6913c73a9\") " pod="openshift-monitoring/node-exporter-ph5tp"
Sep 11 03:41:49.679254 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:41:49.679232    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/addfafec-32a6-4f08-95a8-80909d0e6d73-metrics-client-ca\") pod \"kube-state-metrics-59fbdd77d-b5qx4\" (UID: \"addfafec-32a6-4f08-95a8-80909d0e6d73\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-b5qx4"
Sep 11 03:41:49.679357 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:41:49.679338    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a3b10037-14c1-484f-9733-d0b6913c73a9-metrics-client-ca\") pod \"node-exporter-ph5tp\" (UID: \"a3b10037-14c1-484f-9733-d0b6913c73a9\") " pod="openshift-monitoring/node-exporter-ph5tp"
Sep 11 03:41:49.679409 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:41:49.679356    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/a3b10037-14c1-484f-9733-d0b6913c73a9-node-exporter-accelerators-collector-config\") pod \"node-exporter-ph5tp\" (UID: \"a3b10037-14c1-484f-9733-d0b6913c73a9\") " pod="openshift-monitoring/node-exporter-ph5tp"
Sep 11 03:41:49.679477 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:41:49.679458    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/addfafec-32a6-4f08-95a8-80909d0e6d73-volume-directive-shadow\") pod \"kube-state-metrics-59fbdd77d-b5qx4\" (UID: \"addfafec-32a6-4f08-95a8-80909d0e6d73\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-b5qx4"
Sep 11 03:41:49.681210 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:41:49.681184    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/a3b10037-14c1-484f-9733-d0b6913c73a9-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-ph5tp\" (UID: \"a3b10037-14c1-484f-9733-d0b6913c73a9\") " pod="openshift-monitoring/node-exporter-ph5tp"
Sep 11 03:41:49.681342 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:41:49.681322    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/addfafec-32a6-4f08-95a8-80909d0e6d73-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-59fbdd77d-b5qx4\" (UID: \"addfafec-32a6-4f08-95a8-80909d0e6d73\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-b5qx4"
Sep 11 03:41:49.681460 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:41:49.681442    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/a3b10037-14c1-484f-9733-d0b6913c73a9-node-exporter-tls\") pod \"node-exporter-ph5tp\" (UID: \"a3b10037-14c1-484f-9733-d0b6913c73a9\") " pod="openshift-monitoring/node-exporter-ph5tp"
Sep 11 03:41:49.686952 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:41:49.686928    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hjflz\" (UniqueName: \"kubernetes.io/projected/addfafec-32a6-4f08-95a8-80909d0e6d73-kube-api-access-hjflz\") pod \"kube-state-metrics-59fbdd77d-b5qx4\" (UID: \"addfafec-32a6-4f08-95a8-80909d0e6d73\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-b5qx4"
Sep 11 03:41:49.687971 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:41:49.687951    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-htpxp\" (UniqueName: \"kubernetes.io/projected/a3b10037-14c1-484f-9733-d0b6913c73a9-kube-api-access-htpxp\") pod \"node-exporter-ph5tp\" (UID: \"a3b10037-14c1-484f-9733-d0b6913c73a9\") " pod="openshift-monitoring/node-exporter-ph5tp"
Sep 11 03:41:49.828062 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:41:49.827981    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-ph5tp"
Sep 11 03:41:49.838913 ip-10-0-141-105 kubenswrapper[2579]: W0911 03:41:49.838860    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda3b10037_14c1_484f_9733_d0b6913c73a9.slice/crio-9c33ea911e5aa079e1801562f356b77ac6312cf0a02e900a9871310cb041f4a4 WatchSource:0}: Error finding container 9c33ea911e5aa079e1801562f356b77ac6312cf0a02e900a9871310cb041f4a4: Status 404 returned error can't find the container with id 9c33ea911e5aa079e1801562f356b77ac6312cf0a02e900a9871310cb041f4a4
Sep 11 03:41:50.182715 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:41:50.182671    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/addfafec-32a6-4f08-95a8-80909d0e6d73-kube-state-metrics-tls\") pod \"kube-state-metrics-59fbdd77d-b5qx4\" (UID: \"addfafec-32a6-4f08-95a8-80909d0e6d73\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-b5qx4"
Sep 11 03:41:50.185361 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:41:50.185332    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/addfafec-32a6-4f08-95a8-80909d0e6d73-kube-state-metrics-tls\") pod \"kube-state-metrics-59fbdd77d-b5qx4\" (UID: \"addfafec-32a6-4f08-95a8-80909d0e6d73\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-b5qx4"
Sep 11 03:41:50.337364 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:41:50.337318    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-ph5tp" event={"ID":"a3b10037-14c1-484f-9733-d0b6913c73a9","Type":"ContainerStarted","Data":"9c33ea911e5aa079e1801562f356b77ac6312cf0a02e900a9871310cb041f4a4"}
Sep 11 03:41:50.410947 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:41:50.410699    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-59fbdd77d-b5qx4"
Sep 11 03:41:50.510662 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:41:50.510630    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 11 03:41:50.514265 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:41:50.514246    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Sep 11 03:41:50.517235 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:41:50.516980    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\""
Sep 11 03:41:50.517235 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:41:50.517066    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\""
Sep 11 03:41:50.517858 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:41:50.517603    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\""
Sep 11 03:41:50.517858 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:41:50.517661    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-ffk29\""
Sep 11 03:41:50.517858 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:41:50.517672    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\""
Sep 11 03:41:50.517858 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:41:50.517684    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\""
Sep 11 03:41:50.517858 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:41:50.517198    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\""
Sep 11 03:41:50.517858 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:41:50.517815    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\""
Sep 11 03:41:50.517858 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:41:50.517858    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\""
Sep 11 03:41:50.523450 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:41:50.523427    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\""
Sep 11 03:41:50.529266 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:41:50.529190    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 11 03:41:50.586345 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:41:50.586310    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/ca69377b-efa8-460f-9ea7-c4c297378e53-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"ca69377b-efa8-460f-9ea7-c4c297378e53\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 03:41:50.586502 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:41:50.586360    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/ca69377b-efa8-460f-9ea7-c4c297378e53-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"ca69377b-efa8-460f-9ea7-c4c297378e53\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 03:41:50.586502 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:41:50.586383    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/ca69377b-efa8-460f-9ea7-c4c297378e53-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"ca69377b-efa8-460f-9ea7-c4c297378e53\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 03:41:50.586502 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:41:50.586412    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/ca69377b-efa8-460f-9ea7-c4c297378e53-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"ca69377b-efa8-460f-9ea7-c4c297378e53\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 03:41:50.586502 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:41:50.586437    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ca69377b-efa8-460f-9ea7-c4c297378e53-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"ca69377b-efa8-460f-9ea7-c4c297378e53\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 03:41:50.586502 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:41:50.586452    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pmlj4\" (UniqueName: \"kubernetes.io/projected/ca69377b-efa8-460f-9ea7-c4c297378e53-kube-api-access-pmlj4\") pod \"alertmanager-main-0\" (UID: \"ca69377b-efa8-460f-9ea7-c4c297378e53\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 03:41:50.586502 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:41:50.586469    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/ca69377b-efa8-460f-9ea7-c4c297378e53-config-volume\") pod \"alertmanager-main-0\" (UID: \"ca69377b-efa8-460f-9ea7-c4c297378e53\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 03:41:50.586502 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:41:50.586487    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/ca69377b-efa8-460f-9ea7-c4c297378e53-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"ca69377b-efa8-460f-9ea7-c4c297378e53\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 03:41:50.586849 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:41:50.586506    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/ca69377b-efa8-460f-9ea7-c4c297378e53-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"ca69377b-efa8-460f-9ea7-c4c297378e53\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 03:41:50.586849 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:41:50.586528    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ca69377b-efa8-460f-9ea7-c4c297378e53-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"ca69377b-efa8-460f-9ea7-c4c297378e53\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 03:41:50.586849 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:41:50.586560    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/ca69377b-efa8-460f-9ea7-c4c297378e53-tls-assets\") pod \"alertmanager-main-0\" (UID: \"ca69377b-efa8-460f-9ea7-c4c297378e53\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 03:41:50.586849 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:41:50.586606    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/ca69377b-efa8-460f-9ea7-c4c297378e53-web-config\") pod \"alertmanager-main-0\" (UID: \"ca69377b-efa8-460f-9ea7-c4c297378e53\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 03:41:50.586849 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:41:50.586631    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/ca69377b-efa8-460f-9ea7-c4c297378e53-config-out\") pod \"alertmanager-main-0\" (UID: \"ca69377b-efa8-460f-9ea7-c4c297378e53\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 03:41:50.687454 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:41:50.687417    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/ca69377b-efa8-460f-9ea7-c4c297378e53-tls-assets\") pod \"alertmanager-main-0\" (UID: \"ca69377b-efa8-460f-9ea7-c4c297378e53\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 03:41:50.687591 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:41:50.687480    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/ca69377b-efa8-460f-9ea7-c4c297378e53-web-config\") pod \"alertmanager-main-0\" (UID: \"ca69377b-efa8-460f-9ea7-c4c297378e53\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 03:41:50.687591 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:41:50.687509    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/ca69377b-efa8-460f-9ea7-c4c297378e53-config-out\") pod \"alertmanager-main-0\" (UID: \"ca69377b-efa8-460f-9ea7-c4c297378e53\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 03:41:50.687591 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:41:50.687551    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/ca69377b-efa8-460f-9ea7-c4c297378e53-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"ca69377b-efa8-460f-9ea7-c4c297378e53\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 03:41:50.687591 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:41:50.687586    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/ca69377b-efa8-460f-9ea7-c4c297378e53-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"ca69377b-efa8-460f-9ea7-c4c297378e53\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 03:41:50.687830 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:41:50.687615    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/ca69377b-efa8-460f-9ea7-c4c297378e53-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"ca69377b-efa8-460f-9ea7-c4c297378e53\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 03:41:50.687830 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:41:50.687644    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/ca69377b-efa8-460f-9ea7-c4c297378e53-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"ca69377b-efa8-460f-9ea7-c4c297378e53\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 03:41:50.687830 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:41:50.687668    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ca69377b-efa8-460f-9ea7-c4c297378e53-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"ca69377b-efa8-460f-9ea7-c4c297378e53\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 03:41:50.687830 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:41:50.687693    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pmlj4\" (UniqueName: \"kubernetes.io/projected/ca69377b-efa8-460f-9ea7-c4c297378e53-kube-api-access-pmlj4\") pod \"alertmanager-main-0\" (UID: \"ca69377b-efa8-460f-9ea7-c4c297378e53\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 03:41:50.687830 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:41:50.687721    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/ca69377b-efa8-460f-9ea7-c4c297378e53-config-volume\") pod \"alertmanager-main-0\" (UID: \"ca69377b-efa8-460f-9ea7-c4c297378e53\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 03:41:50.687830 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:41:50.687748    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/ca69377b-efa8-460f-9ea7-c4c297378e53-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"ca69377b-efa8-460f-9ea7-c4c297378e53\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 03:41:50.687830 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:41:50.687776    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/ca69377b-efa8-460f-9ea7-c4c297378e53-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"ca69377b-efa8-460f-9ea7-c4c297378e53\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 03:41:50.687830 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:41:50.687797    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ca69377b-efa8-460f-9ea7-c4c297378e53-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"ca69377b-efa8-460f-9ea7-c4c297378e53\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 03:41:50.688275 ip-10-0-141-105 kubenswrapper[2579]: E0911 03:41:50.688253    2579 secret.go:189] Couldn't get secret openshift-monitoring/alertmanager-main-tls: secret "alertmanager-main-tls" not found
Sep 11 03:41:50.688456 ip-10-0-141-105 kubenswrapper[2579]: E0911 03:41:50.688327    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ca69377b-efa8-460f-9ea7-c4c297378e53-secret-alertmanager-main-tls podName:ca69377b-efa8-460f-9ea7-c4c297378e53 nodeName:}" failed. No retries permitted until 2026-09-11 03:41:51.188307277 +0000 UTC m=+88.843512824 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "secret-alertmanager-main-tls" (UniqueName: "kubernetes.io/secret/ca69377b-efa8-460f-9ea7-c4c297378e53-secret-alertmanager-main-tls") pod "alertmanager-main-0" (UID: "ca69377b-efa8-460f-9ea7-c4c297378e53") : secret "alertmanager-main-tls" not found
Sep 11 03:41:50.688619 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:41:50.688562    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ca69377b-efa8-460f-9ea7-c4c297378e53-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"ca69377b-efa8-460f-9ea7-c4c297378e53\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 03:41:50.688738 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:41:50.688719    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/ca69377b-efa8-460f-9ea7-c4c297378e53-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"ca69377b-efa8-460f-9ea7-c4c297378e53\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 03:41:50.689646 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:41:50.689613    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ca69377b-efa8-460f-9ea7-c4c297378e53-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"ca69377b-efa8-460f-9ea7-c4c297378e53\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 03:41:50.692118 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:41:50.692024    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/ca69377b-efa8-460f-9ea7-c4c297378e53-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"ca69377b-efa8-460f-9ea7-c4c297378e53\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 03:41:50.692118 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:41:50.692063    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/ca69377b-efa8-460f-9ea7-c4c297378e53-web-config\") pod \"alertmanager-main-0\" (UID: \"ca69377b-efa8-460f-9ea7-c4c297378e53\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 03:41:50.692534 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:41:50.692493    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/ca69377b-efa8-460f-9ea7-c4c297378e53-config-volume\") pod \"alertmanager-main-0\" (UID: \"ca69377b-efa8-460f-9ea7-c4c297378e53\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 03:41:50.692693 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:41:50.692676    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/ca69377b-efa8-460f-9ea7-c4c297378e53-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"ca69377b-efa8-460f-9ea7-c4c297378e53\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 03:41:50.693076 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:41:50.693054    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/ca69377b-efa8-460f-9ea7-c4c297378e53-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"ca69377b-efa8-460f-9ea7-c4c297378e53\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 03:41:50.693526 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:41:50.693504    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/ca69377b-efa8-460f-9ea7-c4c297378e53-config-out\") pod \"alertmanager-main-0\" (UID: \"ca69377b-efa8-460f-9ea7-c4c297378e53\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 03:41:50.694138 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:41:50.694119    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/ca69377b-efa8-460f-9ea7-c4c297378e53-tls-assets\") pod \"alertmanager-main-0\" (UID: \"ca69377b-efa8-460f-9ea7-c4c297378e53\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 03:41:50.696100 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:41:50.696023    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/ca69377b-efa8-460f-9ea7-c4c297378e53-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"ca69377b-efa8-460f-9ea7-c4c297378e53\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 03:41:50.696936 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:41:50.696863    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-59fbdd77d-b5qx4"]
Sep 11 03:41:50.701189 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:41:50.701160    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pmlj4\" (UniqueName: \"kubernetes.io/projected/ca69377b-efa8-460f-9ea7-c4c297378e53-kube-api-access-pmlj4\") pod \"alertmanager-main-0\" (UID: \"ca69377b-efa8-460f-9ea7-c4c297378e53\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 03:41:50.815432 ip-10-0-141-105 kubenswrapper[2579]: W0911 03:41:50.815364    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaddfafec_32a6_4f08_95a8_80909d0e6d73.slice/crio-cb7c75dd01153f6759e197c441dd57b7f13e0921d242911efcfebc7269c93758 WatchSource:0}: Error finding container cb7c75dd01153f6759e197c441dd57b7f13e0921d242911efcfebc7269c93758: Status 404 returned error can't find the container with id cb7c75dd01153f6759e197c441dd57b7f13e0921d242911efcfebc7269c93758
Sep 11 03:41:51.192217 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:41:51.192165    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/ca69377b-efa8-460f-9ea7-c4c297378e53-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"ca69377b-efa8-460f-9ea7-c4c297378e53\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 03:41:51.194718 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:41:51.194689    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/ca69377b-efa8-460f-9ea7-c4c297378e53-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"ca69377b-efa8-460f-9ea7-c4c297378e53\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 03:41:51.342781 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:41:51.342738    2579 generic.go:356] "Generic (PLEG): container finished" podID="a3b10037-14c1-484f-9733-d0b6913c73a9" containerID="5d6f6cdffddd19ff97342640e3c76bf198896d89db92db9aa375ceeda30ce89c" exitCode=0
Sep 11 03:41:51.342949 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:41:51.342819    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-ph5tp" event={"ID":"a3b10037-14c1-484f-9733-d0b6913c73a9","Type":"ContainerDied","Data":"5d6f6cdffddd19ff97342640e3c76bf198896d89db92db9aa375ceeda30ce89c"}
Sep 11 03:41:51.344199 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:41:51.344179    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-59fbdd77d-b5qx4" event={"ID":"addfafec-32a6-4f08-95a8-80909d0e6d73","Type":"ContainerStarted","Data":"cb7c75dd01153f6759e197c441dd57b7f13e0921d242911efcfebc7269c93758"}
Sep 11 03:41:51.424804 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:41:51.424771    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Sep 11 03:41:51.714614 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:41:51.714549    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 11 03:41:51.718283 ip-10-0-141-105 kubenswrapper[2579]: W0911 03:41:51.718252    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podca69377b_efa8_460f_9ea7_c4c297378e53.slice/crio-53f3c17f3e002ca74a63eb456f1a4516b7f590f2fbd7606c2fdec4f1d110f137 WatchSource:0}: Error finding container 53f3c17f3e002ca74a63eb456f1a4516b7f590f2fbd7606c2fdec4f1d110f137: Status 404 returned error can't find the container with id 53f3c17f3e002ca74a63eb456f1a4516b7f590f2fbd7606c2fdec4f1d110f137
Sep 11 03:41:52.353818 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:41:52.353268    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"ca69377b-efa8-460f-9ea7-c4c297378e53","Type":"ContainerStarted","Data":"53f3c17f3e002ca74a63eb456f1a4516b7f590f2fbd7606c2fdec4f1d110f137"}
Sep 11 03:41:52.356179 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:41:52.356156    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-ph5tp" event={"ID":"a3b10037-14c1-484f-9733-d0b6913c73a9","Type":"ContainerStarted","Data":"8f1e6eab287f02b84a27e01eb9592c8e58476c8959c57de88783404333bf044f"}
Sep 11 03:41:52.356290 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:41:52.356205    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-ph5tp" event={"ID":"a3b10037-14c1-484f-9733-d0b6913c73a9","Type":"ContainerStarted","Data":"5e1f9707642ca183b7cfe336be04592273f6f4957abacf62f7acdf658491ad6a"}
Sep 11 03:41:52.358717 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:41:52.358677    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-59fbdd77d-b5qx4" event={"ID":"addfafec-32a6-4f08-95a8-80909d0e6d73","Type":"ContainerStarted","Data":"aec3ccbf4c6e7697a36b895da435efd57722fdaadc2419e73d5ebf6bd58de74e"}
Sep 11 03:41:52.358829 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:41:52.358732    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-59fbdd77d-b5qx4" event={"ID":"addfafec-32a6-4f08-95a8-80909d0e6d73","Type":"ContainerStarted","Data":"50f79fb502c0c94f426baa8c951f9d6406e04503461d53526aa459b07c6a4bbc"}
Sep 11 03:41:52.378426 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:41:52.378387    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-ph5tp" podStartSLOduration=2.307874174 podStartE2EDuration="3.378372779s" totalImagesPullingTime="999.72474ms" totalInitContainerRuntime="70.773865ms" isStatefulPod=true podCreationTimestamp="2026-09-11 03:41:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 03:41:52.377249632 +0000 UTC m=+90.032455202" watchObservedRunningTime="2026-09-11 03:41:52.378372779 +0000 UTC m=+90.033578388"
Sep 11 03:41:52.434519 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:41:52.434420    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/thanos-querier-9f6946f8f-5lcb2"]
Sep 11 03:41:52.437227 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:41:52.437205    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-9f6946f8f-5lcb2"
Sep 11 03:41:52.441176 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:41:52.440526    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-web\""
Sep 11 03:41:52.441176 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:41:52.440770    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy\""
Sep 11 03:41:52.441822 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:41:52.440987    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-grpc-tls-4ti0a9m4h99sq\""
Sep 11 03:41:52.442098 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:41:52.441118    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-tls\""
Sep 11 03:41:52.442679 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:41:52.441113    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-metrics\""
Sep 11 03:41:52.442679 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:41:52.441161    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-dockercfg-mzpgj\""
Sep 11 03:41:52.442679 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:41:52.441218    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-rules\""
Sep 11 03:41:52.460303 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:41:52.460275    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-9f6946f8f-5lcb2"]
Sep 11 03:41:52.502925 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:41:52.502884    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/0fdb316e-3b02-4e85-856d-56526c0c1e48-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-9f6946f8f-5lcb2\" (UID: \"0fdb316e-3b02-4e85-856d-56526c0c1e48\") " pod="openshift-monitoring/thanos-querier-9f6946f8f-5lcb2"
Sep 11 03:41:52.503125 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:41:52.502965    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/0fdb316e-3b02-4e85-856d-56526c0c1e48-secret-thanos-querier-tls\") pod \"thanos-querier-9f6946f8f-5lcb2\" (UID: \"0fdb316e-3b02-4e85-856d-56526c0c1e48\") " pod="openshift-monitoring/thanos-querier-9f6946f8f-5lcb2"
Sep 11 03:41:52.503125 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:41:52.503013    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/0fdb316e-3b02-4e85-856d-56526c0c1e48-metrics-client-ca\") pod \"thanos-querier-9f6946f8f-5lcb2\" (UID: \"0fdb316e-3b02-4e85-856d-56526c0c1e48\") " pod="openshift-monitoring/thanos-querier-9f6946f8f-5lcb2"
Sep 11 03:41:52.503125 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:41:52.503101    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/0fdb316e-3b02-4e85-856d-56526c0c1e48-secret-grpc-tls\") pod \"thanos-querier-9f6946f8f-5lcb2\" (UID: \"0fdb316e-3b02-4e85-856d-56526c0c1e48\") " pod="openshift-monitoring/thanos-querier-9f6946f8f-5lcb2"
Sep 11 03:41:52.503296 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:41:52.503142    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/0fdb316e-3b02-4e85-856d-56526c0c1e48-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-9f6946f8f-5lcb2\" (UID: \"0fdb316e-3b02-4e85-856d-56526c0c1e48\") " pod="openshift-monitoring/thanos-querier-9f6946f8f-5lcb2"
Sep 11 03:41:52.503296 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:41:52.503172    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dn9rr\" (UniqueName: \"kubernetes.io/projected/0fdb316e-3b02-4e85-856d-56526c0c1e48-kube-api-access-dn9rr\") pod \"thanos-querier-9f6946f8f-5lcb2\" (UID: \"0fdb316e-3b02-4e85-856d-56526c0c1e48\") " pod="openshift-monitoring/thanos-querier-9f6946f8f-5lcb2"
Sep 11 03:41:52.503296 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:41:52.503199    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/0fdb316e-3b02-4e85-856d-56526c0c1e48-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-9f6946f8f-5lcb2\" (UID: \"0fdb316e-3b02-4e85-856d-56526c0c1e48\") " pod="openshift-monitoring/thanos-querier-9f6946f8f-5lcb2"
Sep 11 03:41:52.503296 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:41:52.503220    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/0fdb316e-3b02-4e85-856d-56526c0c1e48-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-9f6946f8f-5lcb2\" (UID: \"0fdb316e-3b02-4e85-856d-56526c0c1e48\") " pod="openshift-monitoring/thanos-querier-9f6946f8f-5lcb2"
Sep 11 03:41:52.605284 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:41:52.604782    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/0fdb316e-3b02-4e85-856d-56526c0c1e48-secret-grpc-tls\") pod \"thanos-querier-9f6946f8f-5lcb2\" (UID: \"0fdb316e-3b02-4e85-856d-56526c0c1e48\") " pod="openshift-monitoring/thanos-querier-9f6946f8f-5lcb2"
Sep 11 03:41:52.605284 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:41:52.604860    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/0fdb316e-3b02-4e85-856d-56526c0c1e48-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-9f6946f8f-5lcb2\" (UID: \"0fdb316e-3b02-4e85-856d-56526c0c1e48\") " pod="openshift-monitoring/thanos-querier-9f6946f8f-5lcb2"
Sep 11 03:41:52.605284 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:41:52.604913    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dn9rr\" (UniqueName: \"kubernetes.io/projected/0fdb316e-3b02-4e85-856d-56526c0c1e48-kube-api-access-dn9rr\") pod \"thanos-querier-9f6946f8f-5lcb2\" (UID: \"0fdb316e-3b02-4e85-856d-56526c0c1e48\") " pod="openshift-monitoring/thanos-querier-9f6946f8f-5lcb2"
Sep 11 03:41:52.605284 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:41:52.604942    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/0fdb316e-3b02-4e85-856d-56526c0c1e48-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-9f6946f8f-5lcb2\" (UID: \"0fdb316e-3b02-4e85-856d-56526c0c1e48\") " pod="openshift-monitoring/thanos-querier-9f6946f8f-5lcb2"
Sep 11 03:41:52.605284 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:41:52.604985    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/0fdb316e-3b02-4e85-856d-56526c0c1e48-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-9f6946f8f-5lcb2\" (UID: \"0fdb316e-3b02-4e85-856d-56526c0c1e48\") " pod="openshift-monitoring/thanos-querier-9f6946f8f-5lcb2"
Sep 11 03:41:52.605284 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:41:52.605031    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/0fdb316e-3b02-4e85-856d-56526c0c1e48-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-9f6946f8f-5lcb2\" (UID: \"0fdb316e-3b02-4e85-856d-56526c0c1e48\") " pod="openshift-monitoring/thanos-querier-9f6946f8f-5lcb2"
Sep 11 03:41:52.605284 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:41:52.605077    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/0fdb316e-3b02-4e85-856d-56526c0c1e48-secret-thanos-querier-tls\") pod \"thanos-querier-9f6946f8f-5lcb2\" (UID: \"0fdb316e-3b02-4e85-856d-56526c0c1e48\") " pod="openshift-monitoring/thanos-querier-9f6946f8f-5lcb2"
Sep 11 03:41:52.605284 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:41:52.605149    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/0fdb316e-3b02-4e85-856d-56526c0c1e48-metrics-client-ca\") pod \"thanos-querier-9f6946f8f-5lcb2\" (UID: \"0fdb316e-3b02-4e85-856d-56526c0c1e48\") " pod="openshift-monitoring/thanos-querier-9f6946f8f-5lcb2"
Sep 11 03:41:52.607007 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:41:52.606968    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/0fdb316e-3b02-4e85-856d-56526c0c1e48-metrics-client-ca\") pod \"thanos-querier-9f6946f8f-5lcb2\" (UID: \"0fdb316e-3b02-4e85-856d-56526c0c1e48\") " pod="openshift-monitoring/thanos-querier-9f6946f8f-5lcb2"
Sep 11 03:41:52.609545 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:41:52.609515    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/0fdb316e-3b02-4e85-856d-56526c0c1e48-secret-grpc-tls\") pod \"thanos-querier-9f6946f8f-5lcb2\" (UID: \"0fdb316e-3b02-4e85-856d-56526c0c1e48\") " pod="openshift-monitoring/thanos-querier-9f6946f8f-5lcb2"
Sep 11 03:41:52.609645 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:41:52.609562    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/0fdb316e-3b02-4e85-856d-56526c0c1e48-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-9f6946f8f-5lcb2\" (UID: \"0fdb316e-3b02-4e85-856d-56526c0c1e48\") " pod="openshift-monitoring/thanos-querier-9f6946f8f-5lcb2"
Sep 11 03:41:52.609962 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:41:52.609933    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/0fdb316e-3b02-4e85-856d-56526c0c1e48-secret-thanos-querier-tls\") pod \"thanos-querier-9f6946f8f-5lcb2\" (UID: \"0fdb316e-3b02-4e85-856d-56526c0c1e48\") " pod="openshift-monitoring/thanos-querier-9f6946f8f-5lcb2"
Sep 11 03:41:52.610645 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:41:52.610621    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/0fdb316e-3b02-4e85-856d-56526c0c1e48-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-9f6946f8f-5lcb2\" (UID: \"0fdb316e-3b02-4e85-856d-56526c0c1e48\") " pod="openshift-monitoring/thanos-querier-9f6946f8f-5lcb2"
Sep 11 03:41:52.610737 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:41:52.610622    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/0fdb316e-3b02-4e85-856d-56526c0c1e48-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-9f6946f8f-5lcb2\" (UID: \"0fdb316e-3b02-4e85-856d-56526c0c1e48\") " pod="openshift-monitoring/thanos-querier-9f6946f8f-5lcb2"
Sep 11 03:41:52.611150 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:41:52.611126    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/0fdb316e-3b02-4e85-856d-56526c0c1e48-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-9f6946f8f-5lcb2\" (UID: \"0fdb316e-3b02-4e85-856d-56526c0c1e48\") " pod="openshift-monitoring/thanos-querier-9f6946f8f-5lcb2"
Sep 11 03:41:52.614629 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:41:52.614606    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dn9rr\" (UniqueName: \"kubernetes.io/projected/0fdb316e-3b02-4e85-856d-56526c0c1e48-kube-api-access-dn9rr\") pod \"thanos-querier-9f6946f8f-5lcb2\" (UID: \"0fdb316e-3b02-4e85-856d-56526c0c1e48\") " pod="openshift-monitoring/thanos-querier-9f6946f8f-5lcb2"
Sep 11 03:41:52.750042 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:41:52.749961    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-9f6946f8f-5lcb2"
Sep 11 03:41:53.209407 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:41:53.209379    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-9f6946f8f-5lcb2"]
Sep 11 03:41:53.211796 ip-10-0-141-105 kubenswrapper[2579]: W0911 03:41:53.211775    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0fdb316e_3b02_4e85_856d_56526c0c1e48.slice/crio-ed3d5f39bbd047a2015baa8967ac31484af8f3939d0d02ea1e85a48fdf05464f WatchSource:0}: Error finding container ed3d5f39bbd047a2015baa8967ac31484af8f3939d0d02ea1e85a48fdf05464f: Status 404 returned error can't find the container with id ed3d5f39bbd047a2015baa8967ac31484af8f3939d0d02ea1e85a48fdf05464f
Sep 11 03:41:53.363126 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:41:53.363066    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-9f6946f8f-5lcb2" event={"ID":"0fdb316e-3b02-4e85-856d-56526c0c1e48","Type":"ContainerStarted","Data":"ed3d5f39bbd047a2015baa8967ac31484af8f3939d0d02ea1e85a48fdf05464f"}
Sep 11 03:41:53.364712 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:41:53.364680    2579 generic.go:356] "Generic (PLEG): container finished" podID="ca69377b-efa8-460f-9ea7-c4c297378e53" containerID="d4dc50780261eeec829c7d183c5267c294f8cddca6f5adabb65051d3d6c9eb1b" exitCode=0
Sep 11 03:41:53.364834 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:41:53.364770    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"ca69377b-efa8-460f-9ea7-c4c297378e53","Type":"ContainerDied","Data":"d4dc50780261eeec829c7d183c5267c294f8cddca6f5adabb65051d3d6c9eb1b"}
Sep 11 03:41:53.367182 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:41:53.367158    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-59fbdd77d-b5qx4" event={"ID":"addfafec-32a6-4f08-95a8-80909d0e6d73","Type":"ContainerStarted","Data":"f379fd761d154456aa537b4cd1b93dc60e8c7fb89086b2cd01a4d9e4b097e749"}
Sep 11 03:41:53.415973 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:41:53.415921    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/kube-state-metrics-59fbdd77d-b5qx4" podStartSLOduration=3.062063343 podStartE2EDuration="4.41590027s" totalImagesPullingTime="1.353836927s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 03:41:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 03:41:53.415758724 +0000 UTC m=+91.070964330" watchObservedRunningTime="2026-09-11 03:41:53.41590027 +0000 UTC m=+91.071105839"
Sep 11 03:41:53.935271 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:41:53.935233    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/metrics-server-7ff6c96894-qdqpx"]
Sep 11 03:41:53.937673 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:41:53.937644    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-7ff6c96894-qdqpx"
Sep 11 03:41:53.940466 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:41:53.940441    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-client-certs\""
Sep 11 03:41:53.941648 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:41:53.941620    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-server-audit-profiles\""
Sep 11 03:41:53.941771 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:41:53.941683    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-tls\""
Sep 11 03:41:53.941836 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:41:53.941777    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-dockercfg-j869x\""
Sep 11 03:41:53.941893 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:41:53.941835    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\""
Sep 11 03:41:53.941948 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:41:53.941897    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-ab5d9cj6a1av\""
Sep 11 03:41:53.950685 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:41:53.950661    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-7ff6c96894-qdqpx"]
Sep 11 03:41:54.018683 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:41:54.018652    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/4721c4ee-ba92-4cec-98e1-00040dc00315-audit-log\") pod \"metrics-server-7ff6c96894-qdqpx\" (UID: \"4721c4ee-ba92-4cec-98e1-00040dc00315\") " pod="openshift-monitoring/metrics-server-7ff6c96894-qdqpx"
Sep 11 03:41:54.018850 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:41:54.018696    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/4721c4ee-ba92-4cec-98e1-00040dc00315-client-ca-bundle\") pod \"metrics-server-7ff6c96894-qdqpx\" (UID: \"4721c4ee-ba92-4cec-98e1-00040dc00315\") " pod="openshift-monitoring/metrics-server-7ff6c96894-qdqpx"
Sep 11 03:41:54.018850 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:41:54.018726    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5fc9g\" (UniqueName: \"kubernetes.io/projected/4721c4ee-ba92-4cec-98e1-00040dc00315-kube-api-access-5fc9g\") pod \"metrics-server-7ff6c96894-qdqpx\" (UID: \"4721c4ee-ba92-4cec-98e1-00040dc00315\") " pod="openshift-monitoring/metrics-server-7ff6c96894-qdqpx"
Sep 11 03:41:54.018959 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:41:54.018841    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/4721c4ee-ba92-4cec-98e1-00040dc00315-secret-metrics-server-tls\") pod \"metrics-server-7ff6c96894-qdqpx\" (UID: \"4721c4ee-ba92-4cec-98e1-00040dc00315\") " pod="openshift-monitoring/metrics-server-7ff6c96894-qdqpx"
Sep 11 03:41:54.018959 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:41:54.018887    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/4721c4ee-ba92-4cec-98e1-00040dc00315-metrics-server-audit-profiles\") pod \"metrics-server-7ff6c96894-qdqpx\" (UID: \"4721c4ee-ba92-4cec-98e1-00040dc00315\") " pod="openshift-monitoring/metrics-server-7ff6c96894-qdqpx"
Sep 11 03:41:54.019043 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:41:54.018972    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4721c4ee-ba92-4cec-98e1-00040dc00315-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-7ff6c96894-qdqpx\" (UID: \"4721c4ee-ba92-4cec-98e1-00040dc00315\") " pod="openshift-monitoring/metrics-server-7ff6c96894-qdqpx"
Sep 11 03:41:54.019043 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:41:54.019004    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/4721c4ee-ba92-4cec-98e1-00040dc00315-secret-metrics-server-client-certs\") pod \"metrics-server-7ff6c96894-qdqpx\" (UID: \"4721c4ee-ba92-4cec-98e1-00040dc00315\") " pod="openshift-monitoring/metrics-server-7ff6c96894-qdqpx"
Sep 11 03:41:54.120786 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:41:54.120061    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/4721c4ee-ba92-4cec-98e1-00040dc00315-audit-log\") pod \"metrics-server-7ff6c96894-qdqpx\" (UID: \"4721c4ee-ba92-4cec-98e1-00040dc00315\") " pod="openshift-monitoring/metrics-server-7ff6c96894-qdqpx"
Sep 11 03:41:54.120786 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:41:54.120145    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/4721c4ee-ba92-4cec-98e1-00040dc00315-client-ca-bundle\") pod \"metrics-server-7ff6c96894-qdqpx\" (UID: \"4721c4ee-ba92-4cec-98e1-00040dc00315\") " pod="openshift-monitoring/metrics-server-7ff6c96894-qdqpx"
Sep 11 03:41:54.120786 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:41:54.120177    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5fc9g\" (UniqueName: \"kubernetes.io/projected/4721c4ee-ba92-4cec-98e1-00040dc00315-kube-api-access-5fc9g\") pod \"metrics-server-7ff6c96894-qdqpx\" (UID: \"4721c4ee-ba92-4cec-98e1-00040dc00315\") " pod="openshift-monitoring/metrics-server-7ff6c96894-qdqpx"
Sep 11 03:41:54.120786 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:41:54.120242    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/4721c4ee-ba92-4cec-98e1-00040dc00315-secret-metrics-server-tls\") pod \"metrics-server-7ff6c96894-qdqpx\" (UID: \"4721c4ee-ba92-4cec-98e1-00040dc00315\") " pod="openshift-monitoring/metrics-server-7ff6c96894-qdqpx"
Sep 11 03:41:54.120786 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:41:54.120291    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/4721c4ee-ba92-4cec-98e1-00040dc00315-metrics-server-audit-profiles\") pod \"metrics-server-7ff6c96894-qdqpx\" (UID: \"4721c4ee-ba92-4cec-98e1-00040dc00315\") " pod="openshift-monitoring/metrics-server-7ff6c96894-qdqpx"
Sep 11 03:41:54.120786 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:41:54.120340    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4721c4ee-ba92-4cec-98e1-00040dc00315-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-7ff6c96894-qdqpx\" (UID: \"4721c4ee-ba92-4cec-98e1-00040dc00315\") " pod="openshift-monitoring/metrics-server-7ff6c96894-qdqpx"
Sep 11 03:41:54.120786 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:41:54.120381    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/4721c4ee-ba92-4cec-98e1-00040dc00315-secret-metrics-server-client-certs\") pod \"metrics-server-7ff6c96894-qdqpx\" (UID: \"4721c4ee-ba92-4cec-98e1-00040dc00315\") " pod="openshift-monitoring/metrics-server-7ff6c96894-qdqpx"
Sep 11 03:41:54.122226 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:41:54.122184    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4721c4ee-ba92-4cec-98e1-00040dc00315-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-7ff6c96894-qdqpx\" (UID: \"4721c4ee-ba92-4cec-98e1-00040dc00315\") " pod="openshift-monitoring/metrics-server-7ff6c96894-qdqpx"
Sep 11 03:41:54.122411 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:41:54.122388    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/4721c4ee-ba92-4cec-98e1-00040dc00315-audit-log\") pod \"metrics-server-7ff6c96894-qdqpx\" (UID: \"4721c4ee-ba92-4cec-98e1-00040dc00315\") " pod="openshift-monitoring/metrics-server-7ff6c96894-qdqpx"
Sep 11 03:41:54.123101 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:41:54.123041    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/4721c4ee-ba92-4cec-98e1-00040dc00315-metrics-server-audit-profiles\") pod \"metrics-server-7ff6c96894-qdqpx\" (UID: \"4721c4ee-ba92-4cec-98e1-00040dc00315\") " pod="openshift-monitoring/metrics-server-7ff6c96894-qdqpx"
Sep 11 03:41:54.123293 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:41:54.123232    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/4721c4ee-ba92-4cec-98e1-00040dc00315-secret-metrics-server-client-certs\") pod \"metrics-server-7ff6c96894-qdqpx\" (UID: \"4721c4ee-ba92-4cec-98e1-00040dc00315\") " pod="openshift-monitoring/metrics-server-7ff6c96894-qdqpx"
Sep 11 03:41:54.124456 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:41:54.124425    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/4721c4ee-ba92-4cec-98e1-00040dc00315-client-ca-bundle\") pod \"metrics-server-7ff6c96894-qdqpx\" (UID: \"4721c4ee-ba92-4cec-98e1-00040dc00315\") " pod="openshift-monitoring/metrics-server-7ff6c96894-qdqpx"
Sep 11 03:41:54.124936 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:41:54.124896    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/4721c4ee-ba92-4cec-98e1-00040dc00315-secret-metrics-server-tls\") pod \"metrics-server-7ff6c96894-qdqpx\" (UID: \"4721c4ee-ba92-4cec-98e1-00040dc00315\") " pod="openshift-monitoring/metrics-server-7ff6c96894-qdqpx"
Sep 11 03:41:54.129881 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:41:54.129841    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5fc9g\" (UniqueName: \"kubernetes.io/projected/4721c4ee-ba92-4cec-98e1-00040dc00315-kube-api-access-5fc9g\") pod \"metrics-server-7ff6c96894-qdqpx\" (UID: \"4721c4ee-ba92-4cec-98e1-00040dc00315\") " pod="openshift-monitoring/metrics-server-7ff6c96894-qdqpx"
Sep 11 03:41:54.251009 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:41:54.250922    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-7ff6c96894-qdqpx"
Sep 11 03:41:54.545937 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:41:54.545856    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-7ff6c96894-qdqpx"]
Sep 11 03:41:54.690770 ip-10-0-141-105 kubenswrapper[2579]: W0911 03:41:54.690725    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4721c4ee_ba92_4cec_98e1_00040dc00315.slice/crio-409542c4fceb6d7d7ed1462324ea828b8fb272328383212e1d5994849af1236a WatchSource:0}: Error finding container 409542c4fceb6d7d7ed1462324ea828b8fb272328383212e1d5994849af1236a: Status 404 returned error can't find the container with id 409542c4fceb6d7d7ed1462324ea828b8fb272328383212e1d5994849af1236a
Sep 11 03:41:55.380454 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:41:55.380420    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-7ff6c96894-qdqpx" event={"ID":"4721c4ee-ba92-4cec-98e1-00040dc00315","Type":"ContainerStarted","Data":"409542c4fceb6d7d7ed1462324ea828b8fb272328383212e1d5994849af1236a"}
Sep 11 03:41:55.384372 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:41:55.384276    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-9f6946f8f-5lcb2" event={"ID":"0fdb316e-3b02-4e85-856d-56526c0c1e48","Type":"ContainerStarted","Data":"547c2a5f0c563f437689ad6dac887cb4b0ccdf7937a2d0e25fbf929023b995a0"}
Sep 11 03:41:55.387780 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:41:55.387702    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"ca69377b-efa8-460f-9ea7-c4c297378e53","Type":"ContainerStarted","Data":"2c570a50f3e5a8388ad35c05bf9cd57b15800cfeb0d0c3f598216e869b9231d5"}
Sep 11 03:41:55.599218 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:41:55.598810    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 11 03:41:55.602906 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:41:55.602865    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 03:41:55.605763 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:41:55.605738    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\""
Sep 11 03:41:55.606238 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:41:55.606075    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\""
Sep 11 03:41:55.606238 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:41:55.606232    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\""
Sep 11 03:41:55.606717 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:41:55.606467    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\""
Sep 11 03:41:55.606717 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:41:55.606492    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\""
Sep 11 03:41:55.606717 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:41:55.606544    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-6qqj7\""
Sep 11 03:41:55.606717 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:41:55.606613    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\""
Sep 11 03:41:55.607915 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:41:55.607507    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\""
Sep 11 03:41:55.607915 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:41:55.607628    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\""
Sep 11 03:41:55.607915 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:41:55.607652    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\""
Sep 11 03:41:55.607915 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:41:55.607788    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\""
Sep 11 03:41:55.608076 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:41:55.607941    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\""
Sep 11 03:41:55.608076 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:41:55.607966    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\""
Sep 11 03:41:55.608076 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:41:55.608033    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-fesspo8lkis9q\""
Sep 11 03:41:55.608215 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:41:55.608104    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\""
Sep 11 03:41:55.610230 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:41:55.610210    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\""
Sep 11 03:41:55.619404 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:41:55.619319    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 11 03:41:55.737208 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:41:55.737176    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/5208b5bb-0268-4ab8-8e3a-f873bda02436-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"5208b5bb-0268-4ab8-8e3a-f873bda02436\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 03:41:55.737388 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:41:55.737216    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/5208b5bb-0268-4ab8-8e3a-f873bda02436-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"5208b5bb-0268-4ab8-8e3a-f873bda02436\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 03:41:55.737388 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:41:55.737234    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/5208b5bb-0268-4ab8-8e3a-f873bda02436-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"5208b5bb-0268-4ab8-8e3a-f873bda02436\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 03:41:55.737388 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:41:55.737344    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/5208b5bb-0268-4ab8-8e3a-f873bda02436-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"5208b5bb-0268-4ab8-8e3a-f873bda02436\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 03:41:55.737388 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:41:55.737379    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/5208b5bb-0268-4ab8-8e3a-f873bda02436-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"5208b5bb-0268-4ab8-8e3a-f873bda02436\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 03:41:55.737606 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:41:55.737406    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5208b5bb-0268-4ab8-8e3a-f873bda02436-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"5208b5bb-0268-4ab8-8e3a-f873bda02436\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 03:41:55.737606 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:41:55.737433    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rwqlq\" (UniqueName: \"kubernetes.io/projected/5208b5bb-0268-4ab8-8e3a-f873bda02436-kube-api-access-rwqlq\") pod \"prometheus-k8s-0\" (UID: \"5208b5bb-0268-4ab8-8e3a-f873bda02436\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 03:41:55.737606 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:41:55.737485    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/5208b5bb-0268-4ab8-8e3a-f873bda02436-config\") pod \"prometheus-k8s-0\" (UID: \"5208b5bb-0268-4ab8-8e3a-f873bda02436\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 03:41:55.737606 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:41:55.737547    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/5208b5bb-0268-4ab8-8e3a-f873bda02436-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"5208b5bb-0268-4ab8-8e3a-f873bda02436\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 03:41:55.737606 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:41:55.737564    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/5208b5bb-0268-4ab8-8e3a-f873bda02436-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"5208b5bb-0268-4ab8-8e3a-f873bda02436\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 03:41:55.737606 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:41:55.737597    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5208b5bb-0268-4ab8-8e3a-f873bda02436-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"5208b5bb-0268-4ab8-8e3a-f873bda02436\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 03:41:55.737891 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:41:55.737621    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/5208b5bb-0268-4ab8-8e3a-f873bda02436-config-out\") pod \"prometheus-k8s-0\" (UID: \"5208b5bb-0268-4ab8-8e3a-f873bda02436\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 03:41:55.737891 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:41:55.737660    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/5208b5bb-0268-4ab8-8e3a-f873bda02436-web-config\") pod \"prometheus-k8s-0\" (UID: \"5208b5bb-0268-4ab8-8e3a-f873bda02436\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 03:41:55.737891 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:41:55.737696    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/5208b5bb-0268-4ab8-8e3a-f873bda02436-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"5208b5bb-0268-4ab8-8e3a-f873bda02436\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 03:41:55.737891 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:41:55.737727    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/5208b5bb-0268-4ab8-8e3a-f873bda02436-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"5208b5bb-0268-4ab8-8e3a-f873bda02436\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 03:41:55.737891 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:41:55.737748    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/5208b5bb-0268-4ab8-8e3a-f873bda02436-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"5208b5bb-0268-4ab8-8e3a-f873bda02436\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 03:41:55.737891 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:41:55.737795    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/5208b5bb-0268-4ab8-8e3a-f873bda02436-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"5208b5bb-0268-4ab8-8e3a-f873bda02436\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 03:41:55.737891 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:41:55.737847    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/5208b5bb-0268-4ab8-8e3a-f873bda02436-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"5208b5bb-0268-4ab8-8e3a-f873bda02436\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 03:41:55.737891 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:41:55.737876    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/5208b5bb-0268-4ab8-8e3a-f873bda02436-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"5208b5bb-0268-4ab8-8e3a-f873bda02436\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 03:41:55.738286 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:41:55.737926    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5208b5bb-0268-4ab8-8e3a-f873bda02436-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"5208b5bb-0268-4ab8-8e3a-f873bda02436\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 03:41:55.838783 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:41:55.838690    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/5208b5bb-0268-4ab8-8e3a-f873bda02436-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"5208b5bb-0268-4ab8-8e3a-f873bda02436\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 03:41:55.838783 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:41:55.838737    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/5208b5bb-0268-4ab8-8e3a-f873bda02436-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"5208b5bb-0268-4ab8-8e3a-f873bda02436\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 03:41:55.838988 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:41:55.838872    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/5208b5bb-0268-4ab8-8e3a-f873bda02436-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"5208b5bb-0268-4ab8-8e3a-f873bda02436\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 03:41:55.838988 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:41:55.838939    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/5208b5bb-0268-4ab8-8e3a-f873bda02436-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"5208b5bb-0268-4ab8-8e3a-f873bda02436\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 03:41:55.838988 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:41:55.838968    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/5208b5bb-0268-4ab8-8e3a-f873bda02436-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"5208b5bb-0268-4ab8-8e3a-f873bda02436\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 03:41:55.839115 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:41:55.838991    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5208b5bb-0268-4ab8-8e3a-f873bda02436-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"5208b5bb-0268-4ab8-8e3a-f873bda02436\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 03:41:55.839115 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:41:55.839015    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rwqlq\" (UniqueName: \"kubernetes.io/projected/5208b5bb-0268-4ab8-8e3a-f873bda02436-kube-api-access-rwqlq\") pod \"prometheus-k8s-0\" (UID: \"5208b5bb-0268-4ab8-8e3a-f873bda02436\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 03:41:55.839115 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:41:55.839050    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/5208b5bb-0268-4ab8-8e3a-f873bda02436-config\") pod \"prometheus-k8s-0\" (UID: \"5208b5bb-0268-4ab8-8e3a-f873bda02436\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 03:41:55.839251 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:41:55.839116    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/5208b5bb-0268-4ab8-8e3a-f873bda02436-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"5208b5bb-0268-4ab8-8e3a-f873bda02436\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 03:41:55.839251 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:41:55.839141    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/5208b5bb-0268-4ab8-8e3a-f873bda02436-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"5208b5bb-0268-4ab8-8e3a-f873bda02436\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 03:41:55.839251 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:41:55.839181    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5208b5bb-0268-4ab8-8e3a-f873bda02436-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"5208b5bb-0268-4ab8-8e3a-f873bda02436\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 03:41:55.839251 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:41:55.839212    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/5208b5bb-0268-4ab8-8e3a-f873bda02436-config-out\") pod \"prometheus-k8s-0\" (UID: \"5208b5bb-0268-4ab8-8e3a-f873bda02436\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 03:41:55.839251 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:41:55.839234    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/5208b5bb-0268-4ab8-8e3a-f873bda02436-web-config\") pod \"prometheus-k8s-0\" (UID: \"5208b5bb-0268-4ab8-8e3a-f873bda02436\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 03:41:55.839483 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:41:55.839259    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/5208b5bb-0268-4ab8-8e3a-f873bda02436-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"5208b5bb-0268-4ab8-8e3a-f873bda02436\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 03:41:55.839483 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:41:55.839295    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/5208b5bb-0268-4ab8-8e3a-f873bda02436-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"5208b5bb-0268-4ab8-8e3a-f873bda02436\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 03:41:55.839483 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:41:55.839319    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/5208b5bb-0268-4ab8-8e3a-f873bda02436-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"5208b5bb-0268-4ab8-8e3a-f873bda02436\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 03:41:55.839483 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:41:55.839358    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/5208b5bb-0268-4ab8-8e3a-f873bda02436-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"5208b5bb-0268-4ab8-8e3a-f873bda02436\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 03:41:55.839483 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:41:55.839385    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/5208b5bb-0268-4ab8-8e3a-f873bda02436-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"5208b5bb-0268-4ab8-8e3a-f873bda02436\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 03:41:55.839483 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:41:55.839425    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/5208b5bb-0268-4ab8-8e3a-f873bda02436-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"5208b5bb-0268-4ab8-8e3a-f873bda02436\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 03:41:55.839483 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:41:55.839462    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5208b5bb-0268-4ab8-8e3a-f873bda02436-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"5208b5bb-0268-4ab8-8e3a-f873bda02436\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 03:41:55.839898 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:41:55.839871    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/5208b5bb-0268-4ab8-8e3a-f873bda02436-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"5208b5bb-0268-4ab8-8e3a-f873bda02436\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 03:41:55.840100 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:41:55.839607    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/5208b5bb-0268-4ab8-8e3a-f873bda02436-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"5208b5bb-0268-4ab8-8e3a-f873bda02436\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 03:41:55.840230 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:41:55.840200    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5208b5bb-0268-4ab8-8e3a-f873bda02436-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"5208b5bb-0268-4ab8-8e3a-f873bda02436\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 03:41:55.840935 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:41:55.840575    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5208b5bb-0268-4ab8-8e3a-f873bda02436-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"5208b5bb-0268-4ab8-8e3a-f873bda02436\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 03:41:55.841061 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:41:55.840949    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/5208b5bb-0268-4ab8-8e3a-f873bda02436-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"5208b5bb-0268-4ab8-8e3a-f873bda02436\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 03:41:55.841736 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:41:55.841346    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5208b5bb-0268-4ab8-8e3a-f873bda02436-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"5208b5bb-0268-4ab8-8e3a-f873bda02436\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 03:41:55.842709 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:41:55.842166    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/5208b5bb-0268-4ab8-8e3a-f873bda02436-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"5208b5bb-0268-4ab8-8e3a-f873bda02436\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 03:41:55.842709 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:41:55.842663    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/5208b5bb-0268-4ab8-8e3a-f873bda02436-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"5208b5bb-0268-4ab8-8e3a-f873bda02436\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 03:41:55.843033 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:41:55.842711    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/5208b5bb-0268-4ab8-8e3a-f873bda02436-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"5208b5bb-0268-4ab8-8e3a-f873bda02436\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 03:41:55.843328 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:41:55.843305    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/5208b5bb-0268-4ab8-8e3a-f873bda02436-config-out\") pod \"prometheus-k8s-0\" (UID: \"5208b5bb-0268-4ab8-8e3a-f873bda02436\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 03:41:55.844047 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:41:55.843964    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/5208b5bb-0268-4ab8-8e3a-f873bda02436-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"5208b5bb-0268-4ab8-8e3a-f873bda02436\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 03:41:55.845109 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:41:55.845044    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/5208b5bb-0268-4ab8-8e3a-f873bda02436-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"5208b5bb-0268-4ab8-8e3a-f873bda02436\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 03:41:55.845205 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:41:55.845051    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/5208b5bb-0268-4ab8-8e3a-f873bda02436-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"5208b5bb-0268-4ab8-8e3a-f873bda02436\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 03:41:55.845645 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:41:55.845599    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/5208b5bb-0268-4ab8-8e3a-f873bda02436-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"5208b5bb-0268-4ab8-8e3a-f873bda02436\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 03:41:55.845959 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:41:55.845916    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/5208b5bb-0268-4ab8-8e3a-f873bda02436-web-config\") pod \"prometheus-k8s-0\" (UID: \"5208b5bb-0268-4ab8-8e3a-f873bda02436\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 03:41:55.846310 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:41:55.846286    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/5208b5bb-0268-4ab8-8e3a-f873bda02436-config\") pod \"prometheus-k8s-0\" (UID: \"5208b5bb-0268-4ab8-8e3a-f873bda02436\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 03:41:55.846432 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:41:55.846386    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/5208b5bb-0268-4ab8-8e3a-f873bda02436-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"5208b5bb-0268-4ab8-8e3a-f873bda02436\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 03:41:55.847523 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:41:55.847491    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/5208b5bb-0268-4ab8-8e3a-f873bda02436-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"5208b5bb-0268-4ab8-8e3a-f873bda02436\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 03:41:55.848028 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:41:55.848010    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rwqlq\" (UniqueName: \"kubernetes.io/projected/5208b5bb-0268-4ab8-8e3a-f873bda02436-kube-api-access-rwqlq\") pod \"prometheus-k8s-0\" (UID: \"5208b5bb-0268-4ab8-8e3a-f873bda02436\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 03:41:55.848493 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:41:55.848475    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/5208b5bb-0268-4ab8-8e3a-f873bda02436-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"5208b5bb-0268-4ab8-8e3a-f873bda02436\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 03:41:55.918873 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:41:55.918840    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 03:41:56.396734 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:41:56.396696    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-9f6946f8f-5lcb2" event={"ID":"0fdb316e-3b02-4e85-856d-56526c0c1e48","Type":"ContainerStarted","Data":"87fd24491e5ad3ab003a151a019d516c00c3e9d0bdb3909ab86a8676f0144f3c"}
Sep 11 03:41:56.397189 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:41:56.396763    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-9f6946f8f-5lcb2" event={"ID":"0fdb316e-3b02-4e85-856d-56526c0c1e48","Type":"ContainerStarted","Data":"1a5991aa14dcce175939cf1a433adaf511f078ef1630abbaf0f121e9a7a28c6e"}
Sep 11 03:41:56.400100 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:41:56.400057    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"ca69377b-efa8-460f-9ea7-c4c297378e53","Type":"ContainerStarted","Data":"4548395d3c7d772ab4e836acc047517daedca0e165601f7fec1cfab11a9f61bc"}
Sep 11 03:41:56.400234 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:41:56.400149    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"ca69377b-efa8-460f-9ea7-c4c297378e53","Type":"ContainerStarted","Data":"eec2c8da9cd64a88af8b621febcb24a4ae1942cb3e317319e227791c0ab58eb8"}
Sep 11 03:41:56.400234 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:41:56.400191    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"ca69377b-efa8-460f-9ea7-c4c297378e53","Type":"ContainerStarted","Data":"e5510813bc296efd6645a11cb5b66b6d0afa52eb0700b1db2f27070eb05bd782"}
Sep 11 03:41:56.400234 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:41:56.400216    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"ca69377b-efa8-460f-9ea7-c4c297378e53","Type":"ContainerStarted","Data":"2c3779307a5a6e18aebf549c9c29ff64630db41af03856a308431167535dac80"}
Sep 11 03:41:56.401535 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:41:56.401515    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-7ff6c96894-qdqpx" event={"ID":"4721c4ee-ba92-4cec-98e1-00040dc00315","Type":"ContainerStarted","Data":"7fbc1e7312eb3f7ad7b6283e79a7a05f0179502e73e7653dc3e8ca1ceb6d12ce"}
Sep 11 03:41:56.421312 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:41:56.421249    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/metrics-server-7ff6c96894-qdqpx" podStartSLOduration=1.852226328 podStartE2EDuration="3.42122729s" totalImagesPullingTime="1.569000962s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 03:41:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 03:41:56.419877779 +0000 UTC m=+94.075083347" watchObservedRunningTime="2026-09-11 03:41:56.42122729 +0000 UTC m=+94.076432859"
Sep 11 03:41:56.518673 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:41:56.518643    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 11 03:41:56.521777 ip-10-0-141-105 kubenswrapper[2579]: W0911 03:41:56.521745    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5208b5bb_0268_4ab8_8e3a_f873bda02436.slice/crio-7639469849f9e1d544c0b4f34965bae23114a4810a8aeaf093692ffdce290a50 WatchSource:0}: Error finding container 7639469849f9e1d544c0b4f34965bae23114a4810a8aeaf093692ffdce290a50: Status 404 returned error can't find the container with id 7639469849f9e1d544c0b4f34965bae23114a4810a8aeaf093692ffdce290a50
Sep 11 03:41:57.407067 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:41:57.407026    2579 generic.go:356] "Generic (PLEG): container finished" podID="5208b5bb-0268-4ab8-8e3a-f873bda02436" containerID="9ee65c097f8e13ebe68364b4ae0058a6543ca1526d248f62d925296340f5364a" exitCode=0
Sep 11 03:41:57.407642 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:41:57.407120    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"5208b5bb-0268-4ab8-8e3a-f873bda02436","Type":"ContainerDied","Data":"9ee65c097f8e13ebe68364b4ae0058a6543ca1526d248f62d925296340f5364a"}
Sep 11 03:41:57.407642 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:41:57.407194    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"5208b5bb-0268-4ab8-8e3a-f873bda02436","Type":"ContainerStarted","Data":"7639469849f9e1d544c0b4f34965bae23114a4810a8aeaf093692ffdce290a50"}
Sep 11 03:41:57.410778 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:41:57.410748    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"ca69377b-efa8-460f-9ea7-c4c297378e53","Type":"ContainerStarted","Data":"2bf2492ee4a72ec109fc63a156df78b5dfd9c38f4bf02dda04f780ba03163811"}
Sep 11 03:41:57.414045 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:41:57.414021    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-9f6946f8f-5lcb2" event={"ID":"0fdb316e-3b02-4e85-856d-56526c0c1e48","Type":"ContainerStarted","Data":"c95e1f76e8c32ad834e6fc428458c32cc2c16db6c3065c9b2c62a6abc7befa89"}
Sep 11 03:41:57.414152 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:41:57.414095    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-9f6946f8f-5lcb2" event={"ID":"0fdb316e-3b02-4e85-856d-56526c0c1e48","Type":"ContainerStarted","Data":"e59d09cf14ee76ba643c087b58a7496804b0840628dbfd1aaf628a30b5b6e42b"}
Sep 11 03:41:57.414152 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:41:57.414137    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-9f6946f8f-5lcb2" event={"ID":"0fdb316e-3b02-4e85-856d-56526c0c1e48","Type":"ContainerStarted","Data":"5342b28a82535d5957dd16b681d6e17ed30d1492bf2051e15c0cacb533429fe8"}
Sep 11 03:41:57.455843 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:41:57.453978    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/thanos-querier-9f6946f8f-5lcb2" podStartSLOduration=2.15394833 podStartE2EDuration="5.453961088s" totalImagesPullingTime="3.300012758s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 03:41:52 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 03:41:57.4538725 +0000 UTC m=+95.109078068" watchObservedRunningTime="2026-09-11 03:41:57.453961088 +0000 UTC m=+95.109166655"
Sep 11 03:41:57.485049 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:41:57.484985    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=3.2017579290000002 podStartE2EDuration="7.484967735s" totalImagesPullingTime="4.267298064s" totalInitContainerRuntime="15.911742ms" isStatefulPod=false podCreationTimestamp="2026-09-11 03:41:50 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 03:41:57.482602917 +0000 UTC m=+95.137808478" watchObservedRunningTime="2026-09-11 03:41:57.484967735 +0000 UTC m=+95.140173301"
Sep 11 03:41:58.418309 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:41:58.418273    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/thanos-querier-9f6946f8f-5lcb2"
Sep 11 03:42:01.435421 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:42:01.435381    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"5208b5bb-0268-4ab8-8e3a-f873bda02436","Type":"ContainerStarted","Data":"cc15a3445e053dd5656d9fc120f01c37afc3fa0cb6cdc7158a87f9389387200b"}
Sep 11 03:42:01.435786 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:42:01.435458    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"5208b5bb-0268-4ab8-8e3a-f873bda02436","Type":"ContainerStarted","Data":"21fe8a05bab9b83615a8293c808cba67ff07b045fa1ff8abb956d36b8a4548a6"}
Sep 11 03:42:01.435786 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:42:01.435487    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"5208b5bb-0268-4ab8-8e3a-f873bda02436","Type":"ContainerStarted","Data":"4562ba7ff19703d39b6f22517c8811def29f266ad7974c249138ed86e03c7715"}
Sep 11 03:42:01.435786 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:42:01.435518    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"5208b5bb-0268-4ab8-8e3a-f873bda02436","Type":"ContainerStarted","Data":"8f9b2f45cf254555dd48a3598d894fc58b2f165af492fd1ea1df73670f4093e9"}
Sep 11 03:42:01.435786 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:42:01.435547    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"5208b5bb-0268-4ab8-8e3a-f873bda02436","Type":"ContainerStarted","Data":"c34f844b4ee0b9f4eb00ca54e6fbfe8f911ae7f338602dee41d5cfc502683450"}
Sep 11 03:42:01.435786 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:42:01.435572    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"5208b5bb-0268-4ab8-8e3a-f873bda02436","Type":"ContainerStarted","Data":"a5fb41f056f6486570ae6451b1ccc687e68091d1258fa434fbe505339d967aa9"}
Sep 11 03:42:01.464339 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:42:01.464292    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=3.4519118 podStartE2EDuration="6.464276174s" totalImagesPullingTime="2.991690387s" totalInitContainerRuntime="20.673987ms" isStatefulPod=false podCreationTimestamp="2026-09-11 03:41:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 03:42:01.462647344 +0000 UTC m=+99.117852952" watchObservedRunningTime="2026-09-11 03:42:01.464276174 +0000 UTC m=+99.119481741"
Sep 11 03:42:04.428682 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:42:04.428656    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/thanos-querier-9f6946f8f-5lcb2"
Sep 11 03:42:05.919860 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:42:05.919818    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 03:42:06.280272 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:42:06.280185    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-p54z5"
Sep 11 03:42:14.251693 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:42:14.251659    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/metrics-server-7ff6c96894-qdqpx"
Sep 11 03:42:14.252168 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:42:14.251713    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/metrics-server-7ff6c96894-qdqpx"
Sep 11 03:42:24.530575 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:42:24.530536    2579 generic.go:356] "Generic (PLEG): container finished" podID="9dbdbcd2-cfbf-447b-8064-35a1852b1a9d" containerID="b6d5215edd4b3b5889732d43c15828f38bb479cef9d10d18841a268cf27e9ed1" exitCode=0
Sep 11 03:42:24.531058 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:42:24.530603    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-6hjz5" event={"ID":"9dbdbcd2-cfbf-447b-8064-35a1852b1a9d","Type":"ContainerDied","Data":"b6d5215edd4b3b5889732d43c15828f38bb479cef9d10d18841a268cf27e9ed1"}
Sep 11 03:42:24.531058 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:42:24.531037    2579 scope.go:117] "RemoveContainer" containerID="b6d5215edd4b3b5889732d43c15828f38bb479cef9d10d18841a268cf27e9ed1"
Sep 11 03:42:25.537478 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:42:25.537443    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-6hjz5" event={"ID":"9dbdbcd2-cfbf-447b-8064-35a1852b1a9d","Type":"ContainerStarted","Data":"e8b131f1776491e7d4a9d81e98223f7a72a93ac31d143b98f88328eacf8de149"}
Sep 11 03:42:34.257185 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:42:34.257158    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/metrics-server-7ff6c96894-qdqpx"
Sep 11 03:42:34.261240 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:42:34.261214    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/metrics-server-7ff6c96894-qdqpx"
Sep 11 03:42:55.919885 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:42:55.919853    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 03:42:55.936379 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:42:55.936355    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 03:42:56.693392 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:42:56.693368    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 03:43:09.691203 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:43:09.691168    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 11 03:43:09.691639 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:43:09.691415    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="ca69377b-efa8-460f-9ea7-c4c297378e53" containerName="alertmanager" containerID="cri-o://2c570a50f3e5a8388ad35c05bf9cd57b15800cfeb0d0c3f598216e869b9231d5" gracePeriod=120
Sep 11 03:43:09.691639 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:43:09.691487    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="ca69377b-efa8-460f-9ea7-c4c297378e53" containerName="kube-rbac-proxy-metric" containerID="cri-o://4548395d3c7d772ab4e836acc047517daedca0e165601f7fec1cfab11a9f61bc" gracePeriod=120
Sep 11 03:43:09.691639 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:43:09.691525    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="ca69377b-efa8-460f-9ea7-c4c297378e53" containerName="kube-rbac-proxy-web" containerID="cri-o://e5510813bc296efd6645a11cb5b66b6d0afa52eb0700b1db2f27070eb05bd782" gracePeriod=120
Sep 11 03:43:09.691639 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:43:09.691548    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="ca69377b-efa8-460f-9ea7-c4c297378e53" containerName="config-reloader" containerID="cri-o://2c3779307a5a6e18aebf549c9c29ff64630db41af03856a308431167535dac80" gracePeriod=120
Sep 11 03:43:09.691639 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:43:09.691587    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="ca69377b-efa8-460f-9ea7-c4c297378e53" containerName="kube-rbac-proxy" containerID="cri-o://eec2c8da9cd64a88af8b621febcb24a4ae1942cb3e317319e227791c0ab58eb8" gracePeriod=120
Sep 11 03:43:09.691639 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:43:09.691608    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="ca69377b-efa8-460f-9ea7-c4c297378e53" containerName="prom-label-proxy" containerID="cri-o://2bf2492ee4a72ec109fc63a156df78b5dfd9c38f4bf02dda04f780ba03163811" gracePeriod=120
Sep 11 03:43:10.740498 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:43:10.740471    2579 generic.go:356] "Generic (PLEG): container finished" podID="ca69377b-efa8-460f-9ea7-c4c297378e53" containerID="2bf2492ee4a72ec109fc63a156df78b5dfd9c38f4bf02dda04f780ba03163811" exitCode=0
Sep 11 03:43:10.740498 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:43:10.740495    2579 generic.go:356] "Generic (PLEG): container finished" podID="ca69377b-efa8-460f-9ea7-c4c297378e53" containerID="eec2c8da9cd64a88af8b621febcb24a4ae1942cb3e317319e227791c0ab58eb8" exitCode=0
Sep 11 03:43:10.740498 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:43:10.740503    2579 generic.go:356] "Generic (PLEG): container finished" podID="ca69377b-efa8-460f-9ea7-c4c297378e53" containerID="2c3779307a5a6e18aebf549c9c29ff64630db41af03856a308431167535dac80" exitCode=0
Sep 11 03:43:10.740848 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:43:10.740509    2579 generic.go:356] "Generic (PLEG): container finished" podID="ca69377b-efa8-460f-9ea7-c4c297378e53" containerID="2c570a50f3e5a8388ad35c05bf9cd57b15800cfeb0d0c3f598216e869b9231d5" exitCode=0
Sep 11 03:43:10.740848 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:43:10.740537    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"ca69377b-efa8-460f-9ea7-c4c297378e53","Type":"ContainerDied","Data":"2bf2492ee4a72ec109fc63a156df78b5dfd9c38f4bf02dda04f780ba03163811"}
Sep 11 03:43:10.740848 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:43:10.740600    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"ca69377b-efa8-460f-9ea7-c4c297378e53","Type":"ContainerDied","Data":"eec2c8da9cd64a88af8b621febcb24a4ae1942cb3e317319e227791c0ab58eb8"}
Sep 11 03:43:10.740848 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:43:10.740627    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"ca69377b-efa8-460f-9ea7-c4c297378e53","Type":"ContainerDied","Data":"2c3779307a5a6e18aebf549c9c29ff64630db41af03856a308431167535dac80"}
Sep 11 03:43:10.740848 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:43:10.740654    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"ca69377b-efa8-460f-9ea7-c4c297378e53","Type":"ContainerDied","Data":"2c570a50f3e5a8388ad35c05bf9cd57b15800cfeb0d0c3f598216e869b9231d5"}
Sep 11 03:43:10.946070 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:43:10.946045    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Sep 11 03:43:11.084545 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:43:11.084459    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/ca69377b-efa8-460f-9ea7-c4c297378e53-cluster-tls-config\") pod \"ca69377b-efa8-460f-9ea7-c4c297378e53\" (UID: \"ca69377b-efa8-460f-9ea7-c4c297378e53\") "
Sep 11 03:43:11.084684 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:43:11.084553    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/ca69377b-efa8-460f-9ea7-c4c297378e53-secret-alertmanager-main-tls\") pod \"ca69377b-efa8-460f-9ea7-c4c297378e53\" (UID: \"ca69377b-efa8-460f-9ea7-c4c297378e53\") "
Sep 11 03:43:11.084684 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:43:11.084582    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/ca69377b-efa8-460f-9ea7-c4c297378e53-config-volume\") pod \"ca69377b-efa8-460f-9ea7-c4c297378e53\" (UID: \"ca69377b-efa8-460f-9ea7-c4c297378e53\") "
Sep 11 03:43:11.084684 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:43:11.084635    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pmlj4\" (UniqueName: \"kubernetes.io/projected/ca69377b-efa8-460f-9ea7-c4c297378e53-kube-api-access-pmlj4\") pod \"ca69377b-efa8-460f-9ea7-c4c297378e53\" (UID: \"ca69377b-efa8-460f-9ea7-c4c297378e53\") "
Sep 11 03:43:11.084684 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:43:11.084658    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/ca69377b-efa8-460f-9ea7-c4c297378e53-tls-assets\") pod \"ca69377b-efa8-460f-9ea7-c4c297378e53\" (UID: \"ca69377b-efa8-460f-9ea7-c4c297378e53\") "
Sep 11 03:43:11.084882 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:43:11.084686    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/ca69377b-efa8-460f-9ea7-c4c297378e53-web-config\") pod \"ca69377b-efa8-460f-9ea7-c4c297378e53\" (UID: \"ca69377b-efa8-460f-9ea7-c4c297378e53\") "
Sep 11 03:43:11.084882 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:43:11.084717    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ca69377b-efa8-460f-9ea7-c4c297378e53-metrics-client-ca\") pod \"ca69377b-efa8-460f-9ea7-c4c297378e53\" (UID: \"ca69377b-efa8-460f-9ea7-c4c297378e53\") "
Sep 11 03:43:11.084882 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:43:11.084750    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/ca69377b-efa8-460f-9ea7-c4c297378e53-secret-alertmanager-kube-rbac-proxy\") pod \"ca69377b-efa8-460f-9ea7-c4c297378e53\" (UID: \"ca69377b-efa8-460f-9ea7-c4c297378e53\") "
Sep 11 03:43:11.084882 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:43:11.084781    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/ca69377b-efa8-460f-9ea7-c4c297378e53-config-out\") pod \"ca69377b-efa8-460f-9ea7-c4c297378e53\" (UID: \"ca69377b-efa8-460f-9ea7-c4c297378e53\") "
Sep 11 03:43:11.084882 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:43:11.084869    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/ca69377b-efa8-460f-9ea7-c4c297378e53-alertmanager-main-db\") pod \"ca69377b-efa8-460f-9ea7-c4c297378e53\" (UID: \"ca69377b-efa8-460f-9ea7-c4c297378e53\") "
Sep 11 03:43:11.085170 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:43:11.084897    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/ca69377b-efa8-460f-9ea7-c4c297378e53-secret-alertmanager-kube-rbac-proxy-web\") pod \"ca69377b-efa8-460f-9ea7-c4c297378e53\" (UID: \"ca69377b-efa8-460f-9ea7-c4c297378e53\") "
Sep 11 03:43:11.085170 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:43:11.084940    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ca69377b-efa8-460f-9ea7-c4c297378e53-alertmanager-trusted-ca-bundle\") pod \"ca69377b-efa8-460f-9ea7-c4c297378e53\" (UID: \"ca69377b-efa8-460f-9ea7-c4c297378e53\") "
Sep 11 03:43:11.085170 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:43:11.084974    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/ca69377b-efa8-460f-9ea7-c4c297378e53-secret-alertmanager-kube-rbac-proxy-metric\") pod \"ca69377b-efa8-460f-9ea7-c4c297378e53\" (UID: \"ca69377b-efa8-460f-9ea7-c4c297378e53\") "
Sep 11 03:43:11.086426 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:43:11.085391    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ca69377b-efa8-460f-9ea7-c4c297378e53-alertmanager-main-db" (OuterVolumeSpecName: "alertmanager-main-db") pod "ca69377b-efa8-460f-9ea7-c4c297378e53" (UID: "ca69377b-efa8-460f-9ea7-c4c297378e53"). InnerVolumeSpecName "alertmanager-main-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 03:43:11.086426 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:43:11.085942    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ca69377b-efa8-460f-9ea7-c4c297378e53-metrics-client-ca" (OuterVolumeSpecName: "metrics-client-ca") pod "ca69377b-efa8-460f-9ea7-c4c297378e53" (UID: "ca69377b-efa8-460f-9ea7-c4c297378e53"). InnerVolumeSpecName "metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 03:43:11.086426 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:43:11.086377    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ca69377b-efa8-460f-9ea7-c4c297378e53-alertmanager-trusted-ca-bundle" (OuterVolumeSpecName: "alertmanager-trusted-ca-bundle") pod "ca69377b-efa8-460f-9ea7-c4c297378e53" (UID: "ca69377b-efa8-460f-9ea7-c4c297378e53"). InnerVolumeSpecName "alertmanager-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 03:43:11.087675 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:43:11.087612    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ca69377b-efa8-460f-9ea7-c4c297378e53-config-volume" (OuterVolumeSpecName: "config-volume") pod "ca69377b-efa8-460f-9ea7-c4c297378e53" (UID: "ca69377b-efa8-460f-9ea7-c4c297378e53"). InnerVolumeSpecName "config-volume". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 03:43:11.088515 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:43:11.088277    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ca69377b-efa8-460f-9ea7-c4c297378e53-tls-assets" (OuterVolumeSpecName: "tls-assets") pod "ca69377b-efa8-460f-9ea7-c4c297378e53" (UID: "ca69377b-efa8-460f-9ea7-c4c297378e53"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 03:43:11.088515 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:43:11.088409    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ca69377b-efa8-460f-9ea7-c4c297378e53-secret-alertmanager-main-tls" (OuterVolumeSpecName: "secret-alertmanager-main-tls") pod "ca69377b-efa8-460f-9ea7-c4c297378e53" (UID: "ca69377b-efa8-460f-9ea7-c4c297378e53"). InnerVolumeSpecName "secret-alertmanager-main-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 03:43:11.088515 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:43:11.088462    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ca69377b-efa8-460f-9ea7-c4c297378e53-secret-alertmanager-kube-rbac-proxy-metric" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy-metric") pod "ca69377b-efa8-460f-9ea7-c4c297378e53" (UID: "ca69377b-efa8-460f-9ea7-c4c297378e53"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-metric". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 03:43:11.088730 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:43:11.088583    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ca69377b-efa8-460f-9ea7-c4c297378e53-secret-alertmanager-kube-rbac-proxy" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy") pod "ca69377b-efa8-460f-9ea7-c4c297378e53" (UID: "ca69377b-efa8-460f-9ea7-c4c297378e53"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 03:43:11.089408 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:43:11.089381    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ca69377b-efa8-460f-9ea7-c4c297378e53-kube-api-access-pmlj4" (OuterVolumeSpecName: "kube-api-access-pmlj4") pod "ca69377b-efa8-460f-9ea7-c4c297378e53" (UID: "ca69377b-efa8-460f-9ea7-c4c297378e53"). InnerVolumeSpecName "kube-api-access-pmlj4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 03:43:11.089760 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:43:11.089740    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ca69377b-efa8-460f-9ea7-c4c297378e53-config-out" (OuterVolumeSpecName: "config-out") pod "ca69377b-efa8-460f-9ea7-c4c297378e53" (UID: "ca69377b-efa8-460f-9ea7-c4c297378e53"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 03:43:11.089914 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:43:11.089884    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ca69377b-efa8-460f-9ea7-c4c297378e53-secret-alertmanager-kube-rbac-proxy-web" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy-web") pod "ca69377b-efa8-460f-9ea7-c4c297378e53" (UID: "ca69377b-efa8-460f-9ea7-c4c297378e53"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 03:43:11.093302 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:43:11.093048    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ca69377b-efa8-460f-9ea7-c4c297378e53-cluster-tls-config" (OuterVolumeSpecName: "cluster-tls-config") pod "ca69377b-efa8-460f-9ea7-c4c297378e53" (UID: "ca69377b-efa8-460f-9ea7-c4c297378e53"). InnerVolumeSpecName "cluster-tls-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 03:43:11.100517 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:43:11.100494    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ca69377b-efa8-460f-9ea7-c4c297378e53-web-config" (OuterVolumeSpecName: "web-config") pod "ca69377b-efa8-460f-9ea7-c4c297378e53" (UID: "ca69377b-efa8-460f-9ea7-c4c297378e53"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 03:43:11.186224 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:43:11.186191    2579 reconciler_common.go:299] "Volume detached for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/ca69377b-efa8-460f-9ea7-c4c297378e53-alertmanager-main-db\") on node \"ip-10-0-141-105.ec2.internal\" DevicePath \"\""
Sep 11 03:43:11.186224 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:43:11.186222    2579 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/ca69377b-efa8-460f-9ea7-c4c297378e53-secret-alertmanager-kube-rbac-proxy-web\") on node \"ip-10-0-141-105.ec2.internal\" DevicePath \"\""
Sep 11 03:43:11.186416 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:43:11.186237    2579 reconciler_common.go:299] "Volume detached for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ca69377b-efa8-460f-9ea7-c4c297378e53-alertmanager-trusted-ca-bundle\") on node \"ip-10-0-141-105.ec2.internal\" DevicePath \"\""
Sep 11 03:43:11.186416 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:43:11.186251    2579 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/ca69377b-efa8-460f-9ea7-c4c297378e53-secret-alertmanager-kube-rbac-proxy-metric\") on node \"ip-10-0-141-105.ec2.internal\" DevicePath \"\""
Sep 11 03:43:11.186416 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:43:11.186265    2579 reconciler_common.go:299] "Volume detached for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/ca69377b-efa8-460f-9ea7-c4c297378e53-cluster-tls-config\") on node \"ip-10-0-141-105.ec2.internal\" DevicePath \"\""
Sep 11 03:43:11.186416 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:43:11.186276    2579 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/ca69377b-efa8-460f-9ea7-c4c297378e53-secret-alertmanager-main-tls\") on node \"ip-10-0-141-105.ec2.internal\" DevicePath \"\""
Sep 11 03:43:11.186416 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:43:11.186289    2579 reconciler_common.go:299] "Volume detached for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/ca69377b-efa8-460f-9ea7-c4c297378e53-config-volume\") on node \"ip-10-0-141-105.ec2.internal\" DevicePath \"\""
Sep 11 03:43:11.186416 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:43:11.186300    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pmlj4\" (UniqueName: \"kubernetes.io/projected/ca69377b-efa8-460f-9ea7-c4c297378e53-kube-api-access-pmlj4\") on node \"ip-10-0-141-105.ec2.internal\" DevicePath \"\""
Sep 11 03:43:11.186416 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:43:11.186312    2579 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/ca69377b-efa8-460f-9ea7-c4c297378e53-tls-assets\") on node \"ip-10-0-141-105.ec2.internal\" DevicePath \"\""
Sep 11 03:43:11.186416 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:43:11.186324    2579 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/ca69377b-efa8-460f-9ea7-c4c297378e53-web-config\") on node \"ip-10-0-141-105.ec2.internal\" DevicePath \"\""
Sep 11 03:43:11.186416 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:43:11.186335    2579 reconciler_common.go:299] "Volume detached for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ca69377b-efa8-460f-9ea7-c4c297378e53-metrics-client-ca\") on node \"ip-10-0-141-105.ec2.internal\" DevicePath \"\""
Sep 11 03:43:11.186416 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:43:11.186346    2579 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/ca69377b-efa8-460f-9ea7-c4c297378e53-secret-alertmanager-kube-rbac-proxy\") on node \"ip-10-0-141-105.ec2.internal\" DevicePath \"\""
Sep 11 03:43:11.186416 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:43:11.186359    2579 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/ca69377b-efa8-460f-9ea7-c4c297378e53-config-out\") on node \"ip-10-0-141-105.ec2.internal\" DevicePath \"\""
Sep 11 03:43:11.748073 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:43:11.748046    2579 generic.go:356] "Generic (PLEG): container finished" podID="ca69377b-efa8-460f-9ea7-c4c297378e53" containerID="4548395d3c7d772ab4e836acc047517daedca0e165601f7fec1cfab11a9f61bc" exitCode=0
Sep 11 03:43:11.748073 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:43:11.748070    2579 generic.go:356] "Generic (PLEG): container finished" podID="ca69377b-efa8-460f-9ea7-c4c297378e53" containerID="e5510813bc296efd6645a11cb5b66b6d0afa52eb0700b1db2f27070eb05bd782" exitCode=0
Sep 11 03:43:11.748473 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:43:11.748134    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Sep 11 03:43:11.748473 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:43:11.748134    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"ca69377b-efa8-460f-9ea7-c4c297378e53","Type":"ContainerDied","Data":"4548395d3c7d772ab4e836acc047517daedca0e165601f7fec1cfab11a9f61bc"}
Sep 11 03:43:11.748473 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:43:11.748201    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"ca69377b-efa8-460f-9ea7-c4c297378e53","Type":"ContainerDied","Data":"e5510813bc296efd6645a11cb5b66b6d0afa52eb0700b1db2f27070eb05bd782"}
Sep 11 03:43:11.748473 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:43:11.748228    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"ca69377b-efa8-460f-9ea7-c4c297378e53","Type":"ContainerDied","Data":"53f3c17f3e002ca74a63eb456f1a4516b7f590f2fbd7606c2fdec4f1d110f137"}
Sep 11 03:43:11.748473 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:43:11.748260    2579 scope.go:117] "RemoveContainer" containerID="2bf2492ee4a72ec109fc63a156df78b5dfd9c38f4bf02dda04f780ba03163811"
Sep 11 03:43:11.756550 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:43:11.756534    2579 scope.go:117] "RemoveContainer" containerID="4548395d3c7d772ab4e836acc047517daedca0e165601f7fec1cfab11a9f61bc"
Sep 11 03:43:11.763889 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:43:11.763871    2579 scope.go:117] "RemoveContainer" containerID="eec2c8da9cd64a88af8b621febcb24a4ae1942cb3e317319e227791c0ab58eb8"
Sep 11 03:43:11.771011 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:43:11.770971    2579 scope.go:117] "RemoveContainer" containerID="e5510813bc296efd6645a11cb5b66b6d0afa52eb0700b1db2f27070eb05bd782"
Sep 11 03:43:11.772134 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:43:11.772115    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 11 03:43:11.779526 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:43:11.779506    2579 scope.go:117] "RemoveContainer" containerID="2c3779307a5a6e18aebf549c9c29ff64630db41af03856a308431167535dac80"
Sep 11 03:43:11.780200 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:43:11.780180    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 11 03:43:11.786665 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:43:11.786652    2579 scope.go:117] "RemoveContainer" containerID="2c570a50f3e5a8388ad35c05bf9cd57b15800cfeb0d0c3f598216e869b9231d5"
Sep 11 03:43:11.794021 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:43:11.794003    2579 scope.go:117] "RemoveContainer" containerID="d4dc50780261eeec829c7d183c5267c294f8cddca6f5adabb65051d3d6c9eb1b"
Sep 11 03:43:11.801860 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:43:11.801838    2579 scope.go:117] "RemoveContainer" containerID="2bf2492ee4a72ec109fc63a156df78b5dfd9c38f4bf02dda04f780ba03163811"
Sep 11 03:43:11.802337 ip-10-0-141-105 kubenswrapper[2579]: E0911 03:43:11.802313    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2bf2492ee4a72ec109fc63a156df78b5dfd9c38f4bf02dda04f780ba03163811\": container with ID starting with 2bf2492ee4a72ec109fc63a156df78b5dfd9c38f4bf02dda04f780ba03163811 not found: ID does not exist" containerID="2bf2492ee4a72ec109fc63a156df78b5dfd9c38f4bf02dda04f780ba03163811"
Sep 11 03:43:11.802498 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:43:11.802373    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2bf2492ee4a72ec109fc63a156df78b5dfd9c38f4bf02dda04f780ba03163811"} err="failed to get container status \"2bf2492ee4a72ec109fc63a156df78b5dfd9c38f4bf02dda04f780ba03163811\": rpc error: code = NotFound desc = could not find container \"2bf2492ee4a72ec109fc63a156df78b5dfd9c38f4bf02dda04f780ba03163811\": container with ID starting with 2bf2492ee4a72ec109fc63a156df78b5dfd9c38f4bf02dda04f780ba03163811 not found: ID does not exist"
Sep 11 03:43:11.802498 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:43:11.802500    2579 scope.go:117] "RemoveContainer" containerID="4548395d3c7d772ab4e836acc047517daedca0e165601f7fec1cfab11a9f61bc"
Sep 11 03:43:11.802961 ip-10-0-141-105 kubenswrapper[2579]: E0911 03:43:11.802941    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4548395d3c7d772ab4e836acc047517daedca0e165601f7fec1cfab11a9f61bc\": container with ID starting with 4548395d3c7d772ab4e836acc047517daedca0e165601f7fec1cfab11a9f61bc not found: ID does not exist" containerID="4548395d3c7d772ab4e836acc047517daedca0e165601f7fec1cfab11a9f61bc"
Sep 11 03:43:11.803063 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:43:11.802965    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4548395d3c7d772ab4e836acc047517daedca0e165601f7fec1cfab11a9f61bc"} err="failed to get container status \"4548395d3c7d772ab4e836acc047517daedca0e165601f7fec1cfab11a9f61bc\": rpc error: code = NotFound desc = could not find container \"4548395d3c7d772ab4e836acc047517daedca0e165601f7fec1cfab11a9f61bc\": container with ID starting with 4548395d3c7d772ab4e836acc047517daedca0e165601f7fec1cfab11a9f61bc not found: ID does not exist"
Sep 11 03:43:11.803063 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:43:11.802981    2579 scope.go:117] "RemoveContainer" containerID="eec2c8da9cd64a88af8b621febcb24a4ae1942cb3e317319e227791c0ab58eb8"
Sep 11 03:43:11.803232 ip-10-0-141-105 kubenswrapper[2579]: E0911 03:43:11.803214    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"eec2c8da9cd64a88af8b621febcb24a4ae1942cb3e317319e227791c0ab58eb8\": container with ID starting with eec2c8da9cd64a88af8b621febcb24a4ae1942cb3e317319e227791c0ab58eb8 not found: ID does not exist" containerID="eec2c8da9cd64a88af8b621febcb24a4ae1942cb3e317319e227791c0ab58eb8"
Sep 11 03:43:11.803271 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:43:11.803236    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"eec2c8da9cd64a88af8b621febcb24a4ae1942cb3e317319e227791c0ab58eb8"} err="failed to get container status \"eec2c8da9cd64a88af8b621febcb24a4ae1942cb3e317319e227791c0ab58eb8\": rpc error: code = NotFound desc = could not find container \"eec2c8da9cd64a88af8b621febcb24a4ae1942cb3e317319e227791c0ab58eb8\": container with ID starting with eec2c8da9cd64a88af8b621febcb24a4ae1942cb3e317319e227791c0ab58eb8 not found: ID does not exist"
Sep 11 03:43:11.803271 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:43:11.803249    2579 scope.go:117] "RemoveContainer" containerID="e5510813bc296efd6645a11cb5b66b6d0afa52eb0700b1db2f27070eb05bd782"
Sep 11 03:43:11.803519 ip-10-0-141-105 kubenswrapper[2579]: E0911 03:43:11.803501    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e5510813bc296efd6645a11cb5b66b6d0afa52eb0700b1db2f27070eb05bd782\": container with ID starting with e5510813bc296efd6645a11cb5b66b6d0afa52eb0700b1db2f27070eb05bd782 not found: ID does not exist" containerID="e5510813bc296efd6645a11cb5b66b6d0afa52eb0700b1db2f27070eb05bd782"
Sep 11 03:43:11.803583 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:43:11.803528    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e5510813bc296efd6645a11cb5b66b6d0afa52eb0700b1db2f27070eb05bd782"} err="failed to get container status \"e5510813bc296efd6645a11cb5b66b6d0afa52eb0700b1db2f27070eb05bd782\": rpc error: code = NotFound desc = could not find container \"e5510813bc296efd6645a11cb5b66b6d0afa52eb0700b1db2f27070eb05bd782\": container with ID starting with e5510813bc296efd6645a11cb5b66b6d0afa52eb0700b1db2f27070eb05bd782 not found: ID does not exist"
Sep 11 03:43:11.803583 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:43:11.803552    2579 scope.go:117] "RemoveContainer" containerID="2c3779307a5a6e18aebf549c9c29ff64630db41af03856a308431167535dac80"
Sep 11 03:43:11.803835 ip-10-0-141-105 kubenswrapper[2579]: E0911 03:43:11.803818    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2c3779307a5a6e18aebf549c9c29ff64630db41af03856a308431167535dac80\": container with ID starting with 2c3779307a5a6e18aebf549c9c29ff64630db41af03856a308431167535dac80 not found: ID does not exist" containerID="2c3779307a5a6e18aebf549c9c29ff64630db41af03856a308431167535dac80"
Sep 11 03:43:11.803872 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:43:11.803840    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2c3779307a5a6e18aebf549c9c29ff64630db41af03856a308431167535dac80"} err="failed to get container status \"2c3779307a5a6e18aebf549c9c29ff64630db41af03856a308431167535dac80\": rpc error: code = NotFound desc = could not find container \"2c3779307a5a6e18aebf549c9c29ff64630db41af03856a308431167535dac80\": container with ID starting with 2c3779307a5a6e18aebf549c9c29ff64630db41af03856a308431167535dac80 not found: ID does not exist"
Sep 11 03:43:11.803872 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:43:11.803855    2579 scope.go:117] "RemoveContainer" containerID="2c570a50f3e5a8388ad35c05bf9cd57b15800cfeb0d0c3f598216e869b9231d5"
Sep 11 03:43:11.804106 ip-10-0-141-105 kubenswrapper[2579]: E0911 03:43:11.804068    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2c570a50f3e5a8388ad35c05bf9cd57b15800cfeb0d0c3f598216e869b9231d5\": container with ID starting with 2c570a50f3e5a8388ad35c05bf9cd57b15800cfeb0d0c3f598216e869b9231d5 not found: ID does not exist" containerID="2c570a50f3e5a8388ad35c05bf9cd57b15800cfeb0d0c3f598216e869b9231d5"
Sep 11 03:43:11.804177 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:43:11.804160    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2c570a50f3e5a8388ad35c05bf9cd57b15800cfeb0d0c3f598216e869b9231d5"} err="failed to get container status \"2c570a50f3e5a8388ad35c05bf9cd57b15800cfeb0d0c3f598216e869b9231d5\": rpc error: code = NotFound desc = could not find container \"2c570a50f3e5a8388ad35c05bf9cd57b15800cfeb0d0c3f598216e869b9231d5\": container with ID starting with 2c570a50f3e5a8388ad35c05bf9cd57b15800cfeb0d0c3f598216e869b9231d5 not found: ID does not exist"
Sep 11 03:43:11.804213 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:43:11.804178    2579 scope.go:117] "RemoveContainer" containerID="d4dc50780261eeec829c7d183c5267c294f8cddca6f5adabb65051d3d6c9eb1b"
Sep 11 03:43:11.804443 ip-10-0-141-105 kubenswrapper[2579]: E0911 03:43:11.804427    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d4dc50780261eeec829c7d183c5267c294f8cddca6f5adabb65051d3d6c9eb1b\": container with ID starting with d4dc50780261eeec829c7d183c5267c294f8cddca6f5adabb65051d3d6c9eb1b not found: ID does not exist" containerID="d4dc50780261eeec829c7d183c5267c294f8cddca6f5adabb65051d3d6c9eb1b"
Sep 11 03:43:11.804624 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:43:11.804450    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d4dc50780261eeec829c7d183c5267c294f8cddca6f5adabb65051d3d6c9eb1b"} err="failed to get container status \"d4dc50780261eeec829c7d183c5267c294f8cddca6f5adabb65051d3d6c9eb1b\": rpc error: code = NotFound desc = could not find container \"d4dc50780261eeec829c7d183c5267c294f8cddca6f5adabb65051d3d6c9eb1b\": container with ID starting with d4dc50780261eeec829c7d183c5267c294f8cddca6f5adabb65051d3d6c9eb1b not found: ID does not exist"
Sep 11 03:43:11.804624 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:43:11.804470    2579 scope.go:117] "RemoveContainer" containerID="2bf2492ee4a72ec109fc63a156df78b5dfd9c38f4bf02dda04f780ba03163811"
Sep 11 03:43:11.804750 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:43:11.804732    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2bf2492ee4a72ec109fc63a156df78b5dfd9c38f4bf02dda04f780ba03163811"} err="failed to get container status \"2bf2492ee4a72ec109fc63a156df78b5dfd9c38f4bf02dda04f780ba03163811\": rpc error: code = NotFound desc = could not find container \"2bf2492ee4a72ec109fc63a156df78b5dfd9c38f4bf02dda04f780ba03163811\": container with ID starting with 2bf2492ee4a72ec109fc63a156df78b5dfd9c38f4bf02dda04f780ba03163811 not found: ID does not exist"
Sep 11 03:43:11.804795 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:43:11.804752    2579 scope.go:117] "RemoveContainer" containerID="4548395d3c7d772ab4e836acc047517daedca0e165601f7fec1cfab11a9f61bc"
Sep 11 03:43:11.805022 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:43:11.805006    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4548395d3c7d772ab4e836acc047517daedca0e165601f7fec1cfab11a9f61bc"} err="failed to get container status \"4548395d3c7d772ab4e836acc047517daedca0e165601f7fec1cfab11a9f61bc\": rpc error: code = NotFound desc = could not find container \"4548395d3c7d772ab4e836acc047517daedca0e165601f7fec1cfab11a9f61bc\": container with ID starting with 4548395d3c7d772ab4e836acc047517daedca0e165601f7fec1cfab11a9f61bc not found: ID does not exist"
Sep 11 03:43:11.805071 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:43:11.805022    2579 scope.go:117] "RemoveContainer" containerID="eec2c8da9cd64a88af8b621febcb24a4ae1942cb3e317319e227791c0ab58eb8"
Sep 11 03:43:11.805287 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:43:11.805272    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"eec2c8da9cd64a88af8b621febcb24a4ae1942cb3e317319e227791c0ab58eb8"} err="failed to get container status \"eec2c8da9cd64a88af8b621febcb24a4ae1942cb3e317319e227791c0ab58eb8\": rpc error: code = NotFound desc = could not find container \"eec2c8da9cd64a88af8b621febcb24a4ae1942cb3e317319e227791c0ab58eb8\": container with ID starting with eec2c8da9cd64a88af8b621febcb24a4ae1942cb3e317319e227791c0ab58eb8 not found: ID does not exist"
Sep 11 03:43:11.805330 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:43:11.805287    2579 scope.go:117] "RemoveContainer" containerID="e5510813bc296efd6645a11cb5b66b6d0afa52eb0700b1db2f27070eb05bd782"
Sep 11 03:43:11.805540 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:43:11.805519    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e5510813bc296efd6645a11cb5b66b6d0afa52eb0700b1db2f27070eb05bd782"} err="failed to get container status \"e5510813bc296efd6645a11cb5b66b6d0afa52eb0700b1db2f27070eb05bd782\": rpc error: code = NotFound desc = could not find container \"e5510813bc296efd6645a11cb5b66b6d0afa52eb0700b1db2f27070eb05bd782\": container with ID starting with e5510813bc296efd6645a11cb5b66b6d0afa52eb0700b1db2f27070eb05bd782 not found: ID does not exist"
Sep 11 03:43:11.805597 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:43:11.805541    2579 scope.go:117] "RemoveContainer" containerID="2c3779307a5a6e18aebf549c9c29ff64630db41af03856a308431167535dac80"
Sep 11 03:43:11.805737 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:43:11.805722    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2c3779307a5a6e18aebf549c9c29ff64630db41af03856a308431167535dac80"} err="failed to get container status \"2c3779307a5a6e18aebf549c9c29ff64630db41af03856a308431167535dac80\": rpc error: code = NotFound desc = could not find container \"2c3779307a5a6e18aebf549c9c29ff64630db41af03856a308431167535dac80\": container with ID starting with 2c3779307a5a6e18aebf549c9c29ff64630db41af03856a308431167535dac80 not found: ID does not exist"
Sep 11 03:43:11.805779 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:43:11.805738    2579 scope.go:117] "RemoveContainer" containerID="2c570a50f3e5a8388ad35c05bf9cd57b15800cfeb0d0c3f598216e869b9231d5"
Sep 11 03:43:11.805956 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:43:11.805941    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2c570a50f3e5a8388ad35c05bf9cd57b15800cfeb0d0c3f598216e869b9231d5"} err="failed to get container status \"2c570a50f3e5a8388ad35c05bf9cd57b15800cfeb0d0c3f598216e869b9231d5\": rpc error: code = NotFound desc = could not find container \"2c570a50f3e5a8388ad35c05bf9cd57b15800cfeb0d0c3f598216e869b9231d5\": container with ID starting with 2c570a50f3e5a8388ad35c05bf9cd57b15800cfeb0d0c3f598216e869b9231d5 not found: ID does not exist"
Sep 11 03:43:11.806008 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:43:11.805956    2579 scope.go:117] "RemoveContainer" containerID="d4dc50780261eeec829c7d183c5267c294f8cddca6f5adabb65051d3d6c9eb1b"
Sep 11 03:43:11.806207 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:43:11.806189    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d4dc50780261eeec829c7d183c5267c294f8cddca6f5adabb65051d3d6c9eb1b"} err="failed to get container status \"d4dc50780261eeec829c7d183c5267c294f8cddca6f5adabb65051d3d6c9eb1b\": rpc error: code = NotFound desc = could not find container \"d4dc50780261eeec829c7d183c5267c294f8cddca6f5adabb65051d3d6c9eb1b\": container with ID starting with d4dc50780261eeec829c7d183c5267c294f8cddca6f5adabb65051d3d6c9eb1b not found: ID does not exist"
Sep 11 03:43:12.851392 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:43:12.851361    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ca69377b-efa8-460f-9ea7-c4c297378e53" path="/var/lib/kubelet/pods/ca69377b-efa8-460f-9ea7-c4c297378e53/volumes"
Sep 11 03:43:13.918497 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:43:13.918414    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 11 03:43:13.918925 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:43:13.918710    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="5208b5bb-0268-4ab8-8e3a-f873bda02436" containerName="prometheus" containerID="cri-o://a5fb41f056f6486570ae6451b1ccc687e68091d1258fa434fbe505339d967aa9" gracePeriod=600
Sep 11 03:43:13.918925 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:43:13.918738    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="5208b5bb-0268-4ab8-8e3a-f873bda02436" containerName="kube-rbac-proxy" containerID="cri-o://21fe8a05bab9b83615a8293c808cba67ff07b045fa1ff8abb956d36b8a4548a6" gracePeriod=600
Sep 11 03:43:13.918925 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:43:13.918774    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="5208b5bb-0268-4ab8-8e3a-f873bda02436" containerName="config-reloader" containerID="cri-o://c34f844b4ee0b9f4eb00ca54e6fbfe8f911ae7f338602dee41d5cfc502683450" gracePeriod=600
Sep 11 03:43:13.918925 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:43:13.918806    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="5208b5bb-0268-4ab8-8e3a-f873bda02436" containerName="kube-rbac-proxy-web" containerID="cri-o://4562ba7ff19703d39b6f22517c8811def29f266ad7974c249138ed86e03c7715" gracePeriod=600
Sep 11 03:43:13.918925 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:43:13.918775    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="5208b5bb-0268-4ab8-8e3a-f873bda02436" containerName="kube-rbac-proxy-thanos" containerID="cri-o://cc15a3445e053dd5656d9fc120f01c37afc3fa0cb6cdc7158a87f9389387200b" gracePeriod=600
Sep 11 03:43:13.918925 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:43:13.918773    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="5208b5bb-0268-4ab8-8e3a-f873bda02436" containerName="thanos-sidecar" containerID="cri-o://8f9b2f45cf254555dd48a3598d894fc58b2f165af492fd1ea1df73670f4093e9" gracePeriod=600
Sep 11 03:43:14.148860 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:43:14.148830    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 03:43:14.314340 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:43:14.314254    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5208b5bb-0268-4ab8-8e3a-f873bda02436-prometheus-trusted-ca-bundle\") pod \"5208b5bb-0268-4ab8-8e3a-f873bda02436\" (UID: \"5208b5bb-0268-4ab8-8e3a-f873bda02436\") "
Sep 11 03:43:14.314340 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:43:14.314302    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/5208b5bb-0268-4ab8-8e3a-f873bda02436-prometheus-k8s-rulefiles-0\") pod \"5208b5bb-0268-4ab8-8e3a-f873bda02436\" (UID: \"5208b5bb-0268-4ab8-8e3a-f873bda02436\") "
Sep 11 03:43:14.314340 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:43:14.314325    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/5208b5bb-0268-4ab8-8e3a-f873bda02436-secret-prometheus-k8s-tls\") pod \"5208b5bb-0268-4ab8-8e3a-f873bda02436\" (UID: \"5208b5bb-0268-4ab8-8e3a-f873bda02436\") "
Sep 11 03:43:14.314607 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:43:14.314353    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/5208b5bb-0268-4ab8-8e3a-f873bda02436-config\") pod \"5208b5bb-0268-4ab8-8e3a-f873bda02436\" (UID: \"5208b5bb-0268-4ab8-8e3a-f873bda02436\") "
Sep 11 03:43:14.314607 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:43:14.314377    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5208b5bb-0268-4ab8-8e3a-f873bda02436-configmap-serving-certs-ca-bundle\") pod \"5208b5bb-0268-4ab8-8e3a-f873bda02436\" (UID: \"5208b5bb-0268-4ab8-8e3a-f873bda02436\") "
Sep 11 03:43:14.314607 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:43:14.314408    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/5208b5bb-0268-4ab8-8e3a-f873bda02436-secret-metrics-client-certs\") pod \"5208b5bb-0268-4ab8-8e3a-f873bda02436\" (UID: \"5208b5bb-0268-4ab8-8e3a-f873bda02436\") "
Sep 11 03:43:14.314607 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:43:14.314435    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/5208b5bb-0268-4ab8-8e3a-f873bda02436-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"5208b5bb-0268-4ab8-8e3a-f873bda02436\" (UID: \"5208b5bb-0268-4ab8-8e3a-f873bda02436\") "
Sep 11 03:43:14.314607 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:43:14.314458    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5208b5bb-0268-4ab8-8e3a-f873bda02436-configmap-kubelet-serving-ca-bundle\") pod \"5208b5bb-0268-4ab8-8e3a-f873bda02436\" (UID: \"5208b5bb-0268-4ab8-8e3a-f873bda02436\") "
Sep 11 03:43:14.314607 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:43:14.314482    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/5208b5bb-0268-4ab8-8e3a-f873bda02436-secret-kube-rbac-proxy\") pod \"5208b5bb-0268-4ab8-8e3a-f873bda02436\" (UID: \"5208b5bb-0268-4ab8-8e3a-f873bda02436\") "
Sep 11 03:43:14.314607 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:43:14.314534    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/5208b5bb-0268-4ab8-8e3a-f873bda02436-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"5208b5bb-0268-4ab8-8e3a-f873bda02436\" (UID: \"5208b5bb-0268-4ab8-8e3a-f873bda02436\") "
Sep 11 03:43:14.314607 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:43:14.314559    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/5208b5bb-0268-4ab8-8e3a-f873bda02436-prometheus-k8s-rulefiles-1\") pod \"5208b5bb-0268-4ab8-8e3a-f873bda02436\" (UID: \"5208b5bb-0268-4ab8-8e3a-f873bda02436\") "
Sep 11 03:43:14.314607 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:43:14.314597    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/5208b5bb-0268-4ab8-8e3a-f873bda02436-thanos-prometheus-http-client-file\") pod \"5208b5bb-0268-4ab8-8e3a-f873bda02436\" (UID: \"5208b5bb-0268-4ab8-8e3a-f873bda02436\") "
Sep 11 03:43:14.315002 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:43:14.314626    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rwqlq\" (UniqueName: \"kubernetes.io/projected/5208b5bb-0268-4ab8-8e3a-f873bda02436-kube-api-access-rwqlq\") pod \"5208b5bb-0268-4ab8-8e3a-f873bda02436\" (UID: \"5208b5bb-0268-4ab8-8e3a-f873bda02436\") "
Sep 11 03:43:14.315002 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:43:14.314661    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/5208b5bb-0268-4ab8-8e3a-f873bda02436-secret-grpc-tls\") pod \"5208b5bb-0268-4ab8-8e3a-f873bda02436\" (UID: \"5208b5bb-0268-4ab8-8e3a-f873bda02436\") "
Sep 11 03:43:14.315002 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:43:14.314688    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/5208b5bb-0268-4ab8-8e3a-f873bda02436-tls-assets\") pod \"5208b5bb-0268-4ab8-8e3a-f873bda02436\" (UID: \"5208b5bb-0268-4ab8-8e3a-f873bda02436\") "
Sep 11 03:43:14.315002 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:43:14.314724    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/5208b5bb-0268-4ab8-8e3a-f873bda02436-prometheus-k8s-rulefiles-2\") pod \"5208b5bb-0268-4ab8-8e3a-f873bda02436\" (UID: \"5208b5bb-0268-4ab8-8e3a-f873bda02436\") "
Sep 11 03:43:14.315002 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:43:14.314750    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/5208b5bb-0268-4ab8-8e3a-f873bda02436-config-out\") pod \"5208b5bb-0268-4ab8-8e3a-f873bda02436\" (UID: \"5208b5bb-0268-4ab8-8e3a-f873bda02436\") "
Sep 11 03:43:14.315002 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:43:14.314778    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/5208b5bb-0268-4ab8-8e3a-f873bda02436-web-config\") pod \"5208b5bb-0268-4ab8-8e3a-f873bda02436\" (UID: \"5208b5bb-0268-4ab8-8e3a-f873bda02436\") "
Sep 11 03:43:14.315002 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:43:14.314786    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5208b5bb-0268-4ab8-8e3a-f873bda02436-prometheus-trusted-ca-bundle" (OuterVolumeSpecName: "prometheus-trusted-ca-bundle") pod "5208b5bb-0268-4ab8-8e3a-f873bda02436" (UID: "5208b5bb-0268-4ab8-8e3a-f873bda02436"). InnerVolumeSpecName "prometheus-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 03:43:14.315002 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:43:14.314810    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/5208b5bb-0268-4ab8-8e3a-f873bda02436-prometheus-k8s-db\") pod \"5208b5bb-0268-4ab8-8e3a-f873bda02436\" (UID: \"5208b5bb-0268-4ab8-8e3a-f873bda02436\") "
Sep 11 03:43:14.315002 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:43:14.314859    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/5208b5bb-0268-4ab8-8e3a-f873bda02436-configmap-metrics-client-ca\") pod \"5208b5bb-0268-4ab8-8e3a-f873bda02436\" (UID: \"5208b5bb-0268-4ab8-8e3a-f873bda02436\") "
Sep 11 03:43:14.315453 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:43:14.315131    2579 reconciler_common.go:299] "Volume detached for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5208b5bb-0268-4ab8-8e3a-f873bda02436-prometheus-trusted-ca-bundle\") on node \"ip-10-0-141-105.ec2.internal\" DevicePath \"\""
Sep 11 03:43:14.315453 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:43:14.315303    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5208b5bb-0268-4ab8-8e3a-f873bda02436-configmap-serving-certs-ca-bundle" (OuterVolumeSpecName: "configmap-serving-certs-ca-bundle") pod "5208b5bb-0268-4ab8-8e3a-f873bda02436" (UID: "5208b5bb-0268-4ab8-8e3a-f873bda02436"). InnerVolumeSpecName "configmap-serving-certs-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 03:43:14.315554 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:43:14.315497    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5208b5bb-0268-4ab8-8e3a-f873bda02436-configmap-metrics-client-ca" (OuterVolumeSpecName: "configmap-metrics-client-ca") pod "5208b5bb-0268-4ab8-8e3a-f873bda02436" (UID: "5208b5bb-0268-4ab8-8e3a-f873bda02436"). InnerVolumeSpecName "configmap-metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 03:43:14.315972 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:43:14.315939    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5208b5bb-0268-4ab8-8e3a-f873bda02436-configmap-kubelet-serving-ca-bundle" (OuterVolumeSpecName: "configmap-kubelet-serving-ca-bundle") pod "5208b5bb-0268-4ab8-8e3a-f873bda02436" (UID: "5208b5bb-0268-4ab8-8e3a-f873bda02436"). InnerVolumeSpecName "configmap-kubelet-serving-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 03:43:14.317664 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:43:14.316441    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5208b5bb-0268-4ab8-8e3a-f873bda02436-prometheus-k8s-rulefiles-2" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-2") pod "5208b5bb-0268-4ab8-8e3a-f873bda02436" (UID: "5208b5bb-0268-4ab8-8e3a-f873bda02436"). InnerVolumeSpecName "prometheus-k8s-rulefiles-2". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 03:43:14.317664 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:43:14.316591    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5208b5bb-0268-4ab8-8e3a-f873bda02436-prometheus-k8s-rulefiles-0" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-0") pod "5208b5bb-0268-4ab8-8e3a-f873bda02436" (UID: "5208b5bb-0268-4ab8-8e3a-f873bda02436"). InnerVolumeSpecName "prometheus-k8s-rulefiles-0". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 03:43:14.317664 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:43:14.316825    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5208b5bb-0268-4ab8-8e3a-f873bda02436-prometheus-k8s-rulefiles-1" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-1") pod "5208b5bb-0268-4ab8-8e3a-f873bda02436" (UID: "5208b5bb-0268-4ab8-8e3a-f873bda02436"). InnerVolumeSpecName "prometheus-k8s-rulefiles-1". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 03:43:14.317664 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:43:14.317625    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5208b5bb-0268-4ab8-8e3a-f873bda02436-prometheus-k8s-db" (OuterVolumeSpecName: "prometheus-k8s-db") pod "5208b5bb-0268-4ab8-8e3a-f873bda02436" (UID: "5208b5bb-0268-4ab8-8e3a-f873bda02436"). InnerVolumeSpecName "prometheus-k8s-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 03:43:14.318479 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:43:14.318438    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5208b5bb-0268-4ab8-8e3a-f873bda02436-config" (OuterVolumeSpecName: "config") pod "5208b5bb-0268-4ab8-8e3a-f873bda02436" (UID: "5208b5bb-0268-4ab8-8e3a-f873bda02436"). InnerVolumeSpecName "config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 03:43:14.318591 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:43:14.318516    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5208b5bb-0268-4ab8-8e3a-f873bda02436-secret-prometheus-k8s-tls" (OuterVolumeSpecName: "secret-prometheus-k8s-tls") pod "5208b5bb-0268-4ab8-8e3a-f873bda02436" (UID: "5208b5bb-0268-4ab8-8e3a-f873bda02436"). InnerVolumeSpecName "secret-prometheus-k8s-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 03:43:14.318653 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:43:14.318596    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5208b5bb-0268-4ab8-8e3a-f873bda02436-secret-metrics-client-certs" (OuterVolumeSpecName: "secret-metrics-client-certs") pod "5208b5bb-0268-4ab8-8e3a-f873bda02436" (UID: "5208b5bb-0268-4ab8-8e3a-f873bda02436"). InnerVolumeSpecName "secret-metrics-client-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 03:43:14.318653 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:43:14.318605    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5208b5bb-0268-4ab8-8e3a-f873bda02436-kube-api-access-rwqlq" (OuterVolumeSpecName: "kube-api-access-rwqlq") pod "5208b5bb-0268-4ab8-8e3a-f873bda02436" (UID: "5208b5bb-0268-4ab8-8e3a-f873bda02436"). InnerVolumeSpecName "kube-api-access-rwqlq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 03:43:14.318913 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:43:14.318885    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5208b5bb-0268-4ab8-8e3a-f873bda02436-secret-prometheus-k8s-kube-rbac-proxy-web" (OuterVolumeSpecName: "secret-prometheus-k8s-kube-rbac-proxy-web") pod "5208b5bb-0268-4ab8-8e3a-f873bda02436" (UID: "5208b5bb-0268-4ab8-8e3a-f873bda02436"). InnerVolumeSpecName "secret-prometheus-k8s-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 03:43:14.319388 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:43:14.319368    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5208b5bb-0268-4ab8-8e3a-f873bda02436-secret-kube-rbac-proxy" (OuterVolumeSpecName: "secret-kube-rbac-proxy") pod "5208b5bb-0268-4ab8-8e3a-f873bda02436" (UID: "5208b5bb-0268-4ab8-8e3a-f873bda02436"). InnerVolumeSpecName "secret-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 03:43:14.319752 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:43:14.319721    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5208b5bb-0268-4ab8-8e3a-f873bda02436-thanos-prometheus-http-client-file" (OuterVolumeSpecName: "thanos-prometheus-http-client-file") pod "5208b5bb-0268-4ab8-8e3a-f873bda02436" (UID: "5208b5bb-0268-4ab8-8e3a-f873bda02436"). InnerVolumeSpecName "thanos-prometheus-http-client-file". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 03:43:14.320248 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:43:14.320230    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5208b5bb-0268-4ab8-8e3a-f873bda02436-config-out" (OuterVolumeSpecName: "config-out") pod "5208b5bb-0268-4ab8-8e3a-f873bda02436" (UID: "5208b5bb-0268-4ab8-8e3a-f873bda02436"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 03:43:14.320397 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:43:14.320374    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5208b5bb-0268-4ab8-8e3a-f873bda02436-secret-grpc-tls" (OuterVolumeSpecName: "secret-grpc-tls") pod "5208b5bb-0268-4ab8-8e3a-f873bda02436" (UID: "5208b5bb-0268-4ab8-8e3a-f873bda02436"). InnerVolumeSpecName "secret-grpc-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 03:43:14.320477 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:43:14.320441    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5208b5bb-0268-4ab8-8e3a-f873bda02436-tls-assets" (OuterVolumeSpecName: "tls-assets") pod "5208b5bb-0268-4ab8-8e3a-f873bda02436" (UID: "5208b5bb-0268-4ab8-8e3a-f873bda02436"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 03:43:14.321005 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:43:14.320977    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5208b5bb-0268-4ab8-8e3a-f873bda02436-secret-prometheus-k8s-thanos-sidecar-tls" (OuterVolumeSpecName: "secret-prometheus-k8s-thanos-sidecar-tls") pod "5208b5bb-0268-4ab8-8e3a-f873bda02436" (UID: "5208b5bb-0268-4ab8-8e3a-f873bda02436"). InnerVolumeSpecName "secret-prometheus-k8s-thanos-sidecar-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 03:43:14.329588 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:43:14.329566    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5208b5bb-0268-4ab8-8e3a-f873bda02436-web-config" (OuterVolumeSpecName: "web-config") pod "5208b5bb-0268-4ab8-8e3a-f873bda02436" (UID: "5208b5bb-0268-4ab8-8e3a-f873bda02436"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 03:43:14.416000 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:43:14.415962    2579 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/5208b5bb-0268-4ab8-8e3a-f873bda02436-prometheus-k8s-rulefiles-2\") on node \"ip-10-0-141-105.ec2.internal\" DevicePath \"\""
Sep 11 03:43:14.416000 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:43:14.415999    2579 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/5208b5bb-0268-4ab8-8e3a-f873bda02436-config-out\") on node \"ip-10-0-141-105.ec2.internal\" DevicePath \"\""
Sep 11 03:43:14.416000 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:43:14.416009    2579 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/5208b5bb-0268-4ab8-8e3a-f873bda02436-web-config\") on node \"ip-10-0-141-105.ec2.internal\" DevicePath \"\""
Sep 11 03:43:14.416220 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:43:14.416017    2579 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/5208b5bb-0268-4ab8-8e3a-f873bda02436-prometheus-k8s-db\") on node \"ip-10-0-141-105.ec2.internal\" DevicePath \"\""
Sep 11 03:43:14.416220 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:43:14.416027    2579 reconciler_common.go:299] "Volume detached for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/5208b5bb-0268-4ab8-8e3a-f873bda02436-configmap-metrics-client-ca\") on node \"ip-10-0-141-105.ec2.internal\" DevicePath \"\""
Sep 11 03:43:14.416220 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:43:14.416038    2579 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/5208b5bb-0268-4ab8-8e3a-f873bda02436-prometheus-k8s-rulefiles-0\") on node \"ip-10-0-141-105.ec2.internal\" DevicePath \"\""
Sep 11 03:43:14.416220 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:43:14.416045    2579 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/5208b5bb-0268-4ab8-8e3a-f873bda02436-secret-prometheus-k8s-tls\") on node \"ip-10-0-141-105.ec2.internal\" DevicePath \"\""
Sep 11 03:43:14.416220 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:43:14.416054    2579 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/secret/5208b5bb-0268-4ab8-8e3a-f873bda02436-config\") on node \"ip-10-0-141-105.ec2.internal\" DevicePath \"\""
Sep 11 03:43:14.416220 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:43:14.416063    2579 reconciler_common.go:299] "Volume detached for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5208b5bb-0268-4ab8-8e3a-f873bda02436-configmap-serving-certs-ca-bundle\") on node \"ip-10-0-141-105.ec2.internal\" DevicePath \"\""
Sep 11 03:43:14.416220 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:43:14.416071    2579 reconciler_common.go:299] "Volume detached for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/5208b5bb-0268-4ab8-8e3a-f873bda02436-secret-metrics-client-certs\") on node \"ip-10-0-141-105.ec2.internal\" DevicePath \"\""
Sep 11 03:43:14.416220 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:43:14.416100    2579 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/5208b5bb-0268-4ab8-8e3a-f873bda02436-secret-prometheus-k8s-kube-rbac-proxy-web\") on node \"ip-10-0-141-105.ec2.internal\" DevicePath \"\""
Sep 11 03:43:14.416220 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:43:14.416110    2579 reconciler_common.go:299] "Volume detached for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5208b5bb-0268-4ab8-8e3a-f873bda02436-configmap-kubelet-serving-ca-bundle\") on node \"ip-10-0-141-105.ec2.internal\" DevicePath \"\""
Sep 11 03:43:14.416220 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:43:14.416118    2579 reconciler_common.go:299] "Volume detached for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/5208b5bb-0268-4ab8-8e3a-f873bda02436-secret-kube-rbac-proxy\") on node \"ip-10-0-141-105.ec2.internal\" DevicePath \"\""
Sep 11 03:43:14.416220 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:43:14.416128    2579 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/5208b5bb-0268-4ab8-8e3a-f873bda02436-secret-prometheus-k8s-thanos-sidecar-tls\") on node \"ip-10-0-141-105.ec2.internal\" DevicePath \"\""
Sep 11 03:43:14.416220 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:43:14.416136    2579 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/5208b5bb-0268-4ab8-8e3a-f873bda02436-prometheus-k8s-rulefiles-1\") on node \"ip-10-0-141-105.ec2.internal\" DevicePath \"\""
Sep 11 03:43:14.416220 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:43:14.416144    2579 reconciler_common.go:299] "Volume detached for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/5208b5bb-0268-4ab8-8e3a-f873bda02436-thanos-prometheus-http-client-file\") on node \"ip-10-0-141-105.ec2.internal\" DevicePath \"\""
Sep 11 03:43:14.416220 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:43:14.416152    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rwqlq\" (UniqueName: \"kubernetes.io/projected/5208b5bb-0268-4ab8-8e3a-f873bda02436-kube-api-access-rwqlq\") on node \"ip-10-0-141-105.ec2.internal\" DevicePath \"\""
Sep 11 03:43:14.416220 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:43:14.416160    2579 reconciler_common.go:299] "Volume detached for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/5208b5bb-0268-4ab8-8e3a-f873bda02436-secret-grpc-tls\") on node \"ip-10-0-141-105.ec2.internal\" DevicePath \"\""
Sep 11 03:43:14.416220 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:43:14.416167    2579 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/5208b5bb-0268-4ab8-8e3a-f873bda02436-tls-assets\") on node \"ip-10-0-141-105.ec2.internal\" DevicePath \"\""
Sep 11 03:43:14.764521 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:43:14.764485    2579 generic.go:356] "Generic (PLEG): container finished" podID="5208b5bb-0268-4ab8-8e3a-f873bda02436" containerID="cc15a3445e053dd5656d9fc120f01c37afc3fa0cb6cdc7158a87f9389387200b" exitCode=0
Sep 11 03:43:14.764521 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:43:14.764509    2579 generic.go:356] "Generic (PLEG): container finished" podID="5208b5bb-0268-4ab8-8e3a-f873bda02436" containerID="21fe8a05bab9b83615a8293c808cba67ff07b045fa1ff8abb956d36b8a4548a6" exitCode=0
Sep 11 03:43:14.764521 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:43:14.764516    2579 generic.go:356] "Generic (PLEG): container finished" podID="5208b5bb-0268-4ab8-8e3a-f873bda02436" containerID="4562ba7ff19703d39b6f22517c8811def29f266ad7974c249138ed86e03c7715" exitCode=0
Sep 11 03:43:14.764521 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:43:14.764523    2579 generic.go:356] "Generic (PLEG): container finished" podID="5208b5bb-0268-4ab8-8e3a-f873bda02436" containerID="8f9b2f45cf254555dd48a3598d894fc58b2f165af492fd1ea1df73670f4093e9" exitCode=0
Sep 11 03:43:14.764521 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:43:14.764527    2579 generic.go:356] "Generic (PLEG): container finished" podID="5208b5bb-0268-4ab8-8e3a-f873bda02436" containerID="c34f844b4ee0b9f4eb00ca54e6fbfe8f911ae7f338602dee41d5cfc502683450" exitCode=0
Sep 11 03:43:14.764521 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:43:14.764532    2579 generic.go:356] "Generic (PLEG): container finished" podID="5208b5bb-0268-4ab8-8e3a-f873bda02436" containerID="a5fb41f056f6486570ae6451b1ccc687e68091d1258fa434fbe505339d967aa9" exitCode=0
Sep 11 03:43:14.764950 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:43:14.764560    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"5208b5bb-0268-4ab8-8e3a-f873bda02436","Type":"ContainerDied","Data":"cc15a3445e053dd5656d9fc120f01c37afc3fa0cb6cdc7158a87f9389387200b"}
Sep 11 03:43:14.764950 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:43:14.764586    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 03:43:14.764950 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:43:14.764673    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"5208b5bb-0268-4ab8-8e3a-f873bda02436","Type":"ContainerDied","Data":"21fe8a05bab9b83615a8293c808cba67ff07b045fa1ff8abb956d36b8a4548a6"}
Sep 11 03:43:14.764950 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:43:14.764720    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"5208b5bb-0268-4ab8-8e3a-f873bda02436","Type":"ContainerDied","Data":"4562ba7ff19703d39b6f22517c8811def29f266ad7974c249138ed86e03c7715"}
Sep 11 03:43:14.764950 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:43:14.764737    2579 scope.go:117] "RemoveContainer" containerID="cc15a3445e053dd5656d9fc120f01c37afc3fa0cb6cdc7158a87f9389387200b"
Sep 11 03:43:14.764950 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:43:14.764744    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"5208b5bb-0268-4ab8-8e3a-f873bda02436","Type":"ContainerDied","Data":"8f9b2f45cf254555dd48a3598d894fc58b2f165af492fd1ea1df73670f4093e9"}
Sep 11 03:43:14.764950 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:43:14.764769    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"5208b5bb-0268-4ab8-8e3a-f873bda02436","Type":"ContainerDied","Data":"c34f844b4ee0b9f4eb00ca54e6fbfe8f911ae7f338602dee41d5cfc502683450"}
Sep 11 03:43:14.764950 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:43:14.764795    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"5208b5bb-0268-4ab8-8e3a-f873bda02436","Type":"ContainerDied","Data":"a5fb41f056f6486570ae6451b1ccc687e68091d1258fa434fbe505339d967aa9"}
Sep 11 03:43:14.764950 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:43:14.764821    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"5208b5bb-0268-4ab8-8e3a-f873bda02436","Type":"ContainerDied","Data":"7639469849f9e1d544c0b4f34965bae23114a4810a8aeaf093692ffdce290a50"}
Sep 11 03:43:14.773872 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:43:14.773853    2579 scope.go:117] "RemoveContainer" containerID="21fe8a05bab9b83615a8293c808cba67ff07b045fa1ff8abb956d36b8a4548a6"
Sep 11 03:43:14.781544 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:43:14.781524    2579 scope.go:117] "RemoveContainer" containerID="4562ba7ff19703d39b6f22517c8811def29f266ad7974c249138ed86e03c7715"
Sep 11 03:43:14.788770 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:43:14.788750    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 11 03:43:14.788988 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:43:14.788971    2579 scope.go:117] "RemoveContainer" containerID="8f9b2f45cf254555dd48a3598d894fc58b2f165af492fd1ea1df73670f4093e9"
Sep 11 03:43:14.793445 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:43:14.793427    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 11 03:43:14.797014 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:43:14.797000    2579 scope.go:117] "RemoveContainer" containerID="c34f844b4ee0b9f4eb00ca54e6fbfe8f911ae7f338602dee41d5cfc502683450"
Sep 11 03:43:14.803765 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:43:14.803752    2579 scope.go:117] "RemoveContainer" containerID="a5fb41f056f6486570ae6451b1ccc687e68091d1258fa434fbe505339d967aa9"
Sep 11 03:43:14.811226 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:43:14.811174    2579 scope.go:117] "RemoveContainer" containerID="9ee65c097f8e13ebe68364b4ae0058a6543ca1526d248f62d925296340f5364a"
Sep 11 03:43:14.818844 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:43:14.818823    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 11 03:43:14.819129 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:43:14.819108    2579 scope.go:117] "RemoveContainer" containerID="cc15a3445e053dd5656d9fc120f01c37afc3fa0cb6cdc7158a87f9389387200b"
Sep 11 03:43:14.819254 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:43:14.819240    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5208b5bb-0268-4ab8-8e3a-f873bda02436" containerName="kube-rbac-proxy-web"
Sep 11 03:43:14.819313 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:43:14.819255    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="5208b5bb-0268-4ab8-8e3a-f873bda02436" containerName="kube-rbac-proxy-web"
Sep 11 03:43:14.819313 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:43:14.819265    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5208b5bb-0268-4ab8-8e3a-f873bda02436" containerName="kube-rbac-proxy-thanos"
Sep 11 03:43:14.819313 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:43:14.819271    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="5208b5bb-0268-4ab8-8e3a-f873bda02436" containerName="kube-rbac-proxy-thanos"
Sep 11 03:43:14.819313 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:43:14.819278    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ca69377b-efa8-460f-9ea7-c4c297378e53" containerName="kube-rbac-proxy"
Sep 11 03:43:14.819313 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:43:14.819284    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="ca69377b-efa8-460f-9ea7-c4c297378e53" containerName="kube-rbac-proxy"
Sep 11 03:43:14.819313 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:43:14.819291    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5208b5bb-0268-4ab8-8e3a-f873bda02436" containerName="prometheus"
Sep 11 03:43:14.819313 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:43:14.819296    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="5208b5bb-0268-4ab8-8e3a-f873bda02436" containerName="prometheus"
Sep 11 03:43:14.819313 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:43:14.819303    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5208b5bb-0268-4ab8-8e3a-f873bda02436" containerName="config-reloader"
Sep 11 03:43:14.819313 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:43:14.819309    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="5208b5bb-0268-4ab8-8e3a-f873bda02436" containerName="config-reloader"
Sep 11 03:43:14.819313 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:43:14.819315    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ca69377b-efa8-460f-9ea7-c4c297378e53" containerName="alertmanager"
Sep 11 03:43:14.819653 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:43:14.819321    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="ca69377b-efa8-460f-9ea7-c4c297378e53" containerName="alertmanager"
Sep 11 03:43:14.819653 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:43:14.819330    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5208b5bb-0268-4ab8-8e3a-f873bda02436" containerName="thanos-sidecar"
Sep 11 03:43:14.819653 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:43:14.819334    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="5208b5bb-0268-4ab8-8e3a-f873bda02436" containerName="thanos-sidecar"
Sep 11 03:43:14.819653 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:43:14.819339    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5208b5bb-0268-4ab8-8e3a-f873bda02436" containerName="kube-rbac-proxy"
Sep 11 03:43:14.819653 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:43:14.819344    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="5208b5bb-0268-4ab8-8e3a-f873bda02436" containerName="kube-rbac-proxy"
Sep 11 03:43:14.819653 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:43:14.819352    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ca69377b-efa8-460f-9ea7-c4c297378e53" containerName="kube-rbac-proxy-metric"
Sep 11 03:43:14.819653 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:43:14.819356    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="ca69377b-efa8-460f-9ea7-c4c297378e53" containerName="kube-rbac-proxy-metric"
Sep 11 03:43:14.819653 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:43:14.819361    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ca69377b-efa8-460f-9ea7-c4c297378e53" containerName="prom-label-proxy"
Sep 11 03:43:14.819653 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:43:14.819366    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="ca69377b-efa8-460f-9ea7-c4c297378e53" containerName="prom-label-proxy"
Sep 11 03:43:14.819653 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:43:14.819374    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ca69377b-efa8-460f-9ea7-c4c297378e53" containerName="config-reloader"
Sep 11 03:43:14.819653 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:43:14.819381    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="ca69377b-efa8-460f-9ea7-c4c297378e53" containerName="config-reloader"
Sep 11 03:43:14.819653 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:43:14.819392    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ca69377b-efa8-460f-9ea7-c4c297378e53" containerName="init-config-reloader"
Sep 11 03:43:14.819653 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:43:14.819397    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="ca69377b-efa8-460f-9ea7-c4c297378e53" containerName="init-config-reloader"
Sep 11 03:43:14.819653 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:43:14.819403    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ca69377b-efa8-460f-9ea7-c4c297378e53" containerName="kube-rbac-proxy-web"
Sep 11 03:43:14.819653 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:43:14.819408    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="ca69377b-efa8-460f-9ea7-c4c297378e53" containerName="kube-rbac-proxy-web"
Sep 11 03:43:14.819653 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:43:14.819415    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5208b5bb-0268-4ab8-8e3a-f873bda02436" containerName="init-config-reloader"
Sep 11 03:43:14.819653 ip-10-0-141-105 kubenswrapper[2579]: E0911 03:43:14.819413    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cc15a3445e053dd5656d9fc120f01c37afc3fa0cb6cdc7158a87f9389387200b\": container with ID starting with cc15a3445e053dd5656d9fc120f01c37afc3fa0cb6cdc7158a87f9389387200b not found: ID does not exist" containerID="cc15a3445e053dd5656d9fc120f01c37afc3fa0cb6cdc7158a87f9389387200b"
Sep 11 03:43:14.819653 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:43:14.819442    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cc15a3445e053dd5656d9fc120f01c37afc3fa0cb6cdc7158a87f9389387200b"} err="failed to get container status \"cc15a3445e053dd5656d9fc120f01c37afc3fa0cb6cdc7158a87f9389387200b\": rpc error: code = NotFound desc = could not find container \"cc15a3445e053dd5656d9fc120f01c37afc3fa0cb6cdc7158a87f9389387200b\": container with ID starting with cc15a3445e053dd5656d9fc120f01c37afc3fa0cb6cdc7158a87f9389387200b not found: ID does not exist"
Sep 11 03:43:14.819653 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:43:14.819459    2579 scope.go:117] "RemoveContainer" containerID="21fe8a05bab9b83615a8293c808cba67ff07b045fa1ff8abb956d36b8a4548a6"
Sep 11 03:43:14.819653 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:43:14.819420    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="5208b5bb-0268-4ab8-8e3a-f873bda02436" containerName="init-config-reloader"
Sep 11 03:43:14.819653 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:43:14.819550    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="5208b5bb-0268-4ab8-8e3a-f873bda02436" containerName="prometheus"
Sep 11 03:43:14.819653 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:43:14.819564    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="5208b5bb-0268-4ab8-8e3a-f873bda02436" containerName="config-reloader"
Sep 11 03:43:14.819653 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:43:14.819574    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="5208b5bb-0268-4ab8-8e3a-f873bda02436" containerName="thanos-sidecar"
Sep 11 03:43:14.819653 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:43:14.819581    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="5208b5bb-0268-4ab8-8e3a-f873bda02436" containerName="kube-rbac-proxy-thanos"
Sep 11 03:43:14.819653 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:43:14.819589    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="ca69377b-efa8-460f-9ea7-c4c297378e53" containerName="kube-rbac-proxy"
Sep 11 03:43:14.819653 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:43:14.819598    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="ca69377b-efa8-460f-9ea7-c4c297378e53" containerName="kube-rbac-proxy-metric"
Sep 11 03:43:14.819653 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:43:14.819603    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="ca69377b-efa8-460f-9ea7-c4c297378e53" containerName="kube-rbac-proxy-web"
Sep 11 03:43:14.819653 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:43:14.819609    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="5208b5bb-0268-4ab8-8e3a-f873bda02436" containerName="kube-rbac-proxy-web"
Sep 11 03:43:14.819653 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:43:14.819614    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="ca69377b-efa8-460f-9ea7-c4c297378e53" containerName="alertmanager"
Sep 11 03:43:14.819653 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:43:14.819619    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="ca69377b-efa8-460f-9ea7-c4c297378e53" containerName="config-reloader"
Sep 11 03:43:14.819653 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:43:14.819624    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="ca69377b-efa8-460f-9ea7-c4c297378e53" containerName="prom-label-proxy"
Sep 11 03:43:14.819653 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:43:14.819629    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="5208b5bb-0268-4ab8-8e3a-f873bda02436" containerName="kube-rbac-proxy"
Sep 11 03:43:14.820712 ip-10-0-141-105 kubenswrapper[2579]: E0911 03:43:14.819721    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"21fe8a05bab9b83615a8293c808cba67ff07b045fa1ff8abb956d36b8a4548a6\": container with ID starting with 21fe8a05bab9b83615a8293c808cba67ff07b045fa1ff8abb956d36b8a4548a6 not found: ID does not exist" containerID="21fe8a05bab9b83615a8293c808cba67ff07b045fa1ff8abb956d36b8a4548a6"
Sep 11 03:43:14.820712 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:43:14.819741    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"21fe8a05bab9b83615a8293c808cba67ff07b045fa1ff8abb956d36b8a4548a6"} err="failed to get container status \"21fe8a05bab9b83615a8293c808cba67ff07b045fa1ff8abb956d36b8a4548a6\": rpc error: code = NotFound desc = could not find container \"21fe8a05bab9b83615a8293c808cba67ff07b045fa1ff8abb956d36b8a4548a6\": container with ID starting with 21fe8a05bab9b83615a8293c808cba67ff07b045fa1ff8abb956d36b8a4548a6 not found: ID does not exist"
Sep 11 03:43:14.820712 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:43:14.819754    2579 scope.go:117] "RemoveContainer" containerID="4562ba7ff19703d39b6f22517c8811def29f266ad7974c249138ed86e03c7715"
Sep 11 03:43:14.820712 ip-10-0-141-105 kubenswrapper[2579]: E0911 03:43:14.819973    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4562ba7ff19703d39b6f22517c8811def29f266ad7974c249138ed86e03c7715\": container with ID starting with 4562ba7ff19703d39b6f22517c8811def29f266ad7974c249138ed86e03c7715 not found: ID does not exist" containerID="4562ba7ff19703d39b6f22517c8811def29f266ad7974c249138ed86e03c7715"
Sep 11 03:43:14.820712 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:43:14.819995    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4562ba7ff19703d39b6f22517c8811def29f266ad7974c249138ed86e03c7715"} err="failed to get container status \"4562ba7ff19703d39b6f22517c8811def29f266ad7974c249138ed86e03c7715\": rpc error: code = NotFound desc = could not find container \"4562ba7ff19703d39b6f22517c8811def29f266ad7974c249138ed86e03c7715\": container with ID starting with 4562ba7ff19703d39b6f22517c8811def29f266ad7974c249138ed86e03c7715 not found: ID does not exist"
Sep 11 03:43:14.820712 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:43:14.820007    2579 scope.go:117] "RemoveContainer" containerID="8f9b2f45cf254555dd48a3598d894fc58b2f165af492fd1ea1df73670f4093e9"
Sep 11 03:43:14.820712 ip-10-0-141-105 kubenswrapper[2579]: E0911 03:43:14.820286    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8f9b2f45cf254555dd48a3598d894fc58b2f165af492fd1ea1df73670f4093e9\": container with ID starting with 8f9b2f45cf254555dd48a3598d894fc58b2f165af492fd1ea1df73670f4093e9 not found: ID does not exist" containerID="8f9b2f45cf254555dd48a3598d894fc58b2f165af492fd1ea1df73670f4093e9"
Sep 11 03:43:14.820712 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:43:14.820301    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8f9b2f45cf254555dd48a3598d894fc58b2f165af492fd1ea1df73670f4093e9"} err="failed to get container status \"8f9b2f45cf254555dd48a3598d894fc58b2f165af492fd1ea1df73670f4093e9\": rpc error: code = NotFound desc = could not find container \"8f9b2f45cf254555dd48a3598d894fc58b2f165af492fd1ea1df73670f4093e9\": container with ID starting with 8f9b2f45cf254555dd48a3598d894fc58b2f165af492fd1ea1df73670f4093e9 not found: ID does not exist"
Sep 11 03:43:14.820712 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:43:14.820321    2579 scope.go:117] "RemoveContainer" containerID="c34f844b4ee0b9f4eb00ca54e6fbfe8f911ae7f338602dee41d5cfc502683450"
Sep 11 03:43:14.820712 ip-10-0-141-105 kubenswrapper[2579]: E0911 03:43:14.820566    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c34f844b4ee0b9f4eb00ca54e6fbfe8f911ae7f338602dee41d5cfc502683450\": container with ID starting with c34f844b4ee0b9f4eb00ca54e6fbfe8f911ae7f338602dee41d5cfc502683450 not found: ID does not exist" containerID="c34f844b4ee0b9f4eb00ca54e6fbfe8f911ae7f338602dee41d5cfc502683450"
Sep 11 03:43:14.820712 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:43:14.820587    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c34f844b4ee0b9f4eb00ca54e6fbfe8f911ae7f338602dee41d5cfc502683450"} err="failed to get container status \"c34f844b4ee0b9f4eb00ca54e6fbfe8f911ae7f338602dee41d5cfc502683450\": rpc error: code = NotFound desc = could not find container \"c34f844b4ee0b9f4eb00ca54e6fbfe8f911ae7f338602dee41d5cfc502683450\": container with ID starting with c34f844b4ee0b9f4eb00ca54e6fbfe8f911ae7f338602dee41d5cfc502683450 not found: ID does not exist"
Sep 11 03:43:14.820712 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:43:14.820602    2579 scope.go:117] "RemoveContainer" containerID="a5fb41f056f6486570ae6451b1ccc687e68091d1258fa434fbe505339d967aa9"
Sep 11 03:43:14.821292 ip-10-0-141-105 kubenswrapper[2579]: E0911 03:43:14.820861    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a5fb41f056f6486570ae6451b1ccc687e68091d1258fa434fbe505339d967aa9\": container with ID starting with a5fb41f056f6486570ae6451b1ccc687e68091d1258fa434fbe505339d967aa9 not found: ID does not exist" containerID="a5fb41f056f6486570ae6451b1ccc687e68091d1258fa434fbe505339d967aa9"
Sep 11 03:43:14.821292 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:43:14.820877    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a5fb41f056f6486570ae6451b1ccc687e68091d1258fa434fbe505339d967aa9"} err="failed to get container status \"a5fb41f056f6486570ae6451b1ccc687e68091d1258fa434fbe505339d967aa9\": rpc error: code = NotFound desc = could not find container \"a5fb41f056f6486570ae6451b1ccc687e68091d1258fa434fbe505339d967aa9\": container with ID starting with a5fb41f056f6486570ae6451b1ccc687e68091d1258fa434fbe505339d967aa9 not found: ID does not exist"
Sep 11 03:43:14.821292 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:43:14.820892    2579 scope.go:117] "RemoveContainer" containerID="9ee65c097f8e13ebe68364b4ae0058a6543ca1526d248f62d925296340f5364a"
Sep 11 03:43:14.821292 ip-10-0-141-105 kubenswrapper[2579]: E0911 03:43:14.821197    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9ee65c097f8e13ebe68364b4ae0058a6543ca1526d248f62d925296340f5364a\": container with ID starting with 9ee65c097f8e13ebe68364b4ae0058a6543ca1526d248f62d925296340f5364a not found: ID does not exist" containerID="9ee65c097f8e13ebe68364b4ae0058a6543ca1526d248f62d925296340f5364a"
Sep 11 03:43:14.821292 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:43:14.821221    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9ee65c097f8e13ebe68364b4ae0058a6543ca1526d248f62d925296340f5364a"} err="failed to get container status \"9ee65c097f8e13ebe68364b4ae0058a6543ca1526d248f62d925296340f5364a\": rpc error: code = NotFound desc = could not find container \"9ee65c097f8e13ebe68364b4ae0058a6543ca1526d248f62d925296340f5364a\": container with ID starting with 9ee65c097f8e13ebe68364b4ae0058a6543ca1526d248f62d925296340f5364a not found: ID does not exist"
Sep 11 03:43:14.821292 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:43:14.821238    2579 scope.go:117] "RemoveContainer" containerID="cc15a3445e053dd5656d9fc120f01c37afc3fa0cb6cdc7158a87f9389387200b"
Sep 11 03:43:14.821552 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:43:14.821496    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cc15a3445e053dd5656d9fc120f01c37afc3fa0cb6cdc7158a87f9389387200b"} err="failed to get container status \"cc15a3445e053dd5656d9fc120f01c37afc3fa0cb6cdc7158a87f9389387200b\": rpc error: code = NotFound desc = could not find container \"cc15a3445e053dd5656d9fc120f01c37afc3fa0cb6cdc7158a87f9389387200b\": container with ID starting with cc15a3445e053dd5656d9fc120f01c37afc3fa0cb6cdc7158a87f9389387200b not found: ID does not exist"
Sep 11 03:43:14.821552 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:43:14.821519    2579 scope.go:117] "RemoveContainer" containerID="21fe8a05bab9b83615a8293c808cba67ff07b045fa1ff8abb956d36b8a4548a6"
Sep 11 03:43:14.825294 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:43:14.821742    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"21fe8a05bab9b83615a8293c808cba67ff07b045fa1ff8abb956d36b8a4548a6"} err="failed to get container status \"21fe8a05bab9b83615a8293c808cba67ff07b045fa1ff8abb956d36b8a4548a6\": rpc error: code = NotFound desc = could not find container \"21fe8a05bab9b83615a8293c808cba67ff07b045fa1ff8abb956d36b8a4548a6\": container with ID starting with 21fe8a05bab9b83615a8293c808cba67ff07b045fa1ff8abb956d36b8a4548a6 not found: ID does not exist"
Sep 11 03:43:14.825294 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:43:14.821764    2579 scope.go:117] "RemoveContainer" containerID="4562ba7ff19703d39b6f22517c8811def29f266ad7974c249138ed86e03c7715"
Sep 11 03:43:14.825294 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:43:14.821979    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4562ba7ff19703d39b6f22517c8811def29f266ad7974c249138ed86e03c7715"} err="failed to get container status \"4562ba7ff19703d39b6f22517c8811def29f266ad7974c249138ed86e03c7715\": rpc error: code = NotFound desc = could not find container \"4562ba7ff19703d39b6f22517c8811def29f266ad7974c249138ed86e03c7715\": container with ID starting with 4562ba7ff19703d39b6f22517c8811def29f266ad7974c249138ed86e03c7715 not found: ID does not exist"
Sep 11 03:43:14.825294 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:43:14.821997    2579 scope.go:117] "RemoveContainer" containerID="8f9b2f45cf254555dd48a3598d894fc58b2f165af492fd1ea1df73670f4093e9"
Sep 11 03:43:14.825294 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:43:14.822237    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8f9b2f45cf254555dd48a3598d894fc58b2f165af492fd1ea1df73670f4093e9"} err="failed to get container status \"8f9b2f45cf254555dd48a3598d894fc58b2f165af492fd1ea1df73670f4093e9\": rpc error: code = NotFound desc = could not find container \"8f9b2f45cf254555dd48a3598d894fc58b2f165af492fd1ea1df73670f4093e9\": container with ID starting with 8f9b2f45cf254555dd48a3598d894fc58b2f165af492fd1ea1df73670f4093e9 not found: ID does not exist"
Sep 11 03:43:14.825294 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:43:14.822259    2579 scope.go:117] "RemoveContainer" containerID="c34f844b4ee0b9f4eb00ca54e6fbfe8f911ae7f338602dee41d5cfc502683450"
Sep 11 03:43:14.825294 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:43:14.822477    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c34f844b4ee0b9f4eb00ca54e6fbfe8f911ae7f338602dee41d5cfc502683450"} err="failed to get container status \"c34f844b4ee0b9f4eb00ca54e6fbfe8f911ae7f338602dee41d5cfc502683450\": rpc error: code = NotFound desc = could not find container \"c34f844b4ee0b9f4eb00ca54e6fbfe8f911ae7f338602dee41d5cfc502683450\": container with ID starting with c34f844b4ee0b9f4eb00ca54e6fbfe8f911ae7f338602dee41d5cfc502683450 not found: ID does not exist"
Sep 11 03:43:14.825294 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:43:14.822496    2579 scope.go:117] "RemoveContainer" containerID="a5fb41f056f6486570ae6451b1ccc687e68091d1258fa434fbe505339d967aa9"
Sep 11 03:43:14.825294 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:43:14.822696    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a5fb41f056f6486570ae6451b1ccc687e68091d1258fa434fbe505339d967aa9"} err="failed to get container status \"a5fb41f056f6486570ae6451b1ccc687e68091d1258fa434fbe505339d967aa9\": rpc error: code = NotFound desc = could not find container \"a5fb41f056f6486570ae6451b1ccc687e68091d1258fa434fbe505339d967aa9\": container with ID starting with a5fb41f056f6486570ae6451b1ccc687e68091d1258fa434fbe505339d967aa9 not found: ID does not exist"
Sep 11 03:43:14.825294 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:43:14.822714    2579 scope.go:117] "RemoveContainer" containerID="9ee65c097f8e13ebe68364b4ae0058a6543ca1526d248f62d925296340f5364a"
Sep 11 03:43:14.825294 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:43:14.822915    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9ee65c097f8e13ebe68364b4ae0058a6543ca1526d248f62d925296340f5364a"} err="failed to get container status \"9ee65c097f8e13ebe68364b4ae0058a6543ca1526d248f62d925296340f5364a\": rpc error: code = NotFound desc = could not find container \"9ee65c097f8e13ebe68364b4ae0058a6543ca1526d248f62d925296340f5364a\": container with ID starting with 9ee65c097f8e13ebe68364b4ae0058a6543ca1526d248f62d925296340f5364a not found: ID does not exist"
Sep 11 03:43:14.825294 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:43:14.822934    2579 scope.go:117] "RemoveContainer" containerID="cc15a3445e053dd5656d9fc120f01c37afc3fa0cb6cdc7158a87f9389387200b"
Sep 11 03:43:14.825294 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:43:14.823237    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cc15a3445e053dd5656d9fc120f01c37afc3fa0cb6cdc7158a87f9389387200b"} err="failed to get container status \"cc15a3445e053dd5656d9fc120f01c37afc3fa0cb6cdc7158a87f9389387200b\": rpc error: code = NotFound desc = could not find container \"cc15a3445e053dd5656d9fc120f01c37afc3fa0cb6cdc7158a87f9389387200b\": container with ID starting with cc15a3445e053dd5656d9fc120f01c37afc3fa0cb6cdc7158a87f9389387200b not found: ID does not exist"
Sep 11 03:43:14.825294 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:43:14.823259    2579 scope.go:117] "RemoveContainer" containerID="21fe8a05bab9b83615a8293c808cba67ff07b045fa1ff8abb956d36b8a4548a6"
Sep 11 03:43:14.825294 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:43:14.823472    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"21fe8a05bab9b83615a8293c808cba67ff07b045fa1ff8abb956d36b8a4548a6"} err="failed to get container status \"21fe8a05bab9b83615a8293c808cba67ff07b045fa1ff8abb956d36b8a4548a6\": rpc error: code = NotFound desc = could not find container \"21fe8a05bab9b83615a8293c808cba67ff07b045fa1ff8abb956d36b8a4548a6\": container with ID starting with 21fe8a05bab9b83615a8293c808cba67ff07b045fa1ff8abb956d36b8a4548a6 not found: ID does not exist"
Sep 11 03:43:14.825294 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:43:14.823489    2579 scope.go:117] "RemoveContainer" containerID="4562ba7ff19703d39b6f22517c8811def29f266ad7974c249138ed86e03c7715"
Sep 11 03:43:14.825294 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:43:14.823685    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4562ba7ff19703d39b6f22517c8811def29f266ad7974c249138ed86e03c7715"} err="failed to get container status \"4562ba7ff19703d39b6f22517c8811def29f266ad7974c249138ed86e03c7715\": rpc error: code = NotFound desc = could not find container \"4562ba7ff19703d39b6f22517c8811def29f266ad7974c249138ed86e03c7715\": container with ID starting with 4562ba7ff19703d39b6f22517c8811def29f266ad7974c249138ed86e03c7715 not found: ID does not exist"
Sep 11 03:43:14.826007 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:43:14.823701    2579 scope.go:117] "RemoveContainer" containerID="8f9b2f45cf254555dd48a3598d894fc58b2f165af492fd1ea1df73670f4093e9"
Sep 11 03:43:14.826007 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:43:14.823897    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8f9b2f45cf254555dd48a3598d894fc58b2f165af492fd1ea1df73670f4093e9"} err="failed to get container status \"8f9b2f45cf254555dd48a3598d894fc58b2f165af492fd1ea1df73670f4093e9\": rpc error: code = NotFound desc = could not find container \"8f9b2f45cf254555dd48a3598d894fc58b2f165af492fd1ea1df73670f4093e9\": container with ID starting with 8f9b2f45cf254555dd48a3598d894fc58b2f165af492fd1ea1df73670f4093e9 not found: ID does not exist"
Sep 11 03:43:14.826007 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:43:14.823914    2579 scope.go:117] "RemoveContainer" containerID="c34f844b4ee0b9f4eb00ca54e6fbfe8f911ae7f338602dee41d5cfc502683450"
Sep 11 03:43:14.826007 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:43:14.824146    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c34f844b4ee0b9f4eb00ca54e6fbfe8f911ae7f338602dee41d5cfc502683450"} err="failed to get container status \"c34f844b4ee0b9f4eb00ca54e6fbfe8f911ae7f338602dee41d5cfc502683450\": rpc error: code = NotFound desc = could not find container \"c34f844b4ee0b9f4eb00ca54e6fbfe8f911ae7f338602dee41d5cfc502683450\": container with ID starting with c34f844b4ee0b9f4eb00ca54e6fbfe8f911ae7f338602dee41d5cfc502683450 not found: ID does not exist"
Sep 11 03:43:14.826007 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:43:14.824164    2579 scope.go:117] "RemoveContainer" containerID="a5fb41f056f6486570ae6451b1ccc687e68091d1258fa434fbe505339d967aa9"
Sep 11 03:43:14.826007 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:43:14.824433    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a5fb41f056f6486570ae6451b1ccc687e68091d1258fa434fbe505339d967aa9"} err="failed to get container status \"a5fb41f056f6486570ae6451b1ccc687e68091d1258fa434fbe505339d967aa9\": rpc error: code = NotFound desc = could not find container \"a5fb41f056f6486570ae6451b1ccc687e68091d1258fa434fbe505339d967aa9\": container with ID starting with a5fb41f056f6486570ae6451b1ccc687e68091d1258fa434fbe505339d967aa9 not found: ID does not exist"
Sep 11 03:43:14.826007 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:43:14.824454    2579 scope.go:117] "RemoveContainer" containerID="9ee65c097f8e13ebe68364b4ae0058a6543ca1526d248f62d925296340f5364a"
Sep 11 03:43:14.826007 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:43:14.824668    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9ee65c097f8e13ebe68364b4ae0058a6543ca1526d248f62d925296340f5364a"} err="failed to get container status \"9ee65c097f8e13ebe68364b4ae0058a6543ca1526d248f62d925296340f5364a\": rpc error: code = NotFound desc = could not find container \"9ee65c097f8e13ebe68364b4ae0058a6543ca1526d248f62d925296340f5364a\": container with ID starting with 9ee65c097f8e13ebe68364b4ae0058a6543ca1526d248f62d925296340f5364a not found: ID does not exist"
Sep 11 03:43:14.826007 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:43:14.824686    2579 scope.go:117] "RemoveContainer" containerID="cc15a3445e053dd5656d9fc120f01c37afc3fa0cb6cdc7158a87f9389387200b"
Sep 11 03:43:14.826007 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:43:14.824889    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cc15a3445e053dd5656d9fc120f01c37afc3fa0cb6cdc7158a87f9389387200b"} err="failed to get container status \"cc15a3445e053dd5656d9fc120f01c37afc3fa0cb6cdc7158a87f9389387200b\": rpc error: code = NotFound desc = could not find container \"cc15a3445e053dd5656d9fc120f01c37afc3fa0cb6cdc7158a87f9389387200b\": container with ID starting with cc15a3445e053dd5656d9fc120f01c37afc3fa0cb6cdc7158a87f9389387200b not found: ID does not exist"
Sep 11 03:43:14.826007 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:43:14.824906    2579 scope.go:117] "RemoveContainer" containerID="21fe8a05bab9b83615a8293c808cba67ff07b045fa1ff8abb956d36b8a4548a6"
Sep 11 03:43:14.826007 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:43:14.825116    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"21fe8a05bab9b83615a8293c808cba67ff07b045fa1ff8abb956d36b8a4548a6"} err="failed to get container status \"21fe8a05bab9b83615a8293c808cba67ff07b045fa1ff8abb956d36b8a4548a6\": rpc error: code = NotFound desc = could not find container \"21fe8a05bab9b83615a8293c808cba67ff07b045fa1ff8abb956d36b8a4548a6\": container with ID starting with 21fe8a05bab9b83615a8293c808cba67ff07b045fa1ff8abb956d36b8a4548a6 not found: ID does not exist"
Sep 11 03:43:14.826007 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:43:14.825134    2579 scope.go:117] "RemoveContainer" containerID="4562ba7ff19703d39b6f22517c8811def29f266ad7974c249138ed86e03c7715"
Sep 11 03:43:14.826007 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:43:14.825344    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4562ba7ff19703d39b6f22517c8811def29f266ad7974c249138ed86e03c7715"} err="failed to get container status \"4562ba7ff19703d39b6f22517c8811def29f266ad7974c249138ed86e03c7715\": rpc error: code = NotFound desc = could not find container \"4562ba7ff19703d39b6f22517c8811def29f266ad7974c249138ed86e03c7715\": container with ID starting with 4562ba7ff19703d39b6f22517c8811def29f266ad7974c249138ed86e03c7715 not found: ID does not exist"
Sep 11 03:43:14.826007 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:43:14.825366    2579 scope.go:117] "RemoveContainer" containerID="8f9b2f45cf254555dd48a3598d894fc58b2f165af492fd1ea1df73670f4093e9"
Sep 11 03:43:14.826007 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:43:14.825593    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8f9b2f45cf254555dd48a3598d894fc58b2f165af492fd1ea1df73670f4093e9"} err="failed to get container status \"8f9b2f45cf254555dd48a3598d894fc58b2f165af492fd1ea1df73670f4093e9\": rpc error: code = NotFound desc = could not find container \"8f9b2f45cf254555dd48a3598d894fc58b2f165af492fd1ea1df73670f4093e9\": container with ID starting with 8f9b2f45cf254555dd48a3598d894fc58b2f165af492fd1ea1df73670f4093e9 not found: ID does not exist"
Sep 11 03:43:14.826007 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:43:14.825607    2579 scope.go:117] "RemoveContainer" containerID="c34f844b4ee0b9f4eb00ca54e6fbfe8f911ae7f338602dee41d5cfc502683450"
Sep 11 03:43:14.826665 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:43:14.825812    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c34f844b4ee0b9f4eb00ca54e6fbfe8f911ae7f338602dee41d5cfc502683450"} err="failed to get container status \"c34f844b4ee0b9f4eb00ca54e6fbfe8f911ae7f338602dee41d5cfc502683450\": rpc error: code = NotFound desc = could not find container \"c34f844b4ee0b9f4eb00ca54e6fbfe8f911ae7f338602dee41d5cfc502683450\": container with ID starting with c34f844b4ee0b9f4eb00ca54e6fbfe8f911ae7f338602dee41d5cfc502683450 not found: ID does not exist"
Sep 11 03:43:14.826665 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:43:14.825826    2579 scope.go:117] "RemoveContainer" containerID="a5fb41f056f6486570ae6451b1ccc687e68091d1258fa434fbe505339d967aa9"
Sep 11 03:43:14.826665 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:43:14.826033    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a5fb41f056f6486570ae6451b1ccc687e68091d1258fa434fbe505339d967aa9"} err="failed to get container status \"a5fb41f056f6486570ae6451b1ccc687e68091d1258fa434fbe505339d967aa9\": rpc error: code = NotFound desc = could not find container \"a5fb41f056f6486570ae6451b1ccc687e68091d1258fa434fbe505339d967aa9\": container with ID starting with a5fb41f056f6486570ae6451b1ccc687e68091d1258fa434fbe505339d967aa9 not found: ID does not exist"
Sep 11 03:43:14.826665 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:43:14.826051    2579 scope.go:117] "RemoveContainer" containerID="9ee65c097f8e13ebe68364b4ae0058a6543ca1526d248f62d925296340f5364a"
Sep 11 03:43:14.826665 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:43:14.826379    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9ee65c097f8e13ebe68364b4ae0058a6543ca1526d248f62d925296340f5364a"} err="failed to get container status \"9ee65c097f8e13ebe68364b4ae0058a6543ca1526d248f62d925296340f5364a\": rpc error: code = NotFound desc = could not find container \"9ee65c097f8e13ebe68364b4ae0058a6543ca1526d248f62d925296340f5364a\": container with ID starting with 9ee65c097f8e13ebe68364b4ae0058a6543ca1526d248f62d925296340f5364a not found: ID does not exist"
Sep 11 03:43:14.826665 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:43:14.826397    2579 scope.go:117] "RemoveContainer" containerID="cc15a3445e053dd5656d9fc120f01c37afc3fa0cb6cdc7158a87f9389387200b"
Sep 11 03:43:14.826665 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:43:14.826626    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cc15a3445e053dd5656d9fc120f01c37afc3fa0cb6cdc7158a87f9389387200b"} err="failed to get container status \"cc15a3445e053dd5656d9fc120f01c37afc3fa0cb6cdc7158a87f9389387200b\": rpc error: code = NotFound desc = could not find container \"cc15a3445e053dd5656d9fc120f01c37afc3fa0cb6cdc7158a87f9389387200b\": container with ID starting with cc15a3445e053dd5656d9fc120f01c37afc3fa0cb6cdc7158a87f9389387200b not found: ID does not exist"
Sep 11 03:43:14.826665 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:43:14.826645    2579 scope.go:117] "RemoveContainer" containerID="21fe8a05bab9b83615a8293c808cba67ff07b045fa1ff8abb956d36b8a4548a6"
Sep 11 03:43:14.826937 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:43:14.826880    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"21fe8a05bab9b83615a8293c808cba67ff07b045fa1ff8abb956d36b8a4548a6"} err="failed to get container status \"21fe8a05bab9b83615a8293c808cba67ff07b045fa1ff8abb956d36b8a4548a6\": rpc error: code = NotFound desc = could not find container \"21fe8a05bab9b83615a8293c808cba67ff07b045fa1ff8abb956d36b8a4548a6\": container with ID starting with 21fe8a05bab9b83615a8293c808cba67ff07b045fa1ff8abb956d36b8a4548a6 not found: ID does not exist"
Sep 11 03:43:14.826937 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:43:14.826898    2579 scope.go:117] "RemoveContainer" containerID="4562ba7ff19703d39b6f22517c8811def29f266ad7974c249138ed86e03c7715"
Sep 11 03:43:14.827131 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:43:14.827114    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4562ba7ff19703d39b6f22517c8811def29f266ad7974c249138ed86e03c7715"} err="failed to get container status \"4562ba7ff19703d39b6f22517c8811def29f266ad7974c249138ed86e03c7715\": rpc error: code = NotFound desc = could not find container \"4562ba7ff19703d39b6f22517c8811def29f266ad7974c249138ed86e03c7715\": container with ID starting with 4562ba7ff19703d39b6f22517c8811def29f266ad7974c249138ed86e03c7715 not found: ID does not exist"
Sep 11 03:43:14.827185 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:43:14.827131    2579 scope.go:117] "RemoveContainer" containerID="8f9b2f45cf254555dd48a3598d894fc58b2f165af492fd1ea1df73670f4093e9"
Sep 11 03:43:14.827380 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:43:14.827363    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8f9b2f45cf254555dd48a3598d894fc58b2f165af492fd1ea1df73670f4093e9"} err="failed to get container status \"8f9b2f45cf254555dd48a3598d894fc58b2f165af492fd1ea1df73670f4093e9\": rpc error: code = NotFound desc = could not find container \"8f9b2f45cf254555dd48a3598d894fc58b2f165af492fd1ea1df73670f4093e9\": container with ID starting with 8f9b2f45cf254555dd48a3598d894fc58b2f165af492fd1ea1df73670f4093e9 not found: ID does not exist"
Sep 11 03:43:14.827425 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:43:14.827382    2579 scope.go:117] "RemoveContainer" containerID="c34f844b4ee0b9f4eb00ca54e6fbfe8f911ae7f338602dee41d5cfc502683450"
Sep 11 03:43:14.827615 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:43:14.827594    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c34f844b4ee0b9f4eb00ca54e6fbfe8f911ae7f338602dee41d5cfc502683450"} err="failed to get container status \"c34f844b4ee0b9f4eb00ca54e6fbfe8f911ae7f338602dee41d5cfc502683450\": rpc error: code = NotFound desc = could not find container \"c34f844b4ee0b9f4eb00ca54e6fbfe8f911ae7f338602dee41d5cfc502683450\": container with ID starting with c34f844b4ee0b9f4eb00ca54e6fbfe8f911ae7f338602dee41d5cfc502683450 not found: ID does not exist"
Sep 11 03:43:14.827663 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:43:14.827616    2579 scope.go:117] "RemoveContainer" containerID="a5fb41f056f6486570ae6451b1ccc687e68091d1258fa434fbe505339d967aa9"
Sep 11 03:43:14.827852 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:43:14.827832    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a5fb41f056f6486570ae6451b1ccc687e68091d1258fa434fbe505339d967aa9"} err="failed to get container status \"a5fb41f056f6486570ae6451b1ccc687e68091d1258fa434fbe505339d967aa9\": rpc error: code = NotFound desc = could not find container \"a5fb41f056f6486570ae6451b1ccc687e68091d1258fa434fbe505339d967aa9\": container with ID starting with a5fb41f056f6486570ae6451b1ccc687e68091d1258fa434fbe505339d967aa9 not found: ID does not exist"
Sep 11 03:43:14.827909 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:43:14.827853    2579 scope.go:117] "RemoveContainer" containerID="9ee65c097f8e13ebe68364b4ae0058a6543ca1526d248f62d925296340f5364a"
Sep 11 03:43:14.828096 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:43:14.828066    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9ee65c097f8e13ebe68364b4ae0058a6543ca1526d248f62d925296340f5364a"} err="failed to get container status \"9ee65c097f8e13ebe68364b4ae0058a6543ca1526d248f62d925296340f5364a\": rpc error: code = NotFound desc = could not find container \"9ee65c097f8e13ebe68364b4ae0058a6543ca1526d248f62d925296340f5364a\": container with ID starting with 9ee65c097f8e13ebe68364b4ae0058a6543ca1526d248f62d925296340f5364a not found: ID does not exist"
Sep 11 03:43:14.828153 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:43:14.828105    2579 scope.go:117] "RemoveContainer" containerID="cc15a3445e053dd5656d9fc120f01c37afc3fa0cb6cdc7158a87f9389387200b"
Sep 11 03:43:14.828332 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:43:14.828307    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cc15a3445e053dd5656d9fc120f01c37afc3fa0cb6cdc7158a87f9389387200b"} err="failed to get container status \"cc15a3445e053dd5656d9fc120f01c37afc3fa0cb6cdc7158a87f9389387200b\": rpc error: code = NotFound desc = could not find container \"cc15a3445e053dd5656d9fc120f01c37afc3fa0cb6cdc7158a87f9389387200b\": container with ID starting with cc15a3445e053dd5656d9fc120f01c37afc3fa0cb6cdc7158a87f9389387200b not found: ID does not exist"
Sep 11 03:43:14.828395 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:43:14.828333    2579 scope.go:117] "RemoveContainer" containerID="21fe8a05bab9b83615a8293c808cba67ff07b045fa1ff8abb956d36b8a4548a6"
Sep 11 03:43:14.828566 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:43:14.828549    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 03:43:14.828636 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:43:14.828592    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"21fe8a05bab9b83615a8293c808cba67ff07b045fa1ff8abb956d36b8a4548a6"} err="failed to get container status \"21fe8a05bab9b83615a8293c808cba67ff07b045fa1ff8abb956d36b8a4548a6\": rpc error: code = NotFound desc = could not find container \"21fe8a05bab9b83615a8293c808cba67ff07b045fa1ff8abb956d36b8a4548a6\": container with ID starting with 21fe8a05bab9b83615a8293c808cba67ff07b045fa1ff8abb956d36b8a4548a6 not found: ID does not exist"
Sep 11 03:43:14.828636 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:43:14.828607    2579 scope.go:117] "RemoveContainer" containerID="4562ba7ff19703d39b6f22517c8811def29f266ad7974c249138ed86e03c7715"
Sep 11 03:43:14.829148 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:43:14.829109    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4562ba7ff19703d39b6f22517c8811def29f266ad7974c249138ed86e03c7715"} err="failed to get container status \"4562ba7ff19703d39b6f22517c8811def29f266ad7974c249138ed86e03c7715\": rpc error: code = NotFound desc = could not find container \"4562ba7ff19703d39b6f22517c8811def29f266ad7974c249138ed86e03c7715\": container with ID starting with 4562ba7ff19703d39b6f22517c8811def29f266ad7974c249138ed86e03c7715 not found: ID does not exist"
Sep 11 03:43:14.829241 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:43:14.829151    2579 scope.go:117] "RemoveContainer" containerID="8f9b2f45cf254555dd48a3598d894fc58b2f165af492fd1ea1df73670f4093e9"
Sep 11 03:43:14.829467 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:43:14.829450    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8f9b2f45cf254555dd48a3598d894fc58b2f165af492fd1ea1df73670f4093e9"} err="failed to get container status \"8f9b2f45cf254555dd48a3598d894fc58b2f165af492fd1ea1df73670f4093e9\": rpc error: code = NotFound desc = could not find container \"8f9b2f45cf254555dd48a3598d894fc58b2f165af492fd1ea1df73670f4093e9\": container with ID starting with 8f9b2f45cf254555dd48a3598d894fc58b2f165af492fd1ea1df73670f4093e9 not found: ID does not exist"
Sep 11 03:43:14.829536 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:43:14.829469    2579 scope.go:117] "RemoveContainer" containerID="c34f844b4ee0b9f4eb00ca54e6fbfe8f911ae7f338602dee41d5cfc502683450"
Sep 11 03:43:14.829785 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:43:14.829761    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c34f844b4ee0b9f4eb00ca54e6fbfe8f911ae7f338602dee41d5cfc502683450"} err="failed to get container status \"c34f844b4ee0b9f4eb00ca54e6fbfe8f911ae7f338602dee41d5cfc502683450\": rpc error: code = NotFound desc = could not find container \"c34f844b4ee0b9f4eb00ca54e6fbfe8f911ae7f338602dee41d5cfc502683450\": container with ID starting with c34f844b4ee0b9f4eb00ca54e6fbfe8f911ae7f338602dee41d5cfc502683450 not found: ID does not exist"
Sep 11 03:43:14.829785 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:43:14.829784    2579 scope.go:117] "RemoveContainer" containerID="a5fb41f056f6486570ae6451b1ccc687e68091d1258fa434fbe505339d967aa9"
Sep 11 03:43:14.830077 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:43:14.830058    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a5fb41f056f6486570ae6451b1ccc687e68091d1258fa434fbe505339d967aa9"} err="failed to get container status \"a5fb41f056f6486570ae6451b1ccc687e68091d1258fa434fbe505339d967aa9\": rpc error: code = NotFound desc = could not find container \"a5fb41f056f6486570ae6451b1ccc687e68091d1258fa434fbe505339d967aa9\": container with ID starting with a5fb41f056f6486570ae6451b1ccc687e68091d1258fa434fbe505339d967aa9 not found: ID does not exist"
Sep 11 03:43:14.830077 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:43:14.830092    2579 scope.go:117] "RemoveContainer" containerID="9ee65c097f8e13ebe68364b4ae0058a6543ca1526d248f62d925296340f5364a"
Sep 11 03:43:14.830388 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:43:14.830370    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9ee65c097f8e13ebe68364b4ae0058a6543ca1526d248f62d925296340f5364a"} err="failed to get container status \"9ee65c097f8e13ebe68364b4ae0058a6543ca1526d248f62d925296340f5364a\": rpc error: code = NotFound desc = could not find container \"9ee65c097f8e13ebe68364b4ae0058a6543ca1526d248f62d925296340f5364a\": container with ID starting with 9ee65c097f8e13ebe68364b4ae0058a6543ca1526d248f62d925296340f5364a not found: ID does not exist"
Sep 11 03:43:14.831254 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:43:14.831237    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\""
Sep 11 03:43:14.831335 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:43:14.831279    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\""
Sep 11 03:43:14.831335 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:43:14.831285    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\""
Sep 11 03:43:14.831566 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:43:14.831446    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-fesspo8lkis9q\""
Sep 11 03:43:14.831566 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:43:14.831472    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\""
Sep 11 03:43:14.831566 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:43:14.831475    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\""
Sep 11 03:43:14.831764 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:43:14.831657    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\""
Sep 11 03:43:14.831764 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:43:14.831718    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\""
Sep 11 03:43:14.831764 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:43:14.831720    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\""
Sep 11 03:43:14.831892 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:43:14.831753    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\""
Sep 11 03:43:14.831892 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:43:14.831827    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-6qqj7\""
Sep 11 03:43:14.831984 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:43:14.831968    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\""
Sep 11 03:43:14.832381 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:43:14.832352    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\""
Sep 11 03:43:14.832578 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:43:14.832562    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\""
Sep 11 03:43:14.833914 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:43:14.833896    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\""
Sep 11 03:43:14.838267 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:43:14.838240    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 11 03:43:14.838731 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:43:14.838712    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\""
Sep 11 03:43:14.852602 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:43:14.852577    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5208b5bb-0268-4ab8-8e3a-f873bda02436" path="/var/lib/kubelet/pods/5208b5bb-0268-4ab8-8e3a-f873bda02436/volumes"
Sep 11 03:43:14.919027 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:43:14.918994    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/eb03d7bc-abea-4d73-8833-1829d3d7f2d4-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"eb03d7bc-abea-4d73-8833-1829d3d7f2d4\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 03:43:14.919027 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:43:14.919027    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/eb03d7bc-abea-4d73-8833-1829d3d7f2d4-web-config\") pod \"prometheus-k8s-0\" (UID: \"eb03d7bc-abea-4d73-8833-1829d3d7f2d4\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 03:43:14.919478 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:43:14.919046    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5mdq2\" (UniqueName: \"kubernetes.io/projected/eb03d7bc-abea-4d73-8833-1829d3d7f2d4-kube-api-access-5mdq2\") pod \"prometheus-k8s-0\" (UID: \"eb03d7bc-abea-4d73-8833-1829d3d7f2d4\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 03:43:14.919478 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:43:14.919120    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/eb03d7bc-abea-4d73-8833-1829d3d7f2d4-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"eb03d7bc-abea-4d73-8833-1829d3d7f2d4\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 03:43:14.919478 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:43:14.919151    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/eb03d7bc-abea-4d73-8833-1829d3d7f2d4-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"eb03d7bc-abea-4d73-8833-1829d3d7f2d4\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 03:43:14.919478 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:43:14.919174    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/eb03d7bc-abea-4d73-8833-1829d3d7f2d4-config\") pod \"prometheus-k8s-0\" (UID: \"eb03d7bc-abea-4d73-8833-1829d3d7f2d4\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 03:43:14.919478 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:43:14.919192    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/eb03d7bc-abea-4d73-8833-1829d3d7f2d4-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"eb03d7bc-abea-4d73-8833-1829d3d7f2d4\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 03:43:14.919478 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:43:14.919212    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/eb03d7bc-abea-4d73-8833-1829d3d7f2d4-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"eb03d7bc-abea-4d73-8833-1829d3d7f2d4\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 03:43:14.919478 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:43:14.919231    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/eb03d7bc-abea-4d73-8833-1829d3d7f2d4-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"eb03d7bc-abea-4d73-8833-1829d3d7f2d4\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 03:43:14.919478 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:43:14.919246    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/eb03d7bc-abea-4d73-8833-1829d3d7f2d4-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"eb03d7bc-abea-4d73-8833-1829d3d7f2d4\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 03:43:14.919478 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:43:14.919266    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/eb03d7bc-abea-4d73-8833-1829d3d7f2d4-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"eb03d7bc-abea-4d73-8833-1829d3d7f2d4\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 03:43:14.919478 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:43:14.919291    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/eb03d7bc-abea-4d73-8833-1829d3d7f2d4-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"eb03d7bc-abea-4d73-8833-1829d3d7f2d4\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 03:43:14.919478 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:43:14.919307    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/eb03d7bc-abea-4d73-8833-1829d3d7f2d4-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"eb03d7bc-abea-4d73-8833-1829d3d7f2d4\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 03:43:14.919478 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:43:14.919326    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/eb03d7bc-abea-4d73-8833-1829d3d7f2d4-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"eb03d7bc-abea-4d73-8833-1829d3d7f2d4\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 03:43:14.919478 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:43:14.919388    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/eb03d7bc-abea-4d73-8833-1829d3d7f2d4-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"eb03d7bc-abea-4d73-8833-1829d3d7f2d4\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 03:43:14.919478 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:43:14.919411    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/eb03d7bc-abea-4d73-8833-1829d3d7f2d4-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"eb03d7bc-abea-4d73-8833-1829d3d7f2d4\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 03:43:14.919478 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:43:14.919429    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/eb03d7bc-abea-4d73-8833-1829d3d7f2d4-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"eb03d7bc-abea-4d73-8833-1829d3d7f2d4\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 03:43:14.919478 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:43:14.919459    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/eb03d7bc-abea-4d73-8833-1829d3d7f2d4-config-out\") pod \"prometheus-k8s-0\" (UID: \"eb03d7bc-abea-4d73-8833-1829d3d7f2d4\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 03:43:14.919995 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:43:14.919474    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/eb03d7bc-abea-4d73-8833-1829d3d7f2d4-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"eb03d7bc-abea-4d73-8833-1829d3d7f2d4\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 03:43:14.919995 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:43:14.919490    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/eb03d7bc-abea-4d73-8833-1829d3d7f2d4-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"eb03d7bc-abea-4d73-8833-1829d3d7f2d4\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 03:43:15.020502 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:43:15.020423    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/eb03d7bc-abea-4d73-8833-1829d3d7f2d4-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"eb03d7bc-abea-4d73-8833-1829d3d7f2d4\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 03:43:15.020502 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:43:15.020455    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/eb03d7bc-abea-4d73-8833-1829d3d7f2d4-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"eb03d7bc-abea-4d73-8833-1829d3d7f2d4\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 03:43:15.020502 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:43:15.020472    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/eb03d7bc-abea-4d73-8833-1829d3d7f2d4-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"eb03d7bc-abea-4d73-8833-1829d3d7f2d4\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 03:43:15.020502 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:43:15.020494    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/eb03d7bc-abea-4d73-8833-1829d3d7f2d4-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"eb03d7bc-abea-4d73-8833-1829d3d7f2d4\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 03:43:15.020790 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:43:15.020602    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/eb03d7bc-abea-4d73-8833-1829d3d7f2d4-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"eb03d7bc-abea-4d73-8833-1829d3d7f2d4\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 03:43:15.020790 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:43:15.020648    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/eb03d7bc-abea-4d73-8833-1829d3d7f2d4-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"eb03d7bc-abea-4d73-8833-1829d3d7f2d4\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 03:43:15.020790 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:43:15.020687    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/eb03d7bc-abea-4d73-8833-1829d3d7f2d4-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"eb03d7bc-abea-4d73-8833-1829d3d7f2d4\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 03:43:15.020790 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:43:15.020711    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/eb03d7bc-abea-4d73-8833-1829d3d7f2d4-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"eb03d7bc-abea-4d73-8833-1829d3d7f2d4\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 03:43:15.020790 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:43:15.020733    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/eb03d7bc-abea-4d73-8833-1829d3d7f2d4-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"eb03d7bc-abea-4d73-8833-1829d3d7f2d4\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 03:43:15.020790 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:43:15.020762    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/eb03d7bc-abea-4d73-8833-1829d3d7f2d4-config-out\") pod \"prometheus-k8s-0\" (UID: \"eb03d7bc-abea-4d73-8833-1829d3d7f2d4\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 03:43:15.020790 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:43:15.020787    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/eb03d7bc-abea-4d73-8833-1829d3d7f2d4-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"eb03d7bc-abea-4d73-8833-1829d3d7f2d4\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 03:43:15.021156 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:43:15.020817    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/eb03d7bc-abea-4d73-8833-1829d3d7f2d4-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"eb03d7bc-abea-4d73-8833-1829d3d7f2d4\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 03:43:15.021156 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:43:15.020859    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/eb03d7bc-abea-4d73-8833-1829d3d7f2d4-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"eb03d7bc-abea-4d73-8833-1829d3d7f2d4\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 03:43:15.021156 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:43:15.020877    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/eb03d7bc-abea-4d73-8833-1829d3d7f2d4-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"eb03d7bc-abea-4d73-8833-1829d3d7f2d4\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 03:43:15.021156 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:43:15.020886    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/eb03d7bc-abea-4d73-8833-1829d3d7f2d4-web-config\") pod \"prometheus-k8s-0\" (UID: \"eb03d7bc-abea-4d73-8833-1829d3d7f2d4\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 03:43:15.021156 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:43:15.020937    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5mdq2\" (UniqueName: \"kubernetes.io/projected/eb03d7bc-abea-4d73-8833-1829d3d7f2d4-kube-api-access-5mdq2\") pod \"prometheus-k8s-0\" (UID: \"eb03d7bc-abea-4d73-8833-1829d3d7f2d4\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 03:43:15.021156 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:43:15.020982    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/eb03d7bc-abea-4d73-8833-1829d3d7f2d4-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"eb03d7bc-abea-4d73-8833-1829d3d7f2d4\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 03:43:15.021156 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:43:15.021007    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/eb03d7bc-abea-4d73-8833-1829d3d7f2d4-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"eb03d7bc-abea-4d73-8833-1829d3d7f2d4\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 03:43:15.021156 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:43:15.021034    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/eb03d7bc-abea-4d73-8833-1829d3d7f2d4-config\") pod \"prometheus-k8s-0\" (UID: \"eb03d7bc-abea-4d73-8833-1829d3d7f2d4\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 03:43:15.021156 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:43:15.021058    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/eb03d7bc-abea-4d73-8833-1829d3d7f2d4-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"eb03d7bc-abea-4d73-8833-1829d3d7f2d4\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 03:43:15.021156 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:43:15.021125    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/eb03d7bc-abea-4d73-8833-1829d3d7f2d4-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"eb03d7bc-abea-4d73-8833-1829d3d7f2d4\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 03:43:15.021548 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:43:15.021287    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/eb03d7bc-abea-4d73-8833-1829d3d7f2d4-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"eb03d7bc-abea-4d73-8833-1829d3d7f2d4\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 03:43:15.022044 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:43:15.022006    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/eb03d7bc-abea-4d73-8833-1829d3d7f2d4-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"eb03d7bc-abea-4d73-8833-1829d3d7f2d4\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 03:43:15.023785 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:43:15.023003    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/eb03d7bc-abea-4d73-8833-1829d3d7f2d4-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"eb03d7bc-abea-4d73-8833-1829d3d7f2d4\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 03:43:15.023785 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:43:15.023020    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/eb03d7bc-abea-4d73-8833-1829d3d7f2d4-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"eb03d7bc-abea-4d73-8833-1829d3d7f2d4\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 03:43:15.023785 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:43:15.023556    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/eb03d7bc-abea-4d73-8833-1829d3d7f2d4-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"eb03d7bc-abea-4d73-8833-1829d3d7f2d4\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 03:43:15.024660 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:43:15.024037    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/eb03d7bc-abea-4d73-8833-1829d3d7f2d4-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"eb03d7bc-abea-4d73-8833-1829d3d7f2d4\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 03:43:15.024660 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:43:15.024419    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/eb03d7bc-abea-4d73-8833-1829d3d7f2d4-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"eb03d7bc-abea-4d73-8833-1829d3d7f2d4\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 03:43:15.024660 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:43:15.024608    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/eb03d7bc-abea-4d73-8833-1829d3d7f2d4-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"eb03d7bc-abea-4d73-8833-1829d3d7f2d4\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 03:43:15.026001 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:43:15.025111    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/eb03d7bc-abea-4d73-8833-1829d3d7f2d4-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"eb03d7bc-abea-4d73-8833-1829d3d7f2d4\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 03:43:15.026001 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:43:15.025179    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/eb03d7bc-abea-4d73-8833-1829d3d7f2d4-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"eb03d7bc-abea-4d73-8833-1829d3d7f2d4\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 03:43:15.026001 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:43:15.025719    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/eb03d7bc-abea-4d73-8833-1829d3d7f2d4-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"eb03d7bc-abea-4d73-8833-1829d3d7f2d4\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 03:43:15.026001 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:43:15.025856    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/eb03d7bc-abea-4d73-8833-1829d3d7f2d4-web-config\") pod \"prometheus-k8s-0\" (UID: \"eb03d7bc-abea-4d73-8833-1829d3d7f2d4\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 03:43:15.026001 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:43:15.025937    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/eb03d7bc-abea-4d73-8833-1829d3d7f2d4-config-out\") pod \"prometheus-k8s-0\" (UID: \"eb03d7bc-abea-4d73-8833-1829d3d7f2d4\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 03:43:15.026609 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:43:15.026577    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/eb03d7bc-abea-4d73-8833-1829d3d7f2d4-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"eb03d7bc-abea-4d73-8833-1829d3d7f2d4\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 03:43:15.026799 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:43:15.026780    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/eb03d7bc-abea-4d73-8833-1829d3d7f2d4-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"eb03d7bc-abea-4d73-8833-1829d3d7f2d4\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 03:43:15.026990 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:43:15.026972    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/eb03d7bc-abea-4d73-8833-1829d3d7f2d4-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"eb03d7bc-abea-4d73-8833-1829d3d7f2d4\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 03:43:15.027070 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:43:15.026992    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/eb03d7bc-abea-4d73-8833-1829d3d7f2d4-config\") pod \"prometheus-k8s-0\" (UID: \"eb03d7bc-abea-4d73-8833-1829d3d7f2d4\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 03:43:15.027477 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:43:15.027461    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/eb03d7bc-abea-4d73-8833-1829d3d7f2d4-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"eb03d7bc-abea-4d73-8833-1829d3d7f2d4\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 03:43:15.030450 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:43:15.030431    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5mdq2\" (UniqueName: \"kubernetes.io/projected/eb03d7bc-abea-4d73-8833-1829d3d7f2d4-kube-api-access-5mdq2\") pod \"prometheus-k8s-0\" (UID: \"eb03d7bc-abea-4d73-8833-1829d3d7f2d4\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 03:43:15.140612 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:43:15.140581    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 03:43:15.424283 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:43:15.424256    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 11 03:43:15.426456 ip-10-0-141-105 kubenswrapper[2579]: W0911 03:43:15.426432    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeb03d7bc_abea_4d73_8833_1829d3d7f2d4.slice/crio-6d5d37f5b4fa670a4137b14d7a4ad0e14919e3ab02dcd26a4b3da970430a90ce WatchSource:0}: Error finding container 6d5d37f5b4fa670a4137b14d7a4ad0e14919e3ab02dcd26a4b3da970430a90ce: Status 404 returned error can't find the container with id 6d5d37f5b4fa670a4137b14d7a4ad0e14919e3ab02dcd26a4b3da970430a90ce
Sep 11 03:43:15.775685 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:43:15.775586    2579 generic.go:356] "Generic (PLEG): container finished" podID="eb03d7bc-abea-4d73-8833-1829d3d7f2d4" containerID="9a3a0e09ba55cbd71d47922ab27af70cf1afbe034a3d4f1476554fb686d15b05" exitCode=0
Sep 11 03:43:15.775843 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:43:15.775680    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"eb03d7bc-abea-4d73-8833-1829d3d7f2d4","Type":"ContainerDied","Data":"9a3a0e09ba55cbd71d47922ab27af70cf1afbe034a3d4f1476554fb686d15b05"}
Sep 11 03:43:15.775843 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:43:15.775775    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"eb03d7bc-abea-4d73-8833-1829d3d7f2d4","Type":"ContainerStarted","Data":"6d5d37f5b4fa670a4137b14d7a4ad0e14919e3ab02dcd26a4b3da970430a90ce"}
Sep 11 03:43:16.783698 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:43:16.783659    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"eb03d7bc-abea-4d73-8833-1829d3d7f2d4","Type":"ContainerStarted","Data":"154bdc1f7b8203bef5d3d0d57c1c94f1e7162cb96a9ec7426caca292df95a79c"}
Sep 11 03:43:16.784120 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:43:16.783745    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"eb03d7bc-abea-4d73-8833-1829d3d7f2d4","Type":"ContainerStarted","Data":"bbeaeac6d03e8be0904574e531990db4b57b3b7f74287cd9c651d80fc9fcdcc5"}
Sep 11 03:43:16.784120 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:43:16.783789    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"eb03d7bc-abea-4d73-8833-1829d3d7f2d4","Type":"ContainerStarted","Data":"c6c8553072d3df845738a17f6ad64bec428165c2b60ac689a42e4a0c88cb0ec7"}
Sep 11 03:43:16.784120 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:43:16.783827    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"eb03d7bc-abea-4d73-8833-1829d3d7f2d4","Type":"ContainerStarted","Data":"df870bd25085a45e22fb6b450832758c5baa4da3a0e53c064af7b5467f8db4d4"}
Sep 11 03:43:16.784120 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:43:16.783861    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"eb03d7bc-abea-4d73-8833-1829d3d7f2d4","Type":"ContainerStarted","Data":"28927348dbc18508fff900a354e7d17dc933d5960dd472436d2b6b0787f4294e"}
Sep 11 03:43:16.784120 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:43:16.783897    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"eb03d7bc-abea-4d73-8833-1829d3d7f2d4","Type":"ContainerStarted","Data":"4ba2cd359cf74290e0faf29c7e3d78a573d1d217cd83a651ece34a311511b4d9"}
Sep 11 03:43:16.813409 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:43:16.813271    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=2.7946333340000002 podStartE2EDuration="2.813249287s" totalImagesPullingTime="0s" totalInitContainerRuntime="18.615953ms" isStatefulPod=false podCreationTimestamp="2026-09-11 03:43:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 03:43:16.810999145 +0000 UTC m=+174.466204741" watchObservedRunningTime="2026-09-11 03:43:16.813249287 +0000 UTC m=+174.468454855"
Sep 11 03:43:20.141627 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:43:20.141587    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 03:44:15.140861 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:44:15.140820    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 03:44:15.156809 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:44:15.156783    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 03:44:16.024582 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:44:16.024552    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 03:45:14.039848 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:45:14.039812    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-6cf4d46bbf-jv5nx"]
Sep 11 03:45:14.043289 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:45:14.043269    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-6cf4d46bbf-jv5nx"
Sep 11 03:45:14.049683 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:45:14.046167    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\""
Sep 11 03:45:14.049683 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:45:14.046346    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\""
Sep 11 03:45:14.049683 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:45:14.046518    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-8t266\""
Sep 11 03:45:14.049683 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:45:14.046577    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\""
Sep 11 03:45:14.049683 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:45:14.046648    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\""
Sep 11 03:45:14.057537 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:45:14.057514    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-6cf4d46bbf-jv5nx"]
Sep 11 03:45:14.089225 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:45:14.089195    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/584a1759-30fa-436c-b0d0-1aff194707e5-manifests\") pod \"opendatahub-operator-controller-manager-6cf4d46bbf-jv5nx\" (UID: \"584a1759-30fa-436c-b0d0-1aff194707e5\") " pod="opendatahub/opendatahub-operator-controller-manager-6cf4d46bbf-jv5nx"
Sep 11 03:45:14.089412 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:45:14.089241    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/584a1759-30fa-436c-b0d0-1aff194707e5-tmp\") pod \"opendatahub-operator-controller-manager-6cf4d46bbf-jv5nx\" (UID: \"584a1759-30fa-436c-b0d0-1aff194707e5\") " pod="opendatahub/opendatahub-operator-controller-manager-6cf4d46bbf-jv5nx"
Sep 11 03:45:14.089412 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:45:14.089262    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wv44m\" (UniqueName: \"kubernetes.io/projected/584a1759-30fa-436c-b0d0-1aff194707e5-kube-api-access-wv44m\") pod \"opendatahub-operator-controller-manager-6cf4d46bbf-jv5nx\" (UID: \"584a1759-30fa-436c-b0d0-1aff194707e5\") " pod="opendatahub/opendatahub-operator-controller-manager-6cf4d46bbf-jv5nx"
Sep 11 03:45:14.089412 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:45:14.089367    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/584a1759-30fa-436c-b0d0-1aff194707e5-apiservice-cert\") pod \"opendatahub-operator-controller-manager-6cf4d46bbf-jv5nx\" (UID: \"584a1759-30fa-436c-b0d0-1aff194707e5\") " pod="opendatahub/opendatahub-operator-controller-manager-6cf4d46bbf-jv5nx"
Sep 11 03:45:14.089412 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:45:14.089393    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/584a1759-30fa-436c-b0d0-1aff194707e5-webhook-cert\") pod \"opendatahub-operator-controller-manager-6cf4d46bbf-jv5nx\" (UID: \"584a1759-30fa-436c-b0d0-1aff194707e5\") " pod="opendatahub/opendatahub-operator-controller-manager-6cf4d46bbf-jv5nx"
Sep 11 03:45:14.190220 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:45:14.190184    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/584a1759-30fa-436c-b0d0-1aff194707e5-tmp\") pod \"opendatahub-operator-controller-manager-6cf4d46bbf-jv5nx\" (UID: \"584a1759-30fa-436c-b0d0-1aff194707e5\") " pod="opendatahub/opendatahub-operator-controller-manager-6cf4d46bbf-jv5nx"
Sep 11 03:45:14.190220 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:45:14.190225    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wv44m\" (UniqueName: \"kubernetes.io/projected/584a1759-30fa-436c-b0d0-1aff194707e5-kube-api-access-wv44m\") pod \"opendatahub-operator-controller-manager-6cf4d46bbf-jv5nx\" (UID: \"584a1759-30fa-436c-b0d0-1aff194707e5\") " pod="opendatahub/opendatahub-operator-controller-manager-6cf4d46bbf-jv5nx"
Sep 11 03:45:14.190528 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:45:14.190293    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/584a1759-30fa-436c-b0d0-1aff194707e5-apiservice-cert\") pod \"opendatahub-operator-controller-manager-6cf4d46bbf-jv5nx\" (UID: \"584a1759-30fa-436c-b0d0-1aff194707e5\") " pod="opendatahub/opendatahub-operator-controller-manager-6cf4d46bbf-jv5nx"
Sep 11 03:45:14.190528 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:45:14.190313    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/584a1759-30fa-436c-b0d0-1aff194707e5-webhook-cert\") pod \"opendatahub-operator-controller-manager-6cf4d46bbf-jv5nx\" (UID: \"584a1759-30fa-436c-b0d0-1aff194707e5\") " pod="opendatahub/opendatahub-operator-controller-manager-6cf4d46bbf-jv5nx"
Sep 11 03:45:14.190528 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:45:14.190385    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/584a1759-30fa-436c-b0d0-1aff194707e5-manifests\") pod \"opendatahub-operator-controller-manager-6cf4d46bbf-jv5nx\" (UID: \"584a1759-30fa-436c-b0d0-1aff194707e5\") " pod="opendatahub/opendatahub-operator-controller-manager-6cf4d46bbf-jv5nx"
Sep 11 03:45:14.190886 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:45:14.190861    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/584a1759-30fa-436c-b0d0-1aff194707e5-manifests\") pod \"opendatahub-operator-controller-manager-6cf4d46bbf-jv5nx\" (UID: \"584a1759-30fa-436c-b0d0-1aff194707e5\") " pod="opendatahub/opendatahub-operator-controller-manager-6cf4d46bbf-jv5nx"
Sep 11 03:45:14.192981 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:45:14.192959    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/584a1759-30fa-436c-b0d0-1aff194707e5-tmp\") pod \"opendatahub-operator-controller-manager-6cf4d46bbf-jv5nx\" (UID: \"584a1759-30fa-436c-b0d0-1aff194707e5\") " pod="opendatahub/opendatahub-operator-controller-manager-6cf4d46bbf-jv5nx"
Sep 11 03:45:14.193129 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:45:14.193110    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/584a1759-30fa-436c-b0d0-1aff194707e5-apiservice-cert\") pod \"opendatahub-operator-controller-manager-6cf4d46bbf-jv5nx\" (UID: \"584a1759-30fa-436c-b0d0-1aff194707e5\") " pod="opendatahub/opendatahub-operator-controller-manager-6cf4d46bbf-jv5nx"
Sep 11 03:45:14.193312 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:45:14.193291    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/584a1759-30fa-436c-b0d0-1aff194707e5-webhook-cert\") pod \"opendatahub-operator-controller-manager-6cf4d46bbf-jv5nx\" (UID: \"584a1759-30fa-436c-b0d0-1aff194707e5\") " pod="opendatahub/opendatahub-operator-controller-manager-6cf4d46bbf-jv5nx"
Sep 11 03:45:14.214579 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:45:14.214551    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wv44m\" (UniqueName: \"kubernetes.io/projected/584a1759-30fa-436c-b0d0-1aff194707e5-kube-api-access-wv44m\") pod \"opendatahub-operator-controller-manager-6cf4d46bbf-jv5nx\" (UID: \"584a1759-30fa-436c-b0d0-1aff194707e5\") " pod="opendatahub/opendatahub-operator-controller-manager-6cf4d46bbf-jv5nx"
Sep 11 03:45:14.357737 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:45:14.357703    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-6cf4d46bbf-jv5nx"
Sep 11 03:45:14.640149 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:45:14.640050    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-6cf4d46bbf-jv5nx"]
Sep 11 03:45:14.643878 ip-10-0-141-105 kubenswrapper[2579]: W0911 03:45:14.643852    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod584a1759_30fa_436c_b0d0_1aff194707e5.slice/crio-eff1f8ee589a31f9d6a33bd49fca894edad7dcf56272b17846cf861936cbf75d WatchSource:0}: Error finding container eff1f8ee589a31f9d6a33bd49fca894edad7dcf56272b17846cf861936cbf75d: Status 404 returned error can't find the container with id eff1f8ee589a31f9d6a33bd49fca894edad7dcf56272b17846cf861936cbf75d
Sep 11 03:45:15.243132 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:45:15.243039    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-6cf4d46bbf-jv5nx" event={"ID":"584a1759-30fa-436c-b0d0-1aff194707e5","Type":"ContainerStarted","Data":"eff1f8ee589a31f9d6a33bd49fca894edad7dcf56272b17846cf861936cbf75d"}
Sep 11 03:45:18.259642 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:45:18.259608    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-6cf4d46bbf-jv5nx" event={"ID":"584a1759-30fa-436c-b0d0-1aff194707e5","Type":"ContainerStarted","Data":"df23acff8a95a406a683245b309383b8b9c2dc6670be362f85b7093258d8192e"}
Sep 11 03:45:19.266308 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:45:19.266271    2579 generic.go:356] "Generic (PLEG): container finished" podID="584a1759-30fa-436c-b0d0-1aff194707e5" containerID="df23acff8a95a406a683245b309383b8b9c2dc6670be362f85b7093258d8192e" exitCode=0
Sep 11 03:45:19.266646 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:45:19.266319    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-6cf4d46bbf-jv5nx" event={"ID":"584a1759-30fa-436c-b0d0-1aff194707e5","Type":"ContainerDied","Data":"df23acff8a95a406a683245b309383b8b9c2dc6670be362f85b7093258d8192e"}
Sep 11 03:45:20.275564 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:45:20.275518    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-6cf4d46bbf-jv5nx" event={"ID":"584a1759-30fa-436c-b0d0-1aff194707e5","Type":"ContainerStarted","Data":"a426165de8850b3d3567bd089269b4dd9159cf99cebd9a6509c239ff756a0f55"}
Sep 11 03:45:20.275983 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:45:20.275641    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-6cf4d46bbf-jv5nx"
Sep 11 03:45:20.297326 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:45:20.297259    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-6cf4d46bbf-jv5nx" podStartSLOduration=1.192372599 podStartE2EDuration="6.297244915s" totalImagesPullingTime="4.068258046s" totalInitContainerRuntime="1.03661427s" isStatefulPod=false podCreationTimestamp="2026-09-11 03:45:14 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 03:45:20.296888631 +0000 UTC m=+297.952094198" watchObservedRunningTime="2026-09-11 03:45:20.297244915 +0000 UTC m=+297.952450482"
Sep 11 03:45:22.731189 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:45:22.731155    2579 kubelet.go:1648] "Image garbage collection succeeded"
Sep 11 03:45:31.281820 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:45:31.281786    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-6cf4d46bbf-jv5nx"
Sep 11 03:45:33.771556 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:45:33.771523    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/lws-controller-manager-67c8664749-z6zpb"]
Sep 11 03:45:33.773845 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:45:33.773830    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-67c8664749-z6zpb"
Sep 11 03:45:33.778127 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:45:33.778049    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\""
Sep 11 03:45:33.778127 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:45:33.778062    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"webhook-server-cert\""
Sep 11 03:45:33.778127 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:45:33.778059    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"lws-controller-manager-dockercfg-vfnc6\""
Sep 11 03:45:33.778429 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:45:33.778133    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"metrics-server-cert\""
Sep 11 03:45:33.778429 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:45:33.778190    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\""
Sep 11 03:45:33.778429 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:45:33.778171    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"lws-manager-config\""
Sep 11 03:45:33.785587 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:45:33.785564    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-67c8664749-z6zpb"]
Sep 11 03:45:33.867978 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:45:33.867938    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/17e941b0-9899-4103-ba1a-2566a6c2f42d-cert\") pod \"lws-controller-manager-67c8664749-z6zpb\" (UID: \"17e941b0-9899-4103-ba1a-2566a6c2f42d\") " pod="openshift-lws-operator/lws-controller-manager-67c8664749-z6zpb"
Sep 11 03:45:33.868180 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:45:33.868021    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z4gt6\" (UniqueName: \"kubernetes.io/projected/17e941b0-9899-4103-ba1a-2566a6c2f42d-kube-api-access-z4gt6\") pod \"lws-controller-manager-67c8664749-z6zpb\" (UID: \"17e941b0-9899-4103-ba1a-2566a6c2f42d\") " pod="openshift-lws-operator/lws-controller-manager-67c8664749-z6zpb"
Sep 11 03:45:33.868180 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:45:33.868049    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/17e941b0-9899-4103-ba1a-2566a6c2f42d-metrics-cert\") pod \"lws-controller-manager-67c8664749-z6zpb\" (UID: \"17e941b0-9899-4103-ba1a-2566a6c2f42d\") " pod="openshift-lws-operator/lws-controller-manager-67c8664749-z6zpb"
Sep 11 03:45:33.868180 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:45:33.868077    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/17e941b0-9899-4103-ba1a-2566a6c2f42d-manager-config\") pod \"lws-controller-manager-67c8664749-z6zpb\" (UID: \"17e941b0-9899-4103-ba1a-2566a6c2f42d\") " pod="openshift-lws-operator/lws-controller-manager-67c8664749-z6zpb"
Sep 11 03:45:33.969343 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:45:33.969309    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z4gt6\" (UniqueName: \"kubernetes.io/projected/17e941b0-9899-4103-ba1a-2566a6c2f42d-kube-api-access-z4gt6\") pod \"lws-controller-manager-67c8664749-z6zpb\" (UID: \"17e941b0-9899-4103-ba1a-2566a6c2f42d\") " pod="openshift-lws-operator/lws-controller-manager-67c8664749-z6zpb"
Sep 11 03:45:33.969485 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:45:33.969351    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/17e941b0-9899-4103-ba1a-2566a6c2f42d-metrics-cert\") pod \"lws-controller-manager-67c8664749-z6zpb\" (UID: \"17e941b0-9899-4103-ba1a-2566a6c2f42d\") " pod="openshift-lws-operator/lws-controller-manager-67c8664749-z6zpb"
Sep 11 03:45:33.969485 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:45:33.969375    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/17e941b0-9899-4103-ba1a-2566a6c2f42d-manager-config\") pod \"lws-controller-manager-67c8664749-z6zpb\" (UID: \"17e941b0-9899-4103-ba1a-2566a6c2f42d\") " pod="openshift-lws-operator/lws-controller-manager-67c8664749-z6zpb"
Sep 11 03:45:33.969485 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:45:33.969403    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/17e941b0-9899-4103-ba1a-2566a6c2f42d-cert\") pod \"lws-controller-manager-67c8664749-z6zpb\" (UID: \"17e941b0-9899-4103-ba1a-2566a6c2f42d\") " pod="openshift-lws-operator/lws-controller-manager-67c8664749-z6zpb"
Sep 11 03:45:33.970168 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:45:33.970147    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/17e941b0-9899-4103-ba1a-2566a6c2f42d-manager-config\") pod \"lws-controller-manager-67c8664749-z6zpb\" (UID: \"17e941b0-9899-4103-ba1a-2566a6c2f42d\") " pod="openshift-lws-operator/lws-controller-manager-67c8664749-z6zpb"
Sep 11 03:45:33.972173 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:45:33.972149    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/17e941b0-9899-4103-ba1a-2566a6c2f42d-cert\") pod \"lws-controller-manager-67c8664749-z6zpb\" (UID: \"17e941b0-9899-4103-ba1a-2566a6c2f42d\") " pod="openshift-lws-operator/lws-controller-manager-67c8664749-z6zpb"
Sep 11 03:45:33.972173 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:45:33.972167    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/17e941b0-9899-4103-ba1a-2566a6c2f42d-metrics-cert\") pod \"lws-controller-manager-67c8664749-z6zpb\" (UID: \"17e941b0-9899-4103-ba1a-2566a6c2f42d\") " pod="openshift-lws-operator/lws-controller-manager-67c8664749-z6zpb"
Sep 11 03:45:33.977871 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:45:33.977839    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-z4gt6\" (UniqueName: \"kubernetes.io/projected/17e941b0-9899-4103-ba1a-2566a6c2f42d-kube-api-access-z4gt6\") pod \"lws-controller-manager-67c8664749-z6zpb\" (UID: \"17e941b0-9899-4103-ba1a-2566a6c2f42d\") " pod="openshift-lws-operator/lws-controller-manager-67c8664749-z6zpb"
Sep 11 03:45:34.083762 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:45:34.083693    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-67c8664749-z6zpb"
Sep 11 03:45:34.354259 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:45:34.354233    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-67c8664749-z6zpb"]
Sep 11 03:45:34.356819 ip-10-0-141-105 kubenswrapper[2579]: W0911 03:45:34.356789    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod17e941b0_9899_4103_ba1a_2566a6c2f42d.slice/crio-aa5d26576e55000e06a178849d2fddab7eb28d6b212a735251ed88900957c7f9 WatchSource:0}: Error finding container aa5d26576e55000e06a178849d2fddab7eb28d6b212a735251ed88900957c7f9: Status 404 returned error can't find the container with id aa5d26576e55000e06a178849d2fddab7eb28d6b212a735251ed88900957c7f9
Sep 11 03:45:34.358818 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:45:34.358798    2579 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 11 03:45:35.343000 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:45:35.342966    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-67c8664749-z6zpb" event={"ID":"17e941b0-9899-4103-ba1a-2566a6c2f42d","Type":"ContainerStarted","Data":"aa5d26576e55000e06a178849d2fddab7eb28d6b212a735251ed88900957c7f9"}
Sep 11 03:45:37.354974 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:45:37.354938    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-67c8664749-z6zpb" event={"ID":"17e941b0-9899-4103-ba1a-2566a6c2f42d","Type":"ContainerStarted","Data":"afe8e8f87da06381788f633ee4add1cfaa3b2efb350b52cd12c7d4e1d2cb1f2a"}
Sep 11 03:45:37.355380 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:45:37.355014    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-lws-operator/lws-controller-manager-67c8664749-z6zpb"
Sep 11 03:45:37.372443 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:45:37.372398    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/lws-controller-manager-67c8664749-z6zpb" podStartSLOduration=2.175317385 podStartE2EDuration="4.372383109s" totalImagesPullingTime="2.197065724s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 03:45:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 03:45:37.371057763 +0000 UTC m=+315.026263332" watchObservedRunningTime="2026-09-11 03:45:37.372383109 +0000 UTC m=+315.027588675"
Sep 11 03:45:44.294373 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:45:44.294343    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-86748cc5bf-kcpw8"]
Sep 11 03:45:44.297406 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:45:44.297391    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-86748cc5bf-kcpw8"
Sep 11 03:45:44.300327 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:45:44.300302    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\""
Sep 11 03:45:44.300626 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:45:44.300608    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-c2754\""
Sep 11 03:45:44.301393 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:45:44.301378    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\""
Sep 11 03:45:44.309154 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:45:44.309130    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-86748cc5bf-kcpw8"]
Sep 11 03:45:44.462421 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:45:44.462392    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/68ec6bcd-85f8-46e2-bcc6-c70d0ec6e80f-tls-certs\") pod \"kube-auth-proxy-86748cc5bf-kcpw8\" (UID: \"68ec6bcd-85f8-46e2-bcc6-c70d0ec6e80f\") " pod="openshift-ingress/kube-auth-proxy-86748cc5bf-kcpw8"
Sep 11 03:45:44.462597 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:45:44.462428    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wctb8\" (UniqueName: \"kubernetes.io/projected/68ec6bcd-85f8-46e2-bcc6-c70d0ec6e80f-kube-api-access-wctb8\") pod \"kube-auth-proxy-86748cc5bf-kcpw8\" (UID: \"68ec6bcd-85f8-46e2-bcc6-c70d0ec6e80f\") " pod="openshift-ingress/kube-auth-proxy-86748cc5bf-kcpw8"
Sep 11 03:45:44.462597 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:45:44.462464    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/68ec6bcd-85f8-46e2-bcc6-c70d0ec6e80f-tmp\") pod \"kube-auth-proxy-86748cc5bf-kcpw8\" (UID: \"68ec6bcd-85f8-46e2-bcc6-c70d0ec6e80f\") " pod="openshift-ingress/kube-auth-proxy-86748cc5bf-kcpw8"
Sep 11 03:45:44.563064 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:45:44.562991    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/68ec6bcd-85f8-46e2-bcc6-c70d0ec6e80f-tmp\") pod \"kube-auth-proxy-86748cc5bf-kcpw8\" (UID: \"68ec6bcd-85f8-46e2-bcc6-c70d0ec6e80f\") " pod="openshift-ingress/kube-auth-proxy-86748cc5bf-kcpw8"
Sep 11 03:45:44.563064 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:45:44.563059    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/68ec6bcd-85f8-46e2-bcc6-c70d0ec6e80f-tls-certs\") pod \"kube-auth-proxy-86748cc5bf-kcpw8\" (UID: \"68ec6bcd-85f8-46e2-bcc6-c70d0ec6e80f\") " pod="openshift-ingress/kube-auth-proxy-86748cc5bf-kcpw8"
Sep 11 03:45:44.563260 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:45:44.563114    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wctb8\" (UniqueName: \"kubernetes.io/projected/68ec6bcd-85f8-46e2-bcc6-c70d0ec6e80f-kube-api-access-wctb8\") pod \"kube-auth-proxy-86748cc5bf-kcpw8\" (UID: \"68ec6bcd-85f8-46e2-bcc6-c70d0ec6e80f\") " pod="openshift-ingress/kube-auth-proxy-86748cc5bf-kcpw8"
Sep 11 03:45:44.565474 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:45:44.565449    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/68ec6bcd-85f8-46e2-bcc6-c70d0ec6e80f-tmp\") pod \"kube-auth-proxy-86748cc5bf-kcpw8\" (UID: \"68ec6bcd-85f8-46e2-bcc6-c70d0ec6e80f\") " pod="openshift-ingress/kube-auth-proxy-86748cc5bf-kcpw8"
Sep 11 03:45:44.565600 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:45:44.565582    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/68ec6bcd-85f8-46e2-bcc6-c70d0ec6e80f-tls-certs\") pod \"kube-auth-proxy-86748cc5bf-kcpw8\" (UID: \"68ec6bcd-85f8-46e2-bcc6-c70d0ec6e80f\") " pod="openshift-ingress/kube-auth-proxy-86748cc5bf-kcpw8"
Sep 11 03:45:44.579589 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:45:44.579564    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wctb8\" (UniqueName: \"kubernetes.io/projected/68ec6bcd-85f8-46e2-bcc6-c70d0ec6e80f-kube-api-access-wctb8\") pod \"kube-auth-proxy-86748cc5bf-kcpw8\" (UID: \"68ec6bcd-85f8-46e2-bcc6-c70d0ec6e80f\") " pod="openshift-ingress/kube-auth-proxy-86748cc5bf-kcpw8"
Sep 11 03:45:44.608169 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:45:44.608147    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-86748cc5bf-kcpw8"
Sep 11 03:45:44.883101 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:45:44.883039    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-86748cc5bf-kcpw8"]
Sep 11 03:45:44.885303 ip-10-0-141-105 kubenswrapper[2579]: W0911 03:45:44.885278    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod68ec6bcd_85f8_46e2_bcc6_c70d0ec6e80f.slice/crio-fe573b08d765538d7b15a20de3e933e2647a337dacca669707e914834073aab3 WatchSource:0}: Error finding container fe573b08d765538d7b15a20de3e933e2647a337dacca669707e914834073aab3: Status 404 returned error can't find the container with id fe573b08d765538d7b15a20de3e933e2647a337dacca669707e914834073aab3
Sep 11 03:45:45.390700 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:45:45.390666    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-86748cc5bf-kcpw8" event={"ID":"68ec6bcd-85f8-46e2-bcc6-c70d0ec6e80f","Type":"ContainerStarted","Data":"fe573b08d765538d7b15a20de3e933e2647a337dacca669707e914834073aab3"}
Sep 11 03:45:48.361435 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:45:48.361406    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-lws-operator/lws-controller-manager-67c8664749-z6zpb"
Sep 11 03:45:48.407055 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:45:48.407018    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-86748cc5bf-kcpw8" event={"ID":"68ec6bcd-85f8-46e2-bcc6-c70d0ec6e80f","Type":"ContainerStarted","Data":"c08fa2548220a2596406e4c1587f15f9ede4021d3e088b29232dbbd6a9097d2f"}
Sep 11 03:45:48.424020 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:45:48.423970    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-86748cc5bf-kcpw8" podStartSLOduration=1.639375714 podStartE2EDuration="4.423952552s" totalImagesPullingTime="2.784576838s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 03:45:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 03:45:48.422764604 +0000 UTC m=+326.077970171" watchObservedRunningTime="2026-09-11 03:45:48.423952552 +0000 UTC m=+326.079158121"
Sep 11 03:47:11.594237 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:47:11.594202    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-grsb6"]
Sep 11 03:47:11.597552 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:47:11.597531    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-grsb6"
Sep 11 03:47:11.600269 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:47:11.600235    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-env\""
Sep 11 03:47:11.600269 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:47:11.600245    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\""
Sep 11 03:47:11.600468 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:47:11.600444    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\""
Sep 11 03:47:11.601317 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:47:11.601301    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-manager-dockercfg-7mdcs\""
Sep 11 03:47:11.609933 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:47:11.609911    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-grsb6"]
Sep 11 03:47:11.701743 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:47:11.701694    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7fcvb\" (UniqueName: \"kubernetes.io/projected/bee3cb9f-2f86-4ea2-ae3e-a3a0481dc5d3-kube-api-access-7fcvb\") pod \"dns-operator-controller-manager-7b84cd7c9c-grsb6\" (UID: \"bee3cb9f-2f86-4ea2-ae3e-a3a0481dc5d3\") " pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-grsb6"
Sep 11 03:47:11.802443 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:47:11.802399    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7fcvb\" (UniqueName: \"kubernetes.io/projected/bee3cb9f-2f86-4ea2-ae3e-a3a0481dc5d3-kube-api-access-7fcvb\") pod \"dns-operator-controller-manager-7b84cd7c9c-grsb6\" (UID: \"bee3cb9f-2f86-4ea2-ae3e-a3a0481dc5d3\") " pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-grsb6"
Sep 11 03:47:11.811168 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:47:11.811145    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7fcvb\" (UniqueName: \"kubernetes.io/projected/bee3cb9f-2f86-4ea2-ae3e-a3a0481dc5d3-kube-api-access-7fcvb\") pod \"dns-operator-controller-manager-7b84cd7c9c-grsb6\" (UID: \"bee3cb9f-2f86-4ea2-ae3e-a3a0481dc5d3\") " pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-grsb6"
Sep 11 03:47:11.908765 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:47:11.908736    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-grsb6"
Sep 11 03:47:12.182992 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:47:12.182949    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-grsb6"]
Sep 11 03:47:12.185483 ip-10-0-141-105 kubenswrapper[2579]: W0911 03:47:12.185435    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbee3cb9f_2f86_4ea2_ae3e_a3a0481dc5d3.slice/crio-fda33580025dc580c25e1b3817099de8b2305e7191b8d6a642a67e830e10d343 WatchSource:0}: Error finding container fda33580025dc580c25e1b3817099de8b2305e7191b8d6a642a67e830e10d343: Status 404 returned error can't find the container with id fda33580025dc580c25e1b3817099de8b2305e7191b8d6a642a67e830e10d343
Sep 11 03:47:12.778279 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:47:12.778246    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-grsb6" event={"ID":"bee3cb9f-2f86-4ea2-ae3e-a3a0481dc5d3","Type":"ContainerStarted","Data":"fda33580025dc580c25e1b3817099de8b2305e7191b8d6a642a67e830e10d343"}
Sep 11 03:47:14.790189 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:47:14.790138    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-grsb6" event={"ID":"bee3cb9f-2f86-4ea2-ae3e-a3a0481dc5d3","Type":"ContainerStarted","Data":"56080c8da42c299c0fd68fe376f45700c8816f42e85188f74e4283cdae84bc4d"}
Sep 11 03:47:14.790589 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:47:14.790216    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-grsb6"
Sep 11 03:47:14.810099 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:47:14.810045    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-grsb6" podStartSLOduration=1.320357162 podStartE2EDuration="3.81003031s" totalImagesPullingTime="2.489673148s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 03:47:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 03:47:14.808201391 +0000 UTC m=+412.463406958" watchObservedRunningTime="2026-09-11 03:47:14.81003031 +0000 UTC m=+412.465235876"
Sep 11 03:47:24.027029 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:47:24.026995    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-lnwx2"]
Sep 11 03:47:24.030432 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:47:24.030407    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-lnwx2"
Sep 11 03:47:24.033416 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:47:24.033398    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-f82vk\""
Sep 11 03:47:24.057494 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:47:24.057458    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-lnwx2"]
Sep 11 03:47:24.115645 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:47:24.115611    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/453f5a42-4409-48c1-af52-848602e024c7-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5f685c496f-lnwx2\" (UID: \"453f5a42-4409-48c1-af52-848602e024c7\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-lnwx2"
Sep 11 03:47:24.115774 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:47:24.115680    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h82td\" (UniqueName: \"kubernetes.io/projected/453f5a42-4409-48c1-af52-848602e024c7-kube-api-access-h82td\") pod \"kuadrant-operator-controller-manager-5f685c496f-lnwx2\" (UID: \"453f5a42-4409-48c1-af52-848602e024c7\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-lnwx2"
Sep 11 03:47:24.216752 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:47:24.216721    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h82td\" (UniqueName: \"kubernetes.io/projected/453f5a42-4409-48c1-af52-848602e024c7-kube-api-access-h82td\") pod \"kuadrant-operator-controller-manager-5f685c496f-lnwx2\" (UID: \"453f5a42-4409-48c1-af52-848602e024c7\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-lnwx2"
Sep 11 03:47:24.216929 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:47:24.216804    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/453f5a42-4409-48c1-af52-848602e024c7-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5f685c496f-lnwx2\" (UID: \"453f5a42-4409-48c1-af52-848602e024c7\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-lnwx2"
Sep 11 03:47:24.217249 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:47:24.217228    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/453f5a42-4409-48c1-af52-848602e024c7-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5f685c496f-lnwx2\" (UID: \"453f5a42-4409-48c1-af52-848602e024c7\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-lnwx2"
Sep 11 03:47:24.229453 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:47:24.229414    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-h82td\" (UniqueName: \"kubernetes.io/projected/453f5a42-4409-48c1-af52-848602e024c7-kube-api-access-h82td\") pod \"kuadrant-operator-controller-manager-5f685c496f-lnwx2\" (UID: \"453f5a42-4409-48c1-af52-848602e024c7\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-lnwx2"
Sep 11 03:47:24.340833 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:47:24.340750    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-lnwx2"
Sep 11 03:47:24.614728 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:47:24.614702    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-lnwx2"]
Sep 11 03:47:24.616945 ip-10-0-141-105 kubenswrapper[2579]: W0911 03:47:24.616919    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod453f5a42_4409_48c1_af52_848602e024c7.slice/crio-7ef438359de32251b256ef02bf44ff4612eb88d4b312efec4a5e00f75ccbdf59 WatchSource:0}: Error finding container 7ef438359de32251b256ef02bf44ff4612eb88d4b312efec4a5e00f75ccbdf59: Status 404 returned error can't find the container with id 7ef438359de32251b256ef02bf44ff4612eb88d4b312efec4a5e00f75ccbdf59
Sep 11 03:47:24.834510 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:47:24.834470    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-lnwx2" event={"ID":"453f5a42-4409-48c1-af52-848602e024c7","Type":"ContainerStarted","Data":"7ef438359de32251b256ef02bf44ff4612eb88d4b312efec4a5e00f75ccbdf59"}
Sep 11 03:47:24.907149 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:47:24.907042    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-lnwx2"]
Sep 11 03:47:24.912984 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:47:24.912952    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-lnwx2"]
Sep 11 03:47:24.931986 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:47:24.931962    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-9wq2t"]
Sep 11 03:47:24.937103 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:47:24.937066    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-9wq2t"
Sep 11 03:47:24.954682 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:47:24.954653    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-9wq2t"]
Sep 11 03:47:25.024566 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:47:25.024536    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hgxwt\" (UniqueName: \"kubernetes.io/projected/a005b167-d364-40c1-9393-72ce6e5a098c-kube-api-access-hgxwt\") pod \"kuadrant-operator-controller-manager-5f685c496f-9wq2t\" (UID: \"a005b167-d364-40c1-9393-72ce6e5a098c\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-9wq2t"
Sep 11 03:47:25.024726 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:47:25.024585    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/a005b167-d364-40c1-9393-72ce6e5a098c-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5f685c496f-9wq2t\" (UID: \"a005b167-d364-40c1-9393-72ce6e5a098c\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-9wq2t"
Sep 11 03:47:25.125951 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:47:25.125911    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hgxwt\" (UniqueName: \"kubernetes.io/projected/a005b167-d364-40c1-9393-72ce6e5a098c-kube-api-access-hgxwt\") pod \"kuadrant-operator-controller-manager-5f685c496f-9wq2t\" (UID: \"a005b167-d364-40c1-9393-72ce6e5a098c\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-9wq2t"
Sep 11 03:47:25.126359 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:47:25.125971    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/a005b167-d364-40c1-9393-72ce6e5a098c-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5f685c496f-9wq2t\" (UID: \"a005b167-d364-40c1-9393-72ce6e5a098c\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-9wq2t"
Sep 11 03:47:25.126359 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:47:25.126313    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/a005b167-d364-40c1-9393-72ce6e5a098c-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5f685c496f-9wq2t\" (UID: \"a005b167-d364-40c1-9393-72ce6e5a098c\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-9wq2t"
Sep 11 03:47:25.135511 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:47:25.135475    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hgxwt\" (UniqueName: \"kubernetes.io/projected/a005b167-d364-40c1-9393-72ce6e5a098c-kube-api-access-hgxwt\") pod \"kuadrant-operator-controller-manager-5f685c496f-9wq2t\" (UID: \"a005b167-d364-40c1-9393-72ce6e5a098c\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-9wq2t"
Sep 11 03:47:25.248785 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:47:25.248698    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-9wq2t"
Sep 11 03:47:25.557527 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:47:25.557023    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-9wq2t"]
Sep 11 03:47:25.560655 ip-10-0-141-105 kubenswrapper[2579]: W0911 03:47:25.560627    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda005b167_d364_40c1_9393_72ce6e5a098c.slice/crio-5f233c05de661d9d83e9485792748581225f6dc92247c8c31d8aa4a89ec5d538 WatchSource:0}: Error finding container 5f233c05de661d9d83e9485792748581225f6dc92247c8c31d8aa4a89ec5d538: Status 404 returned error can't find the container with id 5f233c05de661d9d83e9485792748581225f6dc92247c8c31d8aa4a89ec5d538
Sep 11 03:47:25.797792 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:47:25.797764    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-grsb6"
Sep 11 03:47:25.843363 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:47:25.843280    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-9wq2t" event={"ID":"a005b167-d364-40c1-9393-72ce6e5a098c","Type":"ContainerStarted","Data":"5f233c05de661d9d83e9485792748581225f6dc92247c8c31d8aa4a89ec5d538"}
Sep 11 03:47:28.862809 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:47:28.862768    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-lnwx2" podUID="453f5a42-4409-48c1-af52-848602e024c7" containerName="manager" containerID="cri-o://ef17000dbb685bf9b0c1b41ceb868cf7898bc4946592c6a56a0fed4824e07447" gracePeriod=2
Sep 11 03:47:28.864619 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:47:28.864592    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-9wq2t" event={"ID":"a005b167-d364-40c1-9393-72ce6e5a098c","Type":"ContainerStarted","Data":"9c16ff0d0256ff1fab849be99d8fb039ab7ad84ed57d931c373d9d3830d39831"}
Sep 11 03:47:28.864737 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:47:28.864697    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-9wq2t"
Sep 11 03:47:28.885016 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:47:28.884967    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-9wq2t" podStartSLOduration=1.866755288 podStartE2EDuration="4.884952246s" totalImagesPullingTime="3.018196958s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 03:47:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 03:47:28.882899144 +0000 UTC m=+426.538104710" watchObservedRunningTime="2026-09-11 03:47:28.884952246 +0000 UTC m=+426.540157812"
Sep 11 03:47:29.104235 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:47:29.104208    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-lnwx2"
Sep 11 03:47:29.107034 ip-10-0-141-105 kubenswrapper[2579]: E0911 03:47:29.107010    2579 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-5f685c496f-lnwx2\" is forbidden: User \"system:node:ip-10-0-141-105.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-141-105.ec2.internal' and this object" podUID="453f5a42-4409-48c1-af52-848602e024c7" pod="kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-lnwx2"
Sep 11 03:47:29.264231 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:47:29.264136    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/453f5a42-4409-48c1-af52-848602e024c7-extensions-socket-volume\") pod \"453f5a42-4409-48c1-af52-848602e024c7\" (UID: \"453f5a42-4409-48c1-af52-848602e024c7\") "
Sep 11 03:47:29.264231 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:47:29.264222    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-h82td\" (UniqueName: \"kubernetes.io/projected/453f5a42-4409-48c1-af52-848602e024c7-kube-api-access-h82td\") pod \"453f5a42-4409-48c1-af52-848602e024c7\" (UID: \"453f5a42-4409-48c1-af52-848602e024c7\") "
Sep 11 03:47:29.264502 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:47:29.264471    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/453f5a42-4409-48c1-af52-848602e024c7-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "453f5a42-4409-48c1-af52-848602e024c7" (UID: "453f5a42-4409-48c1-af52-848602e024c7"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 03:47:29.266569 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:47:29.266548    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/453f5a42-4409-48c1-af52-848602e024c7-kube-api-access-h82td" (OuterVolumeSpecName: "kube-api-access-h82td") pod "453f5a42-4409-48c1-af52-848602e024c7" (UID: "453f5a42-4409-48c1-af52-848602e024c7"). InnerVolumeSpecName "kube-api-access-h82td". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 03:47:29.365383 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:47:29.365346    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h82td\" (UniqueName: \"kubernetes.io/projected/453f5a42-4409-48c1-af52-848602e024c7-kube-api-access-h82td\") on node \"ip-10-0-141-105.ec2.internal\" DevicePath \"\""
Sep 11 03:47:29.365383 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:47:29.365378    2579 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/453f5a42-4409-48c1-af52-848602e024c7-extensions-socket-volume\") on node \"ip-10-0-141-105.ec2.internal\" DevicePath \"\""
Sep 11 03:47:29.870277 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:47:29.870247    2579 generic.go:356] "Generic (PLEG): container finished" podID="453f5a42-4409-48c1-af52-848602e024c7" containerID="ef17000dbb685bf9b0c1b41ceb868cf7898bc4946592c6a56a0fed4824e07447" exitCode=2
Sep 11 03:47:29.870691 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:47:29.870290    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-lnwx2"
Sep 11 03:47:29.870691 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:47:29.870342    2579 scope.go:117] "RemoveContainer" containerID="ef17000dbb685bf9b0c1b41ceb868cf7898bc4946592c6a56a0fed4824e07447"
Sep 11 03:47:29.872808 ip-10-0-141-105 kubenswrapper[2579]: E0911 03:47:29.872781    2579 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-5f685c496f-lnwx2\" is forbidden: User \"system:node:ip-10-0-141-105.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-141-105.ec2.internal' and this object" podUID="453f5a42-4409-48c1-af52-848602e024c7" pod="kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-lnwx2"
Sep 11 03:47:29.879466 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:47:29.879448    2579 scope.go:117] "RemoveContainer" containerID="ef17000dbb685bf9b0c1b41ceb868cf7898bc4946592c6a56a0fed4824e07447"
Sep 11 03:47:29.879736 ip-10-0-141-105 kubenswrapper[2579]: E0911 03:47:29.879719    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ef17000dbb685bf9b0c1b41ceb868cf7898bc4946592c6a56a0fed4824e07447\": container with ID starting with ef17000dbb685bf9b0c1b41ceb868cf7898bc4946592c6a56a0fed4824e07447 not found: ID does not exist" containerID="ef17000dbb685bf9b0c1b41ceb868cf7898bc4946592c6a56a0fed4824e07447"
Sep 11 03:47:29.879795 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:47:29.879748    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ef17000dbb685bf9b0c1b41ceb868cf7898bc4946592c6a56a0fed4824e07447"} err="failed to get container status \"ef17000dbb685bf9b0c1b41ceb868cf7898bc4946592c6a56a0fed4824e07447\": rpc error: code = NotFound desc = could not find container \"ef17000dbb685bf9b0c1b41ceb868cf7898bc4946592c6a56a0fed4824e07447\": container with ID starting with ef17000dbb685bf9b0c1b41ceb868cf7898bc4946592c6a56a0fed4824e07447 not found: ID does not exist"
Sep 11 03:47:29.880306 ip-10-0-141-105 kubenswrapper[2579]: E0911 03:47:29.880285    2579 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-5f685c496f-lnwx2\" is forbidden: User \"system:node:ip-10-0-141-105.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-141-105.ec2.internal' and this object" podUID="453f5a42-4409-48c1-af52-848602e024c7" pod="kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-lnwx2"
Sep 11 03:47:30.851587 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:47:30.851552    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="453f5a42-4409-48c1-af52-848602e024c7" path="/var/lib/kubelet/pods/453f5a42-4409-48c1-af52-848602e024c7/volumes"
Sep 11 03:47:39.872905 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:47:39.872867    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-9wq2t"
Sep 11 03:47:55.468000 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:47:55.467924    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-9wq2t"]
Sep 11 03:47:55.468534 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:47:55.468160    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-9wq2t" podUID="a005b167-d364-40c1-9393-72ce6e5a098c" containerName="manager" containerID="cri-o://9c16ff0d0256ff1fab849be99d8fb039ab7ad84ed57d931c373d9d3830d39831" gracePeriod=10
Sep 11 03:47:55.712747 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:47:55.712722    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-9wq2t"
Sep 11 03:47:55.789046 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:47:55.788958    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/a005b167-d364-40c1-9393-72ce6e5a098c-extensions-socket-volume\") pod \"a005b167-d364-40c1-9393-72ce6e5a098c\" (UID: \"a005b167-d364-40c1-9393-72ce6e5a098c\") "
Sep 11 03:47:55.789254 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:47:55.789068    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hgxwt\" (UniqueName: \"kubernetes.io/projected/a005b167-d364-40c1-9393-72ce6e5a098c-kube-api-access-hgxwt\") pod \"a005b167-d364-40c1-9393-72ce6e5a098c\" (UID: \"a005b167-d364-40c1-9393-72ce6e5a098c\") "
Sep 11 03:47:55.789453 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:47:55.789423    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a005b167-d364-40c1-9393-72ce6e5a098c-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "a005b167-d364-40c1-9393-72ce6e5a098c" (UID: "a005b167-d364-40c1-9393-72ce6e5a098c"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 03:47:55.791595 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:47:55.791563    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a005b167-d364-40c1-9393-72ce6e5a098c-kube-api-access-hgxwt" (OuterVolumeSpecName: "kube-api-access-hgxwt") pod "a005b167-d364-40c1-9393-72ce6e5a098c" (UID: "a005b167-d364-40c1-9393-72ce6e5a098c"). InnerVolumeSpecName "kube-api-access-hgxwt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 03:47:55.890045 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:47:55.889999    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hgxwt\" (UniqueName: \"kubernetes.io/projected/a005b167-d364-40c1-9393-72ce6e5a098c-kube-api-access-hgxwt\") on node \"ip-10-0-141-105.ec2.internal\" DevicePath \"\""
Sep 11 03:47:55.890045 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:47:55.890037    2579 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/a005b167-d364-40c1-9393-72ce6e5a098c-extensions-socket-volume\") on node \"ip-10-0-141-105.ec2.internal\" DevicePath \"\""
Sep 11 03:47:55.990958 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:47:55.990921    2579 generic.go:356] "Generic (PLEG): container finished" podID="a005b167-d364-40c1-9393-72ce6e5a098c" containerID="9c16ff0d0256ff1fab849be99d8fb039ab7ad84ed57d931c373d9d3830d39831" exitCode=0
Sep 11 03:47:55.991175 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:47:55.990966    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-9wq2t"
Sep 11 03:47:55.991175 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:47:55.991008    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-9wq2t" event={"ID":"a005b167-d364-40c1-9393-72ce6e5a098c","Type":"ContainerDied","Data":"9c16ff0d0256ff1fab849be99d8fb039ab7ad84ed57d931c373d9d3830d39831"}
Sep 11 03:47:55.991175 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:47:55.991061    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-9wq2t" event={"ID":"a005b167-d364-40c1-9393-72ce6e5a098c","Type":"ContainerDied","Data":"5f233c05de661d9d83e9485792748581225f6dc92247c8c31d8aa4a89ec5d538"}
Sep 11 03:47:55.991175 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:47:55.991113    2579 scope.go:117] "RemoveContainer" containerID="9c16ff0d0256ff1fab849be99d8fb039ab7ad84ed57d931c373d9d3830d39831"
Sep 11 03:47:56.000418 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:47:56.000400    2579 scope.go:117] "RemoveContainer" containerID="9c16ff0d0256ff1fab849be99d8fb039ab7ad84ed57d931c373d9d3830d39831"
Sep 11 03:47:56.000652 ip-10-0-141-105 kubenswrapper[2579]: E0911 03:47:56.000626    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9c16ff0d0256ff1fab849be99d8fb039ab7ad84ed57d931c373d9d3830d39831\": container with ID starting with 9c16ff0d0256ff1fab849be99d8fb039ab7ad84ed57d931c373d9d3830d39831 not found: ID does not exist" containerID="9c16ff0d0256ff1fab849be99d8fb039ab7ad84ed57d931c373d9d3830d39831"
Sep 11 03:47:56.000705 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:47:56.000660    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9c16ff0d0256ff1fab849be99d8fb039ab7ad84ed57d931c373d9d3830d39831"} err="failed to get container status \"9c16ff0d0256ff1fab849be99d8fb039ab7ad84ed57d931c373d9d3830d39831\": rpc error: code = NotFound desc = could not find container \"9c16ff0d0256ff1fab849be99d8fb039ab7ad84ed57d931c373d9d3830d39831\": container with ID starting with 9c16ff0d0256ff1fab849be99d8fb039ab7ad84ed57d931c373d9d3830d39831 not found: ID does not exist"
Sep 11 03:47:56.012386 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:47:56.012361    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-9wq2t"]
Sep 11 03:47:56.018634 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:47:56.018611    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-9wq2t"]
Sep 11 03:47:56.852349 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:47:56.852314    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a005b167-d364-40c1-9393-72ce6e5a098c" path="/var/lib/kubelet/pods/a005b167-d364-40c1-9393-72ce6e5a098c/volumes"
Sep 11 03:48:14.294916 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:48:14.294880    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-5z4cn"]
Sep 11 03:48:14.295429 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:48:14.295236    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="453f5a42-4409-48c1-af52-848602e024c7" containerName="manager"
Sep 11 03:48:14.295429 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:48:14.295248    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="453f5a42-4409-48c1-af52-848602e024c7" containerName="manager"
Sep 11 03:48:14.295429 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:48:14.295262    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a005b167-d364-40c1-9393-72ce6e5a098c" containerName="manager"
Sep 11 03:48:14.295429 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:48:14.295267    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="a005b167-d364-40c1-9393-72ce6e5a098c" containerName="manager"
Sep 11 03:48:14.295429 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:48:14.295331    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="453f5a42-4409-48c1-af52-848602e024c7" containerName="manager"
Sep 11 03:48:14.295429 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:48:14.295340    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="a005b167-d364-40c1-9393-72ce6e5a098c" containerName="manager"
Sep 11 03:48:14.297952 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:48:14.297935    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-84994f7ffc-5z4cn"
Sep 11 03:48:14.300240 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:48:14.300221    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\""
Sep 11 03:48:14.300432 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:48:14.300415    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-t5r8x\""
Sep 11 03:48:14.305283 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:48:14.305262    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-5z4cn"]
Sep 11 03:48:14.318437 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:48:14.318407    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-5z4cn"]
Sep 11 03:48:14.448593 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:48:14.448554    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mkzwj\" (UniqueName: \"kubernetes.io/projected/cb7d6979-e4f6-469b-8be1-fcf4c05bd737-kube-api-access-mkzwj\") pod \"limitador-limitador-84994f7ffc-5z4cn\" (UID: \"cb7d6979-e4f6-469b-8be1-fcf4c05bd737\") " pod="kuadrant-system/limitador-limitador-84994f7ffc-5z4cn"
Sep 11 03:48:14.448593 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:48:14.448592    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/cb7d6979-e4f6-469b-8be1-fcf4c05bd737-config-file\") pod \"limitador-limitador-84994f7ffc-5z4cn\" (UID: \"cb7d6979-e4f6-469b-8be1-fcf4c05bd737\") " pod="kuadrant-system/limitador-limitador-84994f7ffc-5z4cn"
Sep 11 03:48:14.549917 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:48:14.549842    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mkzwj\" (UniqueName: \"kubernetes.io/projected/cb7d6979-e4f6-469b-8be1-fcf4c05bd737-kube-api-access-mkzwj\") pod \"limitador-limitador-84994f7ffc-5z4cn\" (UID: \"cb7d6979-e4f6-469b-8be1-fcf4c05bd737\") " pod="kuadrant-system/limitador-limitador-84994f7ffc-5z4cn"
Sep 11 03:48:14.549917 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:48:14.549873    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/cb7d6979-e4f6-469b-8be1-fcf4c05bd737-config-file\") pod \"limitador-limitador-84994f7ffc-5z4cn\" (UID: \"cb7d6979-e4f6-469b-8be1-fcf4c05bd737\") " pod="kuadrant-system/limitador-limitador-84994f7ffc-5z4cn"
Sep 11 03:48:14.550444 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:48:14.550425    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/cb7d6979-e4f6-469b-8be1-fcf4c05bd737-config-file\") pod \"limitador-limitador-84994f7ffc-5z4cn\" (UID: \"cb7d6979-e4f6-469b-8be1-fcf4c05bd737\") " pod="kuadrant-system/limitador-limitador-84994f7ffc-5z4cn"
Sep 11 03:48:14.559322 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:48:14.559296    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mkzwj\" (UniqueName: \"kubernetes.io/projected/cb7d6979-e4f6-469b-8be1-fcf4c05bd737-kube-api-access-mkzwj\") pod \"limitador-limitador-84994f7ffc-5z4cn\" (UID: \"cb7d6979-e4f6-469b-8be1-fcf4c05bd737\") " pod="kuadrant-system/limitador-limitador-84994f7ffc-5z4cn"
Sep 11 03:48:14.609287 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:48:14.609247    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-84994f7ffc-5z4cn"
Sep 11 03:48:14.776904 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:48:14.776876    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-llv9j"]
Sep 11 03:48:14.779246 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:48:14.779230    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7bcfd8f9cd-llv9j"
Sep 11 03:48:14.781672 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:48:14.781642    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-cjkt4\""
Sep 11 03:48:14.786865 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:48:14.786846    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-llv9j"]
Sep 11 03:48:14.853175 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:48:14.853149    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6hq4d\" (UniqueName: \"kubernetes.io/projected/d29cd113-0ead-4acd-86c2-824fcd2a0027-kube-api-access-6hq4d\") pod \"authorino-7bcfd8f9cd-llv9j\" (UID: \"d29cd113-0ead-4acd-86c2-824fcd2a0027\") " pod="kuadrant-system/authorino-7bcfd8f9cd-llv9j"
Sep 11 03:48:14.876749 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:48:14.876695    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-5z4cn"]
Sep 11 03:48:14.878910 ip-10-0-141-105 kubenswrapper[2579]: W0911 03:48:14.878885    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcb7d6979_e4f6_469b_8be1_fcf4c05bd737.slice/crio-7cfe8976b7dc312449ba1a4904708299a6ec5ed1a4ac77ec272c642903b947fa WatchSource:0}: Error finding container 7cfe8976b7dc312449ba1a4904708299a6ec5ed1a4ac77ec272c642903b947fa: Status 404 returned error can't find the container with id 7cfe8976b7dc312449ba1a4904708299a6ec5ed1a4ac77ec272c642903b947fa
Sep 11 03:48:14.954643 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:48:14.954607    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6hq4d\" (UniqueName: \"kubernetes.io/projected/d29cd113-0ead-4acd-86c2-824fcd2a0027-kube-api-access-6hq4d\") pod \"authorino-7bcfd8f9cd-llv9j\" (UID: \"d29cd113-0ead-4acd-86c2-824fcd2a0027\") " pod="kuadrant-system/authorino-7bcfd8f9cd-llv9j"
Sep 11 03:48:14.962780 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:48:14.962753    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6hq4d\" (UniqueName: \"kubernetes.io/projected/d29cd113-0ead-4acd-86c2-824fcd2a0027-kube-api-access-6hq4d\") pod \"authorino-7bcfd8f9cd-llv9j\" (UID: \"d29cd113-0ead-4acd-86c2-824fcd2a0027\") " pod="kuadrant-system/authorino-7bcfd8f9cd-llv9j"
Sep 11 03:48:15.074535 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:48:15.074500    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-84994f7ffc-5z4cn" event={"ID":"cb7d6979-e4f6-469b-8be1-fcf4c05bd737","Type":"ContainerStarted","Data":"7cfe8976b7dc312449ba1a4904708299a6ec5ed1a4ac77ec272c642903b947fa"}
Sep 11 03:48:15.089621 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:48:15.089590    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7bcfd8f9cd-llv9j"
Sep 11 03:48:15.356838 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:48:15.356797    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-llv9j"]
Sep 11 03:48:15.360474 ip-10-0-141-105 kubenswrapper[2579]: W0911 03:48:15.360451    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd29cd113_0ead_4acd_86c2_824fcd2a0027.slice/crio-d419e1fec649cf4f7d8c897bedc9d7279124524f12422224b26ea3174572a932 WatchSource:0}: Error finding container d419e1fec649cf4f7d8c897bedc9d7279124524f12422224b26ea3174572a932: Status 404 returned error can't find the container with id d419e1fec649cf4f7d8c897bedc9d7279124524f12422224b26ea3174572a932
Sep 11 03:48:16.081877 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:48:16.081783    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-84994f7ffc-5z4cn" event={"ID":"cb7d6979-e4f6-469b-8be1-fcf4c05bd737","Type":"ContainerStarted","Data":"d7e8f3f1d45ec9ec51643bbeae771e7a60dbdd179d736155624377020f31fbba"}
Sep 11 03:48:16.082067 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:48:16.081908    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-84994f7ffc-5z4cn"
Sep 11 03:48:16.083144 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:48:16.083114    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7bcfd8f9cd-llv9j" event={"ID":"d29cd113-0ead-4acd-86c2-824fcd2a0027","Type":"ContainerStarted","Data":"d419e1fec649cf4f7d8c897bedc9d7279124524f12422224b26ea3174572a932"}
Sep 11 03:48:16.099980 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:48:16.099926    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-84994f7ffc-5z4cn" podStartSLOduration=1.199743213 podStartE2EDuration="2.099911189s" totalImagesPullingTime="900.167976ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 03:48:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 03:48:16.097771925 +0000 UTC m=+473.752977494" watchObservedRunningTime="2026-09-11 03:48:16.099911189 +0000 UTC m=+473.755116755"
Sep 11 03:48:18.093653 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:48:18.093613    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7bcfd8f9cd-llv9j" event={"ID":"d29cd113-0ead-4acd-86c2-824fcd2a0027","Type":"ContainerStarted","Data":"712c4747a53ec85095b585453ed0f623c9cfb57539dba6dfcad9df0e3bb17471"}
Sep 11 03:48:18.107702 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:48:18.107652    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-7bcfd8f9cd-llv9j" podStartSLOduration=2.416251843 podStartE2EDuration="4.107634931s" totalImagesPullingTime="1.691383088s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 03:48:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 03:48:18.106338285 +0000 UTC m=+475.761543853" watchObservedRunningTime="2026-09-11 03:48:18.107634931 +0000 UTC m=+475.762840498"
Sep 11 03:48:18.130474 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:48:18.130441    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-llv9j"]
Sep 11 03:48:20.102843 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:48:20.102773    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-7bcfd8f9cd-llv9j" podUID="d29cd113-0ead-4acd-86c2-824fcd2a0027" containerName="authorino" containerID="cri-o://712c4747a53ec85095b585453ed0f623c9cfb57539dba6dfcad9df0e3bb17471" gracePeriod=30
Sep 11 03:48:20.348317 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:48:20.348294    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7bcfd8f9cd-llv9j"
Sep 11 03:48:20.507996 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:48:20.507910    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6hq4d\" (UniqueName: \"kubernetes.io/projected/d29cd113-0ead-4acd-86c2-824fcd2a0027-kube-api-access-6hq4d\") pod \"d29cd113-0ead-4acd-86c2-824fcd2a0027\" (UID: \"d29cd113-0ead-4acd-86c2-824fcd2a0027\") "
Sep 11 03:48:20.510172 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:48:20.510149    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d29cd113-0ead-4acd-86c2-824fcd2a0027-kube-api-access-6hq4d" (OuterVolumeSpecName: "kube-api-access-6hq4d") pod "d29cd113-0ead-4acd-86c2-824fcd2a0027" (UID: "d29cd113-0ead-4acd-86c2-824fcd2a0027"). InnerVolumeSpecName "kube-api-access-6hq4d". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 03:48:20.608818 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:48:20.608785    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6hq4d\" (UniqueName: \"kubernetes.io/projected/d29cd113-0ead-4acd-86c2-824fcd2a0027-kube-api-access-6hq4d\") on node \"ip-10-0-141-105.ec2.internal\" DevicePath \"\""
Sep 11 03:48:21.109747 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:48:21.109650    2579 generic.go:356] "Generic (PLEG): container finished" podID="d29cd113-0ead-4acd-86c2-824fcd2a0027" containerID="712c4747a53ec85095b585453ed0f623c9cfb57539dba6dfcad9df0e3bb17471" exitCode=0
Sep 11 03:48:21.109747 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:48:21.109705    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7bcfd8f9cd-llv9j"
Sep 11 03:48:21.109747 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:48:21.109709    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7bcfd8f9cd-llv9j" event={"ID":"d29cd113-0ead-4acd-86c2-824fcd2a0027","Type":"ContainerDied","Data":"712c4747a53ec85095b585453ed0f623c9cfb57539dba6dfcad9df0e3bb17471"}
Sep 11 03:48:21.110260 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:48:21.109757    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7bcfd8f9cd-llv9j" event={"ID":"d29cd113-0ead-4acd-86c2-824fcd2a0027","Type":"ContainerDied","Data":"d419e1fec649cf4f7d8c897bedc9d7279124524f12422224b26ea3174572a932"}
Sep 11 03:48:21.110260 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:48:21.109779    2579 scope.go:117] "RemoveContainer" containerID="712c4747a53ec85095b585453ed0f623c9cfb57539dba6dfcad9df0e3bb17471"
Sep 11 03:48:21.118267 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:48:21.118248    2579 scope.go:117] "RemoveContainer" containerID="712c4747a53ec85095b585453ed0f623c9cfb57539dba6dfcad9df0e3bb17471"
Sep 11 03:48:21.118540 ip-10-0-141-105 kubenswrapper[2579]: E0911 03:48:21.118513    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"712c4747a53ec85095b585453ed0f623c9cfb57539dba6dfcad9df0e3bb17471\": container with ID starting with 712c4747a53ec85095b585453ed0f623c9cfb57539dba6dfcad9df0e3bb17471 not found: ID does not exist" containerID="712c4747a53ec85095b585453ed0f623c9cfb57539dba6dfcad9df0e3bb17471"
Sep 11 03:48:21.118600 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:48:21.118540    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"712c4747a53ec85095b585453ed0f623c9cfb57539dba6dfcad9df0e3bb17471"} err="failed to get container status \"712c4747a53ec85095b585453ed0f623c9cfb57539dba6dfcad9df0e3bb17471\": rpc error: code = NotFound desc = could not find container \"712c4747a53ec85095b585453ed0f623c9cfb57539dba6dfcad9df0e3bb17471\": container with ID starting with 712c4747a53ec85095b585453ed0f623c9cfb57539dba6dfcad9df0e3bb17471 not found: ID does not exist"
Sep 11 03:48:21.126945 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:48:21.126923    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-llv9j"]
Sep 11 03:48:21.133827 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:48:21.133795    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-llv9j"]
Sep 11 03:48:22.852263 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:48:22.852228    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d29cd113-0ead-4acd-86c2-824fcd2a0027" path="/var/lib/kubelet/pods/d29cd113-0ead-4acd-86c2-824fcd2a0027/volumes"
Sep 11 03:48:27.089451 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:48:27.089417    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-84994f7ffc-5z4cn"
Sep 11 03:48:40.734994 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:48:40.734956    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-557f876f69-6sjhv"]
Sep 11 03:48:40.735610 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:48:40.735518    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d29cd113-0ead-4acd-86c2-824fcd2a0027" containerName="authorino"
Sep 11 03:48:40.735610 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:48:40.735542    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="d29cd113-0ead-4acd-86c2-824fcd2a0027" containerName="authorino"
Sep 11 03:48:40.735733 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:48:40.735629    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="d29cd113-0ead-4acd-86c2-824fcd2a0027" containerName="authorino"
Sep 11 03:48:40.740010 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:48:40.739974    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-557f876f69-6sjhv"
Sep 11 03:48:40.742577 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:48:40.742554    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-cjkt4\""
Sep 11 03:48:40.745127 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:48:40.745104    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-557f876f69-6sjhv"]
Sep 11 03:48:40.770971 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:48:40.770949    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5vqn4\" (UniqueName: \"kubernetes.io/projected/f39ff180-63f0-4d2d-b22b-b0bf62695c2b-kube-api-access-5vqn4\") pod \"authorino-557f876f69-6sjhv\" (UID: \"f39ff180-63f0-4d2d-b22b-b0bf62695c2b\") " pod="kuadrant-system/authorino-557f876f69-6sjhv"
Sep 11 03:48:40.870593 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:48:40.870560    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-557f876f69-6sjhv"]
Sep 11 03:48:40.870796 ip-10-0-141-105 kubenswrapper[2579]: E0911 03:48:40.870781    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kube-api-access-5vqn4], unattached volumes=[], failed to process volumes=[]: context canceled" pod="kuadrant-system/authorino-557f876f69-6sjhv" podUID="f39ff180-63f0-4d2d-b22b-b0bf62695c2b"
Sep 11 03:48:40.871869 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:48:40.871827    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5vqn4\" (UniqueName: \"kubernetes.io/projected/f39ff180-63f0-4d2d-b22b-b0bf62695c2b-kube-api-access-5vqn4\") pod \"authorino-557f876f69-6sjhv\" (UID: \"f39ff180-63f0-4d2d-b22b-b0bf62695c2b\") " pod="kuadrant-system/authorino-557f876f69-6sjhv"
Sep 11 03:48:40.887954 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:48:40.887924    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5vqn4\" (UniqueName: \"kubernetes.io/projected/f39ff180-63f0-4d2d-b22b-b0bf62695c2b-kube-api-access-5vqn4\") pod \"authorino-557f876f69-6sjhv\" (UID: \"f39ff180-63f0-4d2d-b22b-b0bf62695c2b\") " pod="kuadrant-system/authorino-557f876f69-6sjhv"
Sep 11 03:48:40.895942 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:48:40.895917    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-6d4dfc649f-x49x7"]
Sep 11 03:48:40.899278 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:48:40.899262    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6d4dfc649f-x49x7"
Sep 11 03:48:40.901789 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:48:40.901768    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-server-cert\""
Sep 11 03:48:40.906529 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:48:40.906507    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-6d4dfc649f-x49x7"]
Sep 11 03:48:40.959237 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:48:40.959201    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6d4dfc649f-x49x7"]
Sep 11 03:48:40.959477 ip-10-0-141-105 kubenswrapper[2579]: E0911 03:48:40.959457    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kube-api-access-wngzb tls-cert], unattached volumes=[], failed to process volumes=[kube-api-access-wngzb tls-cert]: context canceled" pod="kuadrant-system/authorino-6d4dfc649f-x49x7" podUID="50495ebe-4a9d-4990-a0ed-60e8ce5bb751"
Sep 11 03:48:40.972764 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:48:40.972736    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wngzb\" (UniqueName: \"kubernetes.io/projected/50495ebe-4a9d-4990-a0ed-60e8ce5bb751-kube-api-access-wngzb\") pod \"authorino-6d4dfc649f-x49x7\" (UID: \"50495ebe-4a9d-4990-a0ed-60e8ce5bb751\") " pod="kuadrant-system/authorino-6d4dfc649f-x49x7"
Sep 11 03:48:40.972912 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:48:40.972795    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/50495ebe-4a9d-4990-a0ed-60e8ce5bb751-tls-cert\") pod \"authorino-6d4dfc649f-x49x7\" (UID: \"50495ebe-4a9d-4990-a0ed-60e8ce5bb751\") " pod="kuadrant-system/authorino-6d4dfc649f-x49x7"
Sep 11 03:48:40.981865 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:48:40.981832    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-7fb557657d-jql4x"]
Sep 11 03:48:40.985031 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:48:40.984981    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7fb557657d-jql4x"
Sep 11 03:48:40.992219 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:48:40.992115    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-7fb557657d-jql4x"]
Sep 11 03:48:41.073624 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:48:41.073594    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/50495ebe-4a9d-4990-a0ed-60e8ce5bb751-tls-cert\") pod \"authorino-6d4dfc649f-x49x7\" (UID: \"50495ebe-4a9d-4990-a0ed-60e8ce5bb751\") " pod="kuadrant-system/authorino-6d4dfc649f-x49x7"
Sep 11 03:48:41.073624 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:48:41.073628    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/6c72db99-14e1-4659-8899-5de07da70884-tls-cert\") pod \"authorino-7fb557657d-jql4x\" (UID: \"6c72db99-14e1-4659-8899-5de07da70884\") " pod="kuadrant-system/authorino-7fb557657d-jql4x"
Sep 11 03:48:41.074067 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:48:41.073656    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-94ln9\" (UniqueName: \"kubernetes.io/projected/6c72db99-14e1-4659-8899-5de07da70884-kube-api-access-94ln9\") pod \"authorino-7fb557657d-jql4x\" (UID: \"6c72db99-14e1-4659-8899-5de07da70884\") " pod="kuadrant-system/authorino-7fb557657d-jql4x"
Sep 11 03:48:41.074067 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:48:41.073739    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wngzb\" (UniqueName: \"kubernetes.io/projected/50495ebe-4a9d-4990-a0ed-60e8ce5bb751-kube-api-access-wngzb\") pod \"authorino-6d4dfc649f-x49x7\" (UID: \"50495ebe-4a9d-4990-a0ed-60e8ce5bb751\") " pod="kuadrant-system/authorino-6d4dfc649f-x49x7"
Sep 11 03:48:41.076241 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:48:41.076220    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/50495ebe-4a9d-4990-a0ed-60e8ce5bb751-tls-cert\") pod \"authorino-6d4dfc649f-x49x7\" (UID: \"50495ebe-4a9d-4990-a0ed-60e8ce5bb751\") " pod="kuadrant-system/authorino-6d4dfc649f-x49x7"
Sep 11 03:48:41.081721 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:48:41.081705    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wngzb\" (UniqueName: \"kubernetes.io/projected/50495ebe-4a9d-4990-a0ed-60e8ce5bb751-kube-api-access-wngzb\") pod \"authorino-6d4dfc649f-x49x7\" (UID: \"50495ebe-4a9d-4990-a0ed-60e8ce5bb751\") " pod="kuadrant-system/authorino-6d4dfc649f-x49x7"
Sep 11 03:48:41.174394 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:48:41.174361    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-94ln9\" (UniqueName: \"kubernetes.io/projected/6c72db99-14e1-4659-8899-5de07da70884-kube-api-access-94ln9\") pod \"authorino-7fb557657d-jql4x\" (UID: \"6c72db99-14e1-4659-8899-5de07da70884\") " pod="kuadrant-system/authorino-7fb557657d-jql4x"
Sep 11 03:48:41.174552 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:48:41.174453    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/6c72db99-14e1-4659-8899-5de07da70884-tls-cert\") pod \"authorino-7fb557657d-jql4x\" (UID: \"6c72db99-14e1-4659-8899-5de07da70884\") " pod="kuadrant-system/authorino-7fb557657d-jql4x"
Sep 11 03:48:41.176796 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:48:41.176778    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/6c72db99-14e1-4659-8899-5de07da70884-tls-cert\") pod \"authorino-7fb557657d-jql4x\" (UID: \"6c72db99-14e1-4659-8899-5de07da70884\") " pod="kuadrant-system/authorino-7fb557657d-jql4x"
Sep 11 03:48:41.182614 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:48:41.182593    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-94ln9\" (UniqueName: \"kubernetes.io/projected/6c72db99-14e1-4659-8899-5de07da70884-kube-api-access-94ln9\") pod \"authorino-7fb557657d-jql4x\" (UID: \"6c72db99-14e1-4659-8899-5de07da70884\") " pod="kuadrant-system/authorino-7fb557657d-jql4x"
Sep 11 03:48:41.197574 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:48:41.197543    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6d4dfc649f-x49x7"
Sep 11 03:48:41.197574 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:48:41.197553    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-557f876f69-6sjhv"
Sep 11 03:48:41.204761 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:48:41.204741    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-557f876f69-6sjhv"
Sep 11 03:48:41.207659 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:48:41.207641    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6d4dfc649f-x49x7"
Sep 11 03:48:41.274986 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:48:41.274892    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wngzb\" (UniqueName: \"kubernetes.io/projected/50495ebe-4a9d-4990-a0ed-60e8ce5bb751-kube-api-access-wngzb\") pod \"50495ebe-4a9d-4990-a0ed-60e8ce5bb751\" (UID: \"50495ebe-4a9d-4990-a0ed-60e8ce5bb751\") "
Sep 11 03:48:41.274986 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:48:41.274937    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/50495ebe-4a9d-4990-a0ed-60e8ce5bb751-tls-cert\") pod \"50495ebe-4a9d-4990-a0ed-60e8ce5bb751\" (UID: \"50495ebe-4a9d-4990-a0ed-60e8ce5bb751\") "
Sep 11 03:48:41.274986 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:48:41.274985    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5vqn4\" (UniqueName: \"kubernetes.io/projected/f39ff180-63f0-4d2d-b22b-b0bf62695c2b-kube-api-access-5vqn4\") pod \"f39ff180-63f0-4d2d-b22b-b0bf62695c2b\" (UID: \"f39ff180-63f0-4d2d-b22b-b0bf62695c2b\") "
Sep 11 03:48:41.277150 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:48:41.277114    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/50495ebe-4a9d-4990-a0ed-60e8ce5bb751-tls-cert" (OuterVolumeSpecName: "tls-cert") pod "50495ebe-4a9d-4990-a0ed-60e8ce5bb751" (UID: "50495ebe-4a9d-4990-a0ed-60e8ce5bb751"). InnerVolumeSpecName "tls-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 03:48:41.277261 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:48:41.277211    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/50495ebe-4a9d-4990-a0ed-60e8ce5bb751-kube-api-access-wngzb" (OuterVolumeSpecName: "kube-api-access-wngzb") pod "50495ebe-4a9d-4990-a0ed-60e8ce5bb751" (UID: "50495ebe-4a9d-4990-a0ed-60e8ce5bb751"). InnerVolumeSpecName "kube-api-access-wngzb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 03:48:41.277261 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:48:41.277237    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f39ff180-63f0-4d2d-b22b-b0bf62695c2b-kube-api-access-5vqn4" (OuterVolumeSpecName: "kube-api-access-5vqn4") pod "f39ff180-63f0-4d2d-b22b-b0bf62695c2b" (UID: "f39ff180-63f0-4d2d-b22b-b0bf62695c2b"). InnerVolumeSpecName "kube-api-access-5vqn4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 03:48:41.295159 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:48:41.295139    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7fb557657d-jql4x"
Sep 11 03:48:41.376155 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:48:41.376127    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wngzb\" (UniqueName: \"kubernetes.io/projected/50495ebe-4a9d-4990-a0ed-60e8ce5bb751-kube-api-access-wngzb\") on node \"ip-10-0-141-105.ec2.internal\" DevicePath \"\""
Sep 11 03:48:41.376155 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:48:41.376154    2579 reconciler_common.go:299] "Volume detached for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/50495ebe-4a9d-4990-a0ed-60e8ce5bb751-tls-cert\") on node \"ip-10-0-141-105.ec2.internal\" DevicePath \"\""
Sep 11 03:48:41.376376 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:48:41.376168    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5vqn4\" (UniqueName: \"kubernetes.io/projected/f39ff180-63f0-4d2d-b22b-b0bf62695c2b-kube-api-access-5vqn4\") on node \"ip-10-0-141-105.ec2.internal\" DevicePath \"\""
Sep 11 03:48:41.564983 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:48:41.564961    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-7fb557657d-jql4x"]
Sep 11 03:48:41.567140 ip-10-0-141-105 kubenswrapper[2579]: W0911 03:48:41.567116    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod6c72db99_14e1_4659_8899_5de07da70884.slice/crio-a1b683a4f648c2d0934722eae018cd8951782b3dd7e348adf43469e92f965fce WatchSource:0}: Error finding container a1b683a4f648c2d0934722eae018cd8951782b3dd7e348adf43469e92f965fce: Status 404 returned error can't find the container with id a1b683a4f648c2d0934722eae018cd8951782b3dd7e348adf43469e92f965fce
Sep 11 03:48:42.204262 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:48:42.204154    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7fb557657d-jql4x" event={"ID":"6c72db99-14e1-4659-8899-5de07da70884","Type":"ContainerStarted","Data":"f20726e66bd658b37ebd83138171a5d218290c0a5db42e897a1331e53e40ced4"}
Sep 11 03:48:42.204262 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:48:42.204183    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6d4dfc649f-x49x7"
Sep 11 03:48:42.204262 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:48:42.204210    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7fb557657d-jql4x" event={"ID":"6c72db99-14e1-4659-8899-5de07da70884","Type":"ContainerStarted","Data":"a1b683a4f648c2d0934722eae018cd8951782b3dd7e348adf43469e92f965fce"}
Sep 11 03:48:42.204262 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:48:42.204169    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-557f876f69-6sjhv"
Sep 11 03:48:42.220620 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:48:42.220576    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-7fb557657d-jql4x" podStartSLOduration=1.856836256 podStartE2EDuration="2.220560554s" totalImagesPullingTime="363.724298ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 03:48:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 03:48:42.218631176 +0000 UTC m=+499.873836742" watchObservedRunningTime="2026-09-11 03:48:42.220560554 +0000 UTC m=+499.875766121"
Sep 11 03:48:42.248519 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:48:42.248488    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-557f876f69-6sjhv"]
Sep 11 03:48:42.251675 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:48:42.251652    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-557f876f69-6sjhv"]
Sep 11 03:48:42.277245 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:48:42.277216    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6d4dfc649f-x49x7"]
Sep 11 03:48:42.280689 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:48:42.280662    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-6d4dfc649f-x49x7"]
Sep 11 03:48:42.852509 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:48:42.852473    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="50495ebe-4a9d-4990-a0ed-60e8ce5bb751" path="/var/lib/kubelet/pods/50495ebe-4a9d-4990-a0ed-60e8ce5bb751/volumes"
Sep 11 03:48:42.852707 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:48:42.852695    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f39ff180-63f0-4d2d-b22b-b0bf62695c2b" path="/var/lib/kubelet/pods/f39ff180-63f0-4d2d-b22b-b0bf62695c2b/volumes"
Sep 11 03:49:27.026790 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:49:27.026712    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-5b54b7fbb7-69ccr"]
Sep 11 03:49:27.030063 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:49:27.030047    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-5b54b7fbb7-69ccr"
Sep 11 03:49:27.032871 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:49:27.032820    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins\""
Sep 11 03:49:27.033071 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:49:27.032978    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-dockercfg-hpjm2\""
Sep 11 03:49:27.039791 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:49:27.039768    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-5b54b7fbb7-69ccr"]
Sep 11 03:49:27.073800 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:49:27.073770    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/598a451c-3387-4674-9c03-6ea4ca648208-plugins-config-volume\") pod \"payload-processing-5b54b7fbb7-69ccr\" (UID: \"598a451c-3387-4674-9c03-6ea4ca648208\") " pod="openshift-ingress/payload-processing-5b54b7fbb7-69ccr"
Sep 11 03:49:27.073935 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:49:27.073853    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9k8zt\" (UniqueName: \"kubernetes.io/projected/598a451c-3387-4674-9c03-6ea4ca648208-kube-api-access-9k8zt\") pod \"payload-processing-5b54b7fbb7-69ccr\" (UID: \"598a451c-3387-4674-9c03-6ea4ca648208\") " pod="openshift-ingress/payload-processing-5b54b7fbb7-69ccr"
Sep 11 03:49:27.131188 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:49:27.131152    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-5z4cn"]
Sep 11 03:49:27.174452 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:49:27.174417    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9k8zt\" (UniqueName: \"kubernetes.io/projected/598a451c-3387-4674-9c03-6ea4ca648208-kube-api-access-9k8zt\") pod \"payload-processing-5b54b7fbb7-69ccr\" (UID: \"598a451c-3387-4674-9c03-6ea4ca648208\") " pod="openshift-ingress/payload-processing-5b54b7fbb7-69ccr"
Sep 11 03:49:27.174649 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:49:27.174500    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/598a451c-3387-4674-9c03-6ea4ca648208-plugins-config-volume\") pod \"payload-processing-5b54b7fbb7-69ccr\" (UID: \"598a451c-3387-4674-9c03-6ea4ca648208\") " pod="openshift-ingress/payload-processing-5b54b7fbb7-69ccr"
Sep 11 03:49:27.175287 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:49:27.175266    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/598a451c-3387-4674-9c03-6ea4ca648208-plugins-config-volume\") pod \"payload-processing-5b54b7fbb7-69ccr\" (UID: \"598a451c-3387-4674-9c03-6ea4ca648208\") " pod="openshift-ingress/payload-processing-5b54b7fbb7-69ccr"
Sep 11 03:49:27.190378 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:49:27.190345    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9k8zt\" (UniqueName: \"kubernetes.io/projected/598a451c-3387-4674-9c03-6ea4ca648208-kube-api-access-9k8zt\") pod \"payload-processing-5b54b7fbb7-69ccr\" (UID: \"598a451c-3387-4674-9c03-6ea4ca648208\") " pod="openshift-ingress/payload-processing-5b54b7fbb7-69ccr"
Sep 11 03:49:27.342391 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:49:27.342302    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-5b54b7fbb7-69ccr"
Sep 11 03:49:27.616824 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:49:27.616798    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-5b54b7fbb7-69ccr"]
Sep 11 03:49:27.619504 ip-10-0-141-105 kubenswrapper[2579]: W0911 03:49:27.619480    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod598a451c_3387_4674_9c03_6ea4ca648208.slice/crio-7c7914656437b434a785a445092c23334d6c8813ad3f6e50e9355127877e0360 WatchSource:0}: Error finding container 7c7914656437b434a785a445092c23334d6c8813ad3f6e50e9355127877e0360: Status 404 returned error can't find the container with id 7c7914656437b434a785a445092c23334d6c8813ad3f6e50e9355127877e0360
Sep 11 03:49:28.419187 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:49:28.419123    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-5b54b7fbb7-69ccr" event={"ID":"598a451c-3387-4674-9c03-6ea4ca648208","Type":"ContainerStarted","Data":"7c7914656437b434a785a445092c23334d6c8813ad3f6e50e9355127877e0360"}
Sep 11 03:49:30.432470 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:49:30.432440    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-5b54b7fbb7-69ccr" event={"ID":"598a451c-3387-4674-9c03-6ea4ca648208","Type":"ContainerStarted","Data":"c57c926958500c2a079c6acaa0cc1ca5abc58dca4c7fbf294bbba3d2d8f67048"}
Sep 11 03:49:30.432869 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:49:30.432521    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-5b54b7fbb7-69ccr"
Sep 11 03:49:30.449651 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:49:30.449593    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-5b54b7fbb7-69ccr" podStartSLOduration=1.110360137 podStartE2EDuration="3.449578113s" totalImagesPullingTime="2.339217976s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 03:49:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 03:49:30.448990834 +0000 UTC m=+548.104196439" watchObservedRunningTime="2026-09-11 03:49:30.449578113 +0000 UTC m=+548.104783680"
Sep 11 03:49:41.439847 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:49:41.439816    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-5b54b7fbb7-69ccr"
Sep 11 03:49:55.206496 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:49:55.206459    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-5z4cn"]
Sep 11 03:50:08.416357 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:50:08.416317    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-5z4cn"]
Sep 11 03:50:14.202104 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:50:14.202049    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-5z4cn"]
Sep 11 03:50:22.100959 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:50:22.100925    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-5z4cn"]
Sep 11 03:50:28.203845 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:50:28.203800    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-5z4cn"]
Sep 11 03:50:36.789777 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:50:36.789737    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-9cg5z"]
Sep 11 03:50:36.792195 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:50:36.792180    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-9cg5z"
Sep 11 03:50:36.795753 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:50:36.795730    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-d5247\""
Sep 11 03:50:36.795883 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:50:36.795771    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\""
Sep 11 03:50:36.795883 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:50:36.795784    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"facebook-opt-125m-simulated-kserve-self-signed-certs\""
Sep 11 03:50:36.795883 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:50:36.795834    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\""
Sep 11 03:50:36.802909 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:50:36.802883    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-9cg5z"]
Sep 11 03:50:36.879810 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:50:36.879782    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d922ae85-71de-4dbe-ac65-75b8ae0d20c9-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-9cg5z\" (UID: \"d922ae85-71de-4dbe-ac65-75b8ae0d20c9\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-9cg5z"
Sep 11 03:50:36.879810 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:50:36.879814    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d922ae85-71de-4dbe-ac65-75b8ae0d20c9-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-9cg5z\" (UID: \"d922ae85-71de-4dbe-ac65-75b8ae0d20c9\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-9cg5z"
Sep 11 03:50:36.880047 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:50:36.879898    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jzvtr\" (UniqueName: \"kubernetes.io/projected/d922ae85-71de-4dbe-ac65-75b8ae0d20c9-kube-api-access-jzvtr\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-9cg5z\" (UID: \"d922ae85-71de-4dbe-ac65-75b8ae0d20c9\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-9cg5z"
Sep 11 03:50:36.880047 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:50:36.879937    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d922ae85-71de-4dbe-ac65-75b8ae0d20c9-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-9cg5z\" (UID: \"d922ae85-71de-4dbe-ac65-75b8ae0d20c9\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-9cg5z"
Sep 11 03:50:36.880047 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:50:36.879965    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d922ae85-71de-4dbe-ac65-75b8ae0d20c9-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-9cg5z\" (UID: \"d922ae85-71de-4dbe-ac65-75b8ae0d20c9\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-9cg5z"
Sep 11 03:50:36.880047 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:50:36.879997    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d922ae85-71de-4dbe-ac65-75b8ae0d20c9-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-9cg5z\" (UID: \"d922ae85-71de-4dbe-ac65-75b8ae0d20c9\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-9cg5z"
Sep 11 03:50:36.980903 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:50:36.980862    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d922ae85-71de-4dbe-ac65-75b8ae0d20c9-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-9cg5z\" (UID: \"d922ae85-71de-4dbe-ac65-75b8ae0d20c9\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-9cg5z"
Sep 11 03:50:36.980903 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:50:36.980910    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d922ae85-71de-4dbe-ac65-75b8ae0d20c9-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-9cg5z\" (UID: \"d922ae85-71de-4dbe-ac65-75b8ae0d20c9\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-9cg5z"
Sep 11 03:50:36.981193 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:50:36.980945    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jzvtr\" (UniqueName: \"kubernetes.io/projected/d922ae85-71de-4dbe-ac65-75b8ae0d20c9-kube-api-access-jzvtr\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-9cg5z\" (UID: \"d922ae85-71de-4dbe-ac65-75b8ae0d20c9\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-9cg5z"
Sep 11 03:50:36.981193 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:50:36.980981    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d922ae85-71de-4dbe-ac65-75b8ae0d20c9-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-9cg5z\" (UID: \"d922ae85-71de-4dbe-ac65-75b8ae0d20c9\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-9cg5z"
Sep 11 03:50:36.981193 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:50:36.981002    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d922ae85-71de-4dbe-ac65-75b8ae0d20c9-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-9cg5z\" (UID: \"d922ae85-71de-4dbe-ac65-75b8ae0d20c9\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-9cg5z"
Sep 11 03:50:36.981193 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:50:36.981021    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d922ae85-71de-4dbe-ac65-75b8ae0d20c9-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-9cg5z\" (UID: \"d922ae85-71de-4dbe-ac65-75b8ae0d20c9\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-9cg5z"
Sep 11 03:50:36.981474 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:50:36.981451    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d922ae85-71de-4dbe-ac65-75b8ae0d20c9-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-9cg5z\" (UID: \"d922ae85-71de-4dbe-ac65-75b8ae0d20c9\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-9cg5z"
Sep 11 03:50:36.981545 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:50:36.981459    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d922ae85-71de-4dbe-ac65-75b8ae0d20c9-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-9cg5z\" (UID: \"d922ae85-71de-4dbe-ac65-75b8ae0d20c9\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-9cg5z"
Sep 11 03:50:36.981545 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:50:36.981500    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d922ae85-71de-4dbe-ac65-75b8ae0d20c9-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-9cg5z\" (UID: \"d922ae85-71de-4dbe-ac65-75b8ae0d20c9\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-9cg5z"
Sep 11 03:50:36.983415 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:50:36.983392    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d922ae85-71de-4dbe-ac65-75b8ae0d20c9-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-9cg5z\" (UID: \"d922ae85-71de-4dbe-ac65-75b8ae0d20c9\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-9cg5z"
Sep 11 03:50:36.983742 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:50:36.983725    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d922ae85-71de-4dbe-ac65-75b8ae0d20c9-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-9cg5z\" (UID: \"d922ae85-71de-4dbe-ac65-75b8ae0d20c9\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-9cg5z"
Sep 11 03:50:36.989360 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:50:36.989338    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jzvtr\" (UniqueName: \"kubernetes.io/projected/d922ae85-71de-4dbe-ac65-75b8ae0d20c9-kube-api-access-jzvtr\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-9cg5z\" (UID: \"d922ae85-71de-4dbe-ac65-75b8ae0d20c9\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-9cg5z"
Sep 11 03:50:37.104490 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:50:37.104458    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-9cg5z"
Sep 11 03:50:37.380125 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:50:37.380099    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-9cg5z"]
Sep 11 03:50:37.382670 ip-10-0-141-105 kubenswrapper[2579]: W0911 03:50:37.382640    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd922ae85_71de_4dbe_ac65_75b8ae0d20c9.slice/crio-a901a430048986077906866a274aae0b1b156bc16aa8c6667dbc4abed46f3b9e WatchSource:0}: Error finding container a901a430048986077906866a274aae0b1b156bc16aa8c6667dbc4abed46f3b9e: Status 404 returned error can't find the container with id a901a430048986077906866a274aae0b1b156bc16aa8c6667dbc4abed46f3b9e
Sep 11 03:50:37.384649 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:50:37.384632    2579 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 11 03:50:37.750616 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:50:37.750536    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-9cg5z" event={"ID":"d922ae85-71de-4dbe-ac65-75b8ae0d20c9","Type":"ContainerStarted","Data":"a901a430048986077906866a274aae0b1b156bc16aa8c6667dbc4abed46f3b9e"}
Sep 11 03:50:38.611312 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:50:38.611275    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-5z4cn"]
Sep 11 03:50:38.758984 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:50:38.758887    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-9cg5z" event={"ID":"d922ae85-71de-4dbe-ac65-75b8ae0d20c9","Type":"ContainerStarted","Data":"c11bc24d7816d0496f154072faa9f39992a49d334e72f87abaf7edd6f53a9ee4"}
Sep 11 03:50:38.780316 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:50:38.780254    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-9cg5z" podStartSLOduration=1.745944958 podStartE2EDuration="2.78023759s" totalImagesPullingTime="1.034292632s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 03:50:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 03:50:38.778010169 +0000 UTC m=+616.433215747" watchObservedRunningTime="2026-09-11 03:50:38.78023759 +0000 UTC m=+616.435443158"
Sep 11 03:50:47.104994 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:50:47.104903    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-9cg5z"
Sep 11 03:50:47.104994 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:50:47.104977    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-9cg5z"
Sep 11 03:50:47.117694 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:50:47.117666    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-9cg5z"
Sep 11 03:50:47.811938 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:50:47.811914    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-9cg5z"
Sep 11 03:50:49.895307 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:50:49.895267    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-57fc66f697-m7krz"]
Sep 11 03:50:49.898777 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:50:49.898757    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-m7krz"
Sep 11 03:50:49.901199 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:50:49.901179    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"premium-simulated-simulated-premium-kserve-self-signed-certs\""
Sep 11 03:50:49.906734 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:50:49.906707    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-57fc66f697-m7krz"]
Sep 11 03:50:49.996732 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:50:49.996689    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0938a98d-0acd-4959-80b0-da59ad7f756c-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-m7krz\" (UID: \"0938a98d-0acd-4959-80b0-da59ad7f756c\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-m7krz"
Sep 11 03:50:49.996957 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:50:49.996736    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n862f\" (UniqueName: \"kubernetes.io/projected/0938a98d-0acd-4959-80b0-da59ad7f756c-kube-api-access-n862f\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-m7krz\" (UID: \"0938a98d-0acd-4959-80b0-da59ad7f756c\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-m7krz"
Sep 11 03:50:49.996957 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:50:49.996771    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0938a98d-0acd-4959-80b0-da59ad7f756c-home\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-m7krz\" (UID: \"0938a98d-0acd-4959-80b0-da59ad7f756c\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-m7krz"
Sep 11 03:50:49.996957 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:50:49.996816    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0938a98d-0acd-4959-80b0-da59ad7f756c-dshm\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-m7krz\" (UID: \"0938a98d-0acd-4959-80b0-da59ad7f756c\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-m7krz"
Sep 11 03:50:49.996957 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:50:49.996836    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0938a98d-0acd-4959-80b0-da59ad7f756c-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-m7krz\" (UID: \"0938a98d-0acd-4959-80b0-da59ad7f756c\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-m7krz"
Sep 11 03:50:49.996957 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:50:49.996903    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0938a98d-0acd-4959-80b0-da59ad7f756c-model-cache\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-m7krz\" (UID: \"0938a98d-0acd-4959-80b0-da59ad7f756c\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-m7krz"
Sep 11 03:50:50.097889 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:50:50.097838    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0938a98d-0acd-4959-80b0-da59ad7f756c-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-m7krz\" (UID: \"0938a98d-0acd-4959-80b0-da59ad7f756c\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-m7krz"
Sep 11 03:50:50.097889 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:50:50.097893    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n862f\" (UniqueName: \"kubernetes.io/projected/0938a98d-0acd-4959-80b0-da59ad7f756c-kube-api-access-n862f\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-m7krz\" (UID: \"0938a98d-0acd-4959-80b0-da59ad7f756c\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-m7krz"
Sep 11 03:50:50.098175 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:50:50.097933    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0938a98d-0acd-4959-80b0-da59ad7f756c-home\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-m7krz\" (UID: \"0938a98d-0acd-4959-80b0-da59ad7f756c\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-m7krz"
Sep 11 03:50:50.098175 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:50:50.097977    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0938a98d-0acd-4959-80b0-da59ad7f756c-dshm\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-m7krz\" (UID: \"0938a98d-0acd-4959-80b0-da59ad7f756c\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-m7krz"
Sep 11 03:50:50.098175 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:50:50.098002    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0938a98d-0acd-4959-80b0-da59ad7f756c-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-m7krz\" (UID: \"0938a98d-0acd-4959-80b0-da59ad7f756c\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-m7krz"
Sep 11 03:50:50.098175 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:50:50.098068    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0938a98d-0acd-4959-80b0-da59ad7f756c-model-cache\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-m7krz\" (UID: \"0938a98d-0acd-4959-80b0-da59ad7f756c\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-m7krz"
Sep 11 03:50:50.098411 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:50:50.098390    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0938a98d-0acd-4959-80b0-da59ad7f756c-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-m7krz\" (UID: \"0938a98d-0acd-4959-80b0-da59ad7f756c\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-m7krz"
Sep 11 03:50:50.098451 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:50:50.098390    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0938a98d-0acd-4959-80b0-da59ad7f756c-home\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-m7krz\" (UID: \"0938a98d-0acd-4959-80b0-da59ad7f756c\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-m7krz"
Sep 11 03:50:50.098549 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:50:50.098530    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0938a98d-0acd-4959-80b0-da59ad7f756c-model-cache\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-m7krz\" (UID: \"0938a98d-0acd-4959-80b0-da59ad7f756c\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-m7krz"
Sep 11 03:50:50.100635 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:50:50.100617    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0938a98d-0acd-4959-80b0-da59ad7f756c-dshm\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-m7krz\" (UID: \"0938a98d-0acd-4959-80b0-da59ad7f756c\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-m7krz"
Sep 11 03:50:50.100891 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:50:50.100870    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0938a98d-0acd-4959-80b0-da59ad7f756c-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-m7krz\" (UID: \"0938a98d-0acd-4959-80b0-da59ad7f756c\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-m7krz"
Sep 11 03:50:50.105881 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:50:50.105856    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-n862f\" (UniqueName: \"kubernetes.io/projected/0938a98d-0acd-4959-80b0-da59ad7f756c-kube-api-access-n862f\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-m7krz\" (UID: \"0938a98d-0acd-4959-80b0-da59ad7f756c\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-m7krz"
Sep 11 03:50:50.208677 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:50:50.208591    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-m7krz"
Sep 11 03:50:50.478390 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:50:50.478365    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-57fc66f697-m7krz"]
Sep 11 03:50:50.481028 ip-10-0-141-105 kubenswrapper[2579]: W0911 03:50:50.481000    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod0938a98d_0acd_4959_80b0_da59ad7f756c.slice/crio-d5e9125508a23e894ae0e3a9f2718f3d4951534708ce89dfabeb12485dd2f09b WatchSource:0}: Error finding container d5e9125508a23e894ae0e3a9f2718f3d4951534708ce89dfabeb12485dd2f09b: Status 404 returned error can't find the container with id d5e9125508a23e894ae0e3a9f2718f3d4951534708ce89dfabeb12485dd2f09b
Sep 11 03:50:50.820863 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:50:50.820776    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-m7krz" event={"ID":"0938a98d-0acd-4959-80b0-da59ad7f756c","Type":"ContainerStarted","Data":"3d406046483bbef3a8e12ad160ff89973eb45a53bad2060a4614800fdff15996"}
Sep 11 03:50:50.820863 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:50:50.820820    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-m7krz" event={"ID":"0938a98d-0acd-4959-80b0-da59ad7f756c","Type":"ContainerStarted","Data":"d5e9125508a23e894ae0e3a9f2718f3d4951534708ce89dfabeb12485dd2f09b"}
Sep 11 03:50:50.838071 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:50:50.838017    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-m7krz" podStartSLOduration=1.6586457430000001 podStartE2EDuration="1.838002219s" totalImagesPullingTime="179.356476ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 03:50:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 03:50:50.836205332 +0000 UTC m=+628.491410902" watchObservedRunningTime="2026-09-11 03:50:50.838002219 +0000 UTC m=+628.493207780"
Sep 11 03:50:52.203742 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:50:52.203700    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-5z4cn"]
Sep 11 03:51:00.209414 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:51:00.209378    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-m7krz"
Sep 11 03:51:00.209783 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:51:00.209436    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-m7krz"
Sep 11 03:51:00.222041 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:51:00.222018    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-m7krz"
Sep 11 03:51:00.879846 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:51:00.879820    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-m7krz"
Sep 11 03:52:28.206165 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:52:28.206070    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-fb4df7b9f-2mznf"]
Sep 11 03:52:28.209454 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:52:28.209435    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-fb4df7b9f-2mznf"
Sep 11 03:52:28.218841 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:52:28.218818    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-fb4df7b9f-2mznf"]
Sep 11 03:52:28.382104 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:52:28.382054    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/0d12c92e-34d3-4af7-9d43-5de9ad14efe2-tls-cert\") pod \"authorino-fb4df7b9f-2mznf\" (UID: \"0d12c92e-34d3-4af7-9d43-5de9ad14efe2\") " pod="kuadrant-system/authorino-fb4df7b9f-2mznf"
Sep 11 03:52:28.382272 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:52:28.382125    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wc5d5\" (UniqueName: \"kubernetes.io/projected/0d12c92e-34d3-4af7-9d43-5de9ad14efe2-kube-api-access-wc5d5\") pod \"authorino-fb4df7b9f-2mznf\" (UID: \"0d12c92e-34d3-4af7-9d43-5de9ad14efe2\") " pod="kuadrant-system/authorino-fb4df7b9f-2mznf"
Sep 11 03:52:28.482842 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:52:28.482768    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/0d12c92e-34d3-4af7-9d43-5de9ad14efe2-tls-cert\") pod \"authorino-fb4df7b9f-2mznf\" (UID: \"0d12c92e-34d3-4af7-9d43-5de9ad14efe2\") " pod="kuadrant-system/authorino-fb4df7b9f-2mznf"
Sep 11 03:52:28.482842 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:52:28.482812    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wc5d5\" (UniqueName: \"kubernetes.io/projected/0d12c92e-34d3-4af7-9d43-5de9ad14efe2-kube-api-access-wc5d5\") pod \"authorino-fb4df7b9f-2mznf\" (UID: \"0d12c92e-34d3-4af7-9d43-5de9ad14efe2\") " pod="kuadrant-system/authorino-fb4df7b9f-2mznf"
Sep 11 03:52:28.485518 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:52:28.485493    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/0d12c92e-34d3-4af7-9d43-5de9ad14efe2-tls-cert\") pod \"authorino-fb4df7b9f-2mznf\" (UID: \"0d12c92e-34d3-4af7-9d43-5de9ad14efe2\") " pod="kuadrant-system/authorino-fb4df7b9f-2mznf"
Sep 11 03:52:28.490874 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:52:28.490848    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wc5d5\" (UniqueName: \"kubernetes.io/projected/0d12c92e-34d3-4af7-9d43-5de9ad14efe2-kube-api-access-wc5d5\") pod \"authorino-fb4df7b9f-2mznf\" (UID: \"0d12c92e-34d3-4af7-9d43-5de9ad14efe2\") " pod="kuadrant-system/authorino-fb4df7b9f-2mznf"
Sep 11 03:52:28.518722 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:52:28.518696    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-fb4df7b9f-2mznf"
Sep 11 03:52:28.788909 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:52:28.788881    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-fb4df7b9f-2mznf"]
Sep 11 03:52:28.791466 ip-10-0-141-105 kubenswrapper[2579]: W0911 03:52:28.791439    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod0d12c92e_34d3_4af7_9d43_5de9ad14efe2.slice/crio-6446372a59c1e1de311060432f1bb9bd4420eeb9e9c4e08e480c5f93d8c7c888 WatchSource:0}: Error finding container 6446372a59c1e1de311060432f1bb9bd4420eeb9e9c4e08e480c5f93d8c7c888: Status 404 returned error can't find the container with id 6446372a59c1e1de311060432f1bb9bd4420eeb9e9c4e08e480c5f93d8c7c888
Sep 11 03:52:29.282379 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:52:29.282346    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-fb4df7b9f-2mznf" event={"ID":"0d12c92e-34d3-4af7-9d43-5de9ad14efe2","Type":"ContainerStarted","Data":"6446372a59c1e1de311060432f1bb9bd4420eeb9e9c4e08e480c5f93d8c7c888"}
Sep 11 03:52:30.291221 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:52:30.291189    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-fb4df7b9f-2mznf" event={"ID":"0d12c92e-34d3-4af7-9d43-5de9ad14efe2","Type":"ContainerStarted","Data":"a23678a52df2496e89493939c16e6976e9d325ead6c432e4cfa588a7765a02ab"}
Sep 11 03:52:30.310457 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:52:30.310396    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-fb4df7b9f-2mznf" podStartSLOduration=1.334398379 podStartE2EDuration="2.310375754s" totalImagesPullingTime="975.977375ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 03:52:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 03:52:30.308842684 +0000 UTC m=+727.964048253" watchObservedRunningTime="2026-09-11 03:52:30.310375754 +0000 UTC m=+727.965581321"
Sep 11 03:52:30.336582 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:52:30.336542    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-7fb557657d-jql4x"]
Sep 11 03:52:30.336832 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:52:30.336802    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-7fb557657d-jql4x" podUID="6c72db99-14e1-4659-8899-5de07da70884" containerName="authorino" containerID="cri-o://f20726e66bd658b37ebd83138171a5d218290c0a5db42e897a1331e53e40ced4" gracePeriod=30
Sep 11 03:52:30.618760 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:52:30.618730    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7fb557657d-jql4x"
Sep 11 03:52:30.702357 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:52:30.702321    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-94ln9\" (UniqueName: \"kubernetes.io/projected/6c72db99-14e1-4659-8899-5de07da70884-kube-api-access-94ln9\") pod \"6c72db99-14e1-4659-8899-5de07da70884\" (UID: \"6c72db99-14e1-4659-8899-5de07da70884\") "
Sep 11 03:52:30.702546 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:52:30.702394    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/6c72db99-14e1-4659-8899-5de07da70884-tls-cert\") pod \"6c72db99-14e1-4659-8899-5de07da70884\" (UID: \"6c72db99-14e1-4659-8899-5de07da70884\") "
Sep 11 03:52:30.704679 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:52:30.704636    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6c72db99-14e1-4659-8899-5de07da70884-kube-api-access-94ln9" (OuterVolumeSpecName: "kube-api-access-94ln9") pod "6c72db99-14e1-4659-8899-5de07da70884" (UID: "6c72db99-14e1-4659-8899-5de07da70884"). InnerVolumeSpecName "kube-api-access-94ln9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 03:52:30.713249 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:52:30.713208    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6c72db99-14e1-4659-8899-5de07da70884-tls-cert" (OuterVolumeSpecName: "tls-cert") pod "6c72db99-14e1-4659-8899-5de07da70884" (UID: "6c72db99-14e1-4659-8899-5de07da70884"). InnerVolumeSpecName "tls-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 03:52:30.803226 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:52:30.803192    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-94ln9\" (UniqueName: \"kubernetes.io/projected/6c72db99-14e1-4659-8899-5de07da70884-kube-api-access-94ln9\") on node \"ip-10-0-141-105.ec2.internal\" DevicePath \"\""
Sep 11 03:52:30.803226 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:52:30.803224    2579 reconciler_common.go:299] "Volume detached for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/6c72db99-14e1-4659-8899-5de07da70884-tls-cert\") on node \"ip-10-0-141-105.ec2.internal\" DevicePath \"\""
Sep 11 03:52:31.298553 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:52:31.298462    2579 generic.go:356] "Generic (PLEG): container finished" podID="6c72db99-14e1-4659-8899-5de07da70884" containerID="f20726e66bd658b37ebd83138171a5d218290c0a5db42e897a1331e53e40ced4" exitCode=0
Sep 11 03:52:31.298553 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:52:31.298498    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7fb557657d-jql4x"
Sep 11 03:52:31.298553 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:52:31.298533    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7fb557657d-jql4x" event={"ID":"6c72db99-14e1-4659-8899-5de07da70884","Type":"ContainerDied","Data":"f20726e66bd658b37ebd83138171a5d218290c0a5db42e897a1331e53e40ced4"}
Sep 11 03:52:31.299174 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:52:31.298576    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7fb557657d-jql4x" event={"ID":"6c72db99-14e1-4659-8899-5de07da70884","Type":"ContainerDied","Data":"a1b683a4f648c2d0934722eae018cd8951782b3dd7e348adf43469e92f965fce"}
Sep 11 03:52:31.299174 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:52:31.298594    2579 scope.go:117] "RemoveContainer" containerID="f20726e66bd658b37ebd83138171a5d218290c0a5db42e897a1331e53e40ced4"
Sep 11 03:52:31.308179 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:52:31.308159    2579 scope.go:117] "RemoveContainer" containerID="f20726e66bd658b37ebd83138171a5d218290c0a5db42e897a1331e53e40ced4"
Sep 11 03:52:31.308519 ip-10-0-141-105 kubenswrapper[2579]: E0911 03:52:31.308482    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f20726e66bd658b37ebd83138171a5d218290c0a5db42e897a1331e53e40ced4\": container with ID starting with f20726e66bd658b37ebd83138171a5d218290c0a5db42e897a1331e53e40ced4 not found: ID does not exist" containerID="f20726e66bd658b37ebd83138171a5d218290c0a5db42e897a1331e53e40ced4"
Sep 11 03:52:31.308606 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:52:31.308526    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f20726e66bd658b37ebd83138171a5d218290c0a5db42e897a1331e53e40ced4"} err="failed to get container status \"f20726e66bd658b37ebd83138171a5d218290c0a5db42e897a1331e53e40ced4\": rpc error: code = NotFound desc = could not find container \"f20726e66bd658b37ebd83138171a5d218290c0a5db42e897a1331e53e40ced4\": container with ID starting with f20726e66bd658b37ebd83138171a5d218290c0a5db42e897a1331e53e40ced4 not found: ID does not exist"
Sep 11 03:52:31.316209 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:52:31.316180    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-7fb557657d-jql4x"]
Sep 11 03:52:31.320828 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:52:31.320807    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-7fb557657d-jql4x"]
Sep 11 03:52:32.852034 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:52:32.851994    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6c72db99-14e1-4659-8899-5de07da70884" path="/var/lib/kubelet/pods/6c72db99-14e1-4659-8899-5de07da70884/volumes"
Sep 11 03:52:50.823032 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:52:50.822992    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7713-b29edd"]
Sep 11 03:52:50.823576 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:52:50.823556    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6c72db99-14e1-4659-8899-5de07da70884" containerName="authorino"
Sep 11 03:52:50.823655 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:52:50.823580    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="6c72db99-14e1-4659-8899-5de07da70884" containerName="authorino"
Sep 11 03:52:50.823707 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:52:50.823666    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="6c72db99-14e1-4659-8899-5de07da70884" containerName="authorino"
Sep 11 03:52:50.826585 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:52:50.826563    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7713-b29edd"
Sep 11 03:52:50.829234 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:52:50.829201    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-fjqk5\""
Sep 11 03:52:50.830969 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:52:50.830939    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7713-b29edd"]
Sep 11 03:52:50.896151 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:52:50.896118    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fkk5f\" (UniqueName: \"kubernetes.io/projected/c0af94f8-8696-42cf-b66d-1249c9e48d5d-kube-api-access-fkk5f\") pod \"test-curl-7713-b29edd\" (UID: \"c0af94f8-8696-42cf-b66d-1249c9e48d5d\") " pod="opendatahub/test-curl-7713-b29edd"
Sep 11 03:52:50.997281 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:52:50.997244    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fkk5f\" (UniqueName: \"kubernetes.io/projected/c0af94f8-8696-42cf-b66d-1249c9e48d5d-kube-api-access-fkk5f\") pod \"test-curl-7713-b29edd\" (UID: \"c0af94f8-8696-42cf-b66d-1249c9e48d5d\") " pod="opendatahub/test-curl-7713-b29edd"
Sep 11 03:52:51.005941 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:52:51.005912    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fkk5f\" (UniqueName: \"kubernetes.io/projected/c0af94f8-8696-42cf-b66d-1249c9e48d5d-kube-api-access-fkk5f\") pod \"test-curl-7713-b29edd\" (UID: \"c0af94f8-8696-42cf-b66d-1249c9e48d5d\") " pod="opendatahub/test-curl-7713-b29edd"
Sep 11 03:52:51.137228 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:52:51.137196    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7713-b29edd"
Sep 11 03:52:51.414126 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:52:51.414100    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7713-b29edd"]
Sep 11 03:52:51.416968 ip-10-0-141-105 kubenswrapper[2579]: W0911 03:52:51.416937    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc0af94f8_8696_42cf_b66d_1249c9e48d5d.slice/crio-d06138fe1a929bc5a6e5082f1a544fbf95b64d7908576957a6d35bc4a454fc52 WatchSource:0}: Error finding container d06138fe1a929bc5a6e5082f1a544fbf95b64d7908576957a6d35bc4a454fc52: Status 404 returned error can't find the container with id d06138fe1a929bc5a6e5082f1a544fbf95b64d7908576957a6d35bc4a454fc52
Sep 11 03:52:52.405221 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:52:52.405186    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7713-b29edd" event={"ID":"c0af94f8-8696-42cf-b66d-1249c9e48d5d","Type":"ContainerStarted","Data":"d06138fe1a929bc5a6e5082f1a544fbf95b64d7908576957a6d35bc4a454fc52"}
Sep 11 03:52:52.450557 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:52:52.450510    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-5z4cn"]
Sep 11 03:52:54.418257 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:52:54.418221    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7713-b29edd" event={"ID":"c0af94f8-8696-42cf-b66d-1249c9e48d5d","Type":"ContainerStarted","Data":"f68342fce64e73be594cc060c8f33ccca22f7311bd5bcee3047d7925d04260c2"}
Sep 11 03:52:54.431864 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:52:54.431816    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-7713-b29edd" podStartSLOduration=2.392887494 podStartE2EDuration="4.431800809s" totalImagesPullingTime="2.038913315s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 03:52:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 03:52:54.430825013 +0000 UTC m=+752.086030581" watchObservedRunningTime="2026-09-11 03:52:54.431800809 +0000 UTC m=+752.087006377"
Sep 11 03:52:54.485167 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:52:54.485135    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-4c62f1-6894c46554-w4b22"]
Sep 11 03:52:54.488857 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:52:54.488839    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-4c62f1-6894c46554-w4b22"
Sep 11 03:52:54.491588 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:52:54.491566    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-4c62f1\""
Sep 11 03:52:54.491672 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:52:54.491566    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-4c62f1-dockercfg-tkh6c\""
Sep 11 03:52:54.501472 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:52:54.501449    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-4c62f1-6894c46554-w4b22"]
Sep 11 03:52:54.631255 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:52:54.631223    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fxklz\" (UniqueName: \"kubernetes.io/projected/1199d453-904f-4a53-aeed-f424a04a0ec6-kube-api-access-fxklz\") pod \"payload-pre-processing-e2e-inf-a-4c62f1-6894c46554-w4b22\" (UID: \"1199d453-904f-4a53-aeed-f424a04a0ec6\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-4c62f1-6894c46554-w4b22"
Sep 11 03:52:54.631420 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:52:54.631361    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1199d453-904f-4a53-aeed-f424a04a0ec6-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-4c62f1-6894c46554-w4b22\" (UID: \"1199d453-904f-4a53-aeed-f424a04a0ec6\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-4c62f1-6894c46554-w4b22"
Sep 11 03:52:54.732070 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:52:54.731975    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1199d453-904f-4a53-aeed-f424a04a0ec6-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-4c62f1-6894c46554-w4b22\" (UID: \"1199d453-904f-4a53-aeed-f424a04a0ec6\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-4c62f1-6894c46554-w4b22"
Sep 11 03:52:54.732070 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:52:54.732028    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fxklz\" (UniqueName: \"kubernetes.io/projected/1199d453-904f-4a53-aeed-f424a04a0ec6-kube-api-access-fxklz\") pod \"payload-pre-processing-e2e-inf-a-4c62f1-6894c46554-w4b22\" (UID: \"1199d453-904f-4a53-aeed-f424a04a0ec6\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-4c62f1-6894c46554-w4b22"
Sep 11 03:52:54.732782 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:52:54.732760    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1199d453-904f-4a53-aeed-f424a04a0ec6-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-4c62f1-6894c46554-w4b22\" (UID: \"1199d453-904f-4a53-aeed-f424a04a0ec6\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-4c62f1-6894c46554-w4b22"
Sep 11 03:52:54.741686 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:52:54.741664    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fxklz\" (UniqueName: \"kubernetes.io/projected/1199d453-904f-4a53-aeed-f424a04a0ec6-kube-api-access-fxklz\") pod \"payload-pre-processing-e2e-inf-a-4c62f1-6894c46554-w4b22\" (UID: \"1199d453-904f-4a53-aeed-f424a04a0ec6\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-4c62f1-6894c46554-w4b22"
Sep 11 03:52:54.799563 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:52:54.799526    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-4c62f1-6894c46554-w4b22"
Sep 11 03:52:55.013094 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:52:55.012998    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7713-b29edd"]
Sep 11 03:52:55.015898 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:52:55.015871    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7713-b29edd"]
Sep 11 03:52:55.074721 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:52:55.074672    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-4c62f1-6894c46554-w4b22"]
Sep 11 03:52:55.077671 ip-10-0-141-105 kubenswrapper[2579]: W0911 03:52:55.077647    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1199d453_904f_4a53_aeed_f424a04a0ec6.slice/crio-162ca54375aeceffcbfbe0d0a7cf9177d284594b1200f3d0963c3e5eeb4ecb41 WatchSource:0}: Error finding container 162ca54375aeceffcbfbe0d0a7cf9177d284594b1200f3d0963c3e5eeb4ecb41: Status 404 returned error can't find the container with id 162ca54375aeceffcbfbe0d0a7cf9177d284594b1200f3d0963c3e5eeb4ecb41
Sep 11 03:52:55.132189 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:52:55.132162    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7713-a99165"]
Sep 11 03:52:55.132690 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:52:55.132669    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c0af94f8-8696-42cf-b66d-1249c9e48d5d" containerName="test-curl-7713-b29edd"
Sep 11 03:52:55.132747 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:52:55.132695    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="c0af94f8-8696-42cf-b66d-1249c9e48d5d" containerName="test-curl-7713-b29edd"
Sep 11 03:52:55.132782 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:52:55.132771    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="c0af94f8-8696-42cf-b66d-1249c9e48d5d" containerName="test-curl-7713-b29edd"
Sep 11 03:52:55.137042 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:52:55.137023    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7713-a99165"
Sep 11 03:52:55.140905 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:52:55.140824    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7713-a99165"]
Sep 11 03:52:55.236246 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:52:55.236213    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gp8kv\" (UniqueName: \"kubernetes.io/projected/62962702-8e94-49f1-ab3e-90f8caec2f75-kube-api-access-gp8kv\") pod \"test-curl-7713-a99165\" (UID: \"62962702-8e94-49f1-ab3e-90f8caec2f75\") " pod="opendatahub/test-curl-7713-a99165"
Sep 11 03:52:55.337658 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:52:55.337576    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gp8kv\" (UniqueName: \"kubernetes.io/projected/62962702-8e94-49f1-ab3e-90f8caec2f75-kube-api-access-gp8kv\") pod \"test-curl-7713-a99165\" (UID: \"62962702-8e94-49f1-ab3e-90f8caec2f75\") " pod="opendatahub/test-curl-7713-a99165"
Sep 11 03:52:55.344781 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:52:55.344754    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gp8kv\" (UniqueName: \"kubernetes.io/projected/62962702-8e94-49f1-ab3e-90f8caec2f75-kube-api-access-gp8kv\") pod \"test-curl-7713-a99165\" (UID: \"62962702-8e94-49f1-ab3e-90f8caec2f75\") " pod="opendatahub/test-curl-7713-a99165"
Sep 11 03:52:55.424449 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:52:55.424417    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-4c62f1-6894c46554-w4b22" event={"ID":"1199d453-904f-4a53-aeed-f424a04a0ec6","Type":"ContainerStarted","Data":"759e73c43e1726a6049cc3ecd688682f76a82831259cfc94727a25a5313b620f"}
Sep 11 03:52:55.424818 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:52:55.424471    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-4c62f1-6894c46554-w4b22" event={"ID":"1199d453-904f-4a53-aeed-f424a04a0ec6","Type":"ContainerStarted","Data":"162ca54375aeceffcbfbe0d0a7cf9177d284594b1200f3d0963c3e5eeb4ecb41"}
Sep 11 03:52:55.424818 ip-10-0-141-105 kubenswrapper[2579]: E0911 03:52:55.424622    2579 kuberuntime_manager.go:649] "Missing actuated resource record" pod="opendatahub/test-curl-7713-b29edd" container="test-curl-7713-b29edd"
Sep 11 03:52:55.440583 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:52:55.440537    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-4c62f1-6894c46554-w4b22" podStartSLOduration=1.44052176 podStartE2EDuration="1.44052176s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 03:52:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 03:52:55.438794481 +0000 UTC m=+753.094000059" watchObservedRunningTime="2026-09-11 03:52:55.44052176 +0000 UTC m=+753.095727327"
Sep 11 03:52:55.449001 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:52:55.448975    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7713-a99165"
Sep 11 03:52:55.716753 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:52:55.716627    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7713-a99165"]
Sep 11 03:52:55.718934 ip-10-0-141-105 kubenswrapper[2579]: W0911 03:52:55.718887    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod62962702_8e94_49f1_ab3e_90f8caec2f75.slice/crio-481702e16c8307d126031bc0299ec8ed9ce24592e1e19b8eea7bd7b9f4612108 WatchSource:0}: Error finding container 481702e16c8307d126031bc0299ec8ed9ce24592e1e19b8eea7bd7b9f4612108: Status 404 returned error can't find the container with id 481702e16c8307d126031bc0299ec8ed9ce24592e1e19b8eea7bd7b9f4612108
Sep 11 03:52:56.431860 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:52:56.431818    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7713-a99165" event={"ID":"62962702-8e94-49f1-ab3e-90f8caec2f75","Type":"ContainerStarted","Data":"32659b99a2017bc3bc8e105824b81fb86002cb90e6b251c70243cbf1ff254be6"}
Sep 11 03:52:56.431860 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:52:56.431867    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7713-a99165" event={"ID":"62962702-8e94-49f1-ab3e-90f8caec2f75","Type":"ContainerStarted","Data":"481702e16c8307d126031bc0299ec8ed9ce24592e1e19b8eea7bd7b9f4612108"}
Sep 11 03:52:56.432434 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:52:56.432114    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-4c62f1-6894c46554-w4b22"
Sep 11 03:52:56.432434 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:52:56.432119    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-7713-b29edd" podUID="c0af94f8-8696-42cf-b66d-1249c9e48d5d" containerName="test-curl-7713-b29edd" containerID="cri-o://f68342fce64e73be594cc060c8f33ccca22f7311bd5bcee3047d7925d04260c2" gracePeriod=2
Sep 11 03:52:56.446452 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:52:56.446409    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-7713-a99165" podStartSLOduration=1.446393838 podStartE2EDuration="1.446393838s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 03:52:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 03:52:56.444805148 +0000 UTC m=+754.100010720" watchObservedRunningTime="2026-09-11 03:52:56.446393838 +0000 UTC m=+754.101599458"
Sep 11 03:52:56.446703 ip-10-0-141-105 kubenswrapper[2579]: E0911 03:52:56.446679    2579 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7713-b29edd\" is forbidden: User \"system:node:ip-10-0-141-105.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-141-105.ec2.internal' and this object" podUID="c0af94f8-8696-42cf-b66d-1249c9e48d5d" pod="opendatahub/test-curl-7713-b29edd"
Sep 11 03:52:57.012406 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:52:57.012371    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7713-a99165"]
Sep 11 03:52:57.019257 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:52:57.019218    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7713-a99165"]
Sep 11 03:52:57.437477 ip-10-0-141-105 kubenswrapper[2579]: E0911 03:52:57.437413    2579 kuberuntime_manager.go:649] "Missing actuated resource record" pod="opendatahub/test-curl-7713-a99165" container="test-curl-7713-a99165"
Sep 11 03:52:57.843247 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:52:57.843159    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-autores-fd0def-openshift-default-6d8dfd864d-wjm48"]
Sep 11 03:52:57.843693 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:52:57.843678    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="62962702-8e94-49f1-ab3e-90f8caec2f75" containerName="test-curl-7713-a99165"
Sep 11 03:52:57.843748 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:52:57.843697    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="62962702-8e94-49f1-ab3e-90f8caec2f75" containerName="test-curl-7713-a99165"
Sep 11 03:52:57.843826 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:52:57.843814    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="62962702-8e94-49f1-ab3e-90f8caec2f75" containerName="test-curl-7713-a99165"
Sep 11 03:52:57.847347 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:52:57.847325    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-autores-fd0def-openshift-default-6d8dfd864d-wjm48"
Sep 11 03:52:57.849888 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:52:57.849865    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\""
Sep 11 03:52:57.850014 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:52:57.849961    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-autores-fd0def-openshift-default-dockercfg-f679k\""
Sep 11 03:52:57.857007 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:52:57.856983    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-autores-fd0def-openshift-default-6d8dfd864d-wjm48"]
Sep 11 03:52:57.962490 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:52:57.962457    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d2a064a0-c84d-4420-9d4b-42e679f412f3-istio-data\") pod \"e2e-autores-fd0def-openshift-default-6d8dfd864d-wjm48\" (UID: \"d2a064a0-c84d-4420-9d4b-42e679f412f3\") " pod="openshift-ingress/e2e-autores-fd0def-openshift-default-6d8dfd864d-wjm48"
Sep 11 03:52:57.962671 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:52:57.962496    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d2a064a0-c84d-4420-9d4b-42e679f412f3-workload-certs\") pod \"e2e-autores-fd0def-openshift-default-6d8dfd864d-wjm48\" (UID: \"d2a064a0-c84d-4420-9d4b-42e679f412f3\") " pod="openshift-ingress/e2e-autores-fd0def-openshift-default-6d8dfd864d-wjm48"
Sep 11 03:52:57.962671 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:52:57.962531    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d2a064a0-c84d-4420-9d4b-42e679f412f3-istio-token\") pod \"e2e-autores-fd0def-openshift-default-6d8dfd864d-wjm48\" (UID: \"d2a064a0-c84d-4420-9d4b-42e679f412f3\") " pod="openshift-ingress/e2e-autores-fd0def-openshift-default-6d8dfd864d-wjm48"
Sep 11 03:52:57.962671 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:52:57.962573    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d2a064a0-c84d-4420-9d4b-42e679f412f3-istio-envoy\") pod \"e2e-autores-fd0def-openshift-default-6d8dfd864d-wjm48\" (UID: \"d2a064a0-c84d-4420-9d4b-42e679f412f3\") " pod="openshift-ingress/e2e-autores-fd0def-openshift-default-6d8dfd864d-wjm48"
Sep 11 03:52:57.962671 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:52:57.962600    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d2a064a0-c84d-4420-9d4b-42e679f412f3-workload-socket\") pod \"e2e-autores-fd0def-openshift-default-6d8dfd864d-wjm48\" (UID: \"d2a064a0-c84d-4420-9d4b-42e679f412f3\") " pod="openshift-ingress/e2e-autores-fd0def-openshift-default-6d8dfd864d-wjm48"
Sep 11 03:52:57.962671 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:52:57.962614    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d2a064a0-c84d-4420-9d4b-42e679f412f3-credential-socket\") pod \"e2e-autores-fd0def-openshift-default-6d8dfd864d-wjm48\" (UID: \"d2a064a0-c84d-4420-9d4b-42e679f412f3\") " pod="openshift-ingress/e2e-autores-fd0def-openshift-default-6d8dfd864d-wjm48"
Sep 11 03:52:57.962671 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:52:57.962653    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xk2ms\" (UniqueName: \"kubernetes.io/projected/d2a064a0-c84d-4420-9d4b-42e679f412f3-kube-api-access-xk2ms\") pod \"e2e-autores-fd0def-openshift-default-6d8dfd864d-wjm48\" (UID: \"d2a064a0-c84d-4420-9d4b-42e679f412f3\") " pod="openshift-ingress/e2e-autores-fd0def-openshift-default-6d8dfd864d-wjm48"
Sep 11 03:52:57.962981 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:52:57.962690    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d2a064a0-c84d-4420-9d4b-42e679f412f3-istiod-ca-cert\") pod \"e2e-autores-fd0def-openshift-default-6d8dfd864d-wjm48\" (UID: \"d2a064a0-c84d-4420-9d4b-42e679f412f3\") " pod="openshift-ingress/e2e-autores-fd0def-openshift-default-6d8dfd864d-wjm48"
Sep 11 03:52:57.962981 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:52:57.962708    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d2a064a0-c84d-4420-9d4b-42e679f412f3-istio-podinfo\") pod \"e2e-autores-fd0def-openshift-default-6d8dfd864d-wjm48\" (UID: \"d2a064a0-c84d-4420-9d4b-42e679f412f3\") " pod="openshift-ingress/e2e-autores-fd0def-openshift-default-6d8dfd864d-wjm48"
Sep 11 03:52:58.063692 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:52:58.063645    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d2a064a0-c84d-4420-9d4b-42e679f412f3-istio-data\") pod \"e2e-autores-fd0def-openshift-default-6d8dfd864d-wjm48\" (UID: \"d2a064a0-c84d-4420-9d4b-42e679f412f3\") " pod="openshift-ingress/e2e-autores-fd0def-openshift-default-6d8dfd864d-wjm48"
Sep 11 03:52:58.063692 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:52:58.063685    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d2a064a0-c84d-4420-9d4b-42e679f412f3-workload-certs\") pod \"e2e-autores-fd0def-openshift-default-6d8dfd864d-wjm48\" (UID: \"d2a064a0-c84d-4420-9d4b-42e679f412f3\") " pod="openshift-ingress/e2e-autores-fd0def-openshift-default-6d8dfd864d-wjm48"
Sep 11 03:52:58.063948 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:52:58.063707    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d2a064a0-c84d-4420-9d4b-42e679f412f3-istio-token\") pod \"e2e-autores-fd0def-openshift-default-6d8dfd864d-wjm48\" (UID: \"d2a064a0-c84d-4420-9d4b-42e679f412f3\") " pod="openshift-ingress/e2e-autores-fd0def-openshift-default-6d8dfd864d-wjm48"
Sep 11 03:52:58.063948 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:52:58.063759    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d2a064a0-c84d-4420-9d4b-42e679f412f3-istio-envoy\") pod \"e2e-autores-fd0def-openshift-default-6d8dfd864d-wjm48\" (UID: \"d2a064a0-c84d-4420-9d4b-42e679f412f3\") " pod="openshift-ingress/e2e-autores-fd0def-openshift-default-6d8dfd864d-wjm48"
Sep 11 03:52:58.063948 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:52:58.063785    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d2a064a0-c84d-4420-9d4b-42e679f412f3-workload-socket\") pod \"e2e-autores-fd0def-openshift-default-6d8dfd864d-wjm48\" (UID: \"d2a064a0-c84d-4420-9d4b-42e679f412f3\") " pod="openshift-ingress/e2e-autores-fd0def-openshift-default-6d8dfd864d-wjm48"
Sep 11 03:52:58.063948 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:52:58.063807    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d2a064a0-c84d-4420-9d4b-42e679f412f3-credential-socket\") pod \"e2e-autores-fd0def-openshift-default-6d8dfd864d-wjm48\" (UID: \"d2a064a0-c84d-4420-9d4b-42e679f412f3\") " pod="openshift-ingress/e2e-autores-fd0def-openshift-default-6d8dfd864d-wjm48"
Sep 11 03:52:58.063948 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:52:58.063860    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xk2ms\" (UniqueName: \"kubernetes.io/projected/d2a064a0-c84d-4420-9d4b-42e679f412f3-kube-api-access-xk2ms\") pod \"e2e-autores-fd0def-openshift-default-6d8dfd864d-wjm48\" (UID: \"d2a064a0-c84d-4420-9d4b-42e679f412f3\") " pod="openshift-ingress/e2e-autores-fd0def-openshift-default-6d8dfd864d-wjm48"
Sep 11 03:52:58.063948 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:52:58.063903    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d2a064a0-c84d-4420-9d4b-42e679f412f3-istiod-ca-cert\") pod \"e2e-autores-fd0def-openshift-default-6d8dfd864d-wjm48\" (UID: \"d2a064a0-c84d-4420-9d4b-42e679f412f3\") " pod="openshift-ingress/e2e-autores-fd0def-openshift-default-6d8dfd864d-wjm48"
Sep 11 03:52:58.063948 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:52:58.063930    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d2a064a0-c84d-4420-9d4b-42e679f412f3-istio-podinfo\") pod \"e2e-autores-fd0def-openshift-default-6d8dfd864d-wjm48\" (UID: \"d2a064a0-c84d-4420-9d4b-42e679f412f3\") " pod="openshift-ingress/e2e-autores-fd0def-openshift-default-6d8dfd864d-wjm48"
Sep 11 03:52:58.064338 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:52:58.064172    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d2a064a0-c84d-4420-9d4b-42e679f412f3-workload-certs\") pod \"e2e-autores-fd0def-openshift-default-6d8dfd864d-wjm48\" (UID: \"d2a064a0-c84d-4420-9d4b-42e679f412f3\") " pod="openshift-ingress/e2e-autores-fd0def-openshift-default-6d8dfd864d-wjm48"
Sep 11 03:52:58.064338 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:52:58.064194    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d2a064a0-c84d-4420-9d4b-42e679f412f3-istio-data\") pod \"e2e-autores-fd0def-openshift-default-6d8dfd864d-wjm48\" (UID: \"d2a064a0-c84d-4420-9d4b-42e679f412f3\") " pod="openshift-ingress/e2e-autores-fd0def-openshift-default-6d8dfd864d-wjm48"
Sep 11 03:52:58.064445 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:52:58.064370    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d2a064a0-c84d-4420-9d4b-42e679f412f3-credential-socket\") pod \"e2e-autores-fd0def-openshift-default-6d8dfd864d-wjm48\" (UID: \"d2a064a0-c84d-4420-9d4b-42e679f412f3\") " pod="openshift-ingress/e2e-autores-fd0def-openshift-default-6d8dfd864d-wjm48"
Sep 11 03:52:58.064498 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:52:58.064481    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d2a064a0-c84d-4420-9d4b-42e679f412f3-workload-socket\") pod \"e2e-autores-fd0def-openshift-default-6d8dfd864d-wjm48\" (UID: \"d2a064a0-c84d-4420-9d4b-42e679f412f3\") " pod="openshift-ingress/e2e-autores-fd0def-openshift-default-6d8dfd864d-wjm48"
Sep 11 03:52:58.065146 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:52:58.065118    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d2a064a0-c84d-4420-9d4b-42e679f412f3-istiod-ca-cert\") pod \"e2e-autores-fd0def-openshift-default-6d8dfd864d-wjm48\" (UID: \"d2a064a0-c84d-4420-9d4b-42e679f412f3\") " pod="openshift-ingress/e2e-autores-fd0def-openshift-default-6d8dfd864d-wjm48"
Sep 11 03:52:58.067137 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:52:58.067112    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d2a064a0-c84d-4420-9d4b-42e679f412f3-istio-envoy\") pod \"e2e-autores-fd0def-openshift-default-6d8dfd864d-wjm48\" (UID: \"d2a064a0-c84d-4420-9d4b-42e679f412f3\") " pod="openshift-ingress/e2e-autores-fd0def-openshift-default-6d8dfd864d-wjm48"
Sep 11 03:52:58.067589 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:52:58.067568    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d2a064a0-c84d-4420-9d4b-42e679f412f3-istio-podinfo\") pod \"e2e-autores-fd0def-openshift-default-6d8dfd864d-wjm48\" (UID: \"d2a064a0-c84d-4420-9d4b-42e679f412f3\") " pod="openshift-ingress/e2e-autores-fd0def-openshift-default-6d8dfd864d-wjm48"
Sep 11 03:52:58.072162 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:52:58.072134    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d2a064a0-c84d-4420-9d4b-42e679f412f3-istio-token\") pod \"e2e-autores-fd0def-openshift-default-6d8dfd864d-wjm48\" (UID: \"d2a064a0-c84d-4420-9d4b-42e679f412f3\") " pod="openshift-ingress/e2e-autores-fd0def-openshift-default-6d8dfd864d-wjm48"
Sep 11 03:52:58.072676 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:52:58.072653    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xk2ms\" (UniqueName: \"kubernetes.io/projected/d2a064a0-c84d-4420-9d4b-42e679f412f3-kube-api-access-xk2ms\") pod \"e2e-autores-fd0def-openshift-default-6d8dfd864d-wjm48\" (UID: \"d2a064a0-c84d-4420-9d4b-42e679f412f3\") " pod="openshift-ingress/e2e-autores-fd0def-openshift-default-6d8dfd864d-wjm48"
Sep 11 03:52:58.161536 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:52:58.161496    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-autores-fd0def-openshift-default-6d8dfd864d-wjm48"
Sep 11 03:52:58.443426 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:52:58.443389    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-7713-a99165" podUID="62962702-8e94-49f1-ab3e-90f8caec2f75" containerName="test-curl-7713-a99165" containerID="cri-o://32659b99a2017bc3bc8e105824b81fb86002cb90e6b251c70243cbf1ff254be6" gracePeriod=2
Sep 11 03:52:58.445401 ip-10-0-141-105 kubenswrapper[2579]: E0911 03:52:58.445368    2579 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7713-a99165\" is forbidden: User \"system:node:ip-10-0-141-105.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-141-105.ec2.internal' and this object" podUID="62962702-8e94-49f1-ab3e-90f8caec2f75" pod="opendatahub/test-curl-7713-a99165"
Sep 11 03:52:58.449632 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:52:58.449601    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-autores-fd0def-openshift-default-6d8dfd864d-wjm48"]
Sep 11 03:52:58.499013 ip-10-0-141-105 kubenswrapper[2579]: W0911 03:52:58.498983    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd2a064a0_c84d_4420_9d4b_42e679f412f3.slice/crio-9b07304cfae5d380d7496054fdfff72459e5733d63d7c6b956e8c699c26e3310 WatchSource:0}: Error finding container 9b07304cfae5d380d7496054fdfff72459e5733d63d7c6b956e8c699c26e3310: Status 404 returned error can't find the container with id 9b07304cfae5d380d7496054fdfff72459e5733d63d7c6b956e8c699c26e3310
Sep 11 03:52:58.634849 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:52:58.634823    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7713-b29edd"
Sep 11 03:52:58.636935 ip-10-0-141-105 kubenswrapper[2579]: E0911 03:52:58.636910    2579 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7713-a99165\" is forbidden: User \"system:node:ip-10-0-141-105.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-141-105.ec2.internal' and this object" podUID="62962702-8e94-49f1-ab3e-90f8caec2f75" pod="opendatahub/test-curl-7713-a99165"
Sep 11 03:52:58.770424 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:52:58.770335    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fkk5f\" (UniqueName: \"kubernetes.io/projected/c0af94f8-8696-42cf-b66d-1249c9e48d5d-kube-api-access-fkk5f\") pod \"c0af94f8-8696-42cf-b66d-1249c9e48d5d\" (UID: \"c0af94f8-8696-42cf-b66d-1249c9e48d5d\") "
Sep 11 03:52:58.772762 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:52:58.772738    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c0af94f8-8696-42cf-b66d-1249c9e48d5d-kube-api-access-fkk5f" (OuterVolumeSpecName: "kube-api-access-fkk5f") pod "c0af94f8-8696-42cf-b66d-1249c9e48d5d" (UID: "c0af94f8-8696-42cf-b66d-1249c9e48d5d"). InnerVolumeSpecName "kube-api-access-fkk5f". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 03:52:58.851872 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:52:58.851832    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c0af94f8-8696-42cf-b66d-1249c9e48d5d" path="/var/lib/kubelet/pods/c0af94f8-8696-42cf-b66d-1249c9e48d5d/volumes"
Sep 11 03:52:58.871859 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:52:58.871832    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fkk5f\" (UniqueName: \"kubernetes.io/projected/c0af94f8-8696-42cf-b66d-1249c9e48d5d-kube-api-access-fkk5f\") on node \"ip-10-0-141-105.ec2.internal\" DevicePath \"\""
Sep 11 03:52:59.450380 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:52:59.450344    2579 generic.go:356] "Generic (PLEG): container finished" podID="c0af94f8-8696-42cf-b66d-1249c9e48d5d" containerID="f68342fce64e73be594cc060c8f33ccca22f7311bd5bcee3047d7925d04260c2" exitCode=137
Sep 11 03:52:59.450922 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:52:59.450402    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7713-b29edd"
Sep 11 03:52:59.450922 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:52:59.450453    2579 scope.go:117] "RemoveContainer" containerID="f68342fce64e73be594cc060c8f33ccca22f7311bd5bcee3047d7925d04260c2"
Sep 11 03:52:59.452034 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:52:59.451966    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-fd0def-openshift-default-6d8dfd864d-wjm48" event={"ID":"d2a064a0-c84d-4420-9d4b-42e679f412f3","Type":"ContainerStarted","Data":"9b07304cfae5d380d7496054fdfff72459e5733d63d7c6b956e8c699c26e3310"}
Sep 11 03:52:59.460735 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:52:59.460714    2579 scope.go:117] "RemoveContainer" containerID="f68342fce64e73be594cc060c8f33ccca22f7311bd5bcee3047d7925d04260c2"
Sep 11 03:52:59.461051 ip-10-0-141-105 kubenswrapper[2579]: E0911 03:52:59.461025    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f68342fce64e73be594cc060c8f33ccca22f7311bd5bcee3047d7925d04260c2\": container with ID starting with f68342fce64e73be594cc060c8f33ccca22f7311bd5bcee3047d7925d04260c2 not found: ID does not exist" containerID="f68342fce64e73be594cc060c8f33ccca22f7311bd5bcee3047d7925d04260c2"
Sep 11 03:52:59.461181 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:52:59.461056    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f68342fce64e73be594cc060c8f33ccca22f7311bd5bcee3047d7925d04260c2"} err="failed to get container status \"f68342fce64e73be594cc060c8f33ccca22f7311bd5bcee3047d7925d04260c2\": rpc error: code = NotFound desc = could not find container \"f68342fce64e73be594cc060c8f33ccca22f7311bd5bcee3047d7925d04260c2\": container with ID starting with f68342fce64e73be594cc060c8f33ccca22f7311bd5bcee3047d7925d04260c2 not found: ID does not exist"
Sep 11 03:53:01.122060 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:53:01.122007    2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 11 03:53:01.122813 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:53:01.122129    2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 11 03:53:01.122813 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:53:01.122179    2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 11 03:53:01.217147 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:53:01.217124    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7713-a99165"
Sep 11 03:53:01.399517 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:53:01.399486    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gp8kv\" (UniqueName: \"kubernetes.io/projected/62962702-8e94-49f1-ab3e-90f8caec2f75-kube-api-access-gp8kv\") pod \"62962702-8e94-49f1-ab3e-90f8caec2f75\" (UID: \"62962702-8e94-49f1-ab3e-90f8caec2f75\") "
Sep 11 03:53:01.401831 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:53:01.401795    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/62962702-8e94-49f1-ab3e-90f8caec2f75-kube-api-access-gp8kv" (OuterVolumeSpecName: "kube-api-access-gp8kv") pod "62962702-8e94-49f1-ab3e-90f8caec2f75" (UID: "62962702-8e94-49f1-ab3e-90f8caec2f75"). InnerVolumeSpecName "kube-api-access-gp8kv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 03:53:01.472043 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:53:01.472007    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-fd0def-openshift-default-6d8dfd864d-wjm48" event={"ID":"d2a064a0-c84d-4420-9d4b-42e679f412f3","Type":"ContainerStarted","Data":"3346bf6dd9e6ecb80e0d9f55889f4bb2f67351fdefd90e8aaf1d86c518682a65"}
Sep 11 03:53:01.473471 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:53:01.473448    2579 generic.go:356] "Generic (PLEG): container finished" podID="62962702-8e94-49f1-ab3e-90f8caec2f75" containerID="32659b99a2017bc3bc8e105824b81fb86002cb90e6b251c70243cbf1ff254be6" exitCode=137
Sep 11 03:53:01.473596 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:53:01.473508    2579 scope.go:117] "RemoveContainer" containerID="32659b99a2017bc3bc8e105824b81fb86002cb90e6b251c70243cbf1ff254be6"
Sep 11 03:53:01.473596 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:53:01.473508    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7713-a99165"
Sep 11 03:53:01.481972 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:53:01.481948    2579 scope.go:117] "RemoveContainer" containerID="32659b99a2017bc3bc8e105824b81fb86002cb90e6b251c70243cbf1ff254be6"
Sep 11 03:53:01.482300 ip-10-0-141-105 kubenswrapper[2579]: E0911 03:53:01.482270    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"32659b99a2017bc3bc8e105824b81fb86002cb90e6b251c70243cbf1ff254be6\": container with ID starting with 32659b99a2017bc3bc8e105824b81fb86002cb90e6b251c70243cbf1ff254be6 not found: ID does not exist" containerID="32659b99a2017bc3bc8e105824b81fb86002cb90e6b251c70243cbf1ff254be6"
Sep 11 03:53:01.482366 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:53:01.482303    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"32659b99a2017bc3bc8e105824b81fb86002cb90e6b251c70243cbf1ff254be6"} err="failed to get container status \"32659b99a2017bc3bc8e105824b81fb86002cb90e6b251c70243cbf1ff254be6\": rpc error: code = NotFound desc = could not find container \"32659b99a2017bc3bc8e105824b81fb86002cb90e6b251c70243cbf1ff254be6\": container with ID starting with 32659b99a2017bc3bc8e105824b81fb86002cb90e6b251c70243cbf1ff254be6 not found: ID does not exist"
Sep 11 03:53:01.491820 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:53:01.491777    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-autores-fd0def-openshift-default-6d8dfd864d-wjm48" podStartSLOduration=1.871572085 podStartE2EDuration="4.491734138s" totalImagesPullingTime="2.620162053s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 03:52:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 03:53:01.490637314 +0000 UTC m=+759.145842883" watchObservedRunningTime="2026-09-11 03:53:01.491734138 +0000 UTC m=+759.146939704"
Sep 11 03:53:01.492570 ip-10-0-141-105 kubenswrapper[2579]: E0911 03:53:01.492543    2579 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7713-a99165\" is forbidden: User \"system:node:ip-10-0-141-105.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-141-105.ec2.internal' and this object" podUID="62962702-8e94-49f1-ab3e-90f8caec2f75" pod="opendatahub/test-curl-7713-a99165"
Sep 11 03:53:01.501012 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:53:01.500984    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gp8kv\" (UniqueName: \"kubernetes.io/projected/62962702-8e94-49f1-ab3e-90f8caec2f75-kube-api-access-gp8kv\") on node \"ip-10-0-141-105.ec2.internal\" DevicePath \"\""
Sep 11 03:53:01.817925 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:53:01.817847    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-1eeb81b2-56c595b47d-v9cgk"]
Sep 11 03:53:01.822964 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:53:01.822944    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-1eeb81b2-56c595b47d-v9cgk"
Sep 11 03:53:01.825453 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:53:01.825421    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-1eeb81b2\""
Sep 11 03:53:01.832774 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:53:01.832750    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-1eeb81b2-56c595b47d-v9cgk"]
Sep 11 03:53:01.834730 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:53:01.834707    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-1eeb81b2-798bd466f5-vxnm2"]
Sep 11 03:53:01.838680 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:53:01.838663    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-1eeb81b2-798bd466f5-vxnm2"
Sep 11 03:53:01.841239 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:53:01.841222    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-1eeb81b2-dockercfg-lzblc\""
Sep 11 03:53:01.845068 ip-10-0-141-105 kubenswrapper[2579]: E0911 03:53:01.845043    2579 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7713-a99165\" is forbidden: User \"system:node:ip-10-0-141-105.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-141-105.ec2.internal' and this object" podUID="62962702-8e94-49f1-ab3e-90f8caec2f75" pod="opendatahub/test-curl-7713-a99165"
Sep 11 03:53:01.845893 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:53:01.845870    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-1eeb81b2-798bd466f5-vxnm2"]
Sep 11 03:53:01.847022 ip-10-0-141-105 kubenswrapper[2579]: E0911 03:53:01.846991    2579 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7713-a99165\" is forbidden: User \"system:node:ip-10-0-141-105.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-141-105.ec2.internal' and this object" podUID="62962702-8e94-49f1-ab3e-90f8caec2f75" pod="opendatahub/test-curl-7713-a99165"
Sep 11 03:53:02.006560 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:53:02.006523    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9ac63561-3bd7-429b-891d-d721c113befc-plugins-config-volume\") pod \"payload-processing-e2e-ait-1eeb81b2-798bd466f5-vxnm2\" (UID: \"9ac63561-3bd7-429b-891d-d721c113befc\") " pod="openshift-ingress/payload-processing-e2e-ait-1eeb81b2-798bd466f5-vxnm2"
Sep 11 03:53:02.006755 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:53:02.006630    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-stgmx\" (UniqueName: \"kubernetes.io/projected/141630fc-f0e0-4672-97e9-406959f90486-kube-api-access-stgmx\") pod \"payload-pre-processing-e2e-ait-1eeb81b2-56c595b47d-v9cgk\" (UID: \"141630fc-f0e0-4672-97e9-406959f90486\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-1eeb81b2-56c595b47d-v9cgk"
Sep 11 03:53:02.006755 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:53:02.006701    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/141630fc-f0e0-4672-97e9-406959f90486-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-1eeb81b2-56c595b47d-v9cgk\" (UID: \"141630fc-f0e0-4672-97e9-406959f90486\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-1eeb81b2-56c595b47d-v9cgk"
Sep 11 03:53:02.006755 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:53:02.006727    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tljzt\" (UniqueName: \"kubernetes.io/projected/9ac63561-3bd7-429b-891d-d721c113befc-kube-api-access-tljzt\") pod \"payload-processing-e2e-ait-1eeb81b2-798bd466f5-vxnm2\" (UID: \"9ac63561-3bd7-429b-891d-d721c113befc\") " pod="openshift-ingress/payload-processing-e2e-ait-1eeb81b2-798bd466f5-vxnm2"
Sep 11 03:53:02.108119 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:53:02.108070    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9ac63561-3bd7-429b-891d-d721c113befc-plugins-config-volume\") pod \"payload-processing-e2e-ait-1eeb81b2-798bd466f5-vxnm2\" (UID: \"9ac63561-3bd7-429b-891d-d721c113befc\") " pod="openshift-ingress/payload-processing-e2e-ait-1eeb81b2-798bd466f5-vxnm2"
Sep 11 03:53:02.108388 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:53:02.108158    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-stgmx\" (UniqueName: \"kubernetes.io/projected/141630fc-f0e0-4672-97e9-406959f90486-kube-api-access-stgmx\") pod \"payload-pre-processing-e2e-ait-1eeb81b2-56c595b47d-v9cgk\" (UID: \"141630fc-f0e0-4672-97e9-406959f90486\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-1eeb81b2-56c595b47d-v9cgk"
Sep 11 03:53:02.108388 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:53:02.108205    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/141630fc-f0e0-4672-97e9-406959f90486-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-1eeb81b2-56c595b47d-v9cgk\" (UID: \"141630fc-f0e0-4672-97e9-406959f90486\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-1eeb81b2-56c595b47d-v9cgk"
Sep 11 03:53:02.108388 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:53:02.108337    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tljzt\" (UniqueName: \"kubernetes.io/projected/9ac63561-3bd7-429b-891d-d721c113befc-kube-api-access-tljzt\") pod \"payload-processing-e2e-ait-1eeb81b2-798bd466f5-vxnm2\" (UID: \"9ac63561-3bd7-429b-891d-d721c113befc\") " pod="openshift-ingress/payload-processing-e2e-ait-1eeb81b2-798bd466f5-vxnm2"
Sep 11 03:53:02.108849 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:53:02.108827    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9ac63561-3bd7-429b-891d-d721c113befc-plugins-config-volume\") pod \"payload-processing-e2e-ait-1eeb81b2-798bd466f5-vxnm2\" (UID: \"9ac63561-3bd7-429b-891d-d721c113befc\") " pod="openshift-ingress/payload-processing-e2e-ait-1eeb81b2-798bd466f5-vxnm2"
Sep 11 03:53:02.108938 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:53:02.108876    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/141630fc-f0e0-4672-97e9-406959f90486-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-1eeb81b2-56c595b47d-v9cgk\" (UID: \"141630fc-f0e0-4672-97e9-406959f90486\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-1eeb81b2-56c595b47d-v9cgk"
Sep 11 03:53:02.116275 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:53:02.116249    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tljzt\" (UniqueName: \"kubernetes.io/projected/9ac63561-3bd7-429b-891d-d721c113befc-kube-api-access-tljzt\") pod \"payload-processing-e2e-ait-1eeb81b2-798bd466f5-vxnm2\" (UID: \"9ac63561-3bd7-429b-891d-d721c113befc\") " pod="openshift-ingress/payload-processing-e2e-ait-1eeb81b2-798bd466f5-vxnm2"
Sep 11 03:53:02.116636 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:53:02.116619    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-stgmx\" (UniqueName: \"kubernetes.io/projected/141630fc-f0e0-4672-97e9-406959f90486-kube-api-access-stgmx\") pod \"payload-pre-processing-e2e-ait-1eeb81b2-56c595b47d-v9cgk\" (UID: \"141630fc-f0e0-4672-97e9-406959f90486\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-1eeb81b2-56c595b47d-v9cgk"
Sep 11 03:53:02.134464 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:53:02.134441    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-1eeb81b2-56c595b47d-v9cgk"
Sep 11 03:53:02.150398 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:53:02.150353    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-1eeb81b2-798bd466f5-vxnm2"
Sep 11 03:53:02.162257 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:53:02.162235    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-autores-fd0def-openshift-default-6d8dfd864d-wjm48"
Sep 11 03:53:02.169672 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:53:02.169649    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-autores-fd0def-openshift-default-6d8dfd864d-wjm48"
Sep 11 03:53:02.171699 ip-10-0-141-105 kubenswrapper[2579]: E0911 03:53:02.171669    2579 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7713-a99165\" is forbidden: User \"system:node:ip-10-0-141-105.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-141-105.ec2.internal' and this object" podUID="62962702-8e94-49f1-ab3e-90f8caec2f75" pod="opendatahub/test-curl-7713-a99165"
Sep 11 03:53:02.408699 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:53:02.408669    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-1eeb81b2-56c595b47d-v9cgk"]
Sep 11 03:53:02.410836 ip-10-0-141-105 kubenswrapper[2579]: W0911 03:53:02.410807    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod141630fc_f0e0_4672_97e9_406959f90486.slice/crio-f764d16aae1815bde5c920f16be147335a5367b49aa40ce4d19d5b58b439b65e WatchSource:0}: Error finding container f764d16aae1815bde5c920f16be147335a5367b49aa40ce4d19d5b58b439b65e: Status 404 returned error can't find the container with id f764d16aae1815bde5c920f16be147335a5367b49aa40ce4d19d5b58b439b65e
Sep 11 03:53:02.430578 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:53:02.430550    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-1eeb81b2-798bd466f5-vxnm2"]
Sep 11 03:53:02.432779 ip-10-0-141-105 kubenswrapper[2579]: W0911 03:53:02.432751    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9ac63561_3bd7_429b_891d_d721c113befc.slice/crio-f1ea2ddb4dd29d5ef97f9aa9946a708013671b5919f335e855fd2d7f6762109d WatchSource:0}: Error finding container f1ea2ddb4dd29d5ef97f9aa9946a708013671b5919f335e855fd2d7f6762109d: Status 404 returned error can't find the container with id f1ea2ddb4dd29d5ef97f9aa9946a708013671b5919f335e855fd2d7f6762109d
Sep 11 03:53:02.482556 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:53:02.482528    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-1eeb81b2-798bd466f5-vxnm2" event={"ID":"9ac63561-3bd7-429b-891d-d721c113befc","Type":"ContainerStarted","Data":"f1ea2ddb4dd29d5ef97f9aa9946a708013671b5919f335e855fd2d7f6762109d"}
Sep 11 03:53:02.484228 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:53:02.484197    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-1eeb81b2-56c595b47d-v9cgk" event={"ID":"141630fc-f0e0-4672-97e9-406959f90486","Type":"ContainerStarted","Data":"f764d16aae1815bde5c920f16be147335a5367b49aa40ce4d19d5b58b439b65e"}
Sep 11 03:53:02.484674 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:53:02.484656    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-autores-fd0def-openshift-default-6d8dfd864d-wjm48"
Sep 11 03:53:02.485971 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:53:02.485952    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-autores-fd0def-openshift-default-6d8dfd864d-wjm48"
Sep 11 03:53:02.499676 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:53:02.499620    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-1eeb81b2-56c595b47d-v9cgk" podStartSLOduration=1.499600254 podStartE2EDuration="1.499600254s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 03:53:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 03:53:02.499243124 +0000 UTC m=+760.154448693" watchObservedRunningTime="2026-09-11 03:53:02.499600254 +0000 UTC m=+760.154805823"
Sep 11 03:53:02.501204 ip-10-0-141-105 kubenswrapper[2579]: E0911 03:53:02.501173    2579 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7713-a99165\" is forbidden: User \"system:node:ip-10-0-141-105.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-141-105.ec2.internal' and this object" podUID="62962702-8e94-49f1-ab3e-90f8caec2f75" pod="opendatahub/test-curl-7713-a99165"
Sep 11 03:53:02.503056 ip-10-0-141-105 kubenswrapper[2579]: E0911 03:53:02.503020    2579 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7713-a99165\" is forbidden: User \"system:node:ip-10-0-141-105.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-141-105.ec2.internal' and this object" podUID="62962702-8e94-49f1-ab3e-90f8caec2f75" pod="opendatahub/test-curl-7713-a99165"
Sep 11 03:53:02.850775 ip-10-0-141-105 kubenswrapper[2579]: E0911 03:53:02.850687    2579 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7713-a99165\" is forbidden: User \"system:node:ip-10-0-141-105.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-141-105.ec2.internal' and this object" podUID="62962702-8e94-49f1-ab3e-90f8caec2f75" pod="opendatahub/test-curl-7713-a99165"
Sep 11 03:53:02.852584 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:53:02.852543    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="62962702-8e94-49f1-ab3e-90f8caec2f75" path="/var/lib/kubelet/pods/62962702-8e94-49f1-ab3e-90f8caec2f75/volumes"
Sep 11 03:53:03.490773 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:53:03.490734    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-1eeb81b2-56c595b47d-v9cgk" event={"ID":"141630fc-f0e0-4672-97e9-406959f90486","Type":"ContainerStarted","Data":"e4ef95fc754e8377726aab0cda76fb53b852690bade0847798979db5d1035125"}
Sep 11 03:53:03.491256 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:53:03.490802    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-1eeb81b2-56c595b47d-v9cgk"
Sep 11 03:53:03.492224 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:53:03.492199    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-1eeb81b2-798bd466f5-vxnm2" event={"ID":"9ac63561-3bd7-429b-891d-d721c113befc","Type":"ContainerStarted","Data":"7dc4aaeb3a7a55c6970a33278b4fddc3ef5e58c60d6d00508f004da168146de1"}
Sep 11 03:53:03.492408 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:53:03.492393    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-1eeb81b2-798bd466f5-vxnm2"
Sep 11 03:53:03.508031 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:53:03.507990    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-1eeb81b2-798bd466f5-vxnm2" podStartSLOduration=2.507975884 podStartE2EDuration="2.507975884s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 03:53:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 03:53:03.506849353 +0000 UTC m=+761.162054933" watchObservedRunningTime="2026-09-11 03:53:03.507975884 +0000 UTC m=+761.163181451"
Sep 11 03:53:04.814557 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:53:04.814520    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-5z4cn"]
Sep 11 03:53:07.345983 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:53:07.345954    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-fd0def-5d59887bb-xb8gk"]
Sep 11 03:53:07.350609 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:53:07.350593    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-autores-fd0def-5d59887bb-xb8gk"
Sep 11 03:53:07.352446 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:53:07.352412    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a173f5c9-2082-4276-99fc-bc121d8c8be3-plugins-config-volume\") pod \"payload-pre-processing-e2e-autores-fd0def-5d59887bb-xb8gk\" (UID: \"a173f5c9-2082-4276-99fc-bc121d8c8be3\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-fd0def-5d59887bb-xb8gk"
Sep 11 03:53:07.352627 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:53:07.352469    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gsw8r\" (UniqueName: \"kubernetes.io/projected/a173f5c9-2082-4276-99fc-bc121d8c8be3-kube-api-access-gsw8r\") pod \"payload-pre-processing-e2e-autores-fd0def-5d59887bb-xb8gk\" (UID: \"a173f5c9-2082-4276-99fc-bc121d8c8be3\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-fd0def-5d59887bb-xb8gk"
Sep 11 03:53:07.353242 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:53:07.353220    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-autores-fd0def\""
Sep 11 03:53:07.358993 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:53:07.358971    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-fd0def-5d59887bb-xb8gk"]
Sep 11 03:53:07.380432 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:53:07.380400    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-fd0def-bb56f6895-nn5m7"]
Sep 11 03:53:07.383753 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:53:07.383738    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-autores-fd0def-bb56f6895-nn5m7"
Sep 11 03:53:07.386476 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:53:07.386443    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-autores-fd0def-dockercfg-pgzt8\""
Sep 11 03:53:07.395626 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:53:07.395600    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-fd0def-bb56f6895-nn5m7"]
Sep 11 03:53:07.439263 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:53:07.439227    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-4c62f1-6894c46554-w4b22"
Sep 11 03:53:07.452925 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:53:07.452894    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a173f5c9-2082-4276-99fc-bc121d8c8be3-plugins-config-volume\") pod \"payload-pre-processing-e2e-autores-fd0def-5d59887bb-xb8gk\" (UID: \"a173f5c9-2082-4276-99fc-bc121d8c8be3\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-fd0def-5d59887bb-xb8gk"
Sep 11 03:53:07.453124 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:53:07.452972    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gsw8r\" (UniqueName: \"kubernetes.io/projected/a173f5c9-2082-4276-99fc-bc121d8c8be3-kube-api-access-gsw8r\") pod \"payload-pre-processing-e2e-autores-fd0def-5d59887bb-xb8gk\" (UID: \"a173f5c9-2082-4276-99fc-bc121d8c8be3\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-fd0def-5d59887bb-xb8gk"
Sep 11 03:53:07.453832 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:53:07.453810    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a173f5c9-2082-4276-99fc-bc121d8c8be3-plugins-config-volume\") pod \"payload-pre-processing-e2e-autores-fd0def-5d59887bb-xb8gk\" (UID: \"a173f5c9-2082-4276-99fc-bc121d8c8be3\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-fd0def-5d59887bb-xb8gk"
Sep 11 03:53:07.478892 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:53:07.478854    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gsw8r\" (UniqueName: \"kubernetes.io/projected/a173f5c9-2082-4276-99fc-bc121d8c8be3-kube-api-access-gsw8r\") pod \"payload-pre-processing-e2e-autores-fd0def-5d59887bb-xb8gk\" (UID: \"a173f5c9-2082-4276-99fc-bc121d8c8be3\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-fd0def-5d59887bb-xb8gk"
Sep 11 03:53:07.554421 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:53:07.554393    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c648f\" (UniqueName: \"kubernetes.io/projected/b1c46d01-52dc-4ef1-83a7-973ac377ef3d-kube-api-access-c648f\") pod \"payload-processing-e2e-autores-fd0def-bb56f6895-nn5m7\" (UID: \"b1c46d01-52dc-4ef1-83a7-973ac377ef3d\") " pod="openshift-ingress/payload-processing-e2e-autores-fd0def-bb56f6895-nn5m7"
Sep 11 03:53:07.554687 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:53:07.554665    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b1c46d01-52dc-4ef1-83a7-973ac377ef3d-plugins-config-volume\") pod \"payload-processing-e2e-autores-fd0def-bb56f6895-nn5m7\" (UID: \"b1c46d01-52dc-4ef1-83a7-973ac377ef3d\") " pod="openshift-ingress/payload-processing-e2e-autores-fd0def-bb56f6895-nn5m7"
Sep 11 03:53:07.655235 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:53:07.655198    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c648f\" (UniqueName: \"kubernetes.io/projected/b1c46d01-52dc-4ef1-83a7-973ac377ef3d-kube-api-access-c648f\") pod \"payload-processing-e2e-autores-fd0def-bb56f6895-nn5m7\" (UID: \"b1c46d01-52dc-4ef1-83a7-973ac377ef3d\") " pod="openshift-ingress/payload-processing-e2e-autores-fd0def-bb56f6895-nn5m7"
Sep 11 03:53:07.655414 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:53:07.655266    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b1c46d01-52dc-4ef1-83a7-973ac377ef3d-plugins-config-volume\") pod \"payload-processing-e2e-autores-fd0def-bb56f6895-nn5m7\" (UID: \"b1c46d01-52dc-4ef1-83a7-973ac377ef3d\") " pod="openshift-ingress/payload-processing-e2e-autores-fd0def-bb56f6895-nn5m7"
Sep 11 03:53:07.655922 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:53:07.655902    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b1c46d01-52dc-4ef1-83a7-973ac377ef3d-plugins-config-volume\") pod \"payload-processing-e2e-autores-fd0def-bb56f6895-nn5m7\" (UID: \"b1c46d01-52dc-4ef1-83a7-973ac377ef3d\") " pod="openshift-ingress/payload-processing-e2e-autores-fd0def-bb56f6895-nn5m7"
Sep 11 03:53:07.663119 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:53:07.663076    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-autores-fd0def-5d59887bb-xb8gk"
Sep 11 03:53:07.691945 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:53:07.691918    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-c648f\" (UniqueName: \"kubernetes.io/projected/b1c46d01-52dc-4ef1-83a7-973ac377ef3d-kube-api-access-c648f\") pod \"payload-processing-e2e-autores-fd0def-bb56f6895-nn5m7\" (UID: \"b1c46d01-52dc-4ef1-83a7-973ac377ef3d\") " pod="openshift-ingress/payload-processing-e2e-autores-fd0def-bb56f6895-nn5m7"
Sep 11 03:53:07.694714 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:53:07.694680    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-autores-fd0def-bb56f6895-nn5m7"
Sep 11 03:53:07.946497 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:53:07.946418    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-fd0def-5d59887bb-xb8gk"]
Sep 11 03:53:07.950651 ip-10-0-141-105 kubenswrapper[2579]: W0911 03:53:07.950529    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda173f5c9_2082_4276_99fc_bc121d8c8be3.slice/crio-043bec5b93e6ce94ea328e168f7c37f5416f530bdc72583c602e170e2e6d26e0 WatchSource:0}: Error finding container 043bec5b93e6ce94ea328e168f7c37f5416f530bdc72583c602e170e2e6d26e0: Status 404 returned error can't find the container with id 043bec5b93e6ce94ea328e168f7c37f5416f530bdc72583c602e170e2e6d26e0
Sep 11 03:53:07.984586 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:53:07.984565    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-fd0def-bb56f6895-nn5m7"]
Sep 11 03:53:07.992343 ip-10-0-141-105 kubenswrapper[2579]: W0911 03:53:07.992319    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb1c46d01_52dc_4ef1_83a7_973ac377ef3d.slice/crio-d7d61a01c032614dce646050580e58966c769021aa4c45f1ba66fd66e8707efd WatchSource:0}: Error finding container d7d61a01c032614dce646050580e58966c769021aa4c45f1ba66fd66e8707efd: Status 404 returned error can't find the container with id d7d61a01c032614dce646050580e58966c769021aa4c45f1ba66fd66e8707efd
Sep 11 03:53:08.522195 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:53:08.522159    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-autores-fd0def-bb56f6895-nn5m7" event={"ID":"b1c46d01-52dc-4ef1-83a7-973ac377ef3d","Type":"ContainerStarted","Data":"9734da4c21acf74093b2a4c6551178ffae02bb5a2cc7102e6d36ded6db1612e1"}
Sep 11 03:53:08.522665 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:53:08.522236    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-autores-fd0def-bb56f6895-nn5m7" event={"ID":"b1c46d01-52dc-4ef1-83a7-973ac377ef3d","Type":"ContainerStarted","Data":"d7d61a01c032614dce646050580e58966c769021aa4c45f1ba66fd66e8707efd"}
Sep 11 03:53:08.522665 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:53:08.522269    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-autores-fd0def-bb56f6895-nn5m7"
Sep 11 03:53:08.523947 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:53:08.523919    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-autores-fd0def-5d59887bb-xb8gk" event={"ID":"a173f5c9-2082-4276-99fc-bc121d8c8be3","Type":"ContainerStarted","Data":"161fff0a23a552b44db59c24de0138c1ecd8ff069dfbd3804edd586289c39590"}
Sep 11 03:53:08.524125 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:53:08.523960    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-autores-fd0def-5d59887bb-xb8gk" event={"ID":"a173f5c9-2082-4276-99fc-bc121d8c8be3","Type":"ContainerStarted","Data":"043bec5b93e6ce94ea328e168f7c37f5416f530bdc72583c602e170e2e6d26e0"}
Sep 11 03:53:08.524125 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:53:08.523997    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-autores-fd0def-5d59887bb-xb8gk"
Sep 11 03:53:08.540682 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:53:08.540631    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-autores-fd0def-bb56f6895-nn5m7" podStartSLOduration=1.5406120909999999 podStartE2EDuration="1.540612091s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 03:53:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 03:53:08.538402631 +0000 UTC m=+766.193608192" watchObservedRunningTime="2026-09-11 03:53:08.540612091 +0000 UTC m=+766.195817655"
Sep 11 03:53:08.555120 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:53:08.555063    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-autores-fd0def-5d59887bb-xb8gk" podStartSLOduration=1.5550487259999999 podStartE2EDuration="1.555048726s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 03:53:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 03:53:08.552516296 +0000 UTC m=+766.207721864" watchObservedRunningTime="2026-09-11 03:53:08.555048726 +0000 UTC m=+766.210254330"
Sep 11 03:53:14.503789 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:53:14.503741    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-1eeb81b2-56c595b47d-v9cgk"
Sep 11 03:53:14.504319 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:53:14.503965    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-1eeb81b2-798bd466f5-vxnm2"
Sep 11 03:53:15.335226 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:53:15.335186    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-1eeb81b2-znk66"]
Sep 11 03:53:15.339833 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:53:15.339797    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-1eeb81b2-znk66"
Sep 11 03:53:15.344622 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:53:15.344591    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 11 03:53:15.344881 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:53:15.344857    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-95ckr\""
Sep 11 03:53:15.349660 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:53:15.349626    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-1eeb81b2-znk66"]
Sep 11 03:53:15.416426 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:53:15.416393    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0b376a5b-903d-4da8-8e0f-25a1590ca210-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-1eeb81b2-znk66\" (UID: \"0b376a5b-903d-4da8-8e0f-25a1590ca210\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-1eeb81b2-znk66"
Sep 11 03:53:15.517369 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:53:15.517332    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0b376a5b-903d-4da8-8e0f-25a1590ca210-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-1eeb81b2-znk66\" (UID: \"0b376a5b-903d-4da8-8e0f-25a1590ca210\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-1eeb81b2-znk66"
Sep 11 03:53:15.517923 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:53:15.517903    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0b376a5b-903d-4da8-8e0f-25a1590ca210-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-1eeb81b2-znk66\" (UID: \"0b376a5b-903d-4da8-8e0f-25a1590ca210\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-1eeb81b2-znk66"
Sep 11 03:53:15.662367 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:53:15.662332    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-1eeb81b2-znk66"
Sep 11 03:53:15.948655 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:53:15.948564    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-1eeb81b2-znk66"]
Sep 11 03:53:15.964992 ip-10-0-141-105 kubenswrapper[2579]: W0911 03:53:15.964951    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0b376a5b_903d_4da8_8e0f_25a1590ca210.slice/crio-19f8e7c70240ac179139ead67254c1e4b0e6116dbd232b4125c0d1eddb758f6b WatchSource:0}: Error finding container 19f8e7c70240ac179139ead67254c1e4b0e6116dbd232b4125c0d1eddb758f6b: Status 404 returned error can't find the container with id 19f8e7c70240ac179139ead67254c1e4b0e6116dbd232b4125c0d1eddb758f6b
Sep 11 03:53:16.581989 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:53:16.581952    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-1eeb81b2-znk66" event={"ID":"0b376a5b-903d-4da8-8e0f-25a1590ca210","Type":"ContainerStarted","Data":"19f8e7c70240ac179139ead67254c1e4b0e6116dbd232b4125c0d1eddb758f6b"}
Sep 11 03:53:17.755105 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:53:17.754722    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-8efac4-68b56cd9d9-5vx4t"]
Sep 11 03:53:17.766125 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:53:17.760970    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-8efac4-68b56cd9d9-5vx4t"
Sep 11 03:53:17.768381 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:53:17.767414    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-8efac4\""
Sep 11 03:53:17.768381 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:53:17.767674    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-b-8efac4-dockercfg-2jvlf\""
Sep 11 03:53:17.769774 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:53:17.769712    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-8efac4-68b56cd9d9-5vx4t"]
Sep 11 03:53:17.862887 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:53:17.862634    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/98e573f4-5d9b-45c4-976a-42f439011e63-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-8efac4-68b56cd9d9-5vx4t\" (UID: \"98e573f4-5d9b-45c4-976a-42f439011e63\") " pod="openshift-ingress/payload-processing-e2e-inf-b-8efac4-68b56cd9d9-5vx4t"
Sep 11 03:53:17.862887 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:53:17.862760    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rj7hz\" (UniqueName: \"kubernetes.io/projected/98e573f4-5d9b-45c4-976a-42f439011e63-kube-api-access-rj7hz\") pod \"payload-processing-e2e-inf-b-8efac4-68b56cd9d9-5vx4t\" (UID: \"98e573f4-5d9b-45c4-976a-42f439011e63\") " pod="openshift-ingress/payload-processing-e2e-inf-b-8efac4-68b56cd9d9-5vx4t"
Sep 11 03:53:17.963581 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:53:17.963539    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rj7hz\" (UniqueName: \"kubernetes.io/projected/98e573f4-5d9b-45c4-976a-42f439011e63-kube-api-access-rj7hz\") pod \"payload-processing-e2e-inf-b-8efac4-68b56cd9d9-5vx4t\" (UID: \"98e573f4-5d9b-45c4-976a-42f439011e63\") " pod="openshift-ingress/payload-processing-e2e-inf-b-8efac4-68b56cd9d9-5vx4t"
Sep 11 03:53:17.963761 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:53:17.963636    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/98e573f4-5d9b-45c4-976a-42f439011e63-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-8efac4-68b56cd9d9-5vx4t\" (UID: \"98e573f4-5d9b-45c4-976a-42f439011e63\") " pod="openshift-ingress/payload-processing-e2e-inf-b-8efac4-68b56cd9d9-5vx4t"
Sep 11 03:53:17.964590 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:53:17.964569    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/98e573f4-5d9b-45c4-976a-42f439011e63-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-8efac4-68b56cd9d9-5vx4t\" (UID: \"98e573f4-5d9b-45c4-976a-42f439011e63\") " pod="openshift-ingress/payload-processing-e2e-inf-b-8efac4-68b56cd9d9-5vx4t"
Sep 11 03:53:17.971700 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:53:17.971653    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rj7hz\" (UniqueName: \"kubernetes.io/projected/98e573f4-5d9b-45c4-976a-42f439011e63-kube-api-access-rj7hz\") pod \"payload-processing-e2e-inf-b-8efac4-68b56cd9d9-5vx4t\" (UID: \"98e573f4-5d9b-45c4-976a-42f439011e63\") " pod="openshift-ingress/payload-processing-e2e-inf-b-8efac4-68b56cd9d9-5vx4t"
Sep 11 03:53:18.009520 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:53:18.009442    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-5z4cn"]
Sep 11 03:53:18.085067 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:53:18.085029    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-8efac4-68b56cd9d9-5vx4t"
Sep 11 03:53:19.234997 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:53:19.234965    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-8efac4-68b56cd9d9-5vx4t"]
Sep 11 03:53:19.530863 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:53:19.530774    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-autores-fd0def-bb56f6895-nn5m7"
Sep 11 03:53:19.530863 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:53:19.530849    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-autores-fd0def-5d59887bb-xb8gk"
Sep 11 03:53:19.603067 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:53:19.603035    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-8efac4-68b56cd9d9-5vx4t" event={"ID":"98e573f4-5d9b-45c4-976a-42f439011e63","Type":"ContainerStarted","Data":"815b5ecd26d546ed10957bde587cf4be85a85c1b25c9809b42cf6eeba3ca13ed"}
Sep 11 03:53:19.603262 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:53:19.603141    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-8efac4-68b56cd9d9-5vx4t" event={"ID":"98e573f4-5d9b-45c4-976a-42f439011e63","Type":"ContainerStarted","Data":"e5eb8b66835547ee2c364621d7cc59cea280622ad892f36447b46efb1bf0ac60"}
Sep 11 03:53:19.603262 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:53:19.603176    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-b-8efac4-68b56cd9d9-5vx4t"
Sep 11 03:53:19.604968 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:53:19.604924    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-1eeb81b2-znk66" event={"ID":"0b376a5b-903d-4da8-8e0f-25a1590ca210","Type":"ContainerStarted","Data":"7d2c23ded6ac69c5e9678d8a214cfc4d2b5c70fad549debed45dc24662239010"}
Sep 11 03:53:19.644109 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:53:19.634722    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-b-8efac4-68b56cd9d9-5vx4t" podStartSLOduration=2.6347014509999998 podStartE2EDuration="2.634701451s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 03:53:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 03:53:19.61949547 +0000 UTC m=+777.274701036" watchObservedRunningTime="2026-09-11 03:53:19.634701451 +0000 UTC m=+777.289907019"
Sep 11 03:53:20.612256 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:53:20.612221    2579 generic.go:356] "Generic (PLEG): container finished" podID="0b376a5b-903d-4da8-8e0f-25a1590ca210" containerID="7d2c23ded6ac69c5e9678d8a214cfc4d2b5c70fad549debed45dc24662239010" exitCode=0
Sep 11 03:53:20.612635 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:53:20.612307    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-1eeb81b2-znk66" event={"ID":"0b376a5b-903d-4da8-8e0f-25a1590ca210","Type":"ContainerDied","Data":"7d2c23ded6ac69c5e9678d8a214cfc4d2b5c70fad549debed45dc24662239010"}
Sep 11 03:53:21.768907 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:53:21.768881    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-1eeb81b2-znk66"
Sep 11 03:53:21.799679 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:53:21.799646    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0b376a5b-903d-4da8-8e0f-25a1590ca210-maas-api-service-ca\") pod \"0b376a5b-903d-4da8-8e0f-25a1590ca210\" (UID: \"0b376a5b-903d-4da8-8e0f-25a1590ca210\") "
Sep 11 03:53:21.800114 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:53:21.800069    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0b376a5b-903d-4da8-8e0f-25a1590ca210-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "0b376a5b-903d-4da8-8e0f-25a1590ca210" (UID: "0b376a5b-903d-4da8-8e0f-25a1590ca210"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 03:53:21.901418 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:53:21.901303    2579 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0b376a5b-903d-4da8-8e0f-25a1590ca210-maas-api-service-ca\") on node \"ip-10-0-141-105.ec2.internal\" DevicePath \"\""
Sep 11 03:53:22.625717 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:53:22.625679    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-1eeb81b2-znk66" event={"ID":"0b376a5b-903d-4da8-8e0f-25a1590ca210","Type":"ContainerDied","Data":"19f8e7c70240ac179139ead67254c1e4b0e6116dbd232b4125c0d1eddb758f6b"}
Sep 11 03:53:22.625717 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:53:22.625702    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-1eeb81b2-znk66"
Sep 11 03:53:22.626036 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:53:22.625733    2579 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="19f8e7c70240ac179139ead67254c1e4b0e6116dbd232b4125c0d1eddb758f6b"
Sep 11 03:53:23.632335 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:53:23.632306    2579 generic.go:356] "Generic (PLEG): container finished" podID="d2a064a0-c84d-4420-9d4b-42e679f412f3" containerID="3346bf6dd9e6ecb80e0d9f55889f4bb2f67351fdefd90e8aaf1d86c518682a65" exitCode=0
Sep 11 03:53:23.632732 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:53:23.632395    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-fd0def-openshift-default-6d8dfd864d-wjm48" event={"ID":"d2a064a0-c84d-4420-9d4b-42e679f412f3","Type":"ContainerDied","Data":"3346bf6dd9e6ecb80e0d9f55889f4bb2f67351fdefd90e8aaf1d86c518682a65"}
Sep 11 03:53:23.632809 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:53:23.632792    2579 scope.go:117] "RemoveContainer" containerID="3346bf6dd9e6ecb80e0d9f55889f4bb2f67351fdefd90e8aaf1d86c518682a65"
Sep 11 03:53:23.633871 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:53:23.633843    2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 11 03:53:23.633929 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:53:23.633914    2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 11 03:53:23.633963 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:53:23.633944    2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 11 03:53:24.640501 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:53:24.640451    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-fd0def-openshift-default-6d8dfd864d-wjm48" event={"ID":"d2a064a0-c84d-4420-9d4b-42e679f412f3","Type":"ContainerStarted","Data":"d7bb9e021bd5332c4afb55cccf722de0ac9fcd1f083d556555229253d2fef27b"}
Sep 11 03:53:25.162661 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:53:25.162622    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-autores-fd0def-openshift-default-6d8dfd864d-wjm48"
Sep 11 03:53:25.162836 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:53:25.162768    2579 patch_prober.go:28] interesting pod/e2e-autores-fd0def-openshift-default-6d8dfd864d-wjm48 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused" start-of-body=
Sep 11 03:53:25.162879 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:53:25.162824    2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-fd0def-openshift-default-6d8dfd864d-wjm48" podUID="d2a064a0-c84d-4420-9d4b-42e679f412f3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused"
Sep 11 03:53:25.972288 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:53:25.972218    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-1eeb81b2-798bd466f5-vxnm2"]
Sep 11 03:53:25.973068 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:53:25.973037    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-1eeb81b2-798bd466f5-vxnm2" podUID="9ac63561-3bd7-429b-891d-d721c113befc" containerName="payload-processing" containerID="cri-o://7dc4aaeb3a7a55c6970a33278b4fddc3ef5e58c60d6d00508f004da168146de1" gracePeriod=30
Sep 11 03:53:25.975541 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:53:25.975465    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-1eeb81b2-56c595b47d-v9cgk"]
Sep 11 03:53:25.975780 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:53:25.975738    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-1eeb81b2-56c595b47d-v9cgk" podUID="141630fc-f0e0-4672-97e9-406959f90486" containerName="payload-pre-processing" containerID="cri-o://e4ef95fc754e8377726aab0cda76fb53b852690bade0847798979db5d1035125" gracePeriod=30
Sep 11 03:53:26.074681 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:53:26.074648    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-1eeb81b2-znk66"]
Sep 11 03:53:26.077610 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:53:26.077584    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-1eeb81b2-znk66"]
Sep 11 03:53:26.162111 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:53:26.162006    2579 patch_prober.go:28] interesting pod/e2e-autores-fd0def-openshift-default-6d8dfd864d-wjm48 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused" start-of-body=
Sep 11 03:53:26.162312 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:53:26.162109    2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-fd0def-openshift-default-6d8dfd864d-wjm48" podUID="d2a064a0-c84d-4420-9d4b-42e679f412f3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused"
Sep 11 03:53:26.310665 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:53:26.310636    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-1eeb81b2-56c595b47d-v9cgk"
Sep 11 03:53:26.313986 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:53:26.313965    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-1eeb81b2-798bd466f5-vxnm2"
Sep 11 03:53:26.368240 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:53:26.363927    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9ac63561-3bd7-429b-891d-d721c113befc-plugins-config-volume\") pod \"9ac63561-3bd7-429b-891d-d721c113befc\" (UID: \"9ac63561-3bd7-429b-891d-d721c113befc\") "
Sep 11 03:53:26.368240 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:53:26.364127    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-stgmx\" (UniqueName: \"kubernetes.io/projected/141630fc-f0e0-4672-97e9-406959f90486-kube-api-access-stgmx\") pod \"141630fc-f0e0-4672-97e9-406959f90486\" (UID: \"141630fc-f0e0-4672-97e9-406959f90486\") "
Sep 11 03:53:26.368240 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:53:26.364218    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/141630fc-f0e0-4672-97e9-406959f90486-plugins-config-volume\") pod \"141630fc-f0e0-4672-97e9-406959f90486\" (UID: \"141630fc-f0e0-4672-97e9-406959f90486\") "
Sep 11 03:53:26.368240 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:53:26.364243    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tljzt\" (UniqueName: \"kubernetes.io/projected/9ac63561-3bd7-429b-891d-d721c113befc-kube-api-access-tljzt\") pod \"9ac63561-3bd7-429b-891d-d721c113befc\" (UID: \"9ac63561-3bd7-429b-891d-d721c113befc\") "
Sep 11 03:53:26.368240 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:53:26.364428    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9ac63561-3bd7-429b-891d-d721c113befc-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "9ac63561-3bd7-429b-891d-d721c113befc" (UID: "9ac63561-3bd7-429b-891d-d721c113befc"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 03:53:26.368240 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:53:26.364559    2579 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9ac63561-3bd7-429b-891d-d721c113befc-plugins-config-volume\") on node \"ip-10-0-141-105.ec2.internal\" DevicePath \"\""
Sep 11 03:53:26.368240 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:53:26.364883    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/141630fc-f0e0-4672-97e9-406959f90486-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "141630fc-f0e0-4672-97e9-406959f90486" (UID: "141630fc-f0e0-4672-97e9-406959f90486"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 03:53:26.368764 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:53:26.368254    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/141630fc-f0e0-4672-97e9-406959f90486-kube-api-access-stgmx" (OuterVolumeSpecName: "kube-api-access-stgmx") pod "141630fc-f0e0-4672-97e9-406959f90486" (UID: "141630fc-f0e0-4672-97e9-406959f90486"). InnerVolumeSpecName "kube-api-access-stgmx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 03:53:26.368832 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:53:26.368782    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9ac63561-3bd7-429b-891d-d721c113befc-kube-api-access-tljzt" (OuterVolumeSpecName: "kube-api-access-tljzt") pod "9ac63561-3bd7-429b-891d-d721c113befc" (UID: "9ac63561-3bd7-429b-891d-d721c113befc"). InnerVolumeSpecName "kube-api-access-tljzt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 03:53:26.465728 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:53:26.465694    2579 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/141630fc-f0e0-4672-97e9-406959f90486-plugins-config-volume\") on node \"ip-10-0-141-105.ec2.internal\" DevicePath \"\""
Sep 11 03:53:26.465728 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:53:26.465722    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tljzt\" (UniqueName: \"kubernetes.io/projected/9ac63561-3bd7-429b-891d-d721c113befc-kube-api-access-tljzt\") on node \"ip-10-0-141-105.ec2.internal\" DevicePath \"\""
Sep 11 03:53:26.465728 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:53:26.465735    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-stgmx\" (UniqueName: \"kubernetes.io/projected/141630fc-f0e0-4672-97e9-406959f90486-kube-api-access-stgmx\") on node \"ip-10-0-141-105.ec2.internal\" DevicePath \"\""
Sep 11 03:53:26.661218 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:53:26.661184    2579 generic.go:356] "Generic (PLEG): container finished" podID="9ac63561-3bd7-429b-891d-d721c113befc" containerID="7dc4aaeb3a7a55c6970a33278b4fddc3ef5e58c60d6d00508f004da168146de1" exitCode=0
Sep 11 03:53:26.661427 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:53:26.661239    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-1eeb81b2-798bd466f5-vxnm2" event={"ID":"9ac63561-3bd7-429b-891d-d721c113befc","Type":"ContainerDied","Data":"7dc4aaeb3a7a55c6970a33278b4fddc3ef5e58c60d6d00508f004da168146de1"}
Sep 11 03:53:26.661427 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:53:26.661309    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-1eeb81b2-798bd466f5-vxnm2" event={"ID":"9ac63561-3bd7-429b-891d-d721c113befc","Type":"ContainerDied","Data":"f1ea2ddb4dd29d5ef97f9aa9946a708013671b5919f335e855fd2d7f6762109d"}
Sep 11 03:53:26.661427 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:53:26.661249    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-1eeb81b2-798bd466f5-vxnm2"
Sep 11 03:53:26.661427 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:53:26.661342    2579 scope.go:117] "RemoveContainer" containerID="7dc4aaeb3a7a55c6970a33278b4fddc3ef5e58c60d6d00508f004da168146de1"
Sep 11 03:53:26.662886 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:53:26.662861    2579 generic.go:356] "Generic (PLEG): container finished" podID="141630fc-f0e0-4672-97e9-406959f90486" containerID="e4ef95fc754e8377726aab0cda76fb53b852690bade0847798979db5d1035125" exitCode=0
Sep 11 03:53:26.663061 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:53:26.662917    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-1eeb81b2-56c595b47d-v9cgk"
Sep 11 03:53:26.663061 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:53:26.662919    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-1eeb81b2-56c595b47d-v9cgk" event={"ID":"141630fc-f0e0-4672-97e9-406959f90486","Type":"ContainerDied","Data":"e4ef95fc754e8377726aab0cda76fb53b852690bade0847798979db5d1035125"}
Sep 11 03:53:26.663061 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:53:26.662966    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-1eeb81b2-56c595b47d-v9cgk" event={"ID":"141630fc-f0e0-4672-97e9-406959f90486","Type":"ContainerDied","Data":"f764d16aae1815bde5c920f16be147335a5367b49aa40ce4d19d5b58b439b65e"}
Sep 11 03:53:26.679974 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:53:26.679950    2579 scope.go:117] "RemoveContainer" containerID="7dc4aaeb3a7a55c6970a33278b4fddc3ef5e58c60d6d00508f004da168146de1"
Sep 11 03:53:26.680301 ip-10-0-141-105 kubenswrapper[2579]: E0911 03:53:26.680273    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7dc4aaeb3a7a55c6970a33278b4fddc3ef5e58c60d6d00508f004da168146de1\": container with ID starting with 7dc4aaeb3a7a55c6970a33278b4fddc3ef5e58c60d6d00508f004da168146de1 not found: ID does not exist" containerID="7dc4aaeb3a7a55c6970a33278b4fddc3ef5e58c60d6d00508f004da168146de1"
Sep 11 03:53:26.680364 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:53:26.680314    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7dc4aaeb3a7a55c6970a33278b4fddc3ef5e58c60d6d00508f004da168146de1"} err="failed to get container status \"7dc4aaeb3a7a55c6970a33278b4fddc3ef5e58c60d6d00508f004da168146de1\": rpc error: code = NotFound desc = could not find container \"7dc4aaeb3a7a55c6970a33278b4fddc3ef5e58c60d6d00508f004da168146de1\": container with ID starting with 7dc4aaeb3a7a55c6970a33278b4fddc3ef5e58c60d6d00508f004da168146de1 not found: ID does not exist"
Sep 11 03:53:26.680364 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:53:26.680331    2579 scope.go:117] "RemoveContainer" containerID="e4ef95fc754e8377726aab0cda76fb53b852690bade0847798979db5d1035125"
Sep 11 03:53:26.688970 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:53:26.688951    2579 scope.go:117] "RemoveContainer" containerID="e4ef95fc754e8377726aab0cda76fb53b852690bade0847798979db5d1035125"
Sep 11 03:53:26.689352 ip-10-0-141-105 kubenswrapper[2579]: E0911 03:53:26.689335    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e4ef95fc754e8377726aab0cda76fb53b852690bade0847798979db5d1035125\": container with ID starting with e4ef95fc754e8377726aab0cda76fb53b852690bade0847798979db5d1035125 not found: ID does not exist" containerID="e4ef95fc754e8377726aab0cda76fb53b852690bade0847798979db5d1035125"
Sep 11 03:53:26.689405 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:53:26.689358    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e4ef95fc754e8377726aab0cda76fb53b852690bade0847798979db5d1035125"} err="failed to get container status \"e4ef95fc754e8377726aab0cda76fb53b852690bade0847798979db5d1035125\": rpc error: code = NotFound desc = could not find container \"e4ef95fc754e8377726aab0cda76fb53b852690bade0847798979db5d1035125\": container with ID starting with e4ef95fc754e8377726aab0cda76fb53b852690bade0847798979db5d1035125 not found: ID does not exist"
Sep 11 03:53:26.695384 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:53:26.695361    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-1eeb81b2-56c595b47d-v9cgk"]
Sep 11 03:53:26.705844 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:53:26.705820    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-1eeb81b2-56c595b47d-v9cgk"]
Sep 11 03:53:26.720547 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:53:26.720523    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-1eeb81b2-798bd466f5-vxnm2"]
Sep 11 03:53:26.745109 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:53:26.734425    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-1eeb81b2-798bd466f5-vxnm2"]
Sep 11 03:53:26.858045 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:53:26.858006    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0b376a5b-903d-4da8-8e0f-25a1590ca210" path="/var/lib/kubelet/pods/0b376a5b-903d-4da8-8e0f-25a1590ca210/volumes"
Sep 11 03:53:26.858610 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:53:26.858585    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="141630fc-f0e0-4672-97e9-406959f90486" path="/var/lib/kubelet/pods/141630fc-f0e0-4672-97e9-406959f90486/volumes"
Sep 11 03:53:26.859100 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:53:26.859062    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9ac63561-3bd7-429b-891d-d721c113befc" path="/var/lib/kubelet/pods/9ac63561-3bd7-429b-891d-d721c113befc/volumes"
Sep 11 03:53:27.162445 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:53:27.162414    2579 patch_prober.go:28] interesting pod/e2e-autores-fd0def-openshift-default-6d8dfd864d-wjm48 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused" start-of-body=
Sep 11 03:53:27.162831 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:53:27.162471    2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-fd0def-openshift-default-6d8dfd864d-wjm48" podUID="d2a064a0-c84d-4420-9d4b-42e679f412f3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused"
Sep 11 03:53:28.162213 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:53:28.162177    2579 patch_prober.go:28] interesting pod/e2e-autores-fd0def-openshift-default-6d8dfd864d-wjm48 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused" start-of-body=
Sep 11 03:53:28.162399 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:53:28.162236    2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-fd0def-openshift-default-6d8dfd864d-wjm48" podUID="d2a064a0-c84d-4420-9d4b-42e679f412f3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused"
Sep 11 03:53:29.162470 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:53:29.162439    2579 patch_prober.go:28] interesting pod/e2e-autores-fd0def-openshift-default-6d8dfd864d-wjm48 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused" start-of-body=
Sep 11 03:53:29.162853 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:53:29.162493    2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-fd0def-openshift-default-6d8dfd864d-wjm48" podUID="d2a064a0-c84d-4420-9d4b-42e679f412f3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused"
Sep 11 03:53:30.162728 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:53:30.162682    2579 patch_prober.go:28] interesting pod/e2e-autores-fd0def-openshift-default-6d8dfd864d-wjm48 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused" start-of-body=
Sep 11 03:53:30.163204 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:53:30.162745    2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-fd0def-openshift-default-6d8dfd864d-wjm48" podUID="d2a064a0-c84d-4420-9d4b-42e679f412f3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused"
Sep 11 03:53:30.614818 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:53:30.614793    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-b-8efac4-68b56cd9d9-5vx4t"
Sep 11 03:53:31.166315 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:53:31.166285    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-autores-fd0def-openshift-default-6d8dfd864d-wjm48"
Sep 11 03:53:31.166703 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:53:31.166687    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-autores-fd0def-openshift-default-6d8dfd864d-wjm48"
Sep 11 03:53:31.167409 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:53:31.167393    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-autores-fd0def-openshift-default-6d8dfd864d-wjm48"
Sep 11 03:53:32.131401 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:53:32.131368    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-2c6bf060-844699d46b-nzdk8"]
Sep 11 03:53:32.131827 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:53:32.131810    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="141630fc-f0e0-4672-97e9-406959f90486" containerName="payload-pre-processing"
Sep 11 03:53:32.131921 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:53:32.131831    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="141630fc-f0e0-4672-97e9-406959f90486" containerName="payload-pre-processing"
Sep 11 03:53:32.131921 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:53:32.131842    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0b376a5b-903d-4da8-8e0f-25a1590ca210" containerName="revoke-keys"
Sep 11 03:53:32.131921 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:53:32.131851    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="0b376a5b-903d-4da8-8e0f-25a1590ca210" containerName="revoke-keys"
Sep 11 03:53:32.131921 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:53:32.131881    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9ac63561-3bd7-429b-891d-d721c113befc" containerName="payload-processing"
Sep 11 03:53:32.131921 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:53:32.131890    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="9ac63561-3bd7-429b-891d-d721c113befc" containerName="payload-processing"
Sep 11 03:53:32.132196 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:53:32.131991    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="0b376a5b-903d-4da8-8e0f-25a1590ca210" containerName="revoke-keys"
Sep 11 03:53:32.132196 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:53:32.132006    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="141630fc-f0e0-4672-97e9-406959f90486" containerName="payload-pre-processing"
Sep 11 03:53:32.132196 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:53:32.132016    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="9ac63561-3bd7-429b-891d-d721c113befc" containerName="payload-processing"
Sep 11 03:53:32.138264 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:53:32.138242    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-2c6bf060-844699d46b-nzdk8"
Sep 11 03:53:32.140908 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:53:32.140884    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-migrate-2c6bf060\""
Sep 11 03:53:32.142250 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:53:32.141992    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-migrate-2c6bf060-dockercfg-mvfxb\""
Sep 11 03:53:32.143403 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:53:32.143377    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-2c6bf060-844699d46b-nzdk8"]
Sep 11 03:53:32.224312 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:53:32.224272    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ee6559ba-a351-4b32-9f54-ac38c4ac5063-plugins-config-volume\") pod \"payload-processing-e2e-migrate-2c6bf060-844699d46b-nzdk8\" (UID: \"ee6559ba-a351-4b32-9f54-ac38c4ac5063\") " pod="openshift-ingress/payload-processing-e2e-migrate-2c6bf060-844699d46b-nzdk8"
Sep 11 03:53:32.224738 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:53:32.224396    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jlss4\" (UniqueName: \"kubernetes.io/projected/ee6559ba-a351-4b32-9f54-ac38c4ac5063-kube-api-access-jlss4\") pod \"payload-processing-e2e-migrate-2c6bf060-844699d46b-nzdk8\" (UID: \"ee6559ba-a351-4b32-9f54-ac38c4ac5063\") " pod="openshift-ingress/payload-processing-e2e-migrate-2c6bf060-844699d46b-nzdk8"
Sep 11 03:53:32.325545 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:53:32.325507    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ee6559ba-a351-4b32-9f54-ac38c4ac5063-plugins-config-volume\") pod \"payload-processing-e2e-migrate-2c6bf060-844699d46b-nzdk8\" (UID: \"ee6559ba-a351-4b32-9f54-ac38c4ac5063\") " pod="openshift-ingress/payload-processing-e2e-migrate-2c6bf060-844699d46b-nzdk8"
Sep 11 03:53:32.325737 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:53:32.325627    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jlss4\" (UniqueName: \"kubernetes.io/projected/ee6559ba-a351-4b32-9f54-ac38c4ac5063-kube-api-access-jlss4\") pod \"payload-processing-e2e-migrate-2c6bf060-844699d46b-nzdk8\" (UID: \"ee6559ba-a351-4b32-9f54-ac38c4ac5063\") " pod="openshift-ingress/payload-processing-e2e-migrate-2c6bf060-844699d46b-nzdk8"
Sep 11 03:53:32.326295 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:53:32.326270    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ee6559ba-a351-4b32-9f54-ac38c4ac5063-plugins-config-volume\") pod \"payload-processing-e2e-migrate-2c6bf060-844699d46b-nzdk8\" (UID: \"ee6559ba-a351-4b32-9f54-ac38c4ac5063\") " pod="openshift-ingress/payload-processing-e2e-migrate-2c6bf060-844699d46b-nzdk8"
Sep 11 03:53:32.332964 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:53:32.332939    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jlss4\" (UniqueName: \"kubernetes.io/projected/ee6559ba-a351-4b32-9f54-ac38c4ac5063-kube-api-access-jlss4\") pod \"payload-processing-e2e-migrate-2c6bf060-844699d46b-nzdk8\" (UID: \"ee6559ba-a351-4b32-9f54-ac38c4ac5063\") " pod="openshift-ingress/payload-processing-e2e-migrate-2c6bf060-844699d46b-nzdk8"
Sep 11 03:53:32.450957 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:53:32.450878    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-2c6bf060-844699d46b-nzdk8"
Sep 11 03:53:32.723946 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:53:32.723754    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-2c6bf060-844699d46b-nzdk8"]
Sep 11 03:53:32.726528 ip-10-0-141-105 kubenswrapper[2579]: W0911 03:53:32.726501    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podee6559ba_a351_4b32_9f54_ac38c4ac5063.slice/crio-59edf1882ce9ab56b5b27f80750e79c2e79d27df9c5b5ce00625cecc65c3223d WatchSource:0}: Error finding container 59edf1882ce9ab56b5b27f80750e79c2e79d27df9c5b5ce00625cecc65c3223d: Status 404 returned error can't find the container with id 59edf1882ce9ab56b5b27f80750e79c2e79d27df9c5b5ce00625cecc65c3223d
Sep 11 03:53:33.704198 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:53:33.704157    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-2c6bf060-844699d46b-nzdk8" event={"ID":"ee6559ba-a351-4b32-9f54-ac38c4ac5063","Type":"ContainerStarted","Data":"53df5f3bfdd9297bbfe668c3cdb72a8269b61407f484f680971092aebfb6ae04"}
Sep 11 03:53:33.704682 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:53:33.704215    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-2c6bf060-844699d46b-nzdk8" event={"ID":"ee6559ba-a351-4b32-9f54-ac38c4ac5063","Type":"ContainerStarted","Data":"59edf1882ce9ab56b5b27f80750e79c2e79d27df9c5b5ce00625cecc65c3223d"}
Sep 11 03:53:33.704682 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:53:33.704243    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-migrate-2c6bf060-844699d46b-nzdk8"
Sep 11 03:53:33.721397 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:53:33.721346    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-migrate-2c6bf060-844699d46b-nzdk8" podStartSLOduration=1.72133146 podStartE2EDuration="1.72133146s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 03:53:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 03:53:33.719382874 +0000 UTC m=+791.374588442" watchObservedRunningTime="2026-09-11 03:53:33.72133146 +0000 UTC m=+791.376537026"
Sep 11 03:53:36.610586 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:53:36.610553    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-5z4cn"]
Sep 11 03:53:44.711803 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:53:44.711728    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-migrate-2c6bf060-844699d46b-nzdk8"
Sep 11 03:53:54.603397 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:53:54.603359    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-2c6bf060-dw9fj"]
Sep 11 03:53:54.611424 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:53:54.611402    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-2c6bf060-dw9fj"
Sep 11 03:53:54.613210 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:53:54.613183    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-2c6bf060-dw9fj"]
Sep 11 03:53:54.613905 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:53:54.613884    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 11 03:53:54.613905 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:53:54.613899    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-95ckr\""
Sep 11 03:53:54.631201 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:53:54.631168    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6a04a611-d300-4d83-b59e-364e236ba06d-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-2c6bf060-dw9fj\" (UID: \"6a04a611-d300-4d83-b59e-364e236ba06d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-2c6bf060-dw9fj"
Sep 11 03:53:54.732557 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:53:54.732530    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6a04a611-d300-4d83-b59e-364e236ba06d-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-2c6bf060-dw9fj\" (UID: \"6a04a611-d300-4d83-b59e-364e236ba06d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-2c6bf060-dw9fj"
Sep 11 03:53:54.733236 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:53:54.733214    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6a04a611-d300-4d83-b59e-364e236ba06d-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-2c6bf060-dw9fj\" (UID: \"6a04a611-d300-4d83-b59e-364e236ba06d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-2c6bf060-dw9fj"
Sep 11 03:53:54.924683 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:53:54.924599    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-2c6bf060-dw9fj"
Sep 11 03:53:55.198505 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:53:55.198481    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-2c6bf060-dw9fj"]
Sep 11 03:53:55.200691 ip-10-0-141-105 kubenswrapper[2579]: W0911 03:53:55.200666    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6a04a611_d300_4d83_b59e_364e236ba06d.slice/crio-2fc1e1f2376433c11489ed757b3eaa0e31a2b1021ad8d5501177e337db51c9fb WatchSource:0}: Error finding container 2fc1e1f2376433c11489ed757b3eaa0e31a2b1021ad8d5501177e337db51c9fb: Status 404 returned error can't find the container with id 2fc1e1f2376433c11489ed757b3eaa0e31a2b1021ad8d5501177e337db51c9fb
Sep 11 03:53:55.815426 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:53:55.815331    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-2c6bf060-dw9fj" event={"ID":"6a04a611-d300-4d83-b59e-364e236ba06d","Type":"ContainerStarted","Data":"34e9d149f56159ae10cc0751a5f95508b6506bb9bfb7c37a7dbea4adde383673"}
Sep 11 03:53:55.815426 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:53:55.815399    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-2c6bf060-dw9fj" event={"ID":"6a04a611-d300-4d83-b59e-364e236ba06d","Type":"ContainerStarted","Data":"2fc1e1f2376433c11489ed757b3eaa0e31a2b1021ad8d5501177e337db51c9fb"}
Sep 11 03:53:55.829993 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:53:55.829945    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-2c6bf060-dw9fj" podStartSLOduration=1.82992859 podStartE2EDuration="1.82992859s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 03:53:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 03:53:55.828393816 +0000 UTC m=+813.483599402" watchObservedRunningTime="2026-09-11 03:53:55.82992859 +0000 UTC m=+813.485134157"
Sep 11 03:53:56.822307 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:53:56.822277    2579 generic.go:356] "Generic (PLEG): container finished" podID="6a04a611-d300-4d83-b59e-364e236ba06d" containerID="34e9d149f56159ae10cc0751a5f95508b6506bb9bfb7c37a7dbea4adde383673" exitCode=0
Sep 11 03:53:56.822747 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:53:56.822340    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-2c6bf060-dw9fj" event={"ID":"6a04a611-d300-4d83-b59e-364e236ba06d","Type":"ContainerDied","Data":"34e9d149f56159ae10cc0751a5f95508b6506bb9bfb7c37a7dbea4adde383673"}
Sep 11 03:53:57.967028 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:53:57.967000    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-2c6bf060-dw9fj"
Sep 11 03:53:58.064734 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:53:58.064696    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6a04a611-d300-4d83-b59e-364e236ba06d-maas-api-service-ca\") pod \"6a04a611-d300-4d83-b59e-364e236ba06d\" (UID: \"6a04a611-d300-4d83-b59e-364e236ba06d\") "
Sep 11 03:53:58.065060 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:53:58.065034    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6a04a611-d300-4d83-b59e-364e236ba06d-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "6a04a611-d300-4d83-b59e-364e236ba06d" (UID: "6a04a611-d300-4d83-b59e-364e236ba06d"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 03:53:58.165716 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:53:58.165684    2579 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6a04a611-d300-4d83-b59e-364e236ba06d-maas-api-service-ca\") on node \"ip-10-0-141-105.ec2.internal\" DevicePath \"\""
Sep 11 03:53:58.834548 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:53:58.834521    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-2c6bf060-dw9fj"
Sep 11 03:53:58.834711 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:53:58.834561    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-2c6bf060-dw9fj" event={"ID":"6a04a611-d300-4d83-b59e-364e236ba06d","Type":"ContainerDied","Data":"2fc1e1f2376433c11489ed757b3eaa0e31a2b1021ad8d5501177e337db51c9fb"}
Sep 11 03:53:58.834711 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:53:58.834634    2579 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2fc1e1f2376433c11489ed757b3eaa0e31a2b1021ad8d5501177e337db51c9fb"
Sep 11 03:54:00.579030 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:54:00.579001    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-conflict-b-08754c30-kserve-7cbdbfdb97-ggm4h"]
Sep 11 03:54:00.579530 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:54:00.579440    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6a04a611-d300-4d83-b59e-364e236ba06d" containerName="revoke-keys"
Sep 11 03:54:00.579530 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:54:00.579458    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="6a04a611-d300-4d83-b59e-364e236ba06d" containerName="revoke-keys"
Sep 11 03:54:00.579661 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:54:00.579550    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="6a04a611-d300-4d83-b59e-364e236ba06d" containerName="revoke-keys"
Sep 11 03:54:00.583071 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:54:00.583050    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-conflict-b-08754c30-kserve-7cbdbfdb97-ggm4h"
Sep 11 03:54:00.585633 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:54:00.585611    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-conflict-b-08754c30-kserve-self-signed-certs\""
Sep 11 03:54:00.595836 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:54:00.595793    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-conflict-b-08754c30-kserve-7cbdbfdb97-ggm4h"]
Sep 11 03:54:00.688693 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:54:00.688657    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/47052321-c2e8-489e-8f54-42b49609fa66-tmp-dir\") pod \"e2e-conflict-b-08754c30-kserve-7cbdbfdb97-ggm4h\" (UID: \"47052321-c2e8-489e-8f54-42b49609fa66\") " pod="llm/e2e-conflict-b-08754c30-kserve-7cbdbfdb97-ggm4h"
Sep 11 03:54:00.688889 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:54:00.688706    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/47052321-c2e8-489e-8f54-42b49609fa66-model-cache\") pod \"e2e-conflict-b-08754c30-kserve-7cbdbfdb97-ggm4h\" (UID: \"47052321-c2e8-489e-8f54-42b49609fa66\") " pod="llm/e2e-conflict-b-08754c30-kserve-7cbdbfdb97-ggm4h"
Sep 11 03:54:00.688889 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:54:00.688741    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mttp6\" (UniqueName: \"kubernetes.io/projected/47052321-c2e8-489e-8f54-42b49609fa66-kube-api-access-mttp6\") pod \"e2e-conflict-b-08754c30-kserve-7cbdbfdb97-ggm4h\" (UID: \"47052321-c2e8-489e-8f54-42b49609fa66\") " pod="llm/e2e-conflict-b-08754c30-kserve-7cbdbfdb97-ggm4h"
Sep 11 03:54:00.688889 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:54:00.688836    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/47052321-c2e8-489e-8f54-42b49609fa66-home\") pod \"e2e-conflict-b-08754c30-kserve-7cbdbfdb97-ggm4h\" (UID: \"47052321-c2e8-489e-8f54-42b49609fa66\") " pod="llm/e2e-conflict-b-08754c30-kserve-7cbdbfdb97-ggm4h"
Sep 11 03:54:00.688889 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:54:00.688870    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/47052321-c2e8-489e-8f54-42b49609fa66-dshm\") pod \"e2e-conflict-b-08754c30-kserve-7cbdbfdb97-ggm4h\" (UID: \"47052321-c2e8-489e-8f54-42b49609fa66\") " pod="llm/e2e-conflict-b-08754c30-kserve-7cbdbfdb97-ggm4h"
Sep 11 03:54:00.689132 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:54:00.688934    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/47052321-c2e8-489e-8f54-42b49609fa66-tls-certs\") pod \"e2e-conflict-b-08754c30-kserve-7cbdbfdb97-ggm4h\" (UID: \"47052321-c2e8-489e-8f54-42b49609fa66\") " pod="llm/e2e-conflict-b-08754c30-kserve-7cbdbfdb97-ggm4h"
Sep 11 03:54:00.790260 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:54:00.790227    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/47052321-c2e8-489e-8f54-42b49609fa66-tmp-dir\") pod \"e2e-conflict-b-08754c30-kserve-7cbdbfdb97-ggm4h\" (UID: \"47052321-c2e8-489e-8f54-42b49609fa66\") " pod="llm/e2e-conflict-b-08754c30-kserve-7cbdbfdb97-ggm4h"
Sep 11 03:54:00.790481 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:54:00.790270    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/47052321-c2e8-489e-8f54-42b49609fa66-model-cache\") pod \"e2e-conflict-b-08754c30-kserve-7cbdbfdb97-ggm4h\" (UID: \"47052321-c2e8-489e-8f54-42b49609fa66\") " pod="llm/e2e-conflict-b-08754c30-kserve-7cbdbfdb97-ggm4h"
Sep 11 03:54:00.790481 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:54:00.790292    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mttp6\" (UniqueName: \"kubernetes.io/projected/47052321-c2e8-489e-8f54-42b49609fa66-kube-api-access-mttp6\") pod \"e2e-conflict-b-08754c30-kserve-7cbdbfdb97-ggm4h\" (UID: \"47052321-c2e8-489e-8f54-42b49609fa66\") " pod="llm/e2e-conflict-b-08754c30-kserve-7cbdbfdb97-ggm4h"
Sep 11 03:54:00.790481 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:54:00.790317    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/47052321-c2e8-489e-8f54-42b49609fa66-home\") pod \"e2e-conflict-b-08754c30-kserve-7cbdbfdb97-ggm4h\" (UID: \"47052321-c2e8-489e-8f54-42b49609fa66\") " pod="llm/e2e-conflict-b-08754c30-kserve-7cbdbfdb97-ggm4h"
Sep 11 03:54:00.790481 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:54:00.790333    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/47052321-c2e8-489e-8f54-42b49609fa66-dshm\") pod \"e2e-conflict-b-08754c30-kserve-7cbdbfdb97-ggm4h\" (UID: \"47052321-c2e8-489e-8f54-42b49609fa66\") " pod="llm/e2e-conflict-b-08754c30-kserve-7cbdbfdb97-ggm4h"
Sep 11 03:54:00.790481 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:54:00.790365    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/47052321-c2e8-489e-8f54-42b49609fa66-tls-certs\") pod \"e2e-conflict-b-08754c30-kserve-7cbdbfdb97-ggm4h\" (UID: \"47052321-c2e8-489e-8f54-42b49609fa66\") " pod="llm/e2e-conflict-b-08754c30-kserve-7cbdbfdb97-ggm4h"
Sep 11 03:54:00.790774 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:54:00.790697    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/47052321-c2e8-489e-8f54-42b49609fa66-model-cache\") pod \"e2e-conflict-b-08754c30-kserve-7cbdbfdb97-ggm4h\" (UID: \"47052321-c2e8-489e-8f54-42b49609fa66\") " pod="llm/e2e-conflict-b-08754c30-kserve-7cbdbfdb97-ggm4h"
Sep 11 03:54:00.790838 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:54:00.790779    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/47052321-c2e8-489e-8f54-42b49609fa66-home\") pod \"e2e-conflict-b-08754c30-kserve-7cbdbfdb97-ggm4h\" (UID: \"47052321-c2e8-489e-8f54-42b49609fa66\") " pod="llm/e2e-conflict-b-08754c30-kserve-7cbdbfdb97-ggm4h"
Sep 11 03:54:00.791110 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:54:00.791051    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/47052321-c2e8-489e-8f54-42b49609fa66-tmp-dir\") pod \"e2e-conflict-b-08754c30-kserve-7cbdbfdb97-ggm4h\" (UID: \"47052321-c2e8-489e-8f54-42b49609fa66\") " pod="llm/e2e-conflict-b-08754c30-kserve-7cbdbfdb97-ggm4h"
Sep 11 03:54:00.792887 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:54:00.792860    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/47052321-c2e8-489e-8f54-42b49609fa66-dshm\") pod \"e2e-conflict-b-08754c30-kserve-7cbdbfdb97-ggm4h\" (UID: \"47052321-c2e8-489e-8f54-42b49609fa66\") " pod="llm/e2e-conflict-b-08754c30-kserve-7cbdbfdb97-ggm4h"
Sep 11 03:54:00.793218 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:54:00.793200    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/47052321-c2e8-489e-8f54-42b49609fa66-tls-certs\") pod \"e2e-conflict-b-08754c30-kserve-7cbdbfdb97-ggm4h\" (UID: \"47052321-c2e8-489e-8f54-42b49609fa66\") " pod="llm/e2e-conflict-b-08754c30-kserve-7cbdbfdb97-ggm4h"
Sep 11 03:54:00.798317 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:54:00.798298    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mttp6\" (UniqueName: \"kubernetes.io/projected/47052321-c2e8-489e-8f54-42b49609fa66-kube-api-access-mttp6\") pod \"e2e-conflict-b-08754c30-kserve-7cbdbfdb97-ggm4h\" (UID: \"47052321-c2e8-489e-8f54-42b49609fa66\") " pod="llm/e2e-conflict-b-08754c30-kserve-7cbdbfdb97-ggm4h"
Sep 11 03:54:00.892206 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:54:00.892182    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-conflict-b-08754c30-kserve-7cbdbfdb97-ggm4h"
Sep 11 03:54:01.165317 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:54:01.165289    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-conflict-b-08754c30-kserve-7cbdbfdb97-ggm4h"]
Sep 11 03:54:01.167128 ip-10-0-141-105 kubenswrapper[2579]: W0911 03:54:01.167073    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod47052321_c2e8_489e_8f54_42b49609fa66.slice/crio-213518e8b5886627ddaa5f1a9c501382823601106b34ed8dfe7586e6f7221d5b WatchSource:0}: Error finding container 213518e8b5886627ddaa5f1a9c501382823601106b34ed8dfe7586e6f7221d5b: Status 404 returned error can't find the container with id 213518e8b5886627ddaa5f1a9c501382823601106b34ed8dfe7586e6f7221d5b
Sep 11 03:54:01.664629 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:54:01.664596    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-2c6bf060-844699d46b-nzdk8"]
Sep 11 03:54:01.665115 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:54:01.664834    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-migrate-2c6bf060-844699d46b-nzdk8" podUID="ee6559ba-a351-4b32-9f54-ac38c4ac5063" containerName="payload-processing" containerID="cri-o://53df5f3bfdd9297bbfe668c3cdb72a8269b61407f484f680971092aebfb6ae04" gracePeriod=30
Sep 11 03:54:01.854609 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:54:01.854209    2579 generic.go:356] "Generic (PLEG): container finished" podID="ee6559ba-a351-4b32-9f54-ac38c4ac5063" containerID="53df5f3bfdd9297bbfe668c3cdb72a8269b61407f484f680971092aebfb6ae04" exitCode=0
Sep 11 03:54:01.854609 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:54:01.854307    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-2c6bf060-844699d46b-nzdk8" event={"ID":"ee6559ba-a351-4b32-9f54-ac38c4ac5063","Type":"ContainerDied","Data":"53df5f3bfdd9297bbfe668c3cdb72a8269b61407f484f680971092aebfb6ae04"}
Sep 11 03:54:01.857012 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:54:01.856962    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-conflict-b-08754c30-kserve-7cbdbfdb97-ggm4h" event={"ID":"47052321-c2e8-489e-8f54-42b49609fa66","Type":"ContainerStarted","Data":"3b2ec372875d89975520fdfb5d6793c4298a42949ccfb0b8fcdb92268cbd7e77"}
Sep 11 03:54:01.857012 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:54:01.857009    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-conflict-b-08754c30-kserve-7cbdbfdb97-ggm4h" event={"ID":"47052321-c2e8-489e-8f54-42b49609fa66","Type":"ContainerStarted","Data":"213518e8b5886627ddaa5f1a9c501382823601106b34ed8dfe7586e6f7221d5b"}
Sep 11 03:54:01.885187 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:54:01.885159    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-2c6bf060-dw9fj"]
Sep 11 03:54:01.896671 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:54:01.896617    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-conflict-b-08754c30-kserve-7cbdbfdb97-ggm4h" podStartSLOduration=1.8965964990000002 podStartE2EDuration="1.896596499s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 03:54:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 03:54:01.894336213 +0000 UTC m=+819.549541791" watchObservedRunningTime="2026-09-11 03:54:01.896596499 +0000 UTC m=+819.551802067"
Sep 11 03:54:01.901732 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:54:01.901702    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-2c6bf060-dw9fj"]
Sep 11 03:54:01.932593 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:54:01.932571    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-2c6bf060-844699d46b-nzdk8"
Sep 11 03:54:02.001808 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:54:02.001775    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ee6559ba-a351-4b32-9f54-ac38c4ac5063-plugins-config-volume\") pod \"ee6559ba-a351-4b32-9f54-ac38c4ac5063\" (UID: \"ee6559ba-a351-4b32-9f54-ac38c4ac5063\") "
Sep 11 03:54:02.001995 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:54:02.001835    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jlss4\" (UniqueName: \"kubernetes.io/projected/ee6559ba-a351-4b32-9f54-ac38c4ac5063-kube-api-access-jlss4\") pod \"ee6559ba-a351-4b32-9f54-ac38c4ac5063\" (UID: \"ee6559ba-a351-4b32-9f54-ac38c4ac5063\") "
Sep 11 03:54:02.002225 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:54:02.002199    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ee6559ba-a351-4b32-9f54-ac38c4ac5063-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "ee6559ba-a351-4b32-9f54-ac38c4ac5063" (UID: "ee6559ba-a351-4b32-9f54-ac38c4ac5063"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 03:54:02.004482 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:54:02.004457    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ee6559ba-a351-4b32-9f54-ac38c4ac5063-kube-api-access-jlss4" (OuterVolumeSpecName: "kube-api-access-jlss4") pod "ee6559ba-a351-4b32-9f54-ac38c4ac5063" (UID: "ee6559ba-a351-4b32-9f54-ac38c4ac5063"). InnerVolumeSpecName "kube-api-access-jlss4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 03:54:02.103561 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:54:02.103527    2579 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ee6559ba-a351-4b32-9f54-ac38c4ac5063-plugins-config-volume\") on node \"ip-10-0-141-105.ec2.internal\" DevicePath \"\""
Sep 11 03:54:02.103561 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:54:02.103557    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jlss4\" (UniqueName: \"kubernetes.io/projected/ee6559ba-a351-4b32-9f54-ac38c4ac5063-kube-api-access-jlss4\") on node \"ip-10-0-141-105.ec2.internal\" DevicePath \"\""
Sep 11 03:54:02.862401 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:54:02.862368    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6a04a611-d300-4d83-b59e-364e236ba06d" path="/var/lib/kubelet/pods/6a04a611-d300-4d83-b59e-364e236ba06d/volumes"
Sep 11 03:54:02.865334 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:54:02.865310    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-2c6bf060-844699d46b-nzdk8"
Sep 11 03:54:02.865479 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:54:02.865340    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-2c6bf060-844699d46b-nzdk8" event={"ID":"ee6559ba-a351-4b32-9f54-ac38c4ac5063","Type":"ContainerDied","Data":"59edf1882ce9ab56b5b27f80750e79c2e79d27df9c5b5ce00625cecc65c3223d"}
Sep 11 03:54:02.865479 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:54:02.865406    2579 scope.go:117] "RemoveContainer" containerID="53df5f3bfdd9297bbfe668c3cdb72a8269b61407f484f680971092aebfb6ae04"
Sep 11 03:54:02.889592 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:54:02.889570    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-2c6bf060-844699d46b-nzdk8"]
Sep 11 03:54:02.892424 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:54:02.892402    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-2c6bf060-844699d46b-nzdk8"]
Sep 11 03:54:04.852450 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:54:04.852417    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ee6559ba-a351-4b32-9f54-ac38c4ac5063" path="/var/lib/kubelet/pods/ee6559ba-a351-4b32-9f54-ac38c4ac5063/volumes"
Sep 11 03:54:08.010871 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:54:08.010836    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-5z4cn"]
Sep 11 03:54:10.893070 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:54:10.893032    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-conflict-b-08754c30-kserve-7cbdbfdb97-ggm4h"
Sep 11 03:54:10.893483 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:54:10.893132    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-conflict-b-08754c30-kserve-7cbdbfdb97-ggm4h"
Sep 11 03:54:10.905777 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:54:10.905747    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-conflict-b-08754c30-kserve-7cbdbfdb97-ggm4h"
Sep 11 03:54:10.918036 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:54:10.918016    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-conflict-b-08754c30-kserve-7cbdbfdb97-ggm4h"
Sep 11 03:54:25.984017 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:54:25.983981    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-autores-fd0def/autores-model-fd0def-kserve-79f55c767-w5wkz"]
Sep 11 03:54:25.984451 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:54:25.984438    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ee6559ba-a351-4b32-9f54-ac38c4ac5063" containerName="payload-processing"
Sep 11 03:54:25.984451 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:54:25.984452    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="ee6559ba-a351-4b32-9f54-ac38c4ac5063" containerName="payload-processing"
Sep 11 03:54:25.984522 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:54:25.984518    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="ee6559ba-a351-4b32-9f54-ac38c4ac5063" containerName="payload-processing"
Sep 11 03:54:25.988589 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:54:25.988569    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-autores-fd0def/autores-model-fd0def-kserve-79f55c767-w5wkz"
Sep 11 03:54:25.990904 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:54:25.990884    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-autores-fd0def\"/\"kube-root-ca.crt\""
Sep 11 03:54:25.992185 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:54:25.992163    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-autores-fd0def\"/\"default-dockercfg-9q9pm\""
Sep 11 03:54:25.992378 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:54:25.992198    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-autores-fd0def\"/\"openshift-service-ca.crt\""
Sep 11 03:54:25.992518 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:54:25.992199    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-autores-fd0def\"/\"autores-model-fd0def-kserve-self-signed-certs\""
Sep 11 03:54:25.994685 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:54:25.994659    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-autores-fd0def/autores-model-fd0def-kserve-79f55c767-w5wkz"]
Sep 11 03:54:26.132810 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:54:26.132775    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6cef66fb-3401-4c33-bfc3-f784d0ff19cd-tmp-dir\") pod \"autores-model-fd0def-kserve-79f55c767-w5wkz\" (UID: \"6cef66fb-3401-4c33-bfc3-f784d0ff19cd\") " pod="ai-tenant-e2e-autores-fd0def/autores-model-fd0def-kserve-79f55c767-w5wkz"
Sep 11 03:54:26.132976 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:54:26.132837    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6cef66fb-3401-4c33-bfc3-f784d0ff19cd-tls-certs\") pod \"autores-model-fd0def-kserve-79f55c767-w5wkz\" (UID: \"6cef66fb-3401-4c33-bfc3-f784d0ff19cd\") " pod="ai-tenant-e2e-autores-fd0def/autores-model-fd0def-kserve-79f55c767-w5wkz"
Sep 11 03:54:26.132976 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:54:26.132874    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6cef66fb-3401-4c33-bfc3-f784d0ff19cd-dshm\") pod \"autores-model-fd0def-kserve-79f55c767-w5wkz\" (UID: \"6cef66fb-3401-4c33-bfc3-f784d0ff19cd\") " pod="ai-tenant-e2e-autores-fd0def/autores-model-fd0def-kserve-79f55c767-w5wkz"
Sep 11 03:54:26.132976 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:54:26.132925    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7tg7f\" (UniqueName: \"kubernetes.io/projected/6cef66fb-3401-4c33-bfc3-f784d0ff19cd-kube-api-access-7tg7f\") pod \"autores-model-fd0def-kserve-79f55c767-w5wkz\" (UID: \"6cef66fb-3401-4c33-bfc3-f784d0ff19cd\") " pod="ai-tenant-e2e-autores-fd0def/autores-model-fd0def-kserve-79f55c767-w5wkz"
Sep 11 03:54:26.133136 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:54:26.132976    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6cef66fb-3401-4c33-bfc3-f784d0ff19cd-model-cache\") pod \"autores-model-fd0def-kserve-79f55c767-w5wkz\" (UID: \"6cef66fb-3401-4c33-bfc3-f784d0ff19cd\") " pod="ai-tenant-e2e-autores-fd0def/autores-model-fd0def-kserve-79f55c767-w5wkz"
Sep 11 03:54:26.133136 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:54:26.133029    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6cef66fb-3401-4c33-bfc3-f784d0ff19cd-home\") pod \"autores-model-fd0def-kserve-79f55c767-w5wkz\" (UID: \"6cef66fb-3401-4c33-bfc3-f784d0ff19cd\") " pod="ai-tenant-e2e-autores-fd0def/autores-model-fd0def-kserve-79f55c767-w5wkz"
Sep 11 03:54:26.233700 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:54:26.233663    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6cef66fb-3401-4c33-bfc3-f784d0ff19cd-home\") pod \"autores-model-fd0def-kserve-79f55c767-w5wkz\" (UID: \"6cef66fb-3401-4c33-bfc3-f784d0ff19cd\") " pod="ai-tenant-e2e-autores-fd0def/autores-model-fd0def-kserve-79f55c767-w5wkz"
Sep 11 03:54:26.233899 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:54:26.233712    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6cef66fb-3401-4c33-bfc3-f784d0ff19cd-tmp-dir\") pod \"autores-model-fd0def-kserve-79f55c767-w5wkz\" (UID: \"6cef66fb-3401-4c33-bfc3-f784d0ff19cd\") " pod="ai-tenant-e2e-autores-fd0def/autores-model-fd0def-kserve-79f55c767-w5wkz"
Sep 11 03:54:26.233899 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:54:26.233764    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6cef66fb-3401-4c33-bfc3-f784d0ff19cd-tls-certs\") pod \"autores-model-fd0def-kserve-79f55c767-w5wkz\" (UID: \"6cef66fb-3401-4c33-bfc3-f784d0ff19cd\") " pod="ai-tenant-e2e-autores-fd0def/autores-model-fd0def-kserve-79f55c767-w5wkz"
Sep 11 03:54:26.233899 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:54:26.233787    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6cef66fb-3401-4c33-bfc3-f784d0ff19cd-dshm\") pod \"autores-model-fd0def-kserve-79f55c767-w5wkz\" (UID: \"6cef66fb-3401-4c33-bfc3-f784d0ff19cd\") " pod="ai-tenant-e2e-autores-fd0def/autores-model-fd0def-kserve-79f55c767-w5wkz"
Sep 11 03:54:26.233899 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:54:26.233826    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7tg7f\" (UniqueName: \"kubernetes.io/projected/6cef66fb-3401-4c33-bfc3-f784d0ff19cd-kube-api-access-7tg7f\") pod \"autores-model-fd0def-kserve-79f55c767-w5wkz\" (UID: \"6cef66fb-3401-4c33-bfc3-f784d0ff19cd\") " pod="ai-tenant-e2e-autores-fd0def/autores-model-fd0def-kserve-79f55c767-w5wkz"
Sep 11 03:54:26.233899 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:54:26.233864    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6cef66fb-3401-4c33-bfc3-f784d0ff19cd-model-cache\") pod \"autores-model-fd0def-kserve-79f55c767-w5wkz\" (UID: \"6cef66fb-3401-4c33-bfc3-f784d0ff19cd\") " pod="ai-tenant-e2e-autores-fd0def/autores-model-fd0def-kserve-79f55c767-w5wkz"
Sep 11 03:54:26.234257 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:54:26.234166    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6cef66fb-3401-4c33-bfc3-f784d0ff19cd-home\") pod \"autores-model-fd0def-kserve-79f55c767-w5wkz\" (UID: \"6cef66fb-3401-4c33-bfc3-f784d0ff19cd\") " pod="ai-tenant-e2e-autores-fd0def/autores-model-fd0def-kserve-79f55c767-w5wkz"
Sep 11 03:54:26.234257 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:54:26.234190    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6cef66fb-3401-4c33-bfc3-f784d0ff19cd-tmp-dir\") pod \"autores-model-fd0def-kserve-79f55c767-w5wkz\" (UID: \"6cef66fb-3401-4c33-bfc3-f784d0ff19cd\") " pod="ai-tenant-e2e-autores-fd0def/autores-model-fd0def-kserve-79f55c767-w5wkz"
Sep 11 03:54:26.234257 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:54:26.234230    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6cef66fb-3401-4c33-bfc3-f784d0ff19cd-model-cache\") pod \"autores-model-fd0def-kserve-79f55c767-w5wkz\" (UID: \"6cef66fb-3401-4c33-bfc3-f784d0ff19cd\") " pod="ai-tenant-e2e-autores-fd0def/autores-model-fd0def-kserve-79f55c767-w5wkz"
Sep 11 03:54:26.236322 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:54:26.236287    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6cef66fb-3401-4c33-bfc3-f784d0ff19cd-dshm\") pod \"autores-model-fd0def-kserve-79f55c767-w5wkz\" (UID: \"6cef66fb-3401-4c33-bfc3-f784d0ff19cd\") " pod="ai-tenant-e2e-autores-fd0def/autores-model-fd0def-kserve-79f55c767-w5wkz"
Sep 11 03:54:26.236451 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:54:26.236371    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6cef66fb-3401-4c33-bfc3-f784d0ff19cd-tls-certs\") pod \"autores-model-fd0def-kserve-79f55c767-w5wkz\" (UID: \"6cef66fb-3401-4c33-bfc3-f784d0ff19cd\") " pod="ai-tenant-e2e-autores-fd0def/autores-model-fd0def-kserve-79f55c767-w5wkz"
Sep 11 03:54:26.241623 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:54:26.241596    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7tg7f\" (UniqueName: \"kubernetes.io/projected/6cef66fb-3401-4c33-bfc3-f784d0ff19cd-kube-api-access-7tg7f\") pod \"autores-model-fd0def-kserve-79f55c767-w5wkz\" (UID: \"6cef66fb-3401-4c33-bfc3-f784d0ff19cd\") " pod="ai-tenant-e2e-autores-fd0def/autores-model-fd0def-kserve-79f55c767-w5wkz"
Sep 11 03:54:26.299095 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:54:26.299059    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-autores-fd0def/autores-model-fd0def-kserve-79f55c767-w5wkz"
Sep 11 03:54:26.571647 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:54:26.571623    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-autores-fd0def/autores-model-fd0def-kserve-79f55c767-w5wkz"]
Sep 11 03:54:26.573918 ip-10-0-141-105 kubenswrapper[2579]: W0911 03:54:26.573889    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod6cef66fb_3401_4c33_bfc3_f784d0ff19cd.slice/crio-09a8e4b4a87822976a74d38d0f5714eac7dca7e7ea7dca09608c99044931c333 WatchSource:0}: Error finding container 09a8e4b4a87822976a74d38d0f5714eac7dca7e7ea7dca09608c99044931c333: Status 404 returned error can't find the container with id 09a8e4b4a87822976a74d38d0f5714eac7dca7e7ea7dca09608c99044931c333
Sep 11 03:54:26.993462 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:54:26.993427    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-autores-fd0def/autores-model-fd0def-kserve-79f55c767-w5wkz" event={"ID":"6cef66fb-3401-4c33-bfc3-f784d0ff19cd","Type":"ContainerStarted","Data":"ba3ceef38065136e2ac324f82883019d804e59c6f1571d7ee140b7f59423de7a"}
Sep 11 03:54:26.993865 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:54:26.993470    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-autores-fd0def/autores-model-fd0def-kserve-79f55c767-w5wkz" event={"ID":"6cef66fb-3401-4c33-bfc3-f784d0ff19cd","Type":"ContainerStarted","Data":"09a8e4b4a87822976a74d38d0f5714eac7dca7e7ea7dca09608c99044931c333"}
Sep 11 03:54:27.010028 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:54:27.009983    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-autores-fd0def/autores-model-fd0def-kserve-79f55c767-w5wkz" podStartSLOduration=2.00996875 podStartE2EDuration="2.00996875s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 03:54:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 03:54:27.009320765 +0000 UTC m=+844.664526333" watchObservedRunningTime="2026-09-11 03:54:27.00996875 +0000 UTC m=+844.665174318"
Sep 11 03:54:27.902138 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:54:27.902100    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-5z4cn"]
Sep 11 03:54:31.450966 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:54:31.450925    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b65989d0-xplk8"]
Sep 11 03:54:31.454640 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:54:31.454624    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b65989d0-xplk8"
Sep 11 03:54:31.457128 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:54:31.457110    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 11 03:54:31.457224 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:54:31.457204    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-95ckr\""
Sep 11 03:54:31.461106 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:54:31.461060    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b65989d0-xplk8"]
Sep 11 03:54:31.481912 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:54:31.481885    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e87dfde9-2a80-4266-84c3-f037355af3c3-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-b65989d0-xplk8\" (UID: \"e87dfde9-2a80-4266-84c3-f037355af3c3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b65989d0-xplk8"
Sep 11 03:54:31.582795 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:54:31.582761    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e87dfde9-2a80-4266-84c3-f037355af3c3-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-b65989d0-xplk8\" (UID: \"e87dfde9-2a80-4266-84c3-f037355af3c3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b65989d0-xplk8"
Sep 11 03:54:31.583432 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:54:31.583404    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e87dfde9-2a80-4266-84c3-f037355af3c3-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-b65989d0-xplk8\" (UID: \"e87dfde9-2a80-4266-84c3-f037355af3c3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b65989d0-xplk8"
Sep 11 03:54:31.766829 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:54:31.766751    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b65989d0-xplk8"
Sep 11 03:54:32.041642 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:54:32.041618    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b65989d0-xplk8"]
Sep 11 03:54:32.043734 ip-10-0-141-105 kubenswrapper[2579]: W0911 03:54:32.043706    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode87dfde9_2a80_4266_84c3_f037355af3c3.slice/crio-34b83f4e2d1557c0cdb9fc2ca7b76306e47587aa9d3de48f12e37a8fd0e6aa35 WatchSource:0}: Error finding container 34b83f4e2d1557c0cdb9fc2ca7b76306e47587aa9d3de48f12e37a8fd0e6aa35: Status 404 returned error can't find the container with id 34b83f4e2d1557c0cdb9fc2ca7b76306e47587aa9d3de48f12e37a8fd0e6aa35
Sep 11 03:54:33.027702 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:54:33.027620    2579 generic.go:356] "Generic (PLEG): container finished" podID="e87dfde9-2a80-4266-84c3-f037355af3c3" containerID="7942a49e797c5d92e65afa7c00cfc58c1399e3abb8e5a1a77fb4e82ffb3fab46" exitCode=0
Sep 11 03:54:33.028046 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:54:33.027707    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b65989d0-xplk8" event={"ID":"e87dfde9-2a80-4266-84c3-f037355af3c3","Type":"ContainerDied","Data":"7942a49e797c5d92e65afa7c00cfc58c1399e3abb8e5a1a77fb4e82ffb3fab46"}
Sep 11 03:54:33.028046 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:54:33.027768    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b65989d0-xplk8" event={"ID":"e87dfde9-2a80-4266-84c3-f037355af3c3","Type":"ContainerStarted","Data":"34b83f4e2d1557c0cdb9fc2ca7b76306e47587aa9d3de48f12e37a8fd0e6aa35"}
Sep 11 03:54:34.170189 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:54:34.170160    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b65989d0-xplk8"
Sep 11 03:54:34.206040 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:54:34.206010    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e87dfde9-2a80-4266-84c3-f037355af3c3-maas-api-service-ca\") pod \"e87dfde9-2a80-4266-84c3-f037355af3c3\" (UID: \"e87dfde9-2a80-4266-84c3-f037355af3c3\") "
Sep 11 03:54:34.206490 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:54:34.206452    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e87dfde9-2a80-4266-84c3-f037355af3c3-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "e87dfde9-2a80-4266-84c3-f037355af3c3" (UID: "e87dfde9-2a80-4266-84c3-f037355af3c3"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 03:54:34.307142 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:54:34.307014    2579 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e87dfde9-2a80-4266-84c3-f037355af3c3-maas-api-service-ca\") on node \"ip-10-0-141-105.ec2.internal\" DevicePath \"\""
Sep 11 03:54:35.038989 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:54:35.038965    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b65989d0-xplk8"
Sep 11 03:54:35.039184 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:54:35.038964    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b65989d0-xplk8" event={"ID":"e87dfde9-2a80-4266-84c3-f037355af3c3","Type":"ContainerDied","Data":"34b83f4e2d1557c0cdb9fc2ca7b76306e47587aa9d3de48f12e37a8fd0e6aa35"}
Sep 11 03:54:35.039184 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:54:35.039112    2579 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="34b83f4e2d1557c0cdb9fc2ca7b76306e47587aa9d3de48f12e37a8fd0e6aa35"
Sep 11 03:54:36.299826 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:54:36.299794    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-autores-fd0def/autores-model-fd0def-kserve-79f55c767-w5wkz"
Sep 11 03:54:36.300321 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:54:36.299844    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-autores-fd0def/autores-model-fd0def-kserve-79f55c767-w5wkz"
Sep 11 03:54:36.312654 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:54:36.312631    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-autores-fd0def/autores-model-fd0def-kserve-79f55c767-w5wkz"
Sep 11 03:54:37.060680 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:54:37.060651    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-autores-fd0def/autores-model-fd0def-kserve-79f55c767-w5wkz"
Sep 11 03:54:37.689573 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:54:37.689537    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b65989d0-xplk8"]
Sep 11 03:54:37.695801 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:54:37.695774    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b65989d0-xplk8"]
Sep 11 03:54:38.852146 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:54:38.852111    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e87dfde9-2a80-4266-84c3-f037355af3c3" path="/var/lib/kubelet/pods/e87dfde9-2a80-4266-84c3-f037355af3c3/volumes"
Sep 11 03:54:49.104757 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:54:49.104718    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-5z4cn"]
Sep 11 03:54:56.937338 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:54:56.937309    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-fd0def-z4cnv"]
Sep 11 03:54:56.937708 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:54:56.937690    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e87dfde9-2a80-4266-84c3-f037355af3c3" containerName="revoke-keys"
Sep 11 03:54:56.937708 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:54:56.937701    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="e87dfde9-2a80-4266-84c3-f037355af3c3" containerName="revoke-keys"
Sep 11 03:54:56.937783 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:54:56.937762    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="e87dfde9-2a80-4266-84c3-f037355af3c3" containerName="revoke-keys"
Sep 11 03:54:56.940533 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:54:56.940517    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-fd0def-z4cnv"
Sep 11 03:54:56.942935 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:54:56.942912    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 11 03:54:56.943344 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:54:56.943328    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-95ckr\""
Sep 11 03:54:56.948246 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:54:56.948218    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-fd0def-z4cnv"]
Sep 11 03:54:57.017514 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:54:57.017484    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/784691c5-2ce6-4822-9e51-1b0542a25e49-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-autores-fd0def-z4cnv\" (UID: \"784691c5-2ce6-4822-9e51-1b0542a25e49\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-fd0def-z4cnv"
Sep 11 03:54:57.118171 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:54:57.118144    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/784691c5-2ce6-4822-9e51-1b0542a25e49-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-autores-fd0def-z4cnv\" (UID: \"784691c5-2ce6-4822-9e51-1b0542a25e49\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-fd0def-z4cnv"
Sep 11 03:54:57.118740 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:54:57.118720    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/784691c5-2ce6-4822-9e51-1b0542a25e49-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-autores-fd0def-z4cnv\" (UID: \"784691c5-2ce6-4822-9e51-1b0542a25e49\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-fd0def-z4cnv"
Sep 11 03:54:57.197527 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:54:57.197453    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-a-4c62f1/test-model-4c62f1-kserve-69cb997789-bgf58"]
Sep 11 03:54:57.201007 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:54:57.200989    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-4c62f1/test-model-4c62f1-kserve-69cb997789-bgf58"
Sep 11 03:54:57.203653 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:54:57.203634    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-4c62f1\"/\"test-model-4c62f1-kserve-self-signed-certs\""
Sep 11 03:54:57.203756 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:54:57.203635    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-4c62f1\"/\"kube-root-ca.crt\""
Sep 11 03:54:57.203756 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:54:57.203635    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-4c62f1\"/\"default-dockercfg-p4hm5\""
Sep 11 03:54:57.203756 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:54:57.203643    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-4c62f1\"/\"openshift-service-ca.crt\""
Sep 11 03:54:57.211284 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:54:57.211263    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-4c62f1/test-model-4c62f1-kserve-69cb997789-bgf58"]
Sep 11 03:54:57.252738 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:54:57.252712    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-fd0def-z4cnv"
Sep 11 03:54:57.320140 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:54:57.319805    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6a03b2ee-3de9-4905-8cc9-b9c1e3a5bfb5-model-cache\") pod \"test-model-4c62f1-kserve-69cb997789-bgf58\" (UID: \"6a03b2ee-3de9-4905-8cc9-b9c1e3a5bfb5\") " pod="ai-tenant-e2e-inf-a-4c62f1/test-model-4c62f1-kserve-69cb997789-bgf58"
Sep 11 03:54:57.320140 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:54:57.319853    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6a03b2ee-3de9-4905-8cc9-b9c1e3a5bfb5-tmp-dir\") pod \"test-model-4c62f1-kserve-69cb997789-bgf58\" (UID: \"6a03b2ee-3de9-4905-8cc9-b9c1e3a5bfb5\") " pod="ai-tenant-e2e-inf-a-4c62f1/test-model-4c62f1-kserve-69cb997789-bgf58"
Sep 11 03:54:57.320140 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:54:57.319887    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6a03b2ee-3de9-4905-8cc9-b9c1e3a5bfb5-home\") pod \"test-model-4c62f1-kserve-69cb997789-bgf58\" (UID: \"6a03b2ee-3de9-4905-8cc9-b9c1e3a5bfb5\") " pod="ai-tenant-e2e-inf-a-4c62f1/test-model-4c62f1-kserve-69cb997789-bgf58"
Sep 11 03:54:57.320140 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:54:57.319912    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cxnt6\" (UniqueName: \"kubernetes.io/projected/6a03b2ee-3de9-4905-8cc9-b9c1e3a5bfb5-kube-api-access-cxnt6\") pod \"test-model-4c62f1-kserve-69cb997789-bgf58\" (UID: \"6a03b2ee-3de9-4905-8cc9-b9c1e3a5bfb5\") " pod="ai-tenant-e2e-inf-a-4c62f1/test-model-4c62f1-kserve-69cb997789-bgf58"
Sep 11 03:54:57.320140 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:54:57.319948    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6a03b2ee-3de9-4905-8cc9-b9c1e3a5bfb5-tls-certs\") pod \"test-model-4c62f1-kserve-69cb997789-bgf58\" (UID: \"6a03b2ee-3de9-4905-8cc9-b9c1e3a5bfb5\") " pod="ai-tenant-e2e-inf-a-4c62f1/test-model-4c62f1-kserve-69cb997789-bgf58"
Sep 11 03:54:57.320140 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:54:57.320003    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6a03b2ee-3de9-4905-8cc9-b9c1e3a5bfb5-dshm\") pod \"test-model-4c62f1-kserve-69cb997789-bgf58\" (UID: \"6a03b2ee-3de9-4905-8cc9-b9c1e3a5bfb5\") " pod="ai-tenant-e2e-inf-a-4c62f1/test-model-4c62f1-kserve-69cb997789-bgf58"
Sep 11 03:54:57.422145 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:54:57.422109    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6a03b2ee-3de9-4905-8cc9-b9c1e3a5bfb5-dshm\") pod \"test-model-4c62f1-kserve-69cb997789-bgf58\" (UID: \"6a03b2ee-3de9-4905-8cc9-b9c1e3a5bfb5\") " pod="ai-tenant-e2e-inf-a-4c62f1/test-model-4c62f1-kserve-69cb997789-bgf58"
Sep 11 03:54:57.422317 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:54:57.422215    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6a03b2ee-3de9-4905-8cc9-b9c1e3a5bfb5-model-cache\") pod \"test-model-4c62f1-kserve-69cb997789-bgf58\" (UID: \"6a03b2ee-3de9-4905-8cc9-b9c1e3a5bfb5\") " pod="ai-tenant-e2e-inf-a-4c62f1/test-model-4c62f1-kserve-69cb997789-bgf58"
Sep 11 03:54:57.422317 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:54:57.422237    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6a03b2ee-3de9-4905-8cc9-b9c1e3a5bfb5-tmp-dir\") pod \"test-model-4c62f1-kserve-69cb997789-bgf58\" (UID: \"6a03b2ee-3de9-4905-8cc9-b9c1e3a5bfb5\") " pod="ai-tenant-e2e-inf-a-4c62f1/test-model-4c62f1-kserve-69cb997789-bgf58"
Sep 11 03:54:57.422317 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:54:57.422259    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6a03b2ee-3de9-4905-8cc9-b9c1e3a5bfb5-home\") pod \"test-model-4c62f1-kserve-69cb997789-bgf58\" (UID: \"6a03b2ee-3de9-4905-8cc9-b9c1e3a5bfb5\") " pod="ai-tenant-e2e-inf-a-4c62f1/test-model-4c62f1-kserve-69cb997789-bgf58"
Sep 11 03:54:57.422317 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:54:57.422277    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cxnt6\" (UniqueName: \"kubernetes.io/projected/6a03b2ee-3de9-4905-8cc9-b9c1e3a5bfb5-kube-api-access-cxnt6\") pod \"test-model-4c62f1-kserve-69cb997789-bgf58\" (UID: \"6a03b2ee-3de9-4905-8cc9-b9c1e3a5bfb5\") " pod="ai-tenant-e2e-inf-a-4c62f1/test-model-4c62f1-kserve-69cb997789-bgf58"
Sep 11 03:54:57.422317 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:54:57.422311    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6a03b2ee-3de9-4905-8cc9-b9c1e3a5bfb5-tls-certs\") pod \"test-model-4c62f1-kserve-69cb997789-bgf58\" (UID: \"6a03b2ee-3de9-4905-8cc9-b9c1e3a5bfb5\") " pod="ai-tenant-e2e-inf-a-4c62f1/test-model-4c62f1-kserve-69cb997789-bgf58"
Sep 11 03:54:57.422643 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:54:57.422615    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6a03b2ee-3de9-4905-8cc9-b9c1e3a5bfb5-tmp-dir\") pod \"test-model-4c62f1-kserve-69cb997789-bgf58\" (UID: \"6a03b2ee-3de9-4905-8cc9-b9c1e3a5bfb5\") " pod="ai-tenant-e2e-inf-a-4c62f1/test-model-4c62f1-kserve-69cb997789-bgf58"
Sep 11 03:54:57.422706 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:54:57.422650    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6a03b2ee-3de9-4905-8cc9-b9c1e3a5bfb5-model-cache\") pod \"test-model-4c62f1-kserve-69cb997789-bgf58\" (UID: \"6a03b2ee-3de9-4905-8cc9-b9c1e3a5bfb5\") " pod="ai-tenant-e2e-inf-a-4c62f1/test-model-4c62f1-kserve-69cb997789-bgf58"
Sep 11 03:54:57.422760 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:54:57.422721    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6a03b2ee-3de9-4905-8cc9-b9c1e3a5bfb5-home\") pod \"test-model-4c62f1-kserve-69cb997789-bgf58\" (UID: \"6a03b2ee-3de9-4905-8cc9-b9c1e3a5bfb5\") " pod="ai-tenant-e2e-inf-a-4c62f1/test-model-4c62f1-kserve-69cb997789-bgf58"
Sep 11 03:54:57.424593 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:54:57.424568    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6a03b2ee-3de9-4905-8cc9-b9c1e3a5bfb5-dshm\") pod \"test-model-4c62f1-kserve-69cb997789-bgf58\" (UID: \"6a03b2ee-3de9-4905-8cc9-b9c1e3a5bfb5\") " pod="ai-tenant-e2e-inf-a-4c62f1/test-model-4c62f1-kserve-69cb997789-bgf58"
Sep 11 03:54:57.424898 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:54:57.424881    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6a03b2ee-3de9-4905-8cc9-b9c1e3a5bfb5-tls-certs\") pod \"test-model-4c62f1-kserve-69cb997789-bgf58\" (UID: \"6a03b2ee-3de9-4905-8cc9-b9c1e3a5bfb5\") " pod="ai-tenant-e2e-inf-a-4c62f1/test-model-4c62f1-kserve-69cb997789-bgf58"
Sep 11 03:54:57.429968 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:54:57.429943    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cxnt6\" (UniqueName: \"kubernetes.io/projected/6a03b2ee-3de9-4905-8cc9-b9c1e3a5bfb5-kube-api-access-cxnt6\") pod \"test-model-4c62f1-kserve-69cb997789-bgf58\" (UID: \"6a03b2ee-3de9-4905-8cc9-b9c1e3a5bfb5\") " pod="ai-tenant-e2e-inf-a-4c62f1/test-model-4c62f1-kserve-69cb997789-bgf58"
Sep 11 03:54:57.510831 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:54:57.510750    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-4c62f1/test-model-4c62f1-kserve-69cb997789-bgf58"
Sep 11 03:54:57.526212 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:54:57.526183    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-fd0def-z4cnv"]
Sep 11 03:54:57.528113 ip-10-0-141-105 kubenswrapper[2579]: W0911 03:54:57.528070    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod784691c5_2ce6_4822_9e51_1b0542a25e49.slice/crio-3e115ca5032b7214d19a0a90604e1c62496e2790730fce2960f9940f24724f86 WatchSource:0}: Error finding container 3e115ca5032b7214d19a0a90604e1c62496e2790730fce2960f9940f24724f86: Status 404 returned error can't find the container with id 3e115ca5032b7214d19a0a90604e1c62496e2790730fce2960f9940f24724f86
Sep 11 03:54:57.786984 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:54:57.786946    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-4c62f1/test-model-4c62f1-kserve-69cb997789-bgf58"]
Sep 11 03:54:57.790131 ip-10-0-141-105 kubenswrapper[2579]: W0911 03:54:57.790070    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod6a03b2ee_3de9_4905_8cc9_b9c1e3a5bfb5.slice/crio-843c7a2f6c06fbb85d219d1ac849e3c4d0f49c556ca76cbf05c81b3cd4276e82 WatchSource:0}: Error finding container 843c7a2f6c06fbb85d219d1ac849e3c4d0f49c556ca76cbf05c81b3cd4276e82: Status 404 returned error can't find the container with id 843c7a2f6c06fbb85d219d1ac849e3c4d0f49c556ca76cbf05c81b3cd4276e82
Sep 11 03:54:58.162366 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:54:58.162326    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-4c62f1/test-model-4c62f1-kserve-69cb997789-bgf58" event={"ID":"6a03b2ee-3de9-4905-8cc9-b9c1e3a5bfb5","Type":"ContainerStarted","Data":"add0cd12901d26602b22b7fb9ac23dde287a9c99610caf185129cbbff7c94196"}
Sep 11 03:54:58.162366 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:54:58.162375    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-4c62f1/test-model-4c62f1-kserve-69cb997789-bgf58" event={"ID":"6a03b2ee-3de9-4905-8cc9-b9c1e3a5bfb5","Type":"ContainerStarted","Data":"843c7a2f6c06fbb85d219d1ac849e3c4d0f49c556ca76cbf05c81b3cd4276e82"}
Sep 11 03:54:58.164046 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:54:58.164022    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-fd0def-z4cnv" event={"ID":"784691c5-2ce6-4822-9e51-1b0542a25e49","Type":"ContainerStarted","Data":"54b5f180ead5fed753e611b80819efc8a32b55193aa407ae50f723c608c507a5"}
Sep 11 03:54:58.164166 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:54:58.164066    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-fd0def-z4cnv" event={"ID":"784691c5-2ce6-4822-9e51-1b0542a25e49","Type":"ContainerStarted","Data":"3e115ca5032b7214d19a0a90604e1c62496e2790730fce2960f9940f24724f86"}
Sep 11 03:54:58.178963 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:54:58.178918    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-inf-a-4c62f1/test-model-4c62f1-kserve-69cb997789-bgf58" podStartSLOduration=1.178904026 podStartE2EDuration="1.178904026s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 03:54:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 03:54:58.177282622 +0000 UTC m=+875.832488203" watchObservedRunningTime="2026-09-11 03:54:58.178904026 +0000 UTC m=+875.834109605"
Sep 11 03:54:58.190132 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:54:58.190063    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-fd0def-z4cnv" podStartSLOduration=2.190049148 podStartE2EDuration="2.190049148s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 03:54:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 03:54:58.189517657 +0000 UTC m=+875.844723238" watchObservedRunningTime="2026-09-11 03:54:58.190049148 +0000 UTC m=+875.845254712"
Sep 11 03:54:59.170731 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:54:59.170692    2579 generic.go:356] "Generic (PLEG): container finished" podID="784691c5-2ce6-4822-9e51-1b0542a25e49" containerID="54b5f180ead5fed753e611b80819efc8a32b55193aa407ae50f723c608c507a5" exitCode=0
Sep 11 03:54:59.171218 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:54:59.170779    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-fd0def-z4cnv" event={"ID":"784691c5-2ce6-4822-9e51-1b0542a25e49","Type":"ContainerDied","Data":"54b5f180ead5fed753e611b80819efc8a32b55193aa407ae50f723c608c507a5"}
Sep 11 03:55:00.323094 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:55:00.323050    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-fd0def-z4cnv"
Sep 11 03:55:00.382025 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:55:00.381989    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-conflict-b-08754c30-kserve-7cbdbfdb97-ggm4h"]
Sep 11 03:55:00.450624 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:55:00.450525    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/784691c5-2ce6-4822-9e51-1b0542a25e49-maas-api-service-ca\") pod \"784691c5-2ce6-4822-9e51-1b0542a25e49\" (UID: \"784691c5-2ce6-4822-9e51-1b0542a25e49\") "
Sep 11 03:55:00.450866 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:55:00.450845    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/784691c5-2ce6-4822-9e51-1b0542a25e49-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "784691c5-2ce6-4822-9e51-1b0542a25e49" (UID: "784691c5-2ce6-4822-9e51-1b0542a25e49"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 03:55:00.552062 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:55:00.552031    2579 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/784691c5-2ce6-4822-9e51-1b0542a25e49-maas-api-service-ca\") on node \"ip-10-0-141-105.ec2.internal\" DevicePath \"\""
Sep 11 03:55:01.182368 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:55:01.182273    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-fd0def-z4cnv"
Sep 11 03:55:01.182368 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:55:01.182277    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-fd0def-z4cnv" event={"ID":"784691c5-2ce6-4822-9e51-1b0542a25e49","Type":"ContainerDied","Data":"3e115ca5032b7214d19a0a90604e1c62496e2790730fce2960f9940f24724f86"}
Sep 11 03:55:01.182567 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:55:01.182402    2579 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3e115ca5032b7214d19a0a90604e1c62496e2790730fce2960f9940f24724f86"
Sep 11 03:55:01.212011 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:55:01.211977    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-autores-fd0def/autores-model-fd0def-kserve-79f55c767-w5wkz"]
Sep 11 03:55:01.663145 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:55:01.663101    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-4ce893dc-hm8mb"]
Sep 11 03:55:01.663520 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:55:01.663508    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="784691c5-2ce6-4822-9e51-1b0542a25e49" containerName="revoke-keys"
Sep 11 03:55:01.663520 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:55:01.663520    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="784691c5-2ce6-4822-9e51-1b0542a25e49" containerName="revoke-keys"
Sep 11 03:55:01.663606 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:55:01.663576    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="784691c5-2ce6-4822-9e51-1b0542a25e49" containerName="revoke-keys"
Sep 11 03:55:01.666586 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:55:01.666567    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-4ce893dc-hm8mb"
Sep 11 03:55:01.668801 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:55:01.668776    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-95ckr\""
Sep 11 03:55:01.668801 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:55:01.668782    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 11 03:55:01.672977 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:55:01.672956    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-4ce893dc-hm8mb"]
Sep 11 03:55:01.761872 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:55:01.761843    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/48f0af88-1d41-4526-a48e-3b3a948c17f6-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-4ce893dc-hm8mb\" (UID: \"48f0af88-1d41-4526-a48e-3b3a948c17f6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-4ce893dc-hm8mb"
Sep 11 03:55:01.863543 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:55:01.863505    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/48f0af88-1d41-4526-a48e-3b3a948c17f6-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-4ce893dc-hm8mb\" (UID: \"48f0af88-1d41-4526-a48e-3b3a948c17f6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-4ce893dc-hm8mb"
Sep 11 03:55:01.864136 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:55:01.864116    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/48f0af88-1d41-4526-a48e-3b3a948c17f6-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-4ce893dc-hm8mb\" (UID: \"48f0af88-1d41-4526-a48e-3b3a948c17f6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-4ce893dc-hm8mb"
Sep 11 03:55:01.980113 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:55:01.980017    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-4ce893dc-hm8mb"
Sep 11 03:55:02.249897 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:55:02.249872    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-4ce893dc-hm8mb"]
Sep 11 03:55:02.252470 ip-10-0-141-105 kubenswrapper[2579]: W0911 03:55:02.252434    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod48f0af88_1d41_4526_a48e_3b3a948c17f6.slice/crio-8fd88f7423b8e25a31102044d747af84fb10ba54998ecbe92b124518ccd20938 WatchSource:0}: Error finding container 8fd88f7423b8e25a31102044d747af84fb10ba54998ecbe92b124518ccd20938: Status 404 returned error can't find the container with id 8fd88f7423b8e25a31102044d747af84fb10ba54998ecbe92b124518ccd20938
Sep 11 03:55:03.199842 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:55:03.199756    2579 generic.go:356] "Generic (PLEG): container finished" podID="48f0af88-1d41-4526-a48e-3b3a948c17f6" containerID="7787147ccf33733827ca6905796604c5d250a61c3ec6296e2009952cd50ae0cb" exitCode=0
Sep 11 03:55:03.199842 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:55:03.199828    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-4ce893dc-hm8mb" event={"ID":"48f0af88-1d41-4526-a48e-3b3a948c17f6","Type":"ContainerDied","Data":"7787147ccf33733827ca6905796604c5d250a61c3ec6296e2009952cd50ae0cb"}
Sep 11 03:55:03.200387 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:55:03.199875    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-4ce893dc-hm8mb" event={"ID":"48f0af88-1d41-4526-a48e-3b3a948c17f6","Type":"ContainerStarted","Data":"8fd88f7423b8e25a31102044d747af84fb10ba54998ecbe92b124518ccd20938"}
Sep 11 03:55:04.341889 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:55:04.341863    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-4ce893dc-hm8mb"
Sep 11 03:55:04.489327 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:55:04.489229    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/48f0af88-1d41-4526-a48e-3b3a948c17f6-maas-api-service-ca\") pod \"48f0af88-1d41-4526-a48e-3b3a948c17f6\" (UID: \"48f0af88-1d41-4526-a48e-3b3a948c17f6\") "
Sep 11 03:55:04.489540 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:55:04.489516    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/48f0af88-1d41-4526-a48e-3b3a948c17f6-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "48f0af88-1d41-4526-a48e-3b3a948c17f6" (UID: "48f0af88-1d41-4526-a48e-3b3a948c17f6"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 03:55:04.590100 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:55:04.590058    2579 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/48f0af88-1d41-4526-a48e-3b3a948c17f6-maas-api-service-ca\") on node \"ip-10-0-141-105.ec2.internal\" DevicePath \"\""
Sep 11 03:55:05.211611 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:55:05.211534    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-4ce893dc-hm8mb"
Sep 11 03:55:05.211611 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:55:05.211558    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-4ce893dc-hm8mb" event={"ID":"48f0af88-1d41-4526-a48e-3b3a948c17f6","Type":"ContainerDied","Data":"8fd88f7423b8e25a31102044d747af84fb10ba54998ecbe92b124518ccd20938"}
Sep 11 03:55:05.211611 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:55:05.211602    2579 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8fd88f7423b8e25a31102044d747af84fb10ba54998ecbe92b124518ccd20938"
Sep 11 03:55:07.511675 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:55:07.511641    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-4c62f1/test-model-4c62f1-kserve-69cb997789-bgf58"
Sep 11 03:55:07.512129 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:55:07.511691    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-a-4c62f1/test-model-4c62f1-kserve-69cb997789-bgf58"
Sep 11 03:55:07.524293 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:55:07.524269    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-a-4c62f1/test-model-4c62f1-kserve-69cb997789-bgf58"
Sep 11 03:55:07.783711 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:55:07.783626    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-fd0def-bb56f6895-nn5m7"]
Sep 11 03:55:07.783924 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:55:07.783897    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-autores-fd0def-bb56f6895-nn5m7" podUID="b1c46d01-52dc-4ef1-83a7-973ac377ef3d" containerName="payload-processing" containerID="cri-o://9734da4c21acf74093b2a4c6551178ffae02bb5a2cc7102e6d36ded6db1612e1" gracePeriod=30
Sep 11 03:55:07.793438 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:55:07.791683    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-fd0def-5d59887bb-xb8gk"]
Sep 11 03:55:07.793438 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:55:07.791914    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-autores-fd0def-5d59887bb-xb8gk" podUID="a173f5c9-2082-4276-99fc-bc121d8c8be3" containerName="payload-pre-processing" containerID="cri-o://161fff0a23a552b44db59c24de0138c1ecd8ff069dfbd3804edd586289c39590" gracePeriod=30
Sep 11 03:55:07.929923 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:55:07.929495    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-fd0def-z4cnv"]
Sep 11 03:55:07.932154 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:55:07.932129    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-fd0def-z4cnv"]
Sep 11 03:55:08.072111 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:55:08.072045    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-autores-fd0def-bb56f6895-nn5m7"
Sep 11 03:55:08.075215 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:55:08.075191    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-autores-fd0def-5d59887bb-xb8gk"
Sep 11 03:55:08.224519 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:55:08.224490    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a173f5c9-2082-4276-99fc-bc121d8c8be3-plugins-config-volume\") pod \"a173f5c9-2082-4276-99fc-bc121d8c8be3\" (UID: \"a173f5c9-2082-4276-99fc-bc121d8c8be3\") "
Sep 11 03:55:08.224708 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:55:08.224538    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b1c46d01-52dc-4ef1-83a7-973ac377ef3d-plugins-config-volume\") pod \"b1c46d01-52dc-4ef1-83a7-973ac377ef3d\" (UID: \"b1c46d01-52dc-4ef1-83a7-973ac377ef3d\") "
Sep 11 03:55:08.224708 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:55:08.224559    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gsw8r\" (UniqueName: \"kubernetes.io/projected/a173f5c9-2082-4276-99fc-bc121d8c8be3-kube-api-access-gsw8r\") pod \"a173f5c9-2082-4276-99fc-bc121d8c8be3\" (UID: \"a173f5c9-2082-4276-99fc-bc121d8c8be3\") "
Sep 11 03:55:08.224708 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:55:08.224660    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-c648f\" (UniqueName: \"kubernetes.io/projected/b1c46d01-52dc-4ef1-83a7-973ac377ef3d-kube-api-access-c648f\") pod \"b1c46d01-52dc-4ef1-83a7-973ac377ef3d\" (UID: \"b1c46d01-52dc-4ef1-83a7-973ac377ef3d\") "
Sep 11 03:55:08.224970 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:55:08.224933    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b1c46d01-52dc-4ef1-83a7-973ac377ef3d-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "b1c46d01-52dc-4ef1-83a7-973ac377ef3d" (UID: "b1c46d01-52dc-4ef1-83a7-973ac377ef3d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 03:55:08.224970 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:55:08.224954    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a173f5c9-2082-4276-99fc-bc121d8c8be3-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "a173f5c9-2082-4276-99fc-bc121d8c8be3" (UID: "a173f5c9-2082-4276-99fc-bc121d8c8be3"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 03:55:08.226988 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:55:08.226960    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a173f5c9-2082-4276-99fc-bc121d8c8be3-kube-api-access-gsw8r" (OuterVolumeSpecName: "kube-api-access-gsw8r") pod "a173f5c9-2082-4276-99fc-bc121d8c8be3" (UID: "a173f5c9-2082-4276-99fc-bc121d8c8be3"). InnerVolumeSpecName "kube-api-access-gsw8r". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 03:55:08.227137 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:55:08.227108    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b1c46d01-52dc-4ef1-83a7-973ac377ef3d-kube-api-access-c648f" (OuterVolumeSpecName: "kube-api-access-c648f") pod "b1c46d01-52dc-4ef1-83a7-973ac377ef3d" (UID: "b1c46d01-52dc-4ef1-83a7-973ac377ef3d"). InnerVolumeSpecName "kube-api-access-c648f". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 03:55:08.229863 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:55:08.229843    2579 generic.go:356] "Generic (PLEG): container finished" podID="b1c46d01-52dc-4ef1-83a7-973ac377ef3d" containerID="9734da4c21acf74093b2a4c6551178ffae02bb5a2cc7102e6d36ded6db1612e1" exitCode=0
Sep 11 03:55:08.229971 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:55:08.229893    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-autores-fd0def-bb56f6895-nn5m7"
Sep 11 03:55:08.229971 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:55:08.229907    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-autores-fd0def-bb56f6895-nn5m7" event={"ID":"b1c46d01-52dc-4ef1-83a7-973ac377ef3d","Type":"ContainerDied","Data":"9734da4c21acf74093b2a4c6551178ffae02bb5a2cc7102e6d36ded6db1612e1"}
Sep 11 03:55:08.229971 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:55:08.229964    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-autores-fd0def-bb56f6895-nn5m7" event={"ID":"b1c46d01-52dc-4ef1-83a7-973ac377ef3d","Type":"ContainerDied","Data":"d7d61a01c032614dce646050580e58966c769021aa4c45f1ba66fd66e8707efd"}
Sep 11 03:55:08.230124 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:55:08.229988    2579 scope.go:117] "RemoveContainer" containerID="9734da4c21acf74093b2a4c6551178ffae02bb5a2cc7102e6d36ded6db1612e1"
Sep 11 03:55:08.231245 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:55:08.231226    2579 generic.go:356] "Generic (PLEG): container finished" podID="a173f5c9-2082-4276-99fc-bc121d8c8be3" containerID="161fff0a23a552b44db59c24de0138c1ecd8ff069dfbd3804edd586289c39590" exitCode=0
Sep 11 03:55:08.231324 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:55:08.231277    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-autores-fd0def-5d59887bb-xb8gk"
Sep 11 03:55:08.231324 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:55:08.231277    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-autores-fd0def-5d59887bb-xb8gk" event={"ID":"a173f5c9-2082-4276-99fc-bc121d8c8be3","Type":"ContainerDied","Data":"161fff0a23a552b44db59c24de0138c1ecd8ff069dfbd3804edd586289c39590"}
Sep 11 03:55:08.231406 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:55:08.231336    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-autores-fd0def-5d59887bb-xb8gk" event={"ID":"a173f5c9-2082-4276-99fc-bc121d8c8be3","Type":"ContainerDied","Data":"043bec5b93e6ce94ea328e168f7c37f5416f530bdc72583c602e170e2e6d26e0"}
Sep 11 03:55:08.248222 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:55:08.248196    2579 scope.go:117] "RemoveContainer" containerID="9734da4c21acf74093b2a4c6551178ffae02bb5a2cc7102e6d36ded6db1612e1"
Sep 11 03:55:08.248368 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:55:08.248291    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-4c62f1/test-model-4c62f1-kserve-69cb997789-bgf58"
Sep 11 03:55:08.248531 ip-10-0-141-105 kubenswrapper[2579]: E0911 03:55:08.248511    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9734da4c21acf74093b2a4c6551178ffae02bb5a2cc7102e6d36ded6db1612e1\": container with ID starting with 9734da4c21acf74093b2a4c6551178ffae02bb5a2cc7102e6d36ded6db1612e1 not found: ID does not exist" containerID="9734da4c21acf74093b2a4c6551178ffae02bb5a2cc7102e6d36ded6db1612e1"
Sep 11 03:55:08.248606 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:55:08.248538    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9734da4c21acf74093b2a4c6551178ffae02bb5a2cc7102e6d36ded6db1612e1"} err="failed to get container status \"9734da4c21acf74093b2a4c6551178ffae02bb5a2cc7102e6d36ded6db1612e1\": rpc error: code = NotFound desc = could not find container \"9734da4c21acf74093b2a4c6551178ffae02bb5a2cc7102e6d36ded6db1612e1\": container with ID starting with 9734da4c21acf74093b2a4c6551178ffae02bb5a2cc7102e6d36ded6db1612e1 not found: ID does not exist"
Sep 11 03:55:08.248606 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:55:08.248556    2579 scope.go:117] "RemoveContainer" containerID="161fff0a23a552b44db59c24de0138c1ecd8ff069dfbd3804edd586289c39590"
Sep 11 03:55:08.257310 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:55:08.257293    2579 scope.go:117] "RemoveContainer" containerID="161fff0a23a552b44db59c24de0138c1ecd8ff069dfbd3804edd586289c39590"
Sep 11 03:55:08.257630 ip-10-0-141-105 kubenswrapper[2579]: E0911 03:55:08.257613    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"161fff0a23a552b44db59c24de0138c1ecd8ff069dfbd3804edd586289c39590\": container with ID starting with 161fff0a23a552b44db59c24de0138c1ecd8ff069dfbd3804edd586289c39590 not found: ID does not exist" containerID="161fff0a23a552b44db59c24de0138c1ecd8ff069dfbd3804edd586289c39590"
Sep 11 03:55:08.257694 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:55:08.257637    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"161fff0a23a552b44db59c24de0138c1ecd8ff069dfbd3804edd586289c39590"} err="failed to get container status \"161fff0a23a552b44db59c24de0138c1ecd8ff069dfbd3804edd586289c39590\": rpc error: code = NotFound desc = could not find container \"161fff0a23a552b44db59c24de0138c1ecd8ff069dfbd3804edd586289c39590\": container with ID starting with 161fff0a23a552b44db59c24de0138c1ecd8ff069dfbd3804edd586289c39590 not found: ID does not exist"
Sep 11 03:55:08.258818 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:55:08.258800    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-fd0def-bb56f6895-nn5m7"]
Sep 11 03:55:08.261785 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:55:08.261766    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-fd0def-bb56f6895-nn5m7"]
Sep 11 03:55:08.289156 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:55:08.289127    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-fd0def-5d59887bb-xb8gk"]
Sep 11 03:55:08.293200 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:55:08.293176    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-fd0def-5d59887bb-xb8gk"]
Sep 11 03:55:08.326663 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:55:08.326562    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c648f\" (UniqueName: \"kubernetes.io/projected/b1c46d01-52dc-4ef1-83a7-973ac377ef3d-kube-api-access-c648f\") on node \"ip-10-0-141-105.ec2.internal\" DevicePath \"\""
Sep 11 03:55:08.326663 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:55:08.326594    2579 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a173f5c9-2082-4276-99fc-bc121d8c8be3-plugins-config-volume\") on node \"ip-10-0-141-105.ec2.internal\" DevicePath \"\""
Sep 11 03:55:08.326663 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:55:08.326603    2579 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b1c46d01-52dc-4ef1-83a7-973ac377ef3d-plugins-config-volume\") on node \"ip-10-0-141-105.ec2.internal\" DevicePath \"\""
Sep 11 03:55:08.326663 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:55:08.326615    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gsw8r\" (UniqueName: \"kubernetes.io/projected/a173f5c9-2082-4276-99fc-bc121d8c8be3-kube-api-access-gsw8r\") on node \"ip-10-0-141-105.ec2.internal\" DevicePath \"\""
Sep 11 03:55:08.852147 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:55:08.852111    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="784691c5-2ce6-4822-9e51-1b0542a25e49" path="/var/lib/kubelet/pods/784691c5-2ce6-4822-9e51-1b0542a25e49/volumes"
Sep 11 03:55:08.852564 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:55:08.852493    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a173f5c9-2082-4276-99fc-bc121d8c8be3" path="/var/lib/kubelet/pods/a173f5c9-2082-4276-99fc-bc121d8c8be3/volumes"
Sep 11 03:55:08.852804 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:55:08.852791    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b1c46d01-52dc-4ef1-83a7-973ac377ef3d" path="/var/lib/kubelet/pods/b1c46d01-52dc-4ef1-83a7-973ac377ef3d/volumes"
Sep 11 03:55:12.479905 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:55:12.479873    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-4ce893dc-hm8mb"]
Sep 11 03:55:12.484939 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:55:12.484915    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-4ce893dc-hm8mb"]
Sep 11 03:55:12.858738 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:55:12.858703    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="48f0af88-1d41-4526-a48e-3b3a948c17f6" path="/var/lib/kubelet/pods/48f0af88-1d41-4526-a48e-3b3a948c17f6/volumes"
Sep 11 03:55:15.403856 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:55:15.403774    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="llm/e2e-conflict-b-08754c30-kserve-7cbdbfdb97-ggm4h" podUID="47052321-c2e8-489e-8f54-42b49609fa66" containerName="main" containerID="cri-o://3b2ec372875d89975520fdfb5d6793c4298a42949ccfb0b8fcdb92268cbd7e77" gracePeriod=45
Sep 11 03:55:15.667639 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:55:15.667576    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-conflict-b-08754c30-kserve-7cbdbfdb97-ggm4h"
Sep 11 03:55:15.699038 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:55:15.699000    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/47052321-c2e8-489e-8f54-42b49609fa66-dshm\") pod \"47052321-c2e8-489e-8f54-42b49609fa66\" (UID: \"47052321-c2e8-489e-8f54-42b49609fa66\") "
Sep 11 03:55:15.699228 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:55:15.699049    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/47052321-c2e8-489e-8f54-42b49609fa66-model-cache\") pod \"47052321-c2e8-489e-8f54-42b49609fa66\" (UID: \"47052321-c2e8-489e-8f54-42b49609fa66\") "
Sep 11 03:55:15.699228 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:55:15.699116    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/47052321-c2e8-489e-8f54-42b49609fa66-tmp-dir\") pod \"47052321-c2e8-489e-8f54-42b49609fa66\" (UID: \"47052321-c2e8-489e-8f54-42b49609fa66\") "
Sep 11 03:55:15.699228 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:55:15.699140    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/47052321-c2e8-489e-8f54-42b49609fa66-tls-certs\") pod \"47052321-c2e8-489e-8f54-42b49609fa66\" (UID: \"47052321-c2e8-489e-8f54-42b49609fa66\") "
Sep 11 03:55:15.699228 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:55:15.699166    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mttp6\" (UniqueName: \"kubernetes.io/projected/47052321-c2e8-489e-8f54-42b49609fa66-kube-api-access-mttp6\") pod \"47052321-c2e8-489e-8f54-42b49609fa66\" (UID: \"47052321-c2e8-489e-8f54-42b49609fa66\") "
Sep 11 03:55:15.699468 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:55:15.699255    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/47052321-c2e8-489e-8f54-42b49609fa66-home\") pod \"47052321-c2e8-489e-8f54-42b49609fa66\" (UID: \"47052321-c2e8-489e-8f54-42b49609fa66\") "
Sep 11 03:55:15.699468 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:55:15.699298    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/47052321-c2e8-489e-8f54-42b49609fa66-model-cache" (OuterVolumeSpecName: "model-cache") pod "47052321-c2e8-489e-8f54-42b49609fa66" (UID: "47052321-c2e8-489e-8f54-42b49609fa66"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 03:55:15.699468 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:55:15.699377    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/47052321-c2e8-489e-8f54-42b49609fa66-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "47052321-c2e8-489e-8f54-42b49609fa66" (UID: "47052321-c2e8-489e-8f54-42b49609fa66"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 03:55:15.699635 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:55:15.699514    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/47052321-c2e8-489e-8f54-42b49609fa66-home" (OuterVolumeSpecName: "home") pod "47052321-c2e8-489e-8f54-42b49609fa66" (UID: "47052321-c2e8-489e-8f54-42b49609fa66"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 03:55:15.699635 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:55:15.699545    2579 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/47052321-c2e8-489e-8f54-42b49609fa66-model-cache\") on node \"ip-10-0-141-105.ec2.internal\" DevicePath \"\""
Sep 11 03:55:15.699635 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:55:15.699560    2579 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/47052321-c2e8-489e-8f54-42b49609fa66-tmp-dir\") on node \"ip-10-0-141-105.ec2.internal\" DevicePath \"\""
Sep 11 03:55:15.701577 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:55:15.701552    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/47052321-c2e8-489e-8f54-42b49609fa66-kube-api-access-mttp6" (OuterVolumeSpecName: "kube-api-access-mttp6") pod "47052321-c2e8-489e-8f54-42b49609fa66" (UID: "47052321-c2e8-489e-8f54-42b49609fa66"). InnerVolumeSpecName "kube-api-access-mttp6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 03:55:15.701838 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:55:15.701790    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/47052321-c2e8-489e-8f54-42b49609fa66-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "47052321-c2e8-489e-8f54-42b49609fa66" (UID: "47052321-c2e8-489e-8f54-42b49609fa66"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 03:55:15.701838 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:55:15.701819    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/47052321-c2e8-489e-8f54-42b49609fa66-dshm" (OuterVolumeSpecName: "dshm") pod "47052321-c2e8-489e-8f54-42b49609fa66" (UID: "47052321-c2e8-489e-8f54-42b49609fa66"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 03:55:15.800003 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:55:15.799973    2579 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/47052321-c2e8-489e-8f54-42b49609fa66-dshm\") on node \"ip-10-0-141-105.ec2.internal\" DevicePath \"\""
Sep 11 03:55:15.800185 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:55:15.800015    2579 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/47052321-c2e8-489e-8f54-42b49609fa66-tls-certs\") on node \"ip-10-0-141-105.ec2.internal\" DevicePath \"\""
Sep 11 03:55:15.800185 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:55:15.800027    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mttp6\" (UniqueName: \"kubernetes.io/projected/47052321-c2e8-489e-8f54-42b49609fa66-kube-api-access-mttp6\") on node \"ip-10-0-141-105.ec2.internal\" DevicePath \"\""
Sep 11 03:55:15.800185 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:55:15.800036    2579 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/47052321-c2e8-489e-8f54-42b49609fa66-home\") on node \"ip-10-0-141-105.ec2.internal\" DevicePath \"\""
Sep 11 03:55:16.099359 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:55:16.099273    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-5z4cn"]
Sep 11 03:55:16.232847 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:55:16.232810    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-autores-fd0def/autores-model-fd0def-kserve-79f55c767-w5wkz" podUID="6cef66fb-3401-4c33-bfc3-f784d0ff19cd" containerName="main" containerID="cri-o://ba3ceef38065136e2ac324f82883019d804e59c6f1571d7ee140b7f59423de7a" gracePeriod=45
Sep 11 03:55:16.273767 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:55:16.273739    2579 generic.go:356] "Generic (PLEG): container finished" podID="47052321-c2e8-489e-8f54-42b49609fa66" containerID="3b2ec372875d89975520fdfb5d6793c4298a42949ccfb0b8fcdb92268cbd7e77" exitCode=0
Sep 11 03:55:16.273910 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:55:16.273795    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-conflict-b-08754c30-kserve-7cbdbfdb97-ggm4h"
Sep 11 03:55:16.273910 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:55:16.273845    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-conflict-b-08754c30-kserve-7cbdbfdb97-ggm4h" event={"ID":"47052321-c2e8-489e-8f54-42b49609fa66","Type":"ContainerDied","Data":"3b2ec372875d89975520fdfb5d6793c4298a42949ccfb0b8fcdb92268cbd7e77"}
Sep 11 03:55:16.273910 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:55:16.273892    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-conflict-b-08754c30-kserve-7cbdbfdb97-ggm4h" event={"ID":"47052321-c2e8-489e-8f54-42b49609fa66","Type":"ContainerDied","Data":"213518e8b5886627ddaa5f1a9c501382823601106b34ed8dfe7586e6f7221d5b"}
Sep 11 03:55:16.273910 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:55:16.273909    2579 scope.go:117] "RemoveContainer" containerID="3b2ec372875d89975520fdfb5d6793c4298a42949ccfb0b8fcdb92268cbd7e77"
Sep 11 03:55:16.301832 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:55:16.301805    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-conflict-b-08754c30-kserve-7cbdbfdb97-ggm4h"]
Sep 11 03:55:16.303407 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:55:16.303387    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["llm/e2e-conflict-b-08754c30-kserve-7cbdbfdb97-ggm4h"]
Sep 11 03:55:16.344872 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:55:16.344851    2579 scope.go:117] "RemoveContainer" containerID="3b2ec372875d89975520fdfb5d6793c4298a42949ccfb0b8fcdb92268cbd7e77"
Sep 11 03:55:16.345254 ip-10-0-141-105 kubenswrapper[2579]: E0911 03:55:16.345230    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3b2ec372875d89975520fdfb5d6793c4298a42949ccfb0b8fcdb92268cbd7e77\": container with ID starting with 3b2ec372875d89975520fdfb5d6793c4298a42949ccfb0b8fcdb92268cbd7e77 not found: ID does not exist" containerID="3b2ec372875d89975520fdfb5d6793c4298a42949ccfb0b8fcdb92268cbd7e77"
Sep 11 03:55:16.345333 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:55:16.345262    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3b2ec372875d89975520fdfb5d6793c4298a42949ccfb0b8fcdb92268cbd7e77"} err="failed to get container status \"3b2ec372875d89975520fdfb5d6793c4298a42949ccfb0b8fcdb92268cbd7e77\": rpc error: code = NotFound desc = could not find container \"3b2ec372875d89975520fdfb5d6793c4298a42949ccfb0b8fcdb92268cbd7e77\": container with ID starting with 3b2ec372875d89975520fdfb5d6793c4298a42949ccfb0b8fcdb92268cbd7e77 not found: ID does not exist"
Sep 11 03:55:16.485658 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:55:16.485633    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-autores-fd0def/autores-model-fd0def-kserve-79f55c767-w5wkz"
Sep 11 03:55:16.506674 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:55:16.506649    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7tg7f\" (UniqueName: \"kubernetes.io/projected/6cef66fb-3401-4c33-bfc3-f784d0ff19cd-kube-api-access-7tg7f\") pod \"6cef66fb-3401-4c33-bfc3-f784d0ff19cd\" (UID: \"6cef66fb-3401-4c33-bfc3-f784d0ff19cd\") "
Sep 11 03:55:16.506838 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:55:16.506739    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6cef66fb-3401-4c33-bfc3-f784d0ff19cd-tmp-dir\") pod \"6cef66fb-3401-4c33-bfc3-f784d0ff19cd\" (UID: \"6cef66fb-3401-4c33-bfc3-f784d0ff19cd\") "
Sep 11 03:55:16.506838 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:55:16.506766    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6cef66fb-3401-4c33-bfc3-f784d0ff19cd-tls-certs\") pod \"6cef66fb-3401-4c33-bfc3-f784d0ff19cd\" (UID: \"6cef66fb-3401-4c33-bfc3-f784d0ff19cd\") "
Sep 11 03:55:16.506838 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:55:16.506803    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6cef66fb-3401-4c33-bfc3-f784d0ff19cd-dshm\") pod \"6cef66fb-3401-4c33-bfc3-f784d0ff19cd\" (UID: \"6cef66fb-3401-4c33-bfc3-f784d0ff19cd\") "
Sep 11 03:55:16.506998 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:55:16.506840    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6cef66fb-3401-4c33-bfc3-f784d0ff19cd-home\") pod \"6cef66fb-3401-4c33-bfc3-f784d0ff19cd\" (UID: \"6cef66fb-3401-4c33-bfc3-f784d0ff19cd\") "
Sep 11 03:55:16.506998 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:55:16.506902    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6cef66fb-3401-4c33-bfc3-f784d0ff19cd-model-cache\") pod \"6cef66fb-3401-4c33-bfc3-f784d0ff19cd\" (UID: \"6cef66fb-3401-4c33-bfc3-f784d0ff19cd\") "
Sep 11 03:55:16.507114 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:55:16.507030    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6cef66fb-3401-4c33-bfc3-f784d0ff19cd-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "6cef66fb-3401-4c33-bfc3-f784d0ff19cd" (UID: "6cef66fb-3401-4c33-bfc3-f784d0ff19cd"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 03:55:16.507317 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:55:16.507271    2579 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6cef66fb-3401-4c33-bfc3-f784d0ff19cd-tmp-dir\") on node \"ip-10-0-141-105.ec2.internal\" DevicePath \"\""
Sep 11 03:55:16.507317 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:55:16.507297    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6cef66fb-3401-4c33-bfc3-f784d0ff19cd-home" (OuterVolumeSpecName: "home") pod "6cef66fb-3401-4c33-bfc3-f784d0ff19cd" (UID: "6cef66fb-3401-4c33-bfc3-f784d0ff19cd"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 03:55:16.507472 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:55:16.507400    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6cef66fb-3401-4c33-bfc3-f784d0ff19cd-model-cache" (OuterVolumeSpecName: "model-cache") pod "6cef66fb-3401-4c33-bfc3-f784d0ff19cd" (UID: "6cef66fb-3401-4c33-bfc3-f784d0ff19cd"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 03:55:16.509235 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:55:16.509207    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6cef66fb-3401-4c33-bfc3-f784d0ff19cd-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "6cef66fb-3401-4c33-bfc3-f784d0ff19cd" (UID: "6cef66fb-3401-4c33-bfc3-f784d0ff19cd"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 03:55:16.509348 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:55:16.509329    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6cef66fb-3401-4c33-bfc3-f784d0ff19cd-dshm" (OuterVolumeSpecName: "dshm") pod "6cef66fb-3401-4c33-bfc3-f784d0ff19cd" (UID: "6cef66fb-3401-4c33-bfc3-f784d0ff19cd"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 03:55:16.509444 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:55:16.509425    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6cef66fb-3401-4c33-bfc3-f784d0ff19cd-kube-api-access-7tg7f" (OuterVolumeSpecName: "kube-api-access-7tg7f") pod "6cef66fb-3401-4c33-bfc3-f784d0ff19cd" (UID: "6cef66fb-3401-4c33-bfc3-f784d0ff19cd"). InnerVolumeSpecName "kube-api-access-7tg7f". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 03:55:16.608403 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:55:16.608361    2579 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6cef66fb-3401-4c33-bfc3-f784d0ff19cd-tls-certs\") on node \"ip-10-0-141-105.ec2.internal\" DevicePath \"\""
Sep 11 03:55:16.608403 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:55:16.608394    2579 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6cef66fb-3401-4c33-bfc3-f784d0ff19cd-dshm\") on node \"ip-10-0-141-105.ec2.internal\" DevicePath \"\""
Sep 11 03:55:16.608403 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:55:16.608405    2579 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6cef66fb-3401-4c33-bfc3-f784d0ff19cd-home\") on node \"ip-10-0-141-105.ec2.internal\" DevicePath \"\""
Sep 11 03:55:16.608403 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:55:16.608415    2579 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6cef66fb-3401-4c33-bfc3-f784d0ff19cd-model-cache\") on node \"ip-10-0-141-105.ec2.internal\" DevicePath \"\""
Sep 11 03:55:16.608674 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:55:16.608427    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7tg7f\" (UniqueName: \"kubernetes.io/projected/6cef66fb-3401-4c33-bfc3-f784d0ff19cd-kube-api-access-7tg7f\") on node \"ip-10-0-141-105.ec2.internal\" DevicePath \"\""
Sep 11 03:55:16.852321 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:55:16.852233    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="47052321-c2e8-489e-8f54-42b49609fa66" path="/var/lib/kubelet/pods/47052321-c2e8-489e-8f54-42b49609fa66/volumes"
Sep 11 03:55:17.281562 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:55:17.281525    2579 generic.go:356] "Generic (PLEG): container finished" podID="6cef66fb-3401-4c33-bfc3-f784d0ff19cd" containerID="ba3ceef38065136e2ac324f82883019d804e59c6f1571d7ee140b7f59423de7a" exitCode=0
Sep 11 03:55:17.281750 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:55:17.281569    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-autores-fd0def/autores-model-fd0def-kserve-79f55c767-w5wkz"
Sep 11 03:55:17.281750 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:55:17.281629    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-autores-fd0def/autores-model-fd0def-kserve-79f55c767-w5wkz" event={"ID":"6cef66fb-3401-4c33-bfc3-f784d0ff19cd","Type":"ContainerDied","Data":"ba3ceef38065136e2ac324f82883019d804e59c6f1571d7ee140b7f59423de7a"}
Sep 11 03:55:17.281750 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:55:17.281681    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-autores-fd0def/autores-model-fd0def-kserve-79f55c767-w5wkz" event={"ID":"6cef66fb-3401-4c33-bfc3-f784d0ff19cd","Type":"ContainerDied","Data":"09a8e4b4a87822976a74d38d0f5714eac7dca7e7ea7dca09608c99044931c333"}
Sep 11 03:55:17.281750 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:55:17.281700    2579 scope.go:117] "RemoveContainer" containerID="ba3ceef38065136e2ac324f82883019d804e59c6f1571d7ee140b7f59423de7a"
Sep 11 03:55:17.290883 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:55:17.290867    2579 scope.go:117] "RemoveContainer" containerID="ba3ceef38065136e2ac324f82883019d804e59c6f1571d7ee140b7f59423de7a"
Sep 11 03:55:17.291191 ip-10-0-141-105 kubenswrapper[2579]: E0911 03:55:17.291165    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ba3ceef38065136e2ac324f82883019d804e59c6f1571d7ee140b7f59423de7a\": container with ID starting with ba3ceef38065136e2ac324f82883019d804e59c6f1571d7ee140b7f59423de7a not found: ID does not exist" containerID="ba3ceef38065136e2ac324f82883019d804e59c6f1571d7ee140b7f59423de7a"
Sep 11 03:55:17.291252 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:55:17.291199    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ba3ceef38065136e2ac324f82883019d804e59c6f1571d7ee140b7f59423de7a"} err="failed to get container status \"ba3ceef38065136e2ac324f82883019d804e59c6f1571d7ee140b7f59423de7a\": rpc error: code = NotFound desc = could not find container \"ba3ceef38065136e2ac324f82883019d804e59c6f1571d7ee140b7f59423de7a\": container with ID starting with ba3ceef38065136e2ac324f82883019d804e59c6f1571d7ee140b7f59423de7a not found: ID does not exist"
Sep 11 03:55:17.300123 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:55:17.300074    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-autores-fd0def/autores-model-fd0def-kserve-79f55c767-w5wkz"]
Sep 11 03:55:17.304395 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:55:17.304365    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-autores-fd0def/autores-model-fd0def-kserve-79f55c767-w5wkz"]
Sep 11 03:55:18.852807 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:55:18.852768    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6cef66fb-3401-4c33-bfc3-f784d0ff19cd" path="/var/lib/kubelet/pods/6cef66fb-3401-4c33-bfc3-f784d0ff19cd/volumes"
Sep 11 03:55:19.619572 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:55:19.619531    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-autores-fd0def-openshift-default-6d8dfd864d-wjm48"]
Sep 11 03:55:19.619831 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:55:19.619796    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-autores-fd0def-openshift-default-6d8dfd864d-wjm48" podUID="d2a064a0-c84d-4420-9d4b-42e679f412f3" containerName="istio-proxy" containerID="cri-o://d7bb9e021bd5332c4afb55cccf722de0ac9fcd1f083d556555229253d2fef27b" gracePeriod=30
Sep 11 03:55:24.883741 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:55:24.883674    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-autores-fd0def-openshift-default-6d8dfd864d-wjm48"
Sep 11 03:55:24.987462 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:55:24.987423    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d2a064a0-c84d-4420-9d4b-42e679f412f3-credential-socket\") pod \"d2a064a0-c84d-4420-9d4b-42e679f412f3\" (UID: \"d2a064a0-c84d-4420-9d4b-42e679f412f3\") "
Sep 11 03:55:24.987462 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:55:24.987464    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xk2ms\" (UniqueName: \"kubernetes.io/projected/d2a064a0-c84d-4420-9d4b-42e679f412f3-kube-api-access-xk2ms\") pod \"d2a064a0-c84d-4420-9d4b-42e679f412f3\" (UID: \"d2a064a0-c84d-4420-9d4b-42e679f412f3\") "
Sep 11 03:55:24.987712 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:55:24.987512    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d2a064a0-c84d-4420-9d4b-42e679f412f3-istio-data\") pod \"d2a064a0-c84d-4420-9d4b-42e679f412f3\" (UID: \"d2a064a0-c84d-4420-9d4b-42e679f412f3\") "
Sep 11 03:55:24.987712 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:55:24.987544    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d2a064a0-c84d-4420-9d4b-42e679f412f3-istio-envoy\") pod \"d2a064a0-c84d-4420-9d4b-42e679f412f3\" (UID: \"d2a064a0-c84d-4420-9d4b-42e679f412f3\") "
Sep 11 03:55:24.987712 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:55:24.987574    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d2a064a0-c84d-4420-9d4b-42e679f412f3-workload-certs\") pod \"d2a064a0-c84d-4420-9d4b-42e679f412f3\" (UID: \"d2a064a0-c84d-4420-9d4b-42e679f412f3\") "
Sep 11 03:55:24.987712 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:55:24.987609    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d2a064a0-c84d-4420-9d4b-42e679f412f3-workload-socket\") pod \"d2a064a0-c84d-4420-9d4b-42e679f412f3\" (UID: \"d2a064a0-c84d-4420-9d4b-42e679f412f3\") "
Sep 11 03:55:24.987712 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:55:24.987648    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d2a064a0-c84d-4420-9d4b-42e679f412f3-istio-token\") pod \"d2a064a0-c84d-4420-9d4b-42e679f412f3\" (UID: \"d2a064a0-c84d-4420-9d4b-42e679f412f3\") "
Sep 11 03:55:24.987712 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:55:24.987680    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d2a064a0-c84d-4420-9d4b-42e679f412f3-istiod-ca-cert\") pod \"d2a064a0-c84d-4420-9d4b-42e679f412f3\" (UID: \"d2a064a0-c84d-4420-9d4b-42e679f412f3\") "
Sep 11 03:55:24.987712 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:55:24.987705    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d2a064a0-c84d-4420-9d4b-42e679f412f3-istio-podinfo\") pod \"d2a064a0-c84d-4420-9d4b-42e679f412f3\" (UID: \"d2a064a0-c84d-4420-9d4b-42e679f412f3\") "
Sep 11 03:55:24.988056 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:55:24.987778    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d2a064a0-c84d-4420-9d4b-42e679f412f3-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "d2a064a0-c84d-4420-9d4b-42e679f412f3" (UID: "d2a064a0-c84d-4420-9d4b-42e679f412f3"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 03:55:24.988056 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:55:24.987813    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d2a064a0-c84d-4420-9d4b-42e679f412f3-istio-data" (OuterVolumeSpecName: "istio-data") pod "d2a064a0-c84d-4420-9d4b-42e679f412f3" (UID: "d2a064a0-c84d-4420-9d4b-42e679f412f3"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 03:55:24.988056 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:55:24.987926    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d2a064a0-c84d-4420-9d4b-42e679f412f3-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "d2a064a0-c84d-4420-9d4b-42e679f412f3" (UID: "d2a064a0-c84d-4420-9d4b-42e679f412f3"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 03:55:24.988304 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:55:24.988113    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d2a064a0-c84d-4420-9d4b-42e679f412f3-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "d2a064a0-c84d-4420-9d4b-42e679f412f3" (UID: "d2a064a0-c84d-4420-9d4b-42e679f412f3"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 03:55:24.988304 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:55:24.988193    2579 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d2a064a0-c84d-4420-9d4b-42e679f412f3-workload-certs\") on node \"ip-10-0-141-105.ec2.internal\" DevicePath \"\""
Sep 11 03:55:24.988304 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:55:24.988216    2579 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d2a064a0-c84d-4420-9d4b-42e679f412f3-credential-socket\") on node \"ip-10-0-141-105.ec2.internal\" DevicePath \"\""
Sep 11 03:55:24.988304 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:55:24.988231    2579 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d2a064a0-c84d-4420-9d4b-42e679f412f3-istio-data\") on node \"ip-10-0-141-105.ec2.internal\" DevicePath \"\""
Sep 11 03:55:24.988687 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:55:24.988535    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d2a064a0-c84d-4420-9d4b-42e679f412f3-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "d2a064a0-c84d-4420-9d4b-42e679f412f3" (UID: "d2a064a0-c84d-4420-9d4b-42e679f412f3"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 03:55:24.990045 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:55:24.990016    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d2a064a0-c84d-4420-9d4b-42e679f412f3-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "d2a064a0-c84d-4420-9d4b-42e679f412f3" (UID: "d2a064a0-c84d-4420-9d4b-42e679f412f3"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 03:55:24.990045 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:55:24.990038    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d2a064a0-c84d-4420-9d4b-42e679f412f3-kube-api-access-xk2ms" (OuterVolumeSpecName: "kube-api-access-xk2ms") pod "d2a064a0-c84d-4420-9d4b-42e679f412f3" (UID: "d2a064a0-c84d-4420-9d4b-42e679f412f3"). InnerVolumeSpecName "kube-api-access-xk2ms". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 03:55:24.990214 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:55:24.990069    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d2a064a0-c84d-4420-9d4b-42e679f412f3-istio-token" (OuterVolumeSpecName: "istio-token") pod "d2a064a0-c84d-4420-9d4b-42e679f412f3" (UID: "d2a064a0-c84d-4420-9d4b-42e679f412f3"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 03:55:24.990214 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:55:24.990201    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d2a064a0-c84d-4420-9d4b-42e679f412f3-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "d2a064a0-c84d-4420-9d4b-42e679f412f3" (UID: "d2a064a0-c84d-4420-9d4b-42e679f412f3"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 11 03:55:25.089481 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:55:25.089439    2579 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d2a064a0-c84d-4420-9d4b-42e679f412f3-istio-envoy\") on node \"ip-10-0-141-105.ec2.internal\" DevicePath \"\""
Sep 11 03:55:25.089481 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:55:25.089470    2579 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d2a064a0-c84d-4420-9d4b-42e679f412f3-workload-socket\") on node \"ip-10-0-141-105.ec2.internal\" DevicePath \"\""
Sep 11 03:55:25.089481 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:55:25.089482    2579 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d2a064a0-c84d-4420-9d4b-42e679f412f3-istio-token\") on node \"ip-10-0-141-105.ec2.internal\" DevicePath \"\""
Sep 11 03:55:25.089716 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:55:25.089494    2579 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d2a064a0-c84d-4420-9d4b-42e679f412f3-istiod-ca-cert\") on node \"ip-10-0-141-105.ec2.internal\" DevicePath \"\""
Sep 11 03:55:25.089716 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:55:25.089506    2579 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d2a064a0-c84d-4420-9d4b-42e679f412f3-istio-podinfo\") on node \"ip-10-0-141-105.ec2.internal\" DevicePath \"\""
Sep 11 03:55:25.089716 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:55:25.089519    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xk2ms\" (UniqueName: \"kubernetes.io/projected/d2a064a0-c84d-4420-9d4b-42e679f412f3-kube-api-access-xk2ms\") on node \"ip-10-0-141-105.ec2.internal\" DevicePath \"\""
Sep 11 03:55:25.327895 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:55:25.327804    2579 generic.go:356] "Generic (PLEG): container finished" podID="d2a064a0-c84d-4420-9d4b-42e679f412f3" containerID="d7bb9e021bd5332c4afb55cccf722de0ac9fcd1f083d556555229253d2fef27b" exitCode=0
Sep 11 03:55:25.327895 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:55:25.327886    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-fd0def-openshift-default-6d8dfd864d-wjm48" event={"ID":"d2a064a0-c84d-4420-9d4b-42e679f412f3","Type":"ContainerDied","Data":"d7bb9e021bd5332c4afb55cccf722de0ac9fcd1f083d556555229253d2fef27b"}
Sep 11 03:55:25.328138 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:55:25.327894    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-autores-fd0def-openshift-default-6d8dfd864d-wjm48"
Sep 11 03:55:25.328138 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:55:25.327977    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-fd0def-openshift-default-6d8dfd864d-wjm48" event={"ID":"d2a064a0-c84d-4420-9d4b-42e679f412f3","Type":"ContainerDied","Data":"9b07304cfae5d380d7496054fdfff72459e5733d63d7c6b956e8c699c26e3310"}
Sep 11 03:55:25.328138 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:55:25.328008    2579 scope.go:117] "RemoveContainer" containerID="d7bb9e021bd5332c4afb55cccf722de0ac9fcd1f083d556555229253d2fef27b"
Sep 11 03:55:25.340166 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:55:25.340148    2579 scope.go:117] "RemoveContainer" containerID="3346bf6dd9e6ecb80e0d9f55889f4bb2f67351fdefd90e8aaf1d86c518682a65"
Sep 11 03:55:25.350263 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:55:25.350241    2579 scope.go:117] "RemoveContainer" containerID="d7bb9e021bd5332c4afb55cccf722de0ac9fcd1f083d556555229253d2fef27b"
Sep 11 03:55:25.350631 ip-10-0-141-105 kubenswrapper[2579]: E0911 03:55:25.350612    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d7bb9e021bd5332c4afb55cccf722de0ac9fcd1f083d556555229253d2fef27b\": container with ID starting with d7bb9e021bd5332c4afb55cccf722de0ac9fcd1f083d556555229253d2fef27b not found: ID does not exist" containerID="d7bb9e021bd5332c4afb55cccf722de0ac9fcd1f083d556555229253d2fef27b"
Sep 11 03:55:25.350719 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:55:25.350644    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d7bb9e021bd5332c4afb55cccf722de0ac9fcd1f083d556555229253d2fef27b"} err="failed to get container status \"d7bb9e021bd5332c4afb55cccf722de0ac9fcd1f083d556555229253d2fef27b\": rpc error: code = NotFound desc = could not find container \"d7bb9e021bd5332c4afb55cccf722de0ac9fcd1f083d556555229253d2fef27b\": container with ID starting with d7bb9e021bd5332c4afb55cccf722de0ac9fcd1f083d556555229253d2fef27b not found: ID does not exist"
Sep 11 03:55:25.350719 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:55:25.350671    2579 scope.go:117] "RemoveContainer" containerID="3346bf6dd9e6ecb80e0d9f55889f4bb2f67351fdefd90e8aaf1d86c518682a65"
Sep 11 03:55:25.351001 ip-10-0-141-105 kubenswrapper[2579]: E0911 03:55:25.350980    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3346bf6dd9e6ecb80e0d9f55889f4bb2f67351fdefd90e8aaf1d86c518682a65\": container with ID starting with 3346bf6dd9e6ecb80e0d9f55889f4bb2f67351fdefd90e8aaf1d86c518682a65 not found: ID does not exist" containerID="3346bf6dd9e6ecb80e0d9f55889f4bb2f67351fdefd90e8aaf1d86c518682a65"
Sep 11 03:55:25.351117 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:55:25.351004    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3346bf6dd9e6ecb80e0d9f55889f4bb2f67351fdefd90e8aaf1d86c518682a65"} err="failed to get container status \"3346bf6dd9e6ecb80e0d9f55889f4bb2f67351fdefd90e8aaf1d86c518682a65\": rpc error: code = NotFound desc = could not find container \"3346bf6dd9e6ecb80e0d9f55889f4bb2f67351fdefd90e8aaf1d86c518682a65\": container with ID starting with 3346bf6dd9e6ecb80e0d9f55889f4bb2f67351fdefd90e8aaf1d86c518682a65 not found: ID does not exist"
Sep 11 03:55:25.351672 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:55:25.351650    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-autores-fd0def-openshift-default-6d8dfd864d-wjm48"]
Sep 11 03:55:25.357474 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:55:25.357451    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-autores-fd0def-openshift-default-6d8dfd864d-wjm48"]
Sep 11 03:55:25.969217 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:55:25.969183    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-459a4c-f97c9d9d-24nbj"]
Sep 11 03:55:25.969688 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:55:25.969669    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b1c46d01-52dc-4ef1-83a7-973ac377ef3d" containerName="payload-processing"
Sep 11 03:55:25.969741 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:55:25.969691    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="b1c46d01-52dc-4ef1-83a7-973ac377ef3d" containerName="payload-processing"
Sep 11 03:55:25.969741 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:55:25.969703    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a173f5c9-2082-4276-99fc-bc121d8c8be3" containerName="payload-pre-processing"
Sep 11 03:55:25.969741 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:55:25.969708    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="a173f5c9-2082-4276-99fc-bc121d8c8be3" containerName="payload-pre-processing"
Sep 11 03:55:25.969741 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:55:25.969716    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6cef66fb-3401-4c33-bfc3-f784d0ff19cd" containerName="main"
Sep 11 03:55:25.969741 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:55:25.969721    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="6cef66fb-3401-4c33-bfc3-f784d0ff19cd" containerName="main"
Sep 11 03:55:25.969741 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:55:25.969732    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d2a064a0-c84d-4420-9d4b-42e679f412f3" containerName="istio-proxy"
Sep 11 03:55:25.969741 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:55:25.969737    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="d2a064a0-c84d-4420-9d4b-42e679f412f3" containerName="istio-proxy"
Sep 11 03:55:25.969974 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:55:25.969748    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="47052321-c2e8-489e-8f54-42b49609fa66" containerName="main"
Sep 11 03:55:25.969974 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:55:25.969754    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="47052321-c2e8-489e-8f54-42b49609fa66" containerName="main"
Sep 11 03:55:25.969974 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:55:25.969764    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="48f0af88-1d41-4526-a48e-3b3a948c17f6" containerName="revoke-keys"
Sep 11 03:55:25.969974 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:55:25.969769    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="48f0af88-1d41-4526-a48e-3b3a948c17f6" containerName="revoke-keys"
Sep 11 03:55:25.969974 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:55:25.969831    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="d2a064a0-c84d-4420-9d4b-42e679f412f3" containerName="istio-proxy"
Sep 11 03:55:25.969974 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:55:25.969840    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="d2a064a0-c84d-4420-9d4b-42e679f412f3" containerName="istio-proxy"
Sep 11 03:55:25.969974 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:55:25.969845    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="48f0af88-1d41-4526-a48e-3b3a948c17f6" containerName="revoke-keys"
Sep 11 03:55:25.969974 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:55:25.969851    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="47052321-c2e8-489e-8f54-42b49609fa66" containerName="main"
Sep 11 03:55:25.969974 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:55:25.969856    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="b1c46d01-52dc-4ef1-83a7-973ac377ef3d" containerName="payload-processing"
Sep 11 03:55:25.969974 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:55:25.969863    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="a173f5c9-2082-4276-99fc-bc121d8c8be3" containerName="payload-pre-processing"
Sep 11 03:55:25.969974 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:55:25.969868    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="6cef66fb-3401-4c33-bfc3-f784d0ff19cd" containerName="main"
Sep 11 03:55:25.974387 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:55:25.974369    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-explicit-459a4c-f97c9d9d-24nbj"
Sep 11 03:55:25.976954 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:55:25.976926    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-explicit-459a4c\""
Sep 11 03:55:25.977116 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:55:25.976979    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-explicit-459a4c-dockercfg-gpl6d\""
Sep 11 03:55:25.981872 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:55:25.981845    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-459a4c-f97c9d9d-24nbj"]
Sep 11 03:55:26.098444 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:55:26.098394    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cgxb6\" (UniqueName: \"kubernetes.io/projected/2e715f91-99db-45d9-ad51-c46cff03ff78-kube-api-access-cgxb6\") pod \"payload-pre-processing-e2e-explicit-459a4c-f97c9d9d-24nbj\" (UID: \"2e715f91-99db-45d9-ad51-c46cff03ff78\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-459a4c-f97c9d9d-24nbj"
Sep 11 03:55:26.098626 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:55:26.098460    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2e715f91-99db-45d9-ad51-c46cff03ff78-plugins-config-volume\") pod \"payload-pre-processing-e2e-explicit-459a4c-f97c9d9d-24nbj\" (UID: \"2e715f91-99db-45d9-ad51-c46cff03ff78\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-459a4c-f97c9d9d-24nbj"
Sep 11 03:55:26.199616 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:55:26.199575    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cgxb6\" (UniqueName: \"kubernetes.io/projected/2e715f91-99db-45d9-ad51-c46cff03ff78-kube-api-access-cgxb6\") pod \"payload-pre-processing-e2e-explicit-459a4c-f97c9d9d-24nbj\" (UID: \"2e715f91-99db-45d9-ad51-c46cff03ff78\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-459a4c-f97c9d9d-24nbj"
Sep 11 03:55:26.199791 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:55:26.199627    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2e715f91-99db-45d9-ad51-c46cff03ff78-plugins-config-volume\") pod \"payload-pre-processing-e2e-explicit-459a4c-f97c9d9d-24nbj\" (UID: \"2e715f91-99db-45d9-ad51-c46cff03ff78\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-459a4c-f97c9d9d-24nbj"
Sep 11 03:55:26.200321 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:55:26.200295    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2e715f91-99db-45d9-ad51-c46cff03ff78-plugins-config-volume\") pod \"payload-pre-processing-e2e-explicit-459a4c-f97c9d9d-24nbj\" (UID: \"2e715f91-99db-45d9-ad51-c46cff03ff78\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-459a4c-f97c9d9d-24nbj"
Sep 11 03:55:26.207224 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:55:26.207199    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cgxb6\" (UniqueName: \"kubernetes.io/projected/2e715f91-99db-45d9-ad51-c46cff03ff78-kube-api-access-cgxb6\") pod \"payload-pre-processing-e2e-explicit-459a4c-f97c9d9d-24nbj\" (UID: \"2e715f91-99db-45d9-ad51-c46cff03ff78\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-459a4c-f97c9d9d-24nbj"
Sep 11 03:55:26.288037 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:55:26.287947    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-explicit-459a4c-f97c9d9d-24nbj"
Sep 11 03:55:26.558210 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:55:26.558185    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-459a4c-f97c9d9d-24nbj"]
Sep 11 03:55:26.559967 ip-10-0-141-105 kubenswrapper[2579]: W0911 03:55:26.559934    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2e715f91_99db_45d9_ad51_c46cff03ff78.slice/crio-424f74d3efd888ec7f243472b0c77c1d9f3359ac9aeabe248a4d83a77479c4f0 WatchSource:0}: Error finding container 424f74d3efd888ec7f243472b0c77c1d9f3359ac9aeabe248a4d83a77479c4f0: Status 404 returned error can't find the container with id 424f74d3efd888ec7f243472b0c77c1d9f3359ac9aeabe248a4d83a77479c4f0
Sep 11 03:55:26.851776 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:55:26.851691    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d2a064a0-c84d-4420-9d4b-42e679f412f3" path="/var/lib/kubelet/pods/d2a064a0-c84d-4420-9d4b-42e679f412f3/volumes"
Sep 11 03:55:27.342574 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:55:27.342537    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-explicit-459a4c-f97c9d9d-24nbj" event={"ID":"2e715f91-99db-45d9-ad51-c46cff03ff78","Type":"ContainerStarted","Data":"7f7af2b33acfe59b36d04e4a51f7c01c0b57473f28b6c1ba1b4be82c5e3add2e"}
Sep 11 03:55:27.342974 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:55:27.342601    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-explicit-459a4c-f97c9d9d-24nbj" event={"ID":"2e715f91-99db-45d9-ad51-c46cff03ff78","Type":"ContainerStarted","Data":"424f74d3efd888ec7f243472b0c77c1d9f3359ac9aeabe248a4d83a77479c4f0"}
Sep 11 03:55:27.342974 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:55:27.342670    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-explicit-459a4c-f97c9d9d-24nbj"
Sep 11 03:55:27.359256 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:55:27.359210    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-explicit-459a4c-f97c9d9d-24nbj" podStartSLOduration=2.359195799 podStartE2EDuration="2.359195799s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 03:55:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 03:55:27.357186956 +0000 UTC m=+905.012392524" watchObservedRunningTime="2026-09-11 03:55:27.359195799 +0000 UTC m=+905.014401369"
Sep 11 03:55:28.280170 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:55:28.280135    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-b-8efac4/test-model-8efac4-kserve-cb9995899-sxkk6"]
Sep 11 03:55:28.280538 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:55:28.280524    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d2a064a0-c84d-4420-9d4b-42e679f412f3" containerName="istio-proxy"
Sep 11 03:55:28.280662 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:55:28.280539    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="d2a064a0-c84d-4420-9d4b-42e679f412f3" containerName="istio-proxy"
Sep 11 03:55:28.283940 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:55:28.283923    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-8efac4/test-model-8efac4-kserve-cb9995899-sxkk6"
Sep 11 03:55:28.287697 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:55:28.287665    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-8efac4\"/\"kube-root-ca.crt\""
Sep 11 03:55:28.287823 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:55:28.287698    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-8efac4\"/\"openshift-service-ca.crt\""
Sep 11 03:55:28.287823 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:55:28.287742    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-8efac4\"/\"test-model-8efac4-kserve-self-signed-certs\""
Sep 11 03:55:28.287823 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:55:28.287699    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-8efac4\"/\"default-dockercfg-nxwfk\""
Sep 11 03:55:28.292543 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:55:28.292520    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-8efac4/test-model-8efac4-kserve-cb9995899-sxkk6"]
Sep 11 03:55:28.317368 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:55:28.317334    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3faa005d-de0c-4c9c-9de6-035473a89077-dshm\") pod \"test-model-8efac4-kserve-cb9995899-sxkk6\" (UID: \"3faa005d-de0c-4c9c-9de6-035473a89077\") " pod="ai-tenant-e2e-inf-b-8efac4/test-model-8efac4-kserve-cb9995899-sxkk6"
Sep 11 03:55:28.317534 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:55:28.317381    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3faa005d-de0c-4c9c-9de6-035473a89077-home\") pod \"test-model-8efac4-kserve-cb9995899-sxkk6\" (UID: \"3faa005d-de0c-4c9c-9de6-035473a89077\") " pod="ai-tenant-e2e-inf-b-8efac4/test-model-8efac4-kserve-cb9995899-sxkk6"
Sep 11 03:55:28.317534 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:55:28.317463    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3faa005d-de0c-4c9c-9de6-035473a89077-model-cache\") pod \"test-model-8efac4-kserve-cb9995899-sxkk6\" (UID: \"3faa005d-de0c-4c9c-9de6-035473a89077\") " pod="ai-tenant-e2e-inf-b-8efac4/test-model-8efac4-kserve-cb9995899-sxkk6"
Sep 11 03:55:28.317534 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:55:28.317493    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3faa005d-de0c-4c9c-9de6-035473a89077-tls-certs\") pod \"test-model-8efac4-kserve-cb9995899-sxkk6\" (UID: \"3faa005d-de0c-4c9c-9de6-035473a89077\") " pod="ai-tenant-e2e-inf-b-8efac4/test-model-8efac4-kserve-cb9995899-sxkk6"
Sep 11 03:55:28.317534 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:55:28.317511    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-95f9w\" (UniqueName: \"kubernetes.io/projected/3faa005d-de0c-4c9c-9de6-035473a89077-kube-api-access-95f9w\") pod \"test-model-8efac4-kserve-cb9995899-sxkk6\" (UID: \"3faa005d-de0c-4c9c-9de6-035473a89077\") " pod="ai-tenant-e2e-inf-b-8efac4/test-model-8efac4-kserve-cb9995899-sxkk6"
Sep 11 03:55:28.317534 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:55:28.317528    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3faa005d-de0c-4c9c-9de6-035473a89077-tmp-dir\") pod \"test-model-8efac4-kserve-cb9995899-sxkk6\" (UID: \"3faa005d-de0c-4c9c-9de6-035473a89077\") " pod="ai-tenant-e2e-inf-b-8efac4/test-model-8efac4-kserve-cb9995899-sxkk6"
Sep 11 03:55:28.418561 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:55:28.418525    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3faa005d-de0c-4c9c-9de6-035473a89077-model-cache\") pod \"test-model-8efac4-kserve-cb9995899-sxkk6\" (UID: \"3faa005d-de0c-4c9c-9de6-035473a89077\") " pod="ai-tenant-e2e-inf-b-8efac4/test-model-8efac4-kserve-cb9995899-sxkk6"
Sep 11 03:55:28.419029 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:55:28.418568    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3faa005d-de0c-4c9c-9de6-035473a89077-tls-certs\") pod \"test-model-8efac4-kserve-cb9995899-sxkk6\" (UID: \"3faa005d-de0c-4c9c-9de6-035473a89077\") " pod="ai-tenant-e2e-inf-b-8efac4/test-model-8efac4-kserve-cb9995899-sxkk6"
Sep 11 03:55:28.419029 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:55:28.418590    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-95f9w\" (UniqueName: \"kubernetes.io/projected/3faa005d-de0c-4c9c-9de6-035473a89077-kube-api-access-95f9w\") pod \"test-model-8efac4-kserve-cb9995899-sxkk6\" (UID: \"3faa005d-de0c-4c9c-9de6-035473a89077\") " pod="ai-tenant-e2e-inf-b-8efac4/test-model-8efac4-kserve-cb9995899-sxkk6"
Sep 11 03:55:28.419029 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:55:28.418633    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3faa005d-de0c-4c9c-9de6-035473a89077-tmp-dir\") pod \"test-model-8efac4-kserve-cb9995899-sxkk6\" (UID: \"3faa005d-de0c-4c9c-9de6-035473a89077\") " pod="ai-tenant-e2e-inf-b-8efac4/test-model-8efac4-kserve-cb9995899-sxkk6"
Sep 11 03:55:28.419029 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:55:28.418677    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3faa005d-de0c-4c9c-9de6-035473a89077-dshm\") pod \"test-model-8efac4-kserve-cb9995899-sxkk6\" (UID: \"3faa005d-de0c-4c9c-9de6-035473a89077\") " pod="ai-tenant-e2e-inf-b-8efac4/test-model-8efac4-kserve-cb9995899-sxkk6"
Sep 11 03:55:28.419029 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:55:28.418750    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3faa005d-de0c-4c9c-9de6-035473a89077-home\") pod \"test-model-8efac4-kserve-cb9995899-sxkk6\" (UID: \"3faa005d-de0c-4c9c-9de6-035473a89077\") " pod="ai-tenant-e2e-inf-b-8efac4/test-model-8efac4-kserve-cb9995899-sxkk6"
Sep 11 03:55:28.419029 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:55:28.419011    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3faa005d-de0c-4c9c-9de6-035473a89077-model-cache\") pod \"test-model-8efac4-kserve-cb9995899-sxkk6\" (UID: \"3faa005d-de0c-4c9c-9de6-035473a89077\") " pod="ai-tenant-e2e-inf-b-8efac4/test-model-8efac4-kserve-cb9995899-sxkk6"
Sep 11 03:55:28.419356 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:55:28.419190    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3faa005d-de0c-4c9c-9de6-035473a89077-tmp-dir\") pod \"test-model-8efac4-kserve-cb9995899-sxkk6\" (UID: \"3faa005d-de0c-4c9c-9de6-035473a89077\") " pod="ai-tenant-e2e-inf-b-8efac4/test-model-8efac4-kserve-cb9995899-sxkk6"
Sep 11 03:55:28.419356 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:55:28.419207    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3faa005d-de0c-4c9c-9de6-035473a89077-home\") pod \"test-model-8efac4-kserve-cb9995899-sxkk6\" (UID: \"3faa005d-de0c-4c9c-9de6-035473a89077\") " pod="ai-tenant-e2e-inf-b-8efac4/test-model-8efac4-kserve-cb9995899-sxkk6"
Sep 11 03:55:28.421256 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:55:28.421236    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3faa005d-de0c-4c9c-9de6-035473a89077-dshm\") pod \"test-model-8efac4-kserve-cb9995899-sxkk6\" (UID: \"3faa005d-de0c-4c9c-9de6-035473a89077\") " pod="ai-tenant-e2e-inf-b-8efac4/test-model-8efac4-kserve-cb9995899-sxkk6"
Sep 11 03:55:28.421348 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:55:28.421288    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3faa005d-de0c-4c9c-9de6-035473a89077-tls-certs\") pod \"test-model-8efac4-kserve-cb9995899-sxkk6\" (UID: \"3faa005d-de0c-4c9c-9de6-035473a89077\") " pod="ai-tenant-e2e-inf-b-8efac4/test-model-8efac4-kserve-cb9995899-sxkk6"
Sep 11 03:55:28.426015 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:55:28.425992    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-95f9w\" (UniqueName: \"kubernetes.io/projected/3faa005d-de0c-4c9c-9de6-035473a89077-kube-api-access-95f9w\") pod \"test-model-8efac4-kserve-cb9995899-sxkk6\" (UID: \"3faa005d-de0c-4c9c-9de6-035473a89077\") " pod="ai-tenant-e2e-inf-b-8efac4/test-model-8efac4-kserve-cb9995899-sxkk6"
Sep 11 03:55:28.594473 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:55:28.594386    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-8efac4/test-model-8efac4-kserve-cb9995899-sxkk6"
Sep 11 03:55:28.869276 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:55:28.869250    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-8efac4/test-model-8efac4-kserve-cb9995899-sxkk6"]
Sep 11 03:55:28.871158 ip-10-0-141-105 kubenswrapper[2579]: W0911 03:55:28.871130    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod3faa005d_de0c_4c9c_9de6_035473a89077.slice/crio-82b3487fd6ff31974b71c7c0e9c17c9bed325b8c978ff78d52b83a70b6465af7 WatchSource:0}: Error finding container 82b3487fd6ff31974b71c7c0e9c17c9bed325b8c978ff78d52b83a70b6465af7: Status 404 returned error can't find the container with id 82b3487fd6ff31974b71c7c0e9c17c9bed325b8c978ff78d52b83a70b6465af7
Sep 11 03:55:29.355160 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:55:29.355121    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-8efac4/test-model-8efac4-kserve-cb9995899-sxkk6" event={"ID":"3faa005d-de0c-4c9c-9de6-035473a89077","Type":"ContainerStarted","Data":"6d34dd50a2612144251ddb12ce7473a02978b54dcb14ae2f94e17815b16044c7"}
Sep 11 03:55:29.355413 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:55:29.355167    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-8efac4/test-model-8efac4-kserve-cb9995899-sxkk6" event={"ID":"3faa005d-de0c-4c9c-9de6-035473a89077","Type":"ContainerStarted","Data":"82b3487fd6ff31974b71c7c0e9c17c9bed325b8c978ff78d52b83a70b6465af7"}
Sep 11 03:55:29.370234 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:55:29.370174    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-inf-b-8efac4/test-model-8efac4-kserve-cb9995899-sxkk6" podStartSLOduration=1.370153104 podStartE2EDuration="1.370153104s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 03:55:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 03:55:29.36978547 +0000 UTC m=+907.024991038" watchObservedRunningTime="2026-09-11 03:55:29.370153104 +0000 UTC m=+907.025358672"
Sep 11 03:55:31.428505 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:55:31.428463    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-5z4cn"]
Sep 11 03:55:38.350939 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:55:38.350901    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-explicit-459a4c-f97c9d9d-24nbj"
Sep 11 03:55:38.595186 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:55:38.595145    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-b-8efac4/test-model-8efac4-kserve-cb9995899-sxkk6"
Sep 11 03:55:38.595382 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:55:38.595233    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-8efac4/test-model-8efac4-kserve-cb9995899-sxkk6"
Sep 11 03:55:38.608458 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:55:38.608437    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-b-8efac4/test-model-8efac4-kserve-cb9995899-sxkk6"
Sep 11 03:55:39.418467 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:55:39.418437    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-8efac4/test-model-8efac4-kserve-cb9995899-sxkk6"
Sep 11 03:55:51.807968 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:55:51.807934    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-5z4cn"]
Sep 11 03:56:04.408002 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:56:04.407957    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-5z4cn"]
Sep 11 03:56:14.199413 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:56:14.199378    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-explicit-459a4c/explicit-model-459a4c-kserve-d9cc98b77-j9nlz"]
Sep 11 03:56:14.203924 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:56:14.203904    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-explicit-459a4c/explicit-model-459a4c-kserve-d9cc98b77-j9nlz"
Sep 11 03:56:14.207756 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:56:14.207719    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-explicit-459a4c\"/\"kube-root-ca.crt\""
Sep 11 03:56:14.207898 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:56:14.207758    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-explicit-459a4c\"/\"explicit-model-459a4c-kserve-self-signed-certs\""
Sep 11 03:56:14.207898 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:56:14.207777    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-explicit-459a4c\"/\"openshift-service-ca.crt\""
Sep 11 03:56:14.207898 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:56:14.207723    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-explicit-459a4c\"/\"default-dockercfg-gkq9h\""
Sep 11 03:56:14.215537 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:56:14.215514    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-explicit-459a4c/explicit-model-459a4c-kserve-d9cc98b77-j9nlz"]
Sep 11 03:56:14.350380 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:56:14.350326    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/789f471b-9be6-4de4-acd3-9be9aa2b24f1-dshm\") pod \"explicit-model-459a4c-kserve-d9cc98b77-j9nlz\" (UID: \"789f471b-9be6-4de4-acd3-9be9aa2b24f1\") " pod="ai-tenant-e2e-explicit-459a4c/explicit-model-459a4c-kserve-d9cc98b77-j9nlz"
Sep 11 03:56:14.350566 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:56:14.350392    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/789f471b-9be6-4de4-acd3-9be9aa2b24f1-tmp-dir\") pod \"explicit-model-459a4c-kserve-d9cc98b77-j9nlz\" (UID: \"789f471b-9be6-4de4-acd3-9be9aa2b24f1\") " pod="ai-tenant-e2e-explicit-459a4c/explicit-model-459a4c-kserve-d9cc98b77-j9nlz"
Sep 11 03:56:14.350566 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:56:14.350491    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/789f471b-9be6-4de4-acd3-9be9aa2b24f1-model-cache\") pod \"explicit-model-459a4c-kserve-d9cc98b77-j9nlz\" (UID: \"789f471b-9be6-4de4-acd3-9be9aa2b24f1\") " pod="ai-tenant-e2e-explicit-459a4c/explicit-model-459a4c-kserve-d9cc98b77-j9nlz"
Sep 11 03:56:14.350566 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:56:14.350542    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4sr6b\" (UniqueName: \"kubernetes.io/projected/789f471b-9be6-4de4-acd3-9be9aa2b24f1-kube-api-access-4sr6b\") pod \"explicit-model-459a4c-kserve-d9cc98b77-j9nlz\" (UID: \"789f471b-9be6-4de4-acd3-9be9aa2b24f1\") " pod="ai-tenant-e2e-explicit-459a4c/explicit-model-459a4c-kserve-d9cc98b77-j9nlz"
Sep 11 03:56:14.350756 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:56:14.350622    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/789f471b-9be6-4de4-acd3-9be9aa2b24f1-home\") pod \"explicit-model-459a4c-kserve-d9cc98b77-j9nlz\" (UID: \"789f471b-9be6-4de4-acd3-9be9aa2b24f1\") " pod="ai-tenant-e2e-explicit-459a4c/explicit-model-459a4c-kserve-d9cc98b77-j9nlz"
Sep 11 03:56:14.350756 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:56:14.350672    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/789f471b-9be6-4de4-acd3-9be9aa2b24f1-tls-certs\") pod \"explicit-model-459a4c-kserve-d9cc98b77-j9nlz\" (UID: \"789f471b-9be6-4de4-acd3-9be9aa2b24f1\") " pod="ai-tenant-e2e-explicit-459a4c/explicit-model-459a4c-kserve-d9cc98b77-j9nlz"
Sep 11 03:56:14.452172 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:56:14.452029    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4sr6b\" (UniqueName: \"kubernetes.io/projected/789f471b-9be6-4de4-acd3-9be9aa2b24f1-kube-api-access-4sr6b\") pod \"explicit-model-459a4c-kserve-d9cc98b77-j9nlz\" (UID: \"789f471b-9be6-4de4-acd3-9be9aa2b24f1\") " pod="ai-tenant-e2e-explicit-459a4c/explicit-model-459a4c-kserve-d9cc98b77-j9nlz"
Sep 11 03:56:14.452172 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:56:14.452149    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/789f471b-9be6-4de4-acd3-9be9aa2b24f1-home\") pod \"explicit-model-459a4c-kserve-d9cc98b77-j9nlz\" (UID: \"789f471b-9be6-4de4-acd3-9be9aa2b24f1\") " pod="ai-tenant-e2e-explicit-459a4c/explicit-model-459a4c-kserve-d9cc98b77-j9nlz"
Sep 11 03:56:14.452431 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:56:14.452183    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/789f471b-9be6-4de4-acd3-9be9aa2b24f1-tls-certs\") pod \"explicit-model-459a4c-kserve-d9cc98b77-j9nlz\" (UID: \"789f471b-9be6-4de4-acd3-9be9aa2b24f1\") " pod="ai-tenant-e2e-explicit-459a4c/explicit-model-459a4c-kserve-d9cc98b77-j9nlz"
Sep 11 03:56:14.452431 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:56:14.452384    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/789f471b-9be6-4de4-acd3-9be9aa2b24f1-dshm\") pod \"explicit-model-459a4c-kserve-d9cc98b77-j9nlz\" (UID: \"789f471b-9be6-4de4-acd3-9be9aa2b24f1\") " pod="ai-tenant-e2e-explicit-459a4c/explicit-model-459a4c-kserve-d9cc98b77-j9nlz"
Sep 11 03:56:14.452542 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:56:14.452439    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/789f471b-9be6-4de4-acd3-9be9aa2b24f1-tmp-dir\") pod \"explicit-model-459a4c-kserve-d9cc98b77-j9nlz\" (UID: \"789f471b-9be6-4de4-acd3-9be9aa2b24f1\") " pod="ai-tenant-e2e-explicit-459a4c/explicit-model-459a4c-kserve-d9cc98b77-j9nlz"
Sep 11 03:56:14.452542 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:56:14.452517    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/789f471b-9be6-4de4-acd3-9be9aa2b24f1-model-cache\") pod \"explicit-model-459a4c-kserve-d9cc98b77-j9nlz\" (UID: \"789f471b-9be6-4de4-acd3-9be9aa2b24f1\") " pod="ai-tenant-e2e-explicit-459a4c/explicit-model-459a4c-kserve-d9cc98b77-j9nlz"
Sep 11 03:56:14.452879 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:56:14.452849    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/789f471b-9be6-4de4-acd3-9be9aa2b24f1-tmp-dir\") pod \"explicit-model-459a4c-kserve-d9cc98b77-j9nlz\" (UID: \"789f471b-9be6-4de4-acd3-9be9aa2b24f1\") " pod="ai-tenant-e2e-explicit-459a4c/explicit-model-459a4c-kserve-d9cc98b77-j9nlz"
Sep 11 03:56:14.452996 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:56:14.452898    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/789f471b-9be6-4de4-acd3-9be9aa2b24f1-home\") pod \"explicit-model-459a4c-kserve-d9cc98b77-j9nlz\" (UID: \"789f471b-9be6-4de4-acd3-9be9aa2b24f1\") " pod="ai-tenant-e2e-explicit-459a4c/explicit-model-459a4c-kserve-d9cc98b77-j9nlz"
Sep 11 03:56:14.452996 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:56:14.452953    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/789f471b-9be6-4de4-acd3-9be9aa2b24f1-model-cache\") pod \"explicit-model-459a4c-kserve-d9cc98b77-j9nlz\" (UID: \"789f471b-9be6-4de4-acd3-9be9aa2b24f1\") " pod="ai-tenant-e2e-explicit-459a4c/explicit-model-459a4c-kserve-d9cc98b77-j9nlz"
Sep 11 03:56:14.454949 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:56:14.454929    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/789f471b-9be6-4de4-acd3-9be9aa2b24f1-dshm\") pod \"explicit-model-459a4c-kserve-d9cc98b77-j9nlz\" (UID: \"789f471b-9be6-4de4-acd3-9be9aa2b24f1\") " pod="ai-tenant-e2e-explicit-459a4c/explicit-model-459a4c-kserve-d9cc98b77-j9nlz"
Sep 11 03:56:14.455021 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:56:14.454946    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/789f471b-9be6-4de4-acd3-9be9aa2b24f1-tls-certs\") pod \"explicit-model-459a4c-kserve-d9cc98b77-j9nlz\" (UID: \"789f471b-9be6-4de4-acd3-9be9aa2b24f1\") " pod="ai-tenant-e2e-explicit-459a4c/explicit-model-459a4c-kserve-d9cc98b77-j9nlz"
Sep 11 03:56:14.459507 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:56:14.459479    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4sr6b\" (UniqueName: \"kubernetes.io/projected/789f471b-9be6-4de4-acd3-9be9aa2b24f1-kube-api-access-4sr6b\") pod \"explicit-model-459a4c-kserve-d9cc98b77-j9nlz\" (UID: \"789f471b-9be6-4de4-acd3-9be9aa2b24f1\") " pod="ai-tenant-e2e-explicit-459a4c/explicit-model-459a4c-kserve-d9cc98b77-j9nlz"
Sep 11 03:56:14.516553 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:56:14.516526    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-explicit-459a4c/explicit-model-459a4c-kserve-d9cc98b77-j9nlz"
Sep 11 03:56:14.788768 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:56:14.788743    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-explicit-459a4c/explicit-model-459a4c-kserve-d9cc98b77-j9nlz"]
Sep 11 03:56:14.790460 ip-10-0-141-105 kubenswrapper[2579]: W0911 03:56:14.790433    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod789f471b_9be6_4de4_acd3_9be9aa2b24f1.slice/crio-6ebccb0ee400dad4473d3faf1fda107f2df1bc43b202ac8b246f53f19e66a302 WatchSource:0}: Error finding container 6ebccb0ee400dad4473d3faf1fda107f2df1bc43b202ac8b246f53f19e66a302: Status 404 returned error can't find the container with id 6ebccb0ee400dad4473d3faf1fda107f2df1bc43b202ac8b246f53f19e66a302
Sep 11 03:56:15.597696 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:56:15.597657    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-explicit-459a4c/explicit-model-459a4c-kserve-d9cc98b77-j9nlz" event={"ID":"789f471b-9be6-4de4-acd3-9be9aa2b24f1","Type":"ContainerStarted","Data":"13390a77ead4b1ac796b3b8bef090141f72e500d3873e35762248e2c235fdd6b"}
Sep 11 03:56:15.598073 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:56:15.597704    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-explicit-459a4c/explicit-model-459a4c-kserve-d9cc98b77-j9nlz" event={"ID":"789f471b-9be6-4de4-acd3-9be9aa2b24f1","Type":"ContainerStarted","Data":"6ebccb0ee400dad4473d3faf1fda107f2df1bc43b202ac8b246f53f19e66a302"}
Sep 11 03:56:15.615584 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:56:15.615533    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-explicit-459a4c/explicit-model-459a4c-kserve-d9cc98b77-j9nlz" podStartSLOduration=1.6155171579999998 podStartE2EDuration="1.615517158s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 03:56:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 03:56:15.613706878 +0000 UTC m=+953.268912457" watchObservedRunningTime="2026-09-11 03:56:15.615517158 +0000 UTC m=+953.270722725"
Sep 11 03:56:16.674465 ip-10-0-141-105 kubenswrapper[2579]: E0911 03:56:16.674416    2579 secret.go:189] Couldn't get secret ai-tenant-e2e-explicit-459a4c/explicit-model-459a4c-kserve-self-signed-certs: secret "explicit-model-459a4c-kserve-self-signed-certs" not found
Sep 11 03:56:16.674922 ip-10-0-141-105 kubenswrapper[2579]: E0911 03:56:16.674503    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/789f471b-9be6-4de4-acd3-9be9aa2b24f1-tls-certs podName:789f471b-9be6-4de4-acd3-9be9aa2b24f1 nodeName:}" failed. No retries permitted until 2026-09-11 03:56:17.174479859 +0000 UTC m=+954.829685421 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/789f471b-9be6-4de4-acd3-9be9aa2b24f1-tls-certs") pod "explicit-model-459a4c-kserve-d9cc98b77-j9nlz" (UID: "789f471b-9be6-4de4-acd3-9be9aa2b24f1") : secret "explicit-model-459a4c-kserve-self-signed-certs" not found
Sep 11 03:56:16.677451 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:56:16.677427    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-explicit-459a4c/explicit-model-459a4c-kserve-d9cc98b77-j9nlz"]
Sep 11 03:56:16.800715 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:56:16.800685    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-459a4c-sttf4"]
Sep 11 03:56:16.804297 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:56:16.804281    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-459a4c-sttf4"
Sep 11 03:56:16.806668 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:56:16.806641    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 11 03:56:16.806831 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:56:16.806814    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-95ckr\""
Sep 11 03:56:16.810840 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:56:16.810802    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-459a4c-sttf4"]
Sep 11 03:56:16.875801 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:56:16.875770    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e2c052a7-eb78-499f-80bb-09921036c607-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-explicit-459a4c-sttf4\" (UID: \"e2c052a7-eb78-499f-80bb-09921036c607\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-459a4c-sttf4"
Sep 11 03:56:16.976639 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:56:16.976554    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e2c052a7-eb78-499f-80bb-09921036c607-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-explicit-459a4c-sttf4\" (UID: \"e2c052a7-eb78-499f-80bb-09921036c607\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-459a4c-sttf4"
Sep 11 03:56:16.977367 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:56:16.977347    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e2c052a7-eb78-499f-80bb-09921036c607-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-explicit-459a4c-sttf4\" (UID: \"e2c052a7-eb78-499f-80bb-09921036c607\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-459a4c-sttf4"
Sep 11 03:56:17.118064 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:56:17.118036    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-459a4c-sttf4"
Sep 11 03:56:17.200137 ip-10-0-141-105 kubenswrapper[2579]: E0911 03:56:17.199611    2579 secret.go:189] Couldn't get secret ai-tenant-e2e-explicit-459a4c/explicit-model-459a4c-kserve-self-signed-certs: secret "explicit-model-459a4c-kserve-self-signed-certs" not found
Sep 11 03:56:17.200137 ip-10-0-141-105 kubenswrapper[2579]: E0911 03:56:17.199697    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/789f471b-9be6-4de4-acd3-9be9aa2b24f1-tls-certs podName:789f471b-9be6-4de4-acd3-9be9aa2b24f1 nodeName:}" failed. No retries permitted until 2026-09-11 03:56:18.1996752 +0000 UTC m=+955.854880768 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/789f471b-9be6-4de4-acd3-9be9aa2b24f1-tls-certs") pod "explicit-model-459a4c-kserve-d9cc98b77-j9nlz" (UID: "789f471b-9be6-4de4-acd3-9be9aa2b24f1") : secret "explicit-model-459a4c-kserve-self-signed-certs" not found
Sep 11 03:56:17.398071 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:56:17.398045    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-459a4c-sttf4"]
Sep 11 03:56:17.400513 ip-10-0-141-105 kubenswrapper[2579]: W0911 03:56:17.400471    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode2c052a7_eb78_499f_80bb_09921036c607.slice/crio-62650ed910079577aedef1c5af2450dedceae3c5d8b690211ffac163604400d8 WatchSource:0}: Error finding container 62650ed910079577aedef1c5af2450dedceae3c5d8b690211ffac163604400d8: Status 404 returned error can't find the container with id 62650ed910079577aedef1c5af2450dedceae3c5d8b690211ffac163604400d8
Sep 11 03:56:17.608976 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:56:17.608937    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-459a4c-sttf4" event={"ID":"e2c052a7-eb78-499f-80bb-09921036c607","Type":"ContainerStarted","Data":"62650ed910079577aedef1c5af2450dedceae3c5d8b690211ffac163604400d8"}
Sep 11 03:56:18.210376 ip-10-0-141-105 kubenswrapper[2579]: E0911 03:56:18.210348    2579 secret.go:189] Couldn't get secret ai-tenant-e2e-explicit-459a4c/explicit-model-459a4c-kserve-self-signed-certs: secret "explicit-model-459a4c-kserve-self-signed-certs" not found
Sep 11 03:56:18.210746 ip-10-0-141-105 kubenswrapper[2579]: E0911 03:56:18.210412    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/789f471b-9be6-4de4-acd3-9be9aa2b24f1-tls-certs podName:789f471b-9be6-4de4-acd3-9be9aa2b24f1 nodeName:}" failed. No retries permitted until 2026-09-11 03:56:20.210397883 +0000 UTC m=+957.865603440 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/789f471b-9be6-4de4-acd3-9be9aa2b24f1-tls-certs") pod "explicit-model-459a4c-kserve-d9cc98b77-j9nlz" (UID: "789f471b-9be6-4de4-acd3-9be9aa2b24f1") : secret "explicit-model-459a4c-kserve-self-signed-certs" not found
Sep 11 03:56:18.615843 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:56:18.615812    2579 generic.go:356] "Generic (PLEG): container finished" podID="e2c052a7-eb78-499f-80bb-09921036c607" containerID="008205caba7da40850508c34cda38e9b5706e1fb6fd163092ff281a5dbc71034" exitCode=0
Sep 11 03:56:18.616024 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:56:18.615925    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-459a4c-sttf4" event={"ID":"e2c052a7-eb78-499f-80bb-09921036c607","Type":"ContainerDied","Data":"008205caba7da40850508c34cda38e9b5706e1fb6fd163092ff281a5dbc71034"}
Sep 11 03:56:19.754905 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:56:19.754879    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-459a4c-sttf4"
Sep 11 03:56:19.912736 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:56:19.912645    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-5z4cn"]
Sep 11 03:56:19.928251 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:56:19.928222    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e2c052a7-eb78-499f-80bb-09921036c607-maas-api-service-ca\") pod \"e2c052a7-eb78-499f-80bb-09921036c607\" (UID: \"e2c052a7-eb78-499f-80bb-09921036c607\") "
Sep 11 03:56:19.928590 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:56:19.928540    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e2c052a7-eb78-499f-80bb-09921036c607-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "e2c052a7-eb78-499f-80bb-09921036c607" (UID: "e2c052a7-eb78-499f-80bb-09921036c607"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 03:56:20.029206 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:56:20.029168    2579 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e2c052a7-eb78-499f-80bb-09921036c607-maas-api-service-ca\") on node \"ip-10-0-141-105.ec2.internal\" DevicePath \"\""
Sep 11 03:56:20.231674 ip-10-0-141-105 kubenswrapper[2579]: E0911 03:56:20.231593    2579 secret.go:189] Couldn't get secret ai-tenant-e2e-explicit-459a4c/explicit-model-459a4c-kserve-self-signed-certs: secret "explicit-model-459a4c-kserve-self-signed-certs" not found
Sep 11 03:56:20.231674 ip-10-0-141-105 kubenswrapper[2579]: E0911 03:56:20.231657    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/789f471b-9be6-4de4-acd3-9be9aa2b24f1-tls-certs podName:789f471b-9be6-4de4-acd3-9be9aa2b24f1 nodeName:}" failed. No retries permitted until 2026-09-11 03:56:24.231644231 +0000 UTC m=+961.886849775 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/789f471b-9be6-4de4-acd3-9be9aa2b24f1-tls-certs") pod "explicit-model-459a4c-kserve-d9cc98b77-j9nlz" (UID: "789f471b-9be6-4de4-acd3-9be9aa2b24f1") : secret "explicit-model-459a4c-kserve-self-signed-certs" not found
Sep 11 03:56:20.629311 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:56:20.629285    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-459a4c-sttf4"
Sep 11 03:56:20.629311 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:56:20.629292    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-459a4c-sttf4" event={"ID":"e2c052a7-eb78-499f-80bb-09921036c607","Type":"ContainerDied","Data":"62650ed910079577aedef1c5af2450dedceae3c5d8b690211ffac163604400d8"}
Sep 11 03:56:20.629525 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:56:20.629336    2579 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="62650ed910079577aedef1c5af2450dedceae3c5d8b690211ffac163604400d8"
Sep 11 03:56:24.267814 ip-10-0-141-105 kubenswrapper[2579]: E0911 03:56:24.267780    2579 secret.go:189] Couldn't get secret ai-tenant-e2e-explicit-459a4c/explicit-model-459a4c-kserve-self-signed-certs: secret "explicit-model-459a4c-kserve-self-signed-certs" not found
Sep 11 03:56:24.268218 ip-10-0-141-105 kubenswrapper[2579]: E0911 03:56:24.267853    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/789f471b-9be6-4de4-acd3-9be9aa2b24f1-tls-certs podName:789f471b-9be6-4de4-acd3-9be9aa2b24f1 nodeName:}" failed. No retries permitted until 2026-09-11 03:56:32.267839297 +0000 UTC m=+969.923044842 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/789f471b-9be6-4de4-acd3-9be9aa2b24f1-tls-certs") pod "explicit-model-459a4c-kserve-d9cc98b77-j9nlz" (UID: "789f471b-9be6-4de4-acd3-9be9aa2b24f1") : secret "explicit-model-459a4c-kserve-self-signed-certs" not found
Sep 11 03:56:24.517131 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:56:24.517059    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-explicit-459a4c/explicit-model-459a4c-kserve-d9cc98b77-j9nlz"
Sep 11 03:56:27.269648 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:56:27.269610    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-459a4c-f97c9d9d-24nbj"]
Sep 11 03:56:27.270141 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:56:27.269867    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-explicit-459a4c-f97c9d9d-24nbj" podUID="2e715f91-99db-45d9-ad51-c46cff03ff78" containerName="payload-pre-processing" containerID="cri-o://7f7af2b33acfe59b36d04e4a51f7c01c0b57473f28b6c1ba1b4be82c5e3add2e" gracePeriod=30
Sep 11 03:56:27.410870 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:56:27.410833    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-459a4c-sttf4"]
Sep 11 03:56:27.413315 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:56:27.413283    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-459a4c-sttf4"]
Sep 11 03:56:27.530742 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:56:27.530680    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-explicit-459a4c-f97c9d9d-24nbj"
Sep 11 03:56:27.598570 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:56:27.598536    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cgxb6\" (UniqueName: \"kubernetes.io/projected/2e715f91-99db-45d9-ad51-c46cff03ff78-kube-api-access-cgxb6\") pod \"2e715f91-99db-45d9-ad51-c46cff03ff78\" (UID: \"2e715f91-99db-45d9-ad51-c46cff03ff78\") "
Sep 11 03:56:27.598730 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:56:27.598587    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2e715f91-99db-45d9-ad51-c46cff03ff78-plugins-config-volume\") pod \"2e715f91-99db-45d9-ad51-c46cff03ff78\" (UID: \"2e715f91-99db-45d9-ad51-c46cff03ff78\") "
Sep 11 03:56:27.598958 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:56:27.598931    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2e715f91-99db-45d9-ad51-c46cff03ff78-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "2e715f91-99db-45d9-ad51-c46cff03ff78" (UID: "2e715f91-99db-45d9-ad51-c46cff03ff78"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 03:56:27.600750 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:56:27.600726    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2e715f91-99db-45d9-ad51-c46cff03ff78-kube-api-access-cgxb6" (OuterVolumeSpecName: "kube-api-access-cgxb6") pod "2e715f91-99db-45d9-ad51-c46cff03ff78" (UID: "2e715f91-99db-45d9-ad51-c46cff03ff78"). InnerVolumeSpecName "kube-api-access-cgxb6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 03:56:27.667561 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:56:27.667527    2579 generic.go:356] "Generic (PLEG): container finished" podID="2e715f91-99db-45d9-ad51-c46cff03ff78" containerID="7f7af2b33acfe59b36d04e4a51f7c01c0b57473f28b6c1ba1b4be82c5e3add2e" exitCode=0
Sep 11 03:56:27.667721 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:56:27.667574    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-explicit-459a4c-f97c9d9d-24nbj"
Sep 11 03:56:27.667721 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:56:27.667611    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-explicit-459a4c-f97c9d9d-24nbj" event={"ID":"2e715f91-99db-45d9-ad51-c46cff03ff78","Type":"ContainerDied","Data":"7f7af2b33acfe59b36d04e4a51f7c01c0b57473f28b6c1ba1b4be82c5e3add2e"}
Sep 11 03:56:27.667721 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:56:27.667665    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-explicit-459a4c-f97c9d9d-24nbj" event={"ID":"2e715f91-99db-45d9-ad51-c46cff03ff78","Type":"ContainerDied","Data":"424f74d3efd888ec7f243472b0c77c1d9f3359ac9aeabe248a4d83a77479c4f0"}
Sep 11 03:56:27.667721 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:56:27.667692    2579 scope.go:117] "RemoveContainer" containerID="7f7af2b33acfe59b36d04e4a51f7c01c0b57473f28b6c1ba1b4be82c5e3add2e"
Sep 11 03:56:27.677229 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:56:27.677211    2579 scope.go:117] "RemoveContainer" containerID="7f7af2b33acfe59b36d04e4a51f7c01c0b57473f28b6c1ba1b4be82c5e3add2e"
Sep 11 03:56:27.677523 ip-10-0-141-105 kubenswrapper[2579]: E0911 03:56:27.677504    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7f7af2b33acfe59b36d04e4a51f7c01c0b57473f28b6c1ba1b4be82c5e3add2e\": container with ID starting with 7f7af2b33acfe59b36d04e4a51f7c01c0b57473f28b6c1ba1b4be82c5e3add2e not found: ID does not exist" containerID="7f7af2b33acfe59b36d04e4a51f7c01c0b57473f28b6c1ba1b4be82c5e3add2e"
Sep 11 03:56:27.677588 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:56:27.677531    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7f7af2b33acfe59b36d04e4a51f7c01c0b57473f28b6c1ba1b4be82c5e3add2e"} err="failed to get container status \"7f7af2b33acfe59b36d04e4a51f7c01c0b57473f28b6c1ba1b4be82c5e3add2e\": rpc error: code = NotFound desc = could not find container \"7f7af2b33acfe59b36d04e4a51f7c01c0b57473f28b6c1ba1b4be82c5e3add2e\": container with ID starting with 7f7af2b33acfe59b36d04e4a51f7c01c0b57473f28b6c1ba1b4be82c5e3add2e not found: ID does not exist"
Sep 11 03:56:27.688631 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:56:27.688604    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-459a4c-f97c9d9d-24nbj"]
Sep 11 03:56:27.693219 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:56:27.693197    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-459a4c-f97c9d9d-24nbj"]
Sep 11 03:56:27.699960 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:56:27.699938    2579 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2e715f91-99db-45d9-ad51-c46cff03ff78-plugins-config-volume\") on node \"ip-10-0-141-105.ec2.internal\" DevicePath \"\""
Sep 11 03:56:27.700028 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:56:27.699969    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cgxb6\" (UniqueName: \"kubernetes.io/projected/2e715f91-99db-45d9-ad51-c46cff03ff78-kube-api-access-cgxb6\") on node \"ip-10-0-141-105.ec2.internal\" DevicePath \"\""
Sep 11 03:56:28.852292 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:56:28.852257    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2e715f91-99db-45d9-ad51-c46cff03ff78" path="/var/lib/kubelet/pods/2e715f91-99db-45d9-ad51-c46cff03ff78/volumes"
Sep 11 03:56:28.852680 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:56:28.852609    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e2c052a7-eb78-499f-80bb-09921036c607" path="/var/lib/kubelet/pods/e2c052a7-eb78-499f-80bb-09921036c607/volumes"
Sep 11 03:56:32.342902 ip-10-0-141-105 kubenswrapper[2579]: E0911 03:56:32.342868    2579 secret.go:189] Couldn't get secret ai-tenant-e2e-explicit-459a4c/explicit-model-459a4c-kserve-self-signed-certs: secret "explicit-model-459a4c-kserve-self-signed-certs" not found
Sep 11 03:56:32.343370 ip-10-0-141-105 kubenswrapper[2579]: E0911 03:56:32.342937    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/789f471b-9be6-4de4-acd3-9be9aa2b24f1-tls-certs podName:789f471b-9be6-4de4-acd3-9be9aa2b24f1 nodeName:}" failed. No retries permitted until 2026-09-11 03:56:48.342922349 +0000 UTC m=+985.998127894 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/789f471b-9be6-4de4-acd3-9be9aa2b24f1-tls-certs") pod "explicit-model-459a4c-kserve-d9cc98b77-j9nlz" (UID: "789f471b-9be6-4de4-acd3-9be9aa2b24f1") : secret "explicit-model-459a4c-kserve-self-signed-certs" not found
Sep 11 03:56:32.629440 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:56:32.629398    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-explicit-459a4c/explicit-model-459a4c-kserve-d9cc98b77-j9nlz" podUID="789f471b-9be6-4de4-acd3-9be9aa2b24f1" containerName="main" containerID="cri-o://13390a77ead4b1ac796b3b8bef090141f72e500d3873e35762248e2c235fdd6b" gracePeriod=45
Sep 11 03:56:32.896343 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:56:32.896310    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-explicit-459a4c/explicit-model-459a4c-kserve-d9cc98b77-j9nlz"
Sep 11 03:56:32.947893 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:56:32.947866    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/789f471b-9be6-4de4-acd3-9be9aa2b24f1-model-cache\") pod \"789f471b-9be6-4de4-acd3-9be9aa2b24f1\" (UID: \"789f471b-9be6-4de4-acd3-9be9aa2b24f1\") "
Sep 11 03:56:32.948066 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:56:32.947919    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/789f471b-9be6-4de4-acd3-9be9aa2b24f1-tmp-dir\") pod \"789f471b-9be6-4de4-acd3-9be9aa2b24f1\" (UID: \"789f471b-9be6-4de4-acd3-9be9aa2b24f1\") "
Sep 11 03:56:32.948066 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:56:32.947950    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4sr6b\" (UniqueName: \"kubernetes.io/projected/789f471b-9be6-4de4-acd3-9be9aa2b24f1-kube-api-access-4sr6b\") pod \"789f471b-9be6-4de4-acd3-9be9aa2b24f1\" (UID: \"789f471b-9be6-4de4-acd3-9be9aa2b24f1\") "
Sep 11 03:56:32.948066 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:56:32.947980    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/789f471b-9be6-4de4-acd3-9be9aa2b24f1-tls-certs\") pod \"789f471b-9be6-4de4-acd3-9be9aa2b24f1\" (UID: \"789f471b-9be6-4de4-acd3-9be9aa2b24f1\") "
Sep 11 03:56:32.948066 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:56:32.948010    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/789f471b-9be6-4de4-acd3-9be9aa2b24f1-home\") pod \"789f471b-9be6-4de4-acd3-9be9aa2b24f1\" (UID: \"789f471b-9be6-4de4-acd3-9be9aa2b24f1\") "
Sep 11 03:56:32.948340 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:56:32.948120    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/789f471b-9be6-4de4-acd3-9be9aa2b24f1-dshm\") pod \"789f471b-9be6-4de4-acd3-9be9aa2b24f1\" (UID: \"789f471b-9be6-4de4-acd3-9be9aa2b24f1\") "
Sep 11 03:56:32.948340 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:56:32.948195    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/789f471b-9be6-4de4-acd3-9be9aa2b24f1-model-cache" (OuterVolumeSpecName: "model-cache") pod "789f471b-9be6-4de4-acd3-9be9aa2b24f1" (UID: "789f471b-9be6-4de4-acd3-9be9aa2b24f1"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 03:56:32.948451 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:56:32.948341    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/789f471b-9be6-4de4-acd3-9be9aa2b24f1-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "789f471b-9be6-4de4-acd3-9be9aa2b24f1" (UID: "789f471b-9be6-4de4-acd3-9be9aa2b24f1"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 03:56:32.948541 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:56:32.948509    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/789f471b-9be6-4de4-acd3-9be9aa2b24f1-home" (OuterVolumeSpecName: "home") pod "789f471b-9be6-4de4-acd3-9be9aa2b24f1" (UID: "789f471b-9be6-4de4-acd3-9be9aa2b24f1"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 03:56:32.948867 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:56:32.948762    2579 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/789f471b-9be6-4de4-acd3-9be9aa2b24f1-model-cache\") on node \"ip-10-0-141-105.ec2.internal\" DevicePath \"\""
Sep 11 03:56:32.948982 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:56:32.948873    2579 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/789f471b-9be6-4de4-acd3-9be9aa2b24f1-tmp-dir\") on node \"ip-10-0-141-105.ec2.internal\" DevicePath \"\""
Sep 11 03:56:32.948982 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:56:32.948887    2579 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/789f471b-9be6-4de4-acd3-9be9aa2b24f1-home\") on node \"ip-10-0-141-105.ec2.internal\" DevicePath \"\""
Sep 11 03:56:32.950535 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:56:32.950512    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/789f471b-9be6-4de4-acd3-9be9aa2b24f1-dshm" (OuterVolumeSpecName: "dshm") pod "789f471b-9be6-4de4-acd3-9be9aa2b24f1" (UID: "789f471b-9be6-4de4-acd3-9be9aa2b24f1"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 03:56:32.950535 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:56:32.950524    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/789f471b-9be6-4de4-acd3-9be9aa2b24f1-kube-api-access-4sr6b" (OuterVolumeSpecName: "kube-api-access-4sr6b") pod "789f471b-9be6-4de4-acd3-9be9aa2b24f1" (UID: "789f471b-9be6-4de4-acd3-9be9aa2b24f1"). InnerVolumeSpecName "kube-api-access-4sr6b". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 03:56:32.950822 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:56:32.950801    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/789f471b-9be6-4de4-acd3-9be9aa2b24f1-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "789f471b-9be6-4de4-acd3-9be9aa2b24f1" (UID: "789f471b-9be6-4de4-acd3-9be9aa2b24f1"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 03:56:33.050149 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:56:33.050122    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4sr6b\" (UniqueName: \"kubernetes.io/projected/789f471b-9be6-4de4-acd3-9be9aa2b24f1-kube-api-access-4sr6b\") on node \"ip-10-0-141-105.ec2.internal\" DevicePath \"\""
Sep 11 03:56:33.050149 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:56:33.050145    2579 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/789f471b-9be6-4de4-acd3-9be9aa2b24f1-tls-certs\") on node \"ip-10-0-141-105.ec2.internal\" DevicePath \"\""
Sep 11 03:56:33.050149 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:56:33.050153    2579 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/789f471b-9be6-4de4-acd3-9be9aa2b24f1-dshm\") on node \"ip-10-0-141-105.ec2.internal\" DevicePath \"\""
Sep 11 03:56:33.702638 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:56:33.702598    2579 generic.go:356] "Generic (PLEG): container finished" podID="789f471b-9be6-4de4-acd3-9be9aa2b24f1" containerID="13390a77ead4b1ac796b3b8bef090141f72e500d3873e35762248e2c235fdd6b" exitCode=0
Sep 11 03:56:33.703020 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:56:33.702644    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-explicit-459a4c/explicit-model-459a4c-kserve-d9cc98b77-j9nlz"
Sep 11 03:56:33.703020 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:56:33.702687    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-explicit-459a4c/explicit-model-459a4c-kserve-d9cc98b77-j9nlz" event={"ID":"789f471b-9be6-4de4-acd3-9be9aa2b24f1","Type":"ContainerDied","Data":"13390a77ead4b1ac796b3b8bef090141f72e500d3873e35762248e2c235fdd6b"}
Sep 11 03:56:33.703020 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:56:33.702738    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-explicit-459a4c/explicit-model-459a4c-kserve-d9cc98b77-j9nlz" event={"ID":"789f471b-9be6-4de4-acd3-9be9aa2b24f1","Type":"ContainerDied","Data":"6ebccb0ee400dad4473d3faf1fda107f2df1bc43b202ac8b246f53f19e66a302"}
Sep 11 03:56:33.703020 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:56:33.702755    2579 scope.go:117] "RemoveContainer" containerID="13390a77ead4b1ac796b3b8bef090141f72e500d3873e35762248e2c235fdd6b"
Sep 11 03:56:33.712669 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:56:33.712650    2579 scope.go:117] "RemoveContainer" containerID="13390a77ead4b1ac796b3b8bef090141f72e500d3873e35762248e2c235fdd6b"
Sep 11 03:56:33.712968 ip-10-0-141-105 kubenswrapper[2579]: E0911 03:56:33.712947    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"13390a77ead4b1ac796b3b8bef090141f72e500d3873e35762248e2c235fdd6b\": container with ID starting with 13390a77ead4b1ac796b3b8bef090141f72e500d3873e35762248e2c235fdd6b not found: ID does not exist" containerID="13390a77ead4b1ac796b3b8bef090141f72e500d3873e35762248e2c235fdd6b"
Sep 11 03:56:33.713015 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:56:33.712978    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"13390a77ead4b1ac796b3b8bef090141f72e500d3873e35762248e2c235fdd6b"} err="failed to get container status \"13390a77ead4b1ac796b3b8bef090141f72e500d3873e35762248e2c235fdd6b\": rpc error: code = NotFound desc = could not find container \"13390a77ead4b1ac796b3b8bef090141f72e500d3873e35762248e2c235fdd6b\": container with ID starting with 13390a77ead4b1ac796b3b8bef090141f72e500d3873e35762248e2c235fdd6b not found: ID does not exist"
Sep 11 03:56:33.723688 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:56:33.723663    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-explicit-459a4c/explicit-model-459a4c-kserve-d9cc98b77-j9nlz"]
Sep 11 03:56:33.729312 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:56:33.729287    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-explicit-459a4c/explicit-model-459a4c-kserve-d9cc98b77-j9nlz"]
Sep 11 03:56:34.002410 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:56:34.002326    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-5z4cn"]
Sep 11 03:56:34.852811 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:56:34.852771    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="789f471b-9be6-4de4-acd3-9be9aa2b24f1" path="/var/lib/kubelet/pods/789f471b-9be6-4de4-acd3-9be9aa2b24f1/volumes"
Sep 11 03:56:35.600125 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:56:35.600073    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-4c62f1/test-model-4c62f1-kserve-69cb997789-bgf58"]
Sep 11 03:56:36.678408 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:56:36.678372    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-8efac4/test-model-8efac4-kserve-cb9995899-sxkk6"]
Sep 11 03:56:36.807486 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:56:36.807453    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-4c62f1-dsg4z"]
Sep 11 03:56:36.807933 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:56:36.807916    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="789f471b-9be6-4de4-acd3-9be9aa2b24f1" containerName="main"
Sep 11 03:56:36.807933 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:56:36.807935    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="789f471b-9be6-4de4-acd3-9be9aa2b24f1" containerName="main"
Sep 11 03:56:36.808062 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:56:36.807958    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e2c052a7-eb78-499f-80bb-09921036c607" containerName="revoke-keys"
Sep 11 03:56:36.808062 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:56:36.807964    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="e2c052a7-eb78-499f-80bb-09921036c607" containerName="revoke-keys"
Sep 11 03:56:36.808062 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:56:36.807971    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2e715f91-99db-45d9-ad51-c46cff03ff78" containerName="payload-pre-processing"
Sep 11 03:56:36.808062 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:56:36.807977    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="2e715f91-99db-45d9-ad51-c46cff03ff78" containerName="payload-pre-processing"
Sep 11 03:56:36.808062 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:56:36.808040    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="e2c052a7-eb78-499f-80bb-09921036c607" containerName="revoke-keys"
Sep 11 03:56:36.808062 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:56:36.808049    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="2e715f91-99db-45d9-ad51-c46cff03ff78" containerName="payload-pre-processing"
Sep 11 03:56:36.808062 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:56:36.808056    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="789f471b-9be6-4de4-acd3-9be9aa2b24f1" containerName="main"
Sep 11 03:56:36.812342 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:56:36.812325    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-4c62f1-dsg4z"
Sep 11 03:56:36.814771 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:56:36.814743    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-95ckr\""
Sep 11 03:56:36.815198 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:56:36.814754    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 11 03:56:36.817435 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:56:36.817411    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-4c62f1-dsg4z"]
Sep 11 03:56:36.886943 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:56:36.886903    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f12ac46f-f13a-4ba5-a0c5-99b5069c8632-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-4c62f1-dsg4z\" (UID: \"f12ac46f-f13a-4ba5-a0c5-99b5069c8632\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-4c62f1-dsg4z"
Sep 11 03:56:36.988567 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:56:36.988442    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f12ac46f-f13a-4ba5-a0c5-99b5069c8632-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-4c62f1-dsg4z\" (UID: \"f12ac46f-f13a-4ba5-a0c5-99b5069c8632\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-4c62f1-dsg4z"
Sep 11 03:56:36.989075 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:56:36.989051    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f12ac46f-f13a-4ba5-a0c5-99b5069c8632-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-4c62f1-dsg4z\" (UID: \"f12ac46f-f13a-4ba5-a0c5-99b5069c8632\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-4c62f1-dsg4z"
Sep 11 03:56:37.124479 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:56:37.124450    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-4c62f1-dsg4z"
Sep 11 03:56:37.396915 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:56:37.396886    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-4c62f1-dsg4z"]
Sep 11 03:56:37.399121 ip-10-0-141-105 kubenswrapper[2579]: W0911 03:56:37.399065    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf12ac46f_f13a_4ba5_a0c5_99b5069c8632.slice/crio-26d52794d51577f6a88d4baac7e43ea7ef99d4b2153437c11de1e1db5eb74034 WatchSource:0}: Error finding container 26d52794d51577f6a88d4baac7e43ea7ef99d4b2153437c11de1e1db5eb74034: Status 404 returned error can't find the container with id 26d52794d51577f6a88d4baac7e43ea7ef99d4b2153437c11de1e1db5eb74034
Sep 11 03:56:37.727569 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:56:37.727483    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-4c62f1-dsg4z" event={"ID":"f12ac46f-f13a-4ba5-a0c5-99b5069c8632","Type":"ContainerStarted","Data":"26d52794d51577f6a88d4baac7e43ea7ef99d4b2153437c11de1e1db5eb74034"}
Sep 11 03:56:38.733862 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:56:38.733814    2579 generic.go:356] "Generic (PLEG): container finished" podID="f12ac46f-f13a-4ba5-a0c5-99b5069c8632" containerID="1fa514b2f3408a13c8a5510758d14341d95a03f16b023a0cb282bed1d82c4cf5" exitCode=0
Sep 11 03:56:38.734284 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:56:38.733903    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-4c62f1-dsg4z" event={"ID":"f12ac46f-f13a-4ba5-a0c5-99b5069c8632","Type":"ContainerDied","Data":"1fa514b2f3408a13c8a5510758d14341d95a03f16b023a0cb282bed1d82c4cf5"}
Sep 11 03:56:39.889485 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:56:39.889454    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-4c62f1-dsg4z"
Sep 11 03:56:40.015257 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:56:40.015172    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f12ac46f-f13a-4ba5-a0c5-99b5069c8632-maas-api-service-ca\") pod \"f12ac46f-f13a-4ba5-a0c5-99b5069c8632\" (UID: \"f12ac46f-f13a-4ba5-a0c5-99b5069c8632\") "
Sep 11 03:56:40.015545 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:56:40.015520    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f12ac46f-f13a-4ba5-a0c5-99b5069c8632-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "f12ac46f-f13a-4ba5-a0c5-99b5069c8632" (UID: "f12ac46f-f13a-4ba5-a0c5-99b5069c8632"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 03:56:40.116675 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:56:40.116634    2579 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f12ac46f-f13a-4ba5-a0c5-99b5069c8632-maas-api-service-ca\") on node \"ip-10-0-141-105.ec2.internal\" DevicePath \"\""
Sep 11 03:56:40.746917 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:56:40.746887    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-4c62f1-dsg4z"
Sep 11 03:56:40.747102 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:56:40.746887    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-4c62f1-dsg4z" event={"ID":"f12ac46f-f13a-4ba5-a0c5-99b5069c8632","Type":"ContainerDied","Data":"26d52794d51577f6a88d4baac7e43ea7ef99d4b2153437c11de1e1db5eb74034"}
Sep 11 03:56:40.747102 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:56:40.747011    2579 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="26d52794d51577f6a88d4baac7e43ea7ef99d4b2153437c11de1e1db5eb74034"
Sep 11 03:56:46.196160 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:56:46.196063    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-5z4cn"]
Sep 11 03:56:47.343461 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:56:47.343422    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-4c62f1-6894c46554-w4b22"]
Sep 11 03:56:47.343994 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:56:47.343650    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-4c62f1-6894c46554-w4b22" podUID="1199d453-904f-4a53-aeed-f424a04a0ec6" containerName="payload-pre-processing" containerID="cri-o://759e73c43e1726a6049cc3ecd688682f76a82831259cfc94727a25a5313b620f" gracePeriod=30
Sep 11 03:56:47.438413 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:56:47.438370    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-4c62f1-6894c46554-w4b22" podUID="1199d453-904f-4a53-aeed-f424a04a0ec6" containerName="payload-pre-processing" probeResult="failure" output="dial tcp 10.134.0.37:9004: connect: connection refused"
Sep 11 03:56:47.480653 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:56:47.480602    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-4c62f1-dsg4z"]
Sep 11 03:56:47.482500 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:56:47.482473    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-4c62f1-dsg4z"]
Sep 11 03:56:47.599919 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:56:47.599856    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-4c62f1-6894c46554-w4b22"
Sep 11 03:56:47.682116 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:56:47.682045    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fxklz\" (UniqueName: \"kubernetes.io/projected/1199d453-904f-4a53-aeed-f424a04a0ec6-kube-api-access-fxklz\") pod \"1199d453-904f-4a53-aeed-f424a04a0ec6\" (UID: \"1199d453-904f-4a53-aeed-f424a04a0ec6\") "
Sep 11 03:56:47.682284 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:56:47.682164    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1199d453-904f-4a53-aeed-f424a04a0ec6-plugins-config-volume\") pod \"1199d453-904f-4a53-aeed-f424a04a0ec6\" (UID: \"1199d453-904f-4a53-aeed-f424a04a0ec6\") "
Sep 11 03:56:47.682509 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:56:47.682488    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1199d453-904f-4a53-aeed-f424a04a0ec6-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "1199d453-904f-4a53-aeed-f424a04a0ec6" (UID: "1199d453-904f-4a53-aeed-f424a04a0ec6"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 03:56:47.684256 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:56:47.684232    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1199d453-904f-4a53-aeed-f424a04a0ec6-kube-api-access-fxklz" (OuterVolumeSpecName: "kube-api-access-fxklz") pod "1199d453-904f-4a53-aeed-f424a04a0ec6" (UID: "1199d453-904f-4a53-aeed-f424a04a0ec6"). InnerVolumeSpecName "kube-api-access-fxklz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 03:56:47.782963 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:56:47.782935    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fxklz\" (UniqueName: \"kubernetes.io/projected/1199d453-904f-4a53-aeed-f424a04a0ec6-kube-api-access-fxklz\") on node \"ip-10-0-141-105.ec2.internal\" DevicePath \"\""
Sep 11 03:56:47.782963 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:56:47.782959    2579 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1199d453-904f-4a53-aeed-f424a04a0ec6-plugins-config-volume\") on node \"ip-10-0-141-105.ec2.internal\" DevicePath \"\""
Sep 11 03:56:47.792752 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:56:47.792722    2579 generic.go:356] "Generic (PLEG): container finished" podID="1199d453-904f-4a53-aeed-f424a04a0ec6" containerID="759e73c43e1726a6049cc3ecd688682f76a82831259cfc94727a25a5313b620f" exitCode=0
Sep 11 03:56:47.792900 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:56:47.792776    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-4c62f1-6894c46554-w4b22"
Sep 11 03:56:47.792900 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:56:47.792811    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-4c62f1-6894c46554-w4b22" event={"ID":"1199d453-904f-4a53-aeed-f424a04a0ec6","Type":"ContainerDied","Data":"759e73c43e1726a6049cc3ecd688682f76a82831259cfc94727a25a5313b620f"}
Sep 11 03:56:47.792900 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:56:47.792881    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-4c62f1-6894c46554-w4b22" event={"ID":"1199d453-904f-4a53-aeed-f424a04a0ec6","Type":"ContainerDied","Data":"162ca54375aeceffcbfbe0d0a7cf9177d284594b1200f3d0963c3e5eeb4ecb41"}
Sep 11 03:56:47.793011 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:56:47.792902    2579 scope.go:117] "RemoveContainer" containerID="759e73c43e1726a6049cc3ecd688682f76a82831259cfc94727a25a5313b620f"
Sep 11 03:56:47.802891 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:56:47.802871    2579 scope.go:117] "RemoveContainer" containerID="759e73c43e1726a6049cc3ecd688682f76a82831259cfc94727a25a5313b620f"
Sep 11 03:56:47.803191 ip-10-0-141-105 kubenswrapper[2579]: E0911 03:56:47.803168    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"759e73c43e1726a6049cc3ecd688682f76a82831259cfc94727a25a5313b620f\": container with ID starting with 759e73c43e1726a6049cc3ecd688682f76a82831259cfc94727a25a5313b620f not found: ID does not exist" containerID="759e73c43e1726a6049cc3ecd688682f76a82831259cfc94727a25a5313b620f"
Sep 11 03:56:47.803272 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:56:47.803199    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"759e73c43e1726a6049cc3ecd688682f76a82831259cfc94727a25a5313b620f"} err="failed to get container status \"759e73c43e1726a6049cc3ecd688682f76a82831259cfc94727a25a5313b620f\": rpc error: code = NotFound desc = could not find container \"759e73c43e1726a6049cc3ecd688682f76a82831259cfc94727a25a5313b620f\": container with ID starting with 759e73c43e1726a6049cc3ecd688682f76a82831259cfc94727a25a5313b620f not found: ID does not exist"
Sep 11 03:56:47.813830 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:56:47.813803    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-4c62f1-6894c46554-w4b22"]
Sep 11 03:56:47.816814 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:56:47.816770    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-4c62f1-6894c46554-w4b22"]
Sep 11 03:56:48.852509 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:56:48.852473    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1199d453-904f-4a53-aeed-f424a04a0ec6" path="/var/lib/kubelet/pods/1199d453-904f-4a53-aeed-f424a04a0ec6/volumes"
Sep 11 03:56:48.852898 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:56:48.852856    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f12ac46f-f13a-4ba5-a0c5-99b5069c8632" path="/var/lib/kubelet/pods/f12ac46f-f13a-4ba5-a0c5-99b5069c8632/volumes"
Sep 11 03:56:50.620278 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:56:50.620221    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-inf-a-4c62f1/test-model-4c62f1-kserve-69cb997789-bgf58" podUID="6a03b2ee-3de9-4905-8cc9-b9c1e3a5bfb5" containerName="main" containerID="cri-o://add0cd12901d26602b22b7fb9ac23dde287a9c99610caf185129cbbff7c94196" gracePeriod=45
Sep 11 03:56:50.813459 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:56:50.813427    2579 generic.go:356] "Generic (PLEG): container finished" podID="6a03b2ee-3de9-4905-8cc9-b9c1e3a5bfb5" containerID="add0cd12901d26602b22b7fb9ac23dde287a9c99610caf185129cbbff7c94196" exitCode=0
Sep 11 03:56:50.813621 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:56:50.813499    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-4c62f1/test-model-4c62f1-kserve-69cb997789-bgf58" event={"ID":"6a03b2ee-3de9-4905-8cc9-b9c1e3a5bfb5","Type":"ContainerDied","Data":"add0cd12901d26602b22b7fb9ac23dde287a9c99610caf185129cbbff7c94196"}
Sep 11 03:56:50.875433 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:56:50.875370    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-4c62f1/test-model-4c62f1-kserve-69cb997789-bgf58"
Sep 11 03:56:51.015228 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:56:51.015198    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cxnt6\" (UniqueName: \"kubernetes.io/projected/6a03b2ee-3de9-4905-8cc9-b9c1e3a5bfb5-kube-api-access-cxnt6\") pod \"6a03b2ee-3de9-4905-8cc9-b9c1e3a5bfb5\" (UID: \"6a03b2ee-3de9-4905-8cc9-b9c1e3a5bfb5\") "
Sep 11 03:56:51.015407 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:56:51.015241    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6a03b2ee-3de9-4905-8cc9-b9c1e3a5bfb5-tmp-dir\") pod \"6a03b2ee-3de9-4905-8cc9-b9c1e3a5bfb5\" (UID: \"6a03b2ee-3de9-4905-8cc9-b9c1e3a5bfb5\") "
Sep 11 03:56:51.015407 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:56:51.015259    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6a03b2ee-3de9-4905-8cc9-b9c1e3a5bfb5-dshm\") pod \"6a03b2ee-3de9-4905-8cc9-b9c1e3a5bfb5\" (UID: \"6a03b2ee-3de9-4905-8cc9-b9c1e3a5bfb5\") "
Sep 11 03:56:51.015407 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:56:51.015278    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6a03b2ee-3de9-4905-8cc9-b9c1e3a5bfb5-model-cache\") pod \"6a03b2ee-3de9-4905-8cc9-b9c1e3a5bfb5\" (UID: \"6a03b2ee-3de9-4905-8cc9-b9c1e3a5bfb5\") "
Sep 11 03:56:51.015407 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:56:51.015310    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6a03b2ee-3de9-4905-8cc9-b9c1e3a5bfb5-tls-certs\") pod \"6a03b2ee-3de9-4905-8cc9-b9c1e3a5bfb5\" (UID: \"6a03b2ee-3de9-4905-8cc9-b9c1e3a5bfb5\") "
Sep 11 03:56:51.015407 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:56:51.015328    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6a03b2ee-3de9-4905-8cc9-b9c1e3a5bfb5-home\") pod \"6a03b2ee-3de9-4905-8cc9-b9c1e3a5bfb5\" (UID: \"6a03b2ee-3de9-4905-8cc9-b9c1e3a5bfb5\") "
Sep 11 03:56:51.015666 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:56:51.015543    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6a03b2ee-3de9-4905-8cc9-b9c1e3a5bfb5-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "6a03b2ee-3de9-4905-8cc9-b9c1e3a5bfb5" (UID: "6a03b2ee-3de9-4905-8cc9-b9c1e3a5bfb5"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 03:56:51.015666 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:56:51.015631    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6a03b2ee-3de9-4905-8cc9-b9c1e3a5bfb5-model-cache" (OuterVolumeSpecName: "model-cache") pod "6a03b2ee-3de9-4905-8cc9-b9c1e3a5bfb5" (UID: "6a03b2ee-3de9-4905-8cc9-b9c1e3a5bfb5"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 03:56:51.015780 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:56:51.015737    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6a03b2ee-3de9-4905-8cc9-b9c1e3a5bfb5-home" (OuterVolumeSpecName: "home") pod "6a03b2ee-3de9-4905-8cc9-b9c1e3a5bfb5" (UID: "6a03b2ee-3de9-4905-8cc9-b9c1e3a5bfb5"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 03:56:51.017666 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:56:51.017640    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6a03b2ee-3de9-4905-8cc9-b9c1e3a5bfb5-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "6a03b2ee-3de9-4905-8cc9-b9c1e3a5bfb5" (UID: "6a03b2ee-3de9-4905-8cc9-b9c1e3a5bfb5"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 03:56:51.017779 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:56:51.017748    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6a03b2ee-3de9-4905-8cc9-b9c1e3a5bfb5-kube-api-access-cxnt6" (OuterVolumeSpecName: "kube-api-access-cxnt6") pod "6a03b2ee-3de9-4905-8cc9-b9c1e3a5bfb5" (UID: "6a03b2ee-3de9-4905-8cc9-b9c1e3a5bfb5"). InnerVolumeSpecName "kube-api-access-cxnt6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 03:56:51.017944 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:56:51.017930    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6a03b2ee-3de9-4905-8cc9-b9c1e3a5bfb5-dshm" (OuterVolumeSpecName: "dshm") pod "6a03b2ee-3de9-4905-8cc9-b9c1e3a5bfb5" (UID: "6a03b2ee-3de9-4905-8cc9-b9c1e3a5bfb5"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 03:56:51.116821 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:56:51.116785    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cxnt6\" (UniqueName: \"kubernetes.io/projected/6a03b2ee-3de9-4905-8cc9-b9c1e3a5bfb5-kube-api-access-cxnt6\") on node \"ip-10-0-141-105.ec2.internal\" DevicePath \"\""
Sep 11 03:56:51.116821 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:56:51.116815    2579 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6a03b2ee-3de9-4905-8cc9-b9c1e3a5bfb5-tmp-dir\") on node \"ip-10-0-141-105.ec2.internal\" DevicePath \"\""
Sep 11 03:56:51.116821 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:56:51.116826    2579 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6a03b2ee-3de9-4905-8cc9-b9c1e3a5bfb5-dshm\") on node \"ip-10-0-141-105.ec2.internal\" DevicePath \"\""
Sep 11 03:56:51.117046 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:56:51.116835    2579 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6a03b2ee-3de9-4905-8cc9-b9c1e3a5bfb5-model-cache\") on node \"ip-10-0-141-105.ec2.internal\" DevicePath \"\""
Sep 11 03:56:51.117046 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:56:51.116843    2579 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6a03b2ee-3de9-4905-8cc9-b9c1e3a5bfb5-tls-certs\") on node \"ip-10-0-141-105.ec2.internal\" DevicePath \"\""
Sep 11 03:56:51.117046 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:56:51.116850    2579 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6a03b2ee-3de9-4905-8cc9-b9c1e3a5bfb5-home\") on node \"ip-10-0-141-105.ec2.internal\" DevicePath \"\""
Sep 11 03:56:51.700650 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:56:51.700603    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-inf-b-8efac4/test-model-8efac4-kserve-cb9995899-sxkk6" podUID="3faa005d-de0c-4c9c-9de6-035473a89077" containerName="main" containerID="cri-o://6d34dd50a2612144251ddb12ce7473a02978b54dcb14ae2f94e17815b16044c7" gracePeriod=45
Sep 11 03:56:51.821098 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:56:51.821057    2579 generic.go:356] "Generic (PLEG): container finished" podID="3faa005d-de0c-4c9c-9de6-035473a89077" containerID="6d34dd50a2612144251ddb12ce7473a02978b54dcb14ae2f94e17815b16044c7" exitCode=0
Sep 11 03:56:51.821224 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:56:51.821116    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-8efac4/test-model-8efac4-kserve-cb9995899-sxkk6" event={"ID":"3faa005d-de0c-4c9c-9de6-035473a89077","Type":"ContainerDied","Data":"6d34dd50a2612144251ddb12ce7473a02978b54dcb14ae2f94e17815b16044c7"}
Sep 11 03:56:51.822703 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:56:51.822681    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-4c62f1/test-model-4c62f1-kserve-69cb997789-bgf58" event={"ID":"6a03b2ee-3de9-4905-8cc9-b9c1e3a5bfb5","Type":"ContainerDied","Data":"843c7a2f6c06fbb85d219d1ac849e3c4d0f49c556ca76cbf05c81b3cd4276e82"}
Sep 11 03:56:51.822821 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:56:51.822725    2579 scope.go:117] "RemoveContainer" containerID="add0cd12901d26602b22b7fb9ac23dde287a9c99610caf185129cbbff7c94196"
Sep 11 03:56:51.822821 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:56:51.822725    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-4c62f1/test-model-4c62f1-kserve-69cb997789-bgf58"
Sep 11 03:56:51.843314 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:56:51.843286    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-4c62f1/test-model-4c62f1-kserve-69cb997789-bgf58"]
Sep 11 03:56:51.845063 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:56:51.845039    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-a-4c62f1/test-model-4c62f1-kserve-69cb997789-bgf58"]
Sep 11 03:56:51.964757 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:56:51.964696    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-8efac4/test-model-8efac4-kserve-cb9995899-sxkk6"
Sep 11 03:56:52.126261 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:56:52.126235    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3faa005d-de0c-4c9c-9de6-035473a89077-tls-certs\") pod \"3faa005d-de0c-4c9c-9de6-035473a89077\" (UID: \"3faa005d-de0c-4c9c-9de6-035473a89077\") "
Sep 11 03:56:52.126437 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:56:52.126321    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-95f9w\" (UniqueName: \"kubernetes.io/projected/3faa005d-de0c-4c9c-9de6-035473a89077-kube-api-access-95f9w\") pod \"3faa005d-de0c-4c9c-9de6-035473a89077\" (UID: \"3faa005d-de0c-4c9c-9de6-035473a89077\") "
Sep 11 03:56:52.126437 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:56:52.126344    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3faa005d-de0c-4c9c-9de6-035473a89077-model-cache\") pod \"3faa005d-de0c-4c9c-9de6-035473a89077\" (UID: \"3faa005d-de0c-4c9c-9de6-035473a89077\") "
Sep 11 03:56:52.126437 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:56:52.126365    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3faa005d-de0c-4c9c-9de6-035473a89077-dshm\") pod \"3faa005d-de0c-4c9c-9de6-035473a89077\" (UID: \"3faa005d-de0c-4c9c-9de6-035473a89077\") "
Sep 11 03:56:52.126437 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:56:52.126384    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3faa005d-de0c-4c9c-9de6-035473a89077-home\") pod \"3faa005d-de0c-4c9c-9de6-035473a89077\" (UID: \"3faa005d-de0c-4c9c-9de6-035473a89077\") "
Sep 11 03:56:52.126437 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:56:52.126437    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3faa005d-de0c-4c9c-9de6-035473a89077-tmp-dir\") pod \"3faa005d-de0c-4c9c-9de6-035473a89077\" (UID: \"3faa005d-de0c-4c9c-9de6-035473a89077\") "
Sep 11 03:56:52.126708 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:56:52.126591    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3faa005d-de0c-4c9c-9de6-035473a89077-model-cache" (OuterVolumeSpecName: "model-cache") pod "3faa005d-de0c-4c9c-9de6-035473a89077" (UID: "3faa005d-de0c-4c9c-9de6-035473a89077"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 03:56:52.126708 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:56:52.126697    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3faa005d-de0c-4c9c-9de6-035473a89077-home" (OuterVolumeSpecName: "home") pod "3faa005d-de0c-4c9c-9de6-035473a89077" (UID: "3faa005d-de0c-4c9c-9de6-035473a89077"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 03:56:52.126816 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:56:52.126727    2579 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3faa005d-de0c-4c9c-9de6-035473a89077-model-cache\") on node \"ip-10-0-141-105.ec2.internal\" DevicePath \"\""
Sep 11 03:56:52.126868 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:56:52.126835    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3faa005d-de0c-4c9c-9de6-035473a89077-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "3faa005d-de0c-4c9c-9de6-035473a89077" (UID: "3faa005d-de0c-4c9c-9de6-035473a89077"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 03:56:52.128655 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:56:52.128630    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3faa005d-de0c-4c9c-9de6-035473a89077-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "3faa005d-de0c-4c9c-9de6-035473a89077" (UID: "3faa005d-de0c-4c9c-9de6-035473a89077"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 03:56:52.128801 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:56:52.128702    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3faa005d-de0c-4c9c-9de6-035473a89077-dshm" (OuterVolumeSpecName: "dshm") pod "3faa005d-de0c-4c9c-9de6-035473a89077" (UID: "3faa005d-de0c-4c9c-9de6-035473a89077"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 03:56:52.128801 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:56:52.128747    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3faa005d-de0c-4c9c-9de6-035473a89077-kube-api-access-95f9w" (OuterVolumeSpecName: "kube-api-access-95f9w") pod "3faa005d-de0c-4c9c-9de6-035473a89077" (UID: "3faa005d-de0c-4c9c-9de6-035473a89077"). InnerVolumeSpecName "kube-api-access-95f9w". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 03:56:52.227562 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:56:52.227473    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-95f9w\" (UniqueName: \"kubernetes.io/projected/3faa005d-de0c-4c9c-9de6-035473a89077-kube-api-access-95f9w\") on node \"ip-10-0-141-105.ec2.internal\" DevicePath \"\""
Sep 11 03:56:52.227562 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:56:52.227503    2579 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3faa005d-de0c-4c9c-9de6-035473a89077-dshm\") on node \"ip-10-0-141-105.ec2.internal\" DevicePath \"\""
Sep 11 03:56:52.227562 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:56:52.227512    2579 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3faa005d-de0c-4c9c-9de6-035473a89077-home\") on node \"ip-10-0-141-105.ec2.internal\" DevicePath \"\""
Sep 11 03:56:52.227562 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:56:52.227520    2579 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3faa005d-de0c-4c9c-9de6-035473a89077-tmp-dir\") on node \"ip-10-0-141-105.ec2.internal\" DevicePath \"\""
Sep 11 03:56:52.227562 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:56:52.227529    2579 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3faa005d-de0c-4c9c-9de6-035473a89077-tls-certs\") on node \"ip-10-0-141-105.ec2.internal\" DevicePath \"\""
Sep 11 03:56:52.831738 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:56:52.831709    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-8efac4/test-model-8efac4-kserve-cb9995899-sxkk6"
Sep 11 03:56:52.832224 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:56:52.831741    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-8efac4/test-model-8efac4-kserve-cb9995899-sxkk6" event={"ID":"3faa005d-de0c-4c9c-9de6-035473a89077","Type":"ContainerDied","Data":"82b3487fd6ff31974b71c7c0e9c17c9bed325b8c978ff78d52b83a70b6465af7"}
Sep 11 03:56:52.832224 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:56:52.831815    2579 scope.go:117] "RemoveContainer" containerID="6d34dd50a2612144251ddb12ce7473a02978b54dcb14ae2f94e17815b16044c7"
Sep 11 03:56:52.852771 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:56:52.852746    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6a03b2ee-3de9-4905-8cc9-b9c1e3a5bfb5" path="/var/lib/kubelet/pods/6a03b2ee-3de9-4905-8cc9-b9c1e3a5bfb5/volumes"
Sep 11 03:56:52.855680 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:56:52.855658    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-8efac4/test-model-8efac4-kserve-cb9995899-sxkk6"]
Sep 11 03:56:52.859277 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:56:52.859249    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-b-8efac4/test-model-8efac4-kserve-cb9995899-sxkk6"]
Sep 11 03:56:54.412570 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:56:54.412537    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-4f6c0189-5857d987bb-5sjm6"]
Sep 11 03:56:54.412925 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:56:54.412880    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1199d453-904f-4a53-aeed-f424a04a0ec6" containerName="payload-pre-processing"
Sep 11 03:56:54.412925 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:56:54.412891    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="1199d453-904f-4a53-aeed-f424a04a0ec6" containerName="payload-pre-processing"
Sep 11 03:56:54.412925 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:56:54.412901    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6a03b2ee-3de9-4905-8cc9-b9c1e3a5bfb5" containerName="main"
Sep 11 03:56:54.413054 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:56:54.412933    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="6a03b2ee-3de9-4905-8cc9-b9c1e3a5bfb5" containerName="main"
Sep 11 03:56:54.413054 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:56:54.412942    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f12ac46f-f13a-4ba5-a0c5-99b5069c8632" containerName="revoke-keys"
Sep 11 03:56:54.413054 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:56:54.412948    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="f12ac46f-f13a-4ba5-a0c5-99b5069c8632" containerName="revoke-keys"
Sep 11 03:56:54.413054 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:56:54.412956    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3faa005d-de0c-4c9c-9de6-035473a89077" containerName="main"
Sep 11 03:56:54.413054 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:56:54.412961    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="3faa005d-de0c-4c9c-9de6-035473a89077" containerName="main"
Sep 11 03:56:54.413054 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:56:54.413024    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="3faa005d-de0c-4c9c-9de6-035473a89077" containerName="main"
Sep 11 03:56:54.413054 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:56:54.413035    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="1199d453-904f-4a53-aeed-f424a04a0ec6" containerName="payload-pre-processing"
Sep 11 03:56:54.413054 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:56:54.413040    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="f12ac46f-f13a-4ba5-a0c5-99b5069c8632" containerName="revoke-keys"
Sep 11 03:56:54.413054 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:56:54.413046    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="6a03b2ee-3de9-4905-8cc9-b9c1e3a5bfb5" containerName="main"
Sep 11 03:56:54.417193 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:56:54.417176    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-4f6c0189-5857d987bb-5sjm6"
Sep 11 03:56:54.419581 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:56:54.419558    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-4f6c0189\""
Sep 11 03:56:54.419581 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:56:54.419576    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-4f6c0189-dockercfg-k2dj9\""
Sep 11 03:56:54.425985 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:56:54.425965    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-4f6c0189-5857d987bb-5sjm6"]
Sep 11 03:56:54.454816 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:56:54.454783    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-4f6c0189-65b88f8bc7-8h9gz"]
Sep 11 03:56:54.458051 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:56:54.458036    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-4f6c0189-65b88f8bc7-8h9gz"
Sep 11 03:56:54.467923 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:56:54.467884    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-4f6c0189-65b88f8bc7-8h9gz"]
Sep 11 03:56:54.549127 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:56:54.549068    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-flmcq\" (UniqueName: \"kubernetes.io/projected/5a8ba387-2b66-43e4-97b1-a2869d382e42-kube-api-access-flmcq\") pod \"payload-processing-e2e-mt-4f6c0189-65b88f8bc7-8h9gz\" (UID: \"5a8ba387-2b66-43e4-97b1-a2869d382e42\") " pod="openshift-ingress/payload-processing-e2e-mt-4f6c0189-65b88f8bc7-8h9gz"
Sep 11 03:56:54.549127 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:56:54.549139    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/70da6004-dc56-44ea-89da-8fedc4f01437-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-4f6c0189-5857d987bb-5sjm6\" (UID: \"70da6004-dc56-44ea-89da-8fedc4f01437\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-4f6c0189-5857d987bb-5sjm6"
Sep 11 03:56:54.549345 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:56:54.549181    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5a8ba387-2b66-43e4-97b1-a2869d382e42-plugins-config-volume\") pod \"payload-processing-e2e-mt-4f6c0189-65b88f8bc7-8h9gz\" (UID: \"5a8ba387-2b66-43e4-97b1-a2869d382e42\") " pod="openshift-ingress/payload-processing-e2e-mt-4f6c0189-65b88f8bc7-8h9gz"
Sep 11 03:56:54.549345 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:56:54.549200    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fxck4\" (UniqueName: \"kubernetes.io/projected/70da6004-dc56-44ea-89da-8fedc4f01437-kube-api-access-fxck4\") pod \"payload-pre-processing-e2e-mt-4f6c0189-5857d987bb-5sjm6\" (UID: \"70da6004-dc56-44ea-89da-8fedc4f01437\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-4f6c0189-5857d987bb-5sjm6"
Sep 11 03:56:54.650419 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:56:54.650385    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5a8ba387-2b66-43e4-97b1-a2869d382e42-plugins-config-volume\") pod \"payload-processing-e2e-mt-4f6c0189-65b88f8bc7-8h9gz\" (UID: \"5a8ba387-2b66-43e4-97b1-a2869d382e42\") " pod="openshift-ingress/payload-processing-e2e-mt-4f6c0189-65b88f8bc7-8h9gz"
Sep 11 03:56:54.650419 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:56:54.650418    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fxck4\" (UniqueName: \"kubernetes.io/projected/70da6004-dc56-44ea-89da-8fedc4f01437-kube-api-access-fxck4\") pod \"payload-pre-processing-e2e-mt-4f6c0189-5857d987bb-5sjm6\" (UID: \"70da6004-dc56-44ea-89da-8fedc4f01437\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-4f6c0189-5857d987bb-5sjm6"
Sep 11 03:56:54.650648 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:56:54.650492    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-flmcq\" (UniqueName: \"kubernetes.io/projected/5a8ba387-2b66-43e4-97b1-a2869d382e42-kube-api-access-flmcq\") pod \"payload-processing-e2e-mt-4f6c0189-65b88f8bc7-8h9gz\" (UID: \"5a8ba387-2b66-43e4-97b1-a2869d382e42\") " pod="openshift-ingress/payload-processing-e2e-mt-4f6c0189-65b88f8bc7-8h9gz"
Sep 11 03:56:54.650648 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:56:54.650509    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/70da6004-dc56-44ea-89da-8fedc4f01437-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-4f6c0189-5857d987bb-5sjm6\" (UID: \"70da6004-dc56-44ea-89da-8fedc4f01437\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-4f6c0189-5857d987bb-5sjm6"
Sep 11 03:56:54.651130 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:56:54.651110    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5a8ba387-2b66-43e4-97b1-a2869d382e42-plugins-config-volume\") pod \"payload-processing-e2e-mt-4f6c0189-65b88f8bc7-8h9gz\" (UID: \"5a8ba387-2b66-43e4-97b1-a2869d382e42\") " pod="openshift-ingress/payload-processing-e2e-mt-4f6c0189-65b88f8bc7-8h9gz"
Sep 11 03:56:54.651241 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:56:54.651212    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/70da6004-dc56-44ea-89da-8fedc4f01437-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-4f6c0189-5857d987bb-5sjm6\" (UID: \"70da6004-dc56-44ea-89da-8fedc4f01437\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-4f6c0189-5857d987bb-5sjm6"
Sep 11 03:56:54.658287 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:56:54.658261    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fxck4\" (UniqueName: \"kubernetes.io/projected/70da6004-dc56-44ea-89da-8fedc4f01437-kube-api-access-fxck4\") pod \"payload-pre-processing-e2e-mt-4f6c0189-5857d987bb-5sjm6\" (UID: \"70da6004-dc56-44ea-89da-8fedc4f01437\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-4f6c0189-5857d987bb-5sjm6"
Sep 11 03:56:54.658378 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:56:54.658295    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-flmcq\" (UniqueName: \"kubernetes.io/projected/5a8ba387-2b66-43e4-97b1-a2869d382e42-kube-api-access-flmcq\") pod \"payload-processing-e2e-mt-4f6c0189-65b88f8bc7-8h9gz\" (UID: \"5a8ba387-2b66-43e4-97b1-a2869d382e42\") " pod="openshift-ingress/payload-processing-e2e-mt-4f6c0189-65b88f8bc7-8h9gz"
Sep 11 03:56:54.667233 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:56:54.667176    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-8efac4-xjcvw"]
Sep 11 03:56:54.670819 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:56:54.670805    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-8efac4-xjcvw"
Sep 11 03:56:54.672914 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:56:54.672898    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 11 03:56:54.673280 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:56:54.673263    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-95ckr\""
Sep 11 03:56:54.677375 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:56:54.677356    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-8efac4-xjcvw"]
Sep 11 03:56:54.729491 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:56:54.729456    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-4f6c0189-5857d987bb-5sjm6"
Sep 11 03:56:54.751889 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:56:54.751853    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4949da2d-5693-4be4-9272-5693400dce40-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-8efac4-xjcvw\" (UID: \"4949da2d-5693-4be4-9272-5693400dce40\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-8efac4-xjcvw"
Sep 11 03:56:54.770052 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:56:54.769760    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-4f6c0189-65b88f8bc7-8h9gz"
Sep 11 03:56:54.852472 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:56:54.852428    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4949da2d-5693-4be4-9272-5693400dce40-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-8efac4-xjcvw\" (UID: \"4949da2d-5693-4be4-9272-5693400dce40\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-8efac4-xjcvw"
Sep 11 03:56:54.853029 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:56:54.853009    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4949da2d-5693-4be4-9272-5693400dce40-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-8efac4-xjcvw\" (UID: \"4949da2d-5693-4be4-9272-5693400dce40\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-8efac4-xjcvw"
Sep 11 03:56:54.853712 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:56:54.853659    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3faa005d-de0c-4c9c-9de6-035473a89077" path="/var/lib/kubelet/pods/3faa005d-de0c-4c9c-9de6-035473a89077/volumes"
Sep 11 03:56:54.982192 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:56:54.982064    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-8efac4-xjcvw"
Sep 11 03:56:55.004588 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:56:55.004566    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-4f6c0189-5857d987bb-5sjm6"]
Sep 11 03:56:55.006630 ip-10-0-141-105 kubenswrapper[2579]: W0911 03:56:55.006603    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod70da6004_dc56_44ea_89da_8fedc4f01437.slice/crio-272e4da97a5863af41d8699b72ec45d88ddea431f39a69fa3b3f2af859fea34e WatchSource:0}: Error finding container 272e4da97a5863af41d8699b72ec45d88ddea431f39a69fa3b3f2af859fea34e: Status 404 returned error can't find the container with id 272e4da97a5863af41d8699b72ec45d88ddea431f39a69fa3b3f2af859fea34e
Sep 11 03:56:55.059329 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:56:55.058921    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-4f6c0189-65b88f8bc7-8h9gz"]
Sep 11 03:56:55.063199 ip-10-0-141-105 kubenswrapper[2579]: W0911 03:56:55.063168    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5a8ba387_2b66_43e4_97b1_a2869d382e42.slice/crio-b40775d095f633b2f9bbf03bf779086100357edd9cf65366e7f2513d2c148c28 WatchSource:0}: Error finding container b40775d095f633b2f9bbf03bf779086100357edd9cf65366e7f2513d2c148c28: Status 404 returned error can't find the container with id b40775d095f633b2f9bbf03bf779086100357edd9cf65366e7f2513d2c148c28
Sep 11 03:56:55.257030 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:56:55.257004    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-8efac4-xjcvw"]
Sep 11 03:56:55.259513 ip-10-0-141-105 kubenswrapper[2579]: W0911 03:56:55.259483    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4949da2d_5693_4be4_9272_5693400dce40.slice/crio-d9f186ab20f256f5e5ee662d85f408a7de8a8b131744df3e437c8a2c47e35dd3 WatchSource:0}: Error finding container d9f186ab20f256f5e5ee662d85f408a7de8a8b131744df3e437c8a2c47e35dd3: Status 404 returned error can't find the container with id d9f186ab20f256f5e5ee662d85f408a7de8a8b131744df3e437c8a2c47e35dd3
Sep 11 03:56:55.856475 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:56:55.856435    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-4f6c0189-65b88f8bc7-8h9gz" event={"ID":"5a8ba387-2b66-43e4-97b1-a2869d382e42","Type":"ContainerStarted","Data":"6fa3d5efa08e90b6e53e479f3e26df333548876e470a777fe4afab17321c75a4"}
Sep 11 03:56:55.856982 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:56:55.856529    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-4f6c0189-65b88f8bc7-8h9gz" event={"ID":"5a8ba387-2b66-43e4-97b1-a2869d382e42","Type":"ContainerStarted","Data":"b40775d095f633b2f9bbf03bf779086100357edd9cf65366e7f2513d2c148c28"}
Sep 11 03:56:55.856982 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:56:55.856560    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-mt-4f6c0189-65b88f8bc7-8h9gz"
Sep 11 03:56:55.858327 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:56:55.858296    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-8efac4-xjcvw" event={"ID":"4949da2d-5693-4be4-9272-5693400dce40","Type":"ContainerStarted","Data":"277a1759d01df140ffe425c366919c8d1c59e3fd3229482a3ae71d9885b734e3"}
Sep 11 03:56:55.858457 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:56:55.858346    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-8efac4-xjcvw" event={"ID":"4949da2d-5693-4be4-9272-5693400dce40","Type":"ContainerStarted","Data":"d9f186ab20f256f5e5ee662d85f408a7de8a8b131744df3e437c8a2c47e35dd3"}
Sep 11 03:56:55.860039 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:56:55.860010    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-4f6c0189-5857d987bb-5sjm6" event={"ID":"70da6004-dc56-44ea-89da-8fedc4f01437","Type":"ContainerStarted","Data":"42e0f2d6b97fac7b7f2bcd1a96487ef432ca1777af18e4774e50e67c6aacbce0"}
Sep 11 03:56:55.860162 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:56:55.860061    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-4f6c0189-5857d987bb-5sjm6" event={"ID":"70da6004-dc56-44ea-89da-8fedc4f01437","Type":"ContainerStarted","Data":"272e4da97a5863af41d8699b72ec45d88ddea431f39a69fa3b3f2af859fea34e"}
Sep 11 03:56:55.860207 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:56:55.860199    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-4f6c0189-5857d987bb-5sjm6"
Sep 11 03:56:55.872965 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:56:55.872913    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-mt-4f6c0189-65b88f8bc7-8h9gz" podStartSLOduration=1.872898298 podStartE2EDuration="1.872898298s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 03:56:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 03:56:55.871243795 +0000 UTC m=+993.526449363" watchObservedRunningTime="2026-09-11 03:56:55.872898298 +0000 UTC m=+993.528103864"
Sep 11 03:56:55.885053 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:56:55.885003    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-mt-4f6c0189-5857d987bb-5sjm6" podStartSLOduration=1.884985543 podStartE2EDuration="1.884985543s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 03:56:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 03:56:55.884834942 +0000 UTC m=+993.540040510" watchObservedRunningTime="2026-09-11 03:56:55.884985543 +0000 UTC m=+993.540191110"
Sep 11 03:56:55.898759 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:56:55.898704    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-8efac4-xjcvw" podStartSLOduration=1.898688267 podStartE2EDuration="1.898688267s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 03:56:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 03:56:55.897604953 +0000 UTC m=+993.552810521" watchObservedRunningTime="2026-09-11 03:56:55.898688267 +0000 UTC m=+993.553893833"
Sep 11 03:56:56.208026 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:56:56.207945    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-5z4cn"]
Sep 11 03:56:56.866259 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:56:56.866227    2579 generic.go:356] "Generic (PLEG): container finished" podID="4949da2d-5693-4be4-9272-5693400dce40" containerID="277a1759d01df140ffe425c366919c8d1c59e3fd3229482a3ae71d9885b734e3" exitCode=0
Sep 11 03:56:56.866692 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:56:56.866303    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-8efac4-xjcvw" event={"ID":"4949da2d-5693-4be4-9272-5693400dce40","Type":"ContainerDied","Data":"277a1759d01df140ffe425c366919c8d1c59e3fd3229482a3ae71d9885b734e3"}
Sep 11 03:56:58.003851 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:56:58.003826    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-8efac4-xjcvw"
Sep 11 03:56:58.182490 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:56:58.182400    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4949da2d-5693-4be4-9272-5693400dce40-maas-api-service-ca\") pod \"4949da2d-5693-4be4-9272-5693400dce40\" (UID: \"4949da2d-5693-4be4-9272-5693400dce40\") "
Sep 11 03:56:58.182761 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:56:58.182738    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4949da2d-5693-4be4-9272-5693400dce40-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "4949da2d-5693-4be4-9272-5693400dce40" (UID: "4949da2d-5693-4be4-9272-5693400dce40"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 03:56:58.283371 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:56:58.283336    2579 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4949da2d-5693-4be4-9272-5693400dce40-maas-api-service-ca\") on node \"ip-10-0-141-105.ec2.internal\" DevicePath \"\""
Sep 11 03:56:58.877633 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:56:58.877606    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-8efac4-xjcvw"
Sep 11 03:56:58.877791 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:56:58.877653    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-8efac4-xjcvw" event={"ID":"4949da2d-5693-4be4-9272-5693400dce40","Type":"ContainerDied","Data":"d9f186ab20f256f5e5ee662d85f408a7de8a8b131744df3e437c8a2c47e35dd3"}
Sep 11 03:56:58.877791 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:56:58.877697    2579 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d9f186ab20f256f5e5ee662d85f408a7de8a8b131744df3e437c8a2c47e35dd3"
Sep 11 03:57:05.325909 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:57:05.325874    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-8efac4-68b56cd9d9-5vx4t"]
Sep 11 03:57:05.326435 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:57:05.326200    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-b-8efac4-68b56cd9d9-5vx4t" podUID="98e573f4-5d9b-45c4-976a-42f439011e63" containerName="payload-processing" containerID="cri-o://815b5ecd26d546ed10957bde587cf4be85a85c1b25c9809b42cf6eeba3ca13ed" gracePeriod=30
Sep 11 03:57:05.471916 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:57:05.471882    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-8efac4-xjcvw"]
Sep 11 03:57:05.474946 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:57:05.474913    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-8efac4-xjcvw"]
Sep 11 03:57:05.584907 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:57:05.584851    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-8efac4-68b56cd9d9-5vx4t"
Sep 11 03:57:05.751878 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:57:05.751842    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rj7hz\" (UniqueName: \"kubernetes.io/projected/98e573f4-5d9b-45c4-976a-42f439011e63-kube-api-access-rj7hz\") pod \"98e573f4-5d9b-45c4-976a-42f439011e63\" (UID: \"98e573f4-5d9b-45c4-976a-42f439011e63\") "
Sep 11 03:57:05.752056 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:57:05.751954    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/98e573f4-5d9b-45c4-976a-42f439011e63-plugins-config-volume\") pod \"98e573f4-5d9b-45c4-976a-42f439011e63\" (UID: \"98e573f4-5d9b-45c4-976a-42f439011e63\") "
Sep 11 03:57:05.752383 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:57:05.752358    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/98e573f4-5d9b-45c4-976a-42f439011e63-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "98e573f4-5d9b-45c4-976a-42f439011e63" (UID: "98e573f4-5d9b-45c4-976a-42f439011e63"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 03:57:05.754425 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:57:05.754405    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/98e573f4-5d9b-45c4-976a-42f439011e63-kube-api-access-rj7hz" (OuterVolumeSpecName: "kube-api-access-rj7hz") pod "98e573f4-5d9b-45c4-976a-42f439011e63" (UID: "98e573f4-5d9b-45c4-976a-42f439011e63"). InnerVolumeSpecName "kube-api-access-rj7hz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 03:57:05.852671 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:57:05.852590    2579 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/98e573f4-5d9b-45c4-976a-42f439011e63-plugins-config-volume\") on node \"ip-10-0-141-105.ec2.internal\" DevicePath \"\""
Sep 11 03:57:05.852671 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:57:05.852615    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rj7hz\" (UniqueName: \"kubernetes.io/projected/98e573f4-5d9b-45c4-976a-42f439011e63-kube-api-access-rj7hz\") on node \"ip-10-0-141-105.ec2.internal\" DevicePath \"\""
Sep 11 03:57:05.913321 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:57:05.913286    2579 generic.go:356] "Generic (PLEG): container finished" podID="98e573f4-5d9b-45c4-976a-42f439011e63" containerID="815b5ecd26d546ed10957bde587cf4be85a85c1b25c9809b42cf6eeba3ca13ed" exitCode=0
Sep 11 03:57:05.913484 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:57:05.913325    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-8efac4-68b56cd9d9-5vx4t"
Sep 11 03:57:05.913484 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:57:05.913367    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-8efac4-68b56cd9d9-5vx4t" event={"ID":"98e573f4-5d9b-45c4-976a-42f439011e63","Type":"ContainerDied","Data":"815b5ecd26d546ed10957bde587cf4be85a85c1b25c9809b42cf6eeba3ca13ed"}
Sep 11 03:57:05.913484 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:57:05.913419    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-8efac4-68b56cd9d9-5vx4t" event={"ID":"98e573f4-5d9b-45c4-976a-42f439011e63","Type":"ContainerDied","Data":"e5eb8b66835547ee2c364621d7cc59cea280622ad892f36447b46efb1bf0ac60"}
Sep 11 03:57:05.913484 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:57:05.913444    2579 scope.go:117] "RemoveContainer" containerID="815b5ecd26d546ed10957bde587cf4be85a85c1b25c9809b42cf6eeba3ca13ed"
Sep 11 03:57:05.924764 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:57:05.924729    2579 scope.go:117] "RemoveContainer" containerID="815b5ecd26d546ed10957bde587cf4be85a85c1b25c9809b42cf6eeba3ca13ed"
Sep 11 03:57:05.925289 ip-10-0-141-105 kubenswrapper[2579]: E0911 03:57:05.925257    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"815b5ecd26d546ed10957bde587cf4be85a85c1b25c9809b42cf6eeba3ca13ed\": container with ID starting with 815b5ecd26d546ed10957bde587cf4be85a85c1b25c9809b42cf6eeba3ca13ed not found: ID does not exist" containerID="815b5ecd26d546ed10957bde587cf4be85a85c1b25c9809b42cf6eeba3ca13ed"
Sep 11 03:57:05.925406 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:57:05.925300    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"815b5ecd26d546ed10957bde587cf4be85a85c1b25c9809b42cf6eeba3ca13ed"} err="failed to get container status \"815b5ecd26d546ed10957bde587cf4be85a85c1b25c9809b42cf6eeba3ca13ed\": rpc error: code = NotFound desc = could not find container \"815b5ecd26d546ed10957bde587cf4be85a85c1b25c9809b42cf6eeba3ca13ed\": container with ID starting with 815b5ecd26d546ed10957bde587cf4be85a85c1b25c9809b42cf6eeba3ca13ed not found: ID does not exist"
Sep 11 03:57:05.950958 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:57:05.950927    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-8efac4-68b56cd9d9-5vx4t"]
Sep 11 03:57:05.954652 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:57:05.954623    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-8efac4-68b56cd9d9-5vx4t"]
Sep 11 03:57:06.852055 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:57:06.852019    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4949da2d-5693-4be4-9272-5693400dce40" path="/var/lib/kubelet/pods/4949da2d-5693-4be4-9272-5693400dce40/volumes"
Sep 11 03:57:06.852485 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:57:06.852441    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="98e573f4-5d9b-45c4-976a-42f439011e63" path="/var/lib/kubelet/pods/98e573f4-5d9b-45c4-976a-42f439011e63/volumes"
Sep 11 03:57:06.868832 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:57:06.868810    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-mt-4f6c0189-65b88f8bc7-8h9gz"
Sep 11 03:57:06.868989 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:57:06.868941    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-4f6c0189-5857d987bb-5sjm6"
Sep 11 03:57:07.797407 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:57:07.797365    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-5z4cn"]
Sep 11 03:57:11.895565 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:57:11.895532    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-noten-96c7d4/noten-model-96c7d4-kserve-7bf94b96f5-gfvbq"]
Sep 11 03:57:11.895957 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:57:11.895935    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="98e573f4-5d9b-45c4-976a-42f439011e63" containerName="payload-processing"
Sep 11 03:57:11.895957 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:57:11.895948    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="98e573f4-5d9b-45c4-976a-42f439011e63" containerName="payload-processing"
Sep 11 03:57:11.896040 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:57:11.895960    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4949da2d-5693-4be4-9272-5693400dce40" containerName="revoke-keys"
Sep 11 03:57:11.896040 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:57:11.895966    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="4949da2d-5693-4be4-9272-5693400dce40" containerName="revoke-keys"
Sep 11 03:57:11.896040 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:57:11.896020    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="4949da2d-5693-4be4-9272-5693400dce40" containerName="revoke-keys"
Sep 11 03:57:11.896040 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:57:11.896033    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="98e573f4-5d9b-45c4-976a-42f439011e63" containerName="payload-processing"
Sep 11 03:57:11.900570 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:57:11.900534    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-noten-96c7d4/noten-model-96c7d4-kserve-7bf94b96f5-gfvbq"
Sep 11 03:57:11.902829 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:57:11.902801    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/21600213-6b07-4c47-a178-6fc211e22a1f-tls-certs\") pod \"noten-model-96c7d4-kserve-7bf94b96f5-gfvbq\" (UID: \"21600213-6b07-4c47-a178-6fc211e22a1f\") " pod="ai-tenant-e2e-noten-96c7d4/noten-model-96c7d4-kserve-7bf94b96f5-gfvbq"
Sep 11 03:57:11.902975 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:57:11.902865    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6cqtt\" (UniqueName: \"kubernetes.io/projected/21600213-6b07-4c47-a178-6fc211e22a1f-kube-api-access-6cqtt\") pod \"noten-model-96c7d4-kserve-7bf94b96f5-gfvbq\" (UID: \"21600213-6b07-4c47-a178-6fc211e22a1f\") " pod="ai-tenant-e2e-noten-96c7d4/noten-model-96c7d4-kserve-7bf94b96f5-gfvbq"
Sep 11 03:57:11.902975 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:57:11.902899    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/21600213-6b07-4c47-a178-6fc211e22a1f-model-cache\") pod \"noten-model-96c7d4-kserve-7bf94b96f5-gfvbq\" (UID: \"21600213-6b07-4c47-a178-6fc211e22a1f\") " pod="ai-tenant-e2e-noten-96c7d4/noten-model-96c7d4-kserve-7bf94b96f5-gfvbq"
Sep 11 03:57:11.902975 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:57:11.902926    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/21600213-6b07-4c47-a178-6fc211e22a1f-home\") pod \"noten-model-96c7d4-kserve-7bf94b96f5-gfvbq\" (UID: \"21600213-6b07-4c47-a178-6fc211e22a1f\") " pod="ai-tenant-e2e-noten-96c7d4/noten-model-96c7d4-kserve-7bf94b96f5-gfvbq"
Sep 11 03:57:11.902975 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:57:11.902951    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/21600213-6b07-4c47-a178-6fc211e22a1f-tmp-dir\") pod \"noten-model-96c7d4-kserve-7bf94b96f5-gfvbq\" (UID: \"21600213-6b07-4c47-a178-6fc211e22a1f\") " pod="ai-tenant-e2e-noten-96c7d4/noten-model-96c7d4-kserve-7bf94b96f5-gfvbq"
Sep 11 03:57:11.903194 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:57:11.903018    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/21600213-6b07-4c47-a178-6fc211e22a1f-dshm\") pod \"noten-model-96c7d4-kserve-7bf94b96f5-gfvbq\" (UID: \"21600213-6b07-4c47-a178-6fc211e22a1f\") " pod="ai-tenant-e2e-noten-96c7d4/noten-model-96c7d4-kserve-7bf94b96f5-gfvbq"
Sep 11 03:57:11.903194 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:57:11.903103    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-noten-96c7d4\"/\"noten-model-96c7d4-kserve-self-signed-certs\""
Sep 11 03:57:11.904184 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:57:11.904163    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-noten-96c7d4\"/\"kube-root-ca.crt\""
Sep 11 03:57:11.904303 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:57:11.904180    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-noten-96c7d4\"/\"openshift-service-ca.crt\""
Sep 11 03:57:11.904303 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:57:11.904189    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-noten-96c7d4\"/\"default-dockercfg-hn857\""
Sep 11 03:57:11.910196 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:57:11.910166    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-noten-96c7d4/noten-model-96c7d4-kserve-7bf94b96f5-gfvbq"]
Sep 11 03:57:12.003661 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:57:12.003622    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/21600213-6b07-4c47-a178-6fc211e22a1f-home\") pod \"noten-model-96c7d4-kserve-7bf94b96f5-gfvbq\" (UID: \"21600213-6b07-4c47-a178-6fc211e22a1f\") " pod="ai-tenant-e2e-noten-96c7d4/noten-model-96c7d4-kserve-7bf94b96f5-gfvbq"
Sep 11 03:57:12.003852 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:57:12.003671    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/21600213-6b07-4c47-a178-6fc211e22a1f-tmp-dir\") pod \"noten-model-96c7d4-kserve-7bf94b96f5-gfvbq\" (UID: \"21600213-6b07-4c47-a178-6fc211e22a1f\") " pod="ai-tenant-e2e-noten-96c7d4/noten-model-96c7d4-kserve-7bf94b96f5-gfvbq"
Sep 11 03:57:12.003852 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:57:12.003714    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/21600213-6b07-4c47-a178-6fc211e22a1f-dshm\") pod \"noten-model-96c7d4-kserve-7bf94b96f5-gfvbq\" (UID: \"21600213-6b07-4c47-a178-6fc211e22a1f\") " pod="ai-tenant-e2e-noten-96c7d4/noten-model-96c7d4-kserve-7bf94b96f5-gfvbq"
Sep 11 03:57:12.003852 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:57:12.003743    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/21600213-6b07-4c47-a178-6fc211e22a1f-tls-certs\") pod \"noten-model-96c7d4-kserve-7bf94b96f5-gfvbq\" (UID: \"21600213-6b07-4c47-a178-6fc211e22a1f\") " pod="ai-tenant-e2e-noten-96c7d4/noten-model-96c7d4-kserve-7bf94b96f5-gfvbq"
Sep 11 03:57:12.003852 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:57:12.003794    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6cqtt\" (UniqueName: \"kubernetes.io/projected/21600213-6b07-4c47-a178-6fc211e22a1f-kube-api-access-6cqtt\") pod \"noten-model-96c7d4-kserve-7bf94b96f5-gfvbq\" (UID: \"21600213-6b07-4c47-a178-6fc211e22a1f\") " pod="ai-tenant-e2e-noten-96c7d4/noten-model-96c7d4-kserve-7bf94b96f5-gfvbq"
Sep 11 03:57:12.003852 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:57:12.003842    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/21600213-6b07-4c47-a178-6fc211e22a1f-model-cache\") pod \"noten-model-96c7d4-kserve-7bf94b96f5-gfvbq\" (UID: \"21600213-6b07-4c47-a178-6fc211e22a1f\") " pod="ai-tenant-e2e-noten-96c7d4/noten-model-96c7d4-kserve-7bf94b96f5-gfvbq"
Sep 11 03:57:12.004183 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:57:12.004162    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/21600213-6b07-4c47-a178-6fc211e22a1f-tmp-dir\") pod \"noten-model-96c7d4-kserve-7bf94b96f5-gfvbq\" (UID: \"21600213-6b07-4c47-a178-6fc211e22a1f\") " pod="ai-tenant-e2e-noten-96c7d4/noten-model-96c7d4-kserve-7bf94b96f5-gfvbq"
Sep 11 03:57:12.004408 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:57:12.004384    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/21600213-6b07-4c47-a178-6fc211e22a1f-home\") pod \"noten-model-96c7d4-kserve-7bf94b96f5-gfvbq\" (UID: \"21600213-6b07-4c47-a178-6fc211e22a1f\") " pod="ai-tenant-e2e-noten-96c7d4/noten-model-96c7d4-kserve-7bf94b96f5-gfvbq"
Sep 11 03:57:12.004633 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:57:12.004605    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/21600213-6b07-4c47-a178-6fc211e22a1f-model-cache\") pod \"noten-model-96c7d4-kserve-7bf94b96f5-gfvbq\" (UID: \"21600213-6b07-4c47-a178-6fc211e22a1f\") " pod="ai-tenant-e2e-noten-96c7d4/noten-model-96c7d4-kserve-7bf94b96f5-gfvbq"
Sep 11 03:57:12.006497 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:57:12.006469    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/21600213-6b07-4c47-a178-6fc211e22a1f-dshm\") pod \"noten-model-96c7d4-kserve-7bf94b96f5-gfvbq\" (UID: \"21600213-6b07-4c47-a178-6fc211e22a1f\") " pod="ai-tenant-e2e-noten-96c7d4/noten-model-96c7d4-kserve-7bf94b96f5-gfvbq"
Sep 11 03:57:12.006732 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:57:12.006709    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/21600213-6b07-4c47-a178-6fc211e22a1f-tls-certs\") pod \"noten-model-96c7d4-kserve-7bf94b96f5-gfvbq\" (UID: \"21600213-6b07-4c47-a178-6fc211e22a1f\") " pod="ai-tenant-e2e-noten-96c7d4/noten-model-96c7d4-kserve-7bf94b96f5-gfvbq"
Sep 11 03:57:12.012158 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:57:12.012136    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6cqtt\" (UniqueName: \"kubernetes.io/projected/21600213-6b07-4c47-a178-6fc211e22a1f-kube-api-access-6cqtt\") pod \"noten-model-96c7d4-kserve-7bf94b96f5-gfvbq\" (UID: \"21600213-6b07-4c47-a178-6fc211e22a1f\") " pod="ai-tenant-e2e-noten-96c7d4/noten-model-96c7d4-kserve-7bf94b96f5-gfvbq"
Sep 11 03:57:12.211536 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:57:12.211437    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-noten-96c7d4/noten-model-96c7d4-kserve-7bf94b96f5-gfvbq"
Sep 11 03:57:12.490667 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:57:12.490638    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-noten-96c7d4/noten-model-96c7d4-kserve-7bf94b96f5-gfvbq"]
Sep 11 03:57:12.491651 ip-10-0-141-105 kubenswrapper[2579]: W0911 03:57:12.491623    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod21600213_6b07_4c47_a178_6fc211e22a1f.slice/crio-5cb1f011e98e41891b3f3d619d0abd01001daa26ea0f746ac060637ab152e5d1 WatchSource:0}: Error finding container 5cb1f011e98e41891b3f3d619d0abd01001daa26ea0f746ac060637ab152e5d1: Status 404 returned error can't find the container with id 5cb1f011e98e41891b3f3d619d0abd01001daa26ea0f746ac060637ab152e5d1
Sep 11 03:57:12.954942 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:57:12.954911    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-noten-96c7d4/noten-model-96c7d4-kserve-7bf94b96f5-gfvbq" event={"ID":"21600213-6b07-4c47-a178-6fc211e22a1f","Type":"ContainerStarted","Data":"72031f9e4544752e7127d372d9291b30efd8192f6daf5c5ec5cb20c6e9b0b98a"}
Sep 11 03:57:12.954942 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:57:12.954950    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-noten-96c7d4/noten-model-96c7d4-kserve-7bf94b96f5-gfvbq" event={"ID":"21600213-6b07-4c47-a178-6fc211e22a1f","Type":"ContainerStarted","Data":"5cb1f011e98e41891b3f3d619d0abd01001daa26ea0f746ac060637ab152e5d1"}
Sep 11 03:57:12.978861 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:57:12.978811    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-noten-96c7d4/noten-model-96c7d4-kserve-7bf94b96f5-gfvbq" podStartSLOduration=1.9787947799999999 podStartE2EDuration="1.97879478s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 03:57:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 03:57:12.977358045 +0000 UTC m=+1010.632563611" watchObservedRunningTime="2026-09-11 03:57:12.97879478 +0000 UTC m=+1010.634000407"
Sep 11 03:57:19.621900 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:57:19.621859    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-4f6c0189-2hw2c"]
Sep 11 03:57:19.625399 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:57:19.625378    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-4f6c0189-2hw2c"
Sep 11 03:57:19.627774 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:57:19.627748    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 11 03:57:19.627774 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:57:19.627768    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-95ckr\""
Sep 11 03:57:19.632234 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:57:19.632192    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-4f6c0189-2hw2c"]
Sep 11 03:57:19.675736 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:57:19.675701    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3f56f626-4825-4724-9d5c-1cc85f8885e6-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-4f6c0189-2hw2c\" (UID: \"3f56f626-4825-4724-9d5c-1cc85f8885e6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-4f6c0189-2hw2c"
Sep 11 03:57:19.776568 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:57:19.776520    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3f56f626-4825-4724-9d5c-1cc85f8885e6-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-4f6c0189-2hw2c\" (UID: \"3f56f626-4825-4724-9d5c-1cc85f8885e6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-4f6c0189-2hw2c"
Sep 11 03:57:19.777296 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:57:19.777258    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3f56f626-4825-4724-9d5c-1cc85f8885e6-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-4f6c0189-2hw2c\" (UID: \"3f56f626-4825-4724-9d5c-1cc85f8885e6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-4f6c0189-2hw2c"
Sep 11 03:57:19.938417 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:57:19.938325    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-4f6c0189-2hw2c"
Sep 11 03:57:20.208220 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:57:20.208193    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-4f6c0189-2hw2c"]
Sep 11 03:57:20.210947 ip-10-0-141-105 kubenswrapper[2579]: W0911 03:57:20.210919    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3f56f626_4825_4724_9d5c_1cc85f8885e6.slice/crio-8a02917dc358b10910aeca97088a4e16f2e2e982fc2818b73fc2fe529d6fa424 WatchSource:0}: Error finding container 8a02917dc358b10910aeca97088a4e16f2e2e982fc2818b73fc2fe529d6fa424: Status 404 returned error can't find the container with id 8a02917dc358b10910aeca97088a4e16f2e2e982fc2818b73fc2fe529d6fa424
Sep 11 03:57:20.903043 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:57:20.903004    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-5z4cn"]
Sep 11 03:57:21.001652 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:57:21.001621    2579 generic.go:356] "Generic (PLEG): container finished" podID="3f56f626-4825-4724-9d5c-1cc85f8885e6" containerID="432b0e24a23b2f81c75b86a0cf2d1df72a1d2b85e6d275d5a380fbea832c5ea9" exitCode=0
Sep 11 03:57:21.001830 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:57:21.001716    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-4f6c0189-2hw2c" event={"ID":"3f56f626-4825-4724-9d5c-1cc85f8885e6","Type":"ContainerDied","Data":"432b0e24a23b2f81c75b86a0cf2d1df72a1d2b85e6d275d5a380fbea832c5ea9"}
Sep 11 03:57:21.001830 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:57:21.001760    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-4f6c0189-2hw2c" event={"ID":"3f56f626-4825-4724-9d5c-1cc85f8885e6","Type":"ContainerStarted","Data":"8a02917dc358b10910aeca97088a4e16f2e2e982fc2818b73fc2fe529d6fa424"}
Sep 11 03:57:22.139256 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:57:22.139229    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-4f6c0189-2hw2c"
Sep 11 03:57:22.199962 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:57:22.199932    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3f56f626-4825-4724-9d5c-1cc85f8885e6-maas-api-service-ca\") pod \"3f56f626-4825-4724-9d5c-1cc85f8885e6\" (UID: \"3f56f626-4825-4724-9d5c-1cc85f8885e6\") "
Sep 11 03:57:22.200361 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:57:22.200337    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3f56f626-4825-4724-9d5c-1cc85f8885e6-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "3f56f626-4825-4724-9d5c-1cc85f8885e6" (UID: "3f56f626-4825-4724-9d5c-1cc85f8885e6"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 03:57:22.211939 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:57:22.211918    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-noten-96c7d4/noten-model-96c7d4-kserve-7bf94b96f5-gfvbq"
Sep 11 03:57:22.211992 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:57:22.211978    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-noten-96c7d4/noten-model-96c7d4-kserve-7bf94b96f5-gfvbq"
Sep 11 03:57:22.224270 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:57:22.224250    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-noten-96c7d4/noten-model-96c7d4-kserve-7bf94b96f5-gfvbq"
Sep 11 03:57:22.300963 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:57:22.300888    2579 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3f56f626-4825-4724-9d5c-1cc85f8885e6-maas-api-service-ca\") on node \"ip-10-0-141-105.ec2.internal\" DevicePath \"\""
Sep 11 03:57:23.013136 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:57:23.013065    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-4f6c0189-2hw2c"
Sep 11 03:57:23.013136 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:57:23.013112    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-4f6c0189-2hw2c" event={"ID":"3f56f626-4825-4724-9d5c-1cc85f8885e6","Type":"ContainerDied","Data":"8a02917dc358b10910aeca97088a4e16f2e2e982fc2818b73fc2fe529d6fa424"}
Sep 11 03:57:23.013384 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:57:23.013161    2579 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8a02917dc358b10910aeca97088a4e16f2e2e982fc2818b73fc2fe529d6fa424"
Sep 11 03:57:23.026715 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:57:23.026692    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-noten-96c7d4/noten-model-96c7d4-kserve-7bf94b96f5-gfvbq"
Sep 11 03:57:24.282232 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:57:24.282197    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-4f6c0189-65b88f8bc7-8h9gz"]
Sep 11 03:57:24.282726 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:57:24.282506    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-mt-4f6c0189-65b88f8bc7-8h9gz" podUID="5a8ba387-2b66-43e4-97b1-a2869d382e42" containerName="payload-processing" containerID="cri-o://6fa3d5efa08e90b6e53e479f3e26df333548876e470a777fe4afab17321c75a4" gracePeriod=30
Sep 11 03:57:24.297413 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:57:24.297381    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-4f6c0189-5857d987bb-5sjm6"]
Sep 11 03:57:24.297906 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:57:24.297869    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-mt-4f6c0189-5857d987bb-5sjm6" podUID="70da6004-dc56-44ea-89da-8fedc4f01437" containerName="payload-pre-processing" containerID="cri-o://42e0f2d6b97fac7b7f2bcd1a96487ef432ca1777af18e4774e50e67c6aacbce0" gracePeriod=30
Sep 11 03:57:24.449635 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:57:24.449559    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-4f6c0189-2hw2c"]
Sep 11 03:57:24.450470 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:57:24.450442    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-4f6c0189-2hw2c"]
Sep 11 03:57:24.569602 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:57:24.569582    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-4f6c0189-65b88f8bc7-8h9gz"
Sep 11 03:57:24.572853 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:57:24.572827    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-4f6c0189-5857d987bb-5sjm6"
Sep 11 03:57:24.624401 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:57:24.624372    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/70da6004-dc56-44ea-89da-8fedc4f01437-plugins-config-volume\") pod \"70da6004-dc56-44ea-89da-8fedc4f01437\" (UID: \"70da6004-dc56-44ea-89da-8fedc4f01437\") "
Sep 11 03:57:24.624566 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:57:24.624444    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-flmcq\" (UniqueName: \"kubernetes.io/projected/5a8ba387-2b66-43e4-97b1-a2869d382e42-kube-api-access-flmcq\") pod \"5a8ba387-2b66-43e4-97b1-a2869d382e42\" (UID: \"5a8ba387-2b66-43e4-97b1-a2869d382e42\") "
Sep 11 03:57:24.624566 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:57:24.624477    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fxck4\" (UniqueName: \"kubernetes.io/projected/70da6004-dc56-44ea-89da-8fedc4f01437-kube-api-access-fxck4\") pod \"70da6004-dc56-44ea-89da-8fedc4f01437\" (UID: \"70da6004-dc56-44ea-89da-8fedc4f01437\") "
Sep 11 03:57:24.624566 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:57:24.624497    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5a8ba387-2b66-43e4-97b1-a2869d382e42-plugins-config-volume\") pod \"5a8ba387-2b66-43e4-97b1-a2869d382e42\" (UID: \"5a8ba387-2b66-43e4-97b1-a2869d382e42\") "
Sep 11 03:57:24.624978 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:57:24.624947    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5a8ba387-2b66-43e4-97b1-a2869d382e42-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "5a8ba387-2b66-43e4-97b1-a2869d382e42" (UID: "5a8ba387-2b66-43e4-97b1-a2869d382e42"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 03:57:24.624978 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:57:24.624968    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/70da6004-dc56-44ea-89da-8fedc4f01437-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "70da6004-dc56-44ea-89da-8fedc4f01437" (UID: "70da6004-dc56-44ea-89da-8fedc4f01437"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 03:57:24.626838 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:57:24.626811    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/70da6004-dc56-44ea-89da-8fedc4f01437-kube-api-access-fxck4" (OuterVolumeSpecName: "kube-api-access-fxck4") pod "70da6004-dc56-44ea-89da-8fedc4f01437" (UID: "70da6004-dc56-44ea-89da-8fedc4f01437"). InnerVolumeSpecName "kube-api-access-fxck4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 03:57:24.626965 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:57:24.626885    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5a8ba387-2b66-43e4-97b1-a2869d382e42-kube-api-access-flmcq" (OuterVolumeSpecName: "kube-api-access-flmcq") pod "5a8ba387-2b66-43e4-97b1-a2869d382e42" (UID: "5a8ba387-2b66-43e4-97b1-a2869d382e42"). InnerVolumeSpecName "kube-api-access-flmcq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 03:57:24.725614 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:57:24.725577    2579 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/70da6004-dc56-44ea-89da-8fedc4f01437-plugins-config-volume\") on node \"ip-10-0-141-105.ec2.internal\" DevicePath \"\""
Sep 11 03:57:24.725614 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:57:24.725607    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-flmcq\" (UniqueName: \"kubernetes.io/projected/5a8ba387-2b66-43e4-97b1-a2869d382e42-kube-api-access-flmcq\") on node \"ip-10-0-141-105.ec2.internal\" DevicePath \"\""
Sep 11 03:57:24.725614 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:57:24.725618    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fxck4\" (UniqueName: \"kubernetes.io/projected/70da6004-dc56-44ea-89da-8fedc4f01437-kube-api-access-fxck4\") on node \"ip-10-0-141-105.ec2.internal\" DevicePath \"\""
Sep 11 03:57:24.725869 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:57:24.725626    2579 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5a8ba387-2b66-43e4-97b1-a2869d382e42-plugins-config-volume\") on node \"ip-10-0-141-105.ec2.internal\" DevicePath \"\""
Sep 11 03:57:24.851750 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:57:24.851638    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3f56f626-4825-4724-9d5c-1cc85f8885e6" path="/var/lib/kubelet/pods/3f56f626-4825-4724-9d5c-1cc85f8885e6/volumes"
Sep 11 03:57:25.025188 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:57:25.025147    2579 generic.go:356] "Generic (PLEG): container finished" podID="5a8ba387-2b66-43e4-97b1-a2869d382e42" containerID="6fa3d5efa08e90b6e53e479f3e26df333548876e470a777fe4afab17321c75a4" exitCode=0
Sep 11 03:57:25.025374 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:57:25.025199    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-4f6c0189-65b88f8bc7-8h9gz"
Sep 11 03:57:25.025374 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:57:25.025204    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-4f6c0189-65b88f8bc7-8h9gz" event={"ID":"5a8ba387-2b66-43e4-97b1-a2869d382e42","Type":"ContainerDied","Data":"6fa3d5efa08e90b6e53e479f3e26df333548876e470a777fe4afab17321c75a4"}
Sep 11 03:57:25.025374 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:57:25.025259    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-4f6c0189-65b88f8bc7-8h9gz" event={"ID":"5a8ba387-2b66-43e4-97b1-a2869d382e42","Type":"ContainerDied","Data":"b40775d095f633b2f9bbf03bf779086100357edd9cf65366e7f2513d2c148c28"}
Sep 11 03:57:25.025374 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:57:25.025280    2579 scope.go:117] "RemoveContainer" containerID="6fa3d5efa08e90b6e53e479f3e26df333548876e470a777fe4afab17321c75a4"
Sep 11 03:57:25.026794 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:57:25.026773    2579 generic.go:356] "Generic (PLEG): container finished" podID="70da6004-dc56-44ea-89da-8fedc4f01437" containerID="42e0f2d6b97fac7b7f2bcd1a96487ef432ca1777af18e4774e50e67c6aacbce0" exitCode=0
Sep 11 03:57:25.026904 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:57:25.026816    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-4f6c0189-5857d987bb-5sjm6"
Sep 11 03:57:25.026904 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:57:25.026850    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-4f6c0189-5857d987bb-5sjm6" event={"ID":"70da6004-dc56-44ea-89da-8fedc4f01437","Type":"ContainerDied","Data":"42e0f2d6b97fac7b7f2bcd1a96487ef432ca1777af18e4774e50e67c6aacbce0"}
Sep 11 03:57:25.026986 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:57:25.026906    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-4f6c0189-5857d987bb-5sjm6" event={"ID":"70da6004-dc56-44ea-89da-8fedc4f01437","Type":"ContainerDied","Data":"272e4da97a5863af41d8699b72ec45d88ddea431f39a69fa3b3f2af859fea34e"}
Sep 11 03:57:25.034516 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:57:25.034482    2579 scope.go:117] "RemoveContainer" containerID="6fa3d5efa08e90b6e53e479f3e26df333548876e470a777fe4afab17321c75a4"
Sep 11 03:57:25.034741 ip-10-0-141-105 kubenswrapper[2579]: E0911 03:57:25.034724    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6fa3d5efa08e90b6e53e479f3e26df333548876e470a777fe4afab17321c75a4\": container with ID starting with 6fa3d5efa08e90b6e53e479f3e26df333548876e470a777fe4afab17321c75a4 not found: ID does not exist" containerID="6fa3d5efa08e90b6e53e479f3e26df333548876e470a777fe4afab17321c75a4"
Sep 11 03:57:25.034799 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:57:25.034747    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6fa3d5efa08e90b6e53e479f3e26df333548876e470a777fe4afab17321c75a4"} err="failed to get container status \"6fa3d5efa08e90b6e53e479f3e26df333548876e470a777fe4afab17321c75a4\": rpc error: code = NotFound desc = could not find container \"6fa3d5efa08e90b6e53e479f3e26df333548876e470a777fe4afab17321c75a4\": container with ID starting with 6fa3d5efa08e90b6e53e479f3e26df333548876e470a777fe4afab17321c75a4 not found: ID does not exist"
Sep 11 03:57:25.034799 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:57:25.034762    2579 scope.go:117] "RemoveContainer" containerID="42e0f2d6b97fac7b7f2bcd1a96487ef432ca1777af18e4774e50e67c6aacbce0"
Sep 11 03:57:25.046260 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:57:25.043892    2579 scope.go:117] "RemoveContainer" containerID="42e0f2d6b97fac7b7f2bcd1a96487ef432ca1777af18e4774e50e67c6aacbce0"
Sep 11 03:57:25.046260 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:57:25.044263    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-4f6c0189-65b88f8bc7-8h9gz"]
Sep 11 03:57:25.046260 ip-10-0-141-105 kubenswrapper[2579]: E0911 03:57:25.045326    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"42e0f2d6b97fac7b7f2bcd1a96487ef432ca1777af18e4774e50e67c6aacbce0\": container with ID starting with 42e0f2d6b97fac7b7f2bcd1a96487ef432ca1777af18e4774e50e67c6aacbce0 not found: ID does not exist" containerID="42e0f2d6b97fac7b7f2bcd1a96487ef432ca1777af18e4774e50e67c6aacbce0"
Sep 11 03:57:25.046260 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:57:25.045361    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"42e0f2d6b97fac7b7f2bcd1a96487ef432ca1777af18e4774e50e67c6aacbce0"} err="failed to get container status \"42e0f2d6b97fac7b7f2bcd1a96487ef432ca1777af18e4774e50e67c6aacbce0\": rpc error: code = NotFound desc = could not find container \"42e0f2d6b97fac7b7f2bcd1a96487ef432ca1777af18e4774e50e67c6aacbce0\": container with ID starting with 42e0f2d6b97fac7b7f2bcd1a96487ef432ca1777af18e4774e50e67c6aacbce0 not found: ID does not exist"
Sep 11 03:57:25.050308 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:57:25.050273    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-4f6c0189-65b88f8bc7-8h9gz"]
Sep 11 03:57:25.057485 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:57:25.057465    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-4f6c0189-5857d987bb-5sjm6"]
Sep 11 03:57:25.060593 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:57:25.060572    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-4f6c0189-5857d987bb-5sjm6"]
Sep 11 03:57:26.851524 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:57:26.851495    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5a8ba387-2b66-43e4-97b1-a2869d382e42" path="/var/lib/kubelet/pods/5a8ba387-2b66-43e4-97b1-a2869d382e42/volumes"
Sep 11 03:57:26.851887 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:57:26.851822    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="70da6004-dc56-44ea-89da-8fedc4f01437" path="/var/lib/kubelet/pods/70da6004-dc56-44ea-89da-8fedc4f01437/volumes"
Sep 11 03:57:31.478949 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:57:31.478918    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w3-dd5b30-openshift-default-86dfc4597d-b7w6c"]
Sep 11 03:57:31.479350 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:57:31.479298    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5a8ba387-2b66-43e4-97b1-a2869d382e42" containerName="payload-processing"
Sep 11 03:57:31.479350 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:57:31.479313    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="5a8ba387-2b66-43e4-97b1-a2869d382e42" containerName="payload-processing"
Sep 11 03:57:31.479350 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:57:31.479326    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="70da6004-dc56-44ea-89da-8fedc4f01437" containerName="payload-pre-processing"
Sep 11 03:57:31.479350 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:57:31.479331    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="70da6004-dc56-44ea-89da-8fedc4f01437" containerName="payload-pre-processing"
Sep 11 03:57:31.479350 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:57:31.479337    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3f56f626-4825-4724-9d5c-1cc85f8885e6" containerName="revoke-keys"
Sep 11 03:57:31.479350 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:57:31.479343    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="3f56f626-4825-4724-9d5c-1cc85f8885e6" containerName="revoke-keys"
Sep 11 03:57:31.479550 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:57:31.479401    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="5a8ba387-2b66-43e4-97b1-a2869d382e42" containerName="payload-processing"
Sep 11 03:57:31.479550 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:57:31.479409    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="70da6004-dc56-44ea-89da-8fedc4f01437" containerName="payload-pre-processing"
Sep 11 03:57:31.479550 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:57:31.479418    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="3f56f626-4825-4724-9d5c-1cc85f8885e6" containerName="revoke-keys"
Sep 11 03:57:31.483937 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:57:31.483918    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-dd5b30-openshift-default-86dfc4597d-b7w6c"
Sep 11 03:57:31.486408 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:57:31.486383    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\""
Sep 11 03:57:31.486546 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:57:31.486447    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w3-dd5b30-openshift-default-dockercfg-b9vbq\""
Sep 11 03:57:31.492692 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:57:31.492667    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-dd5b30-openshift-default-86dfc4597d-b7w6c"]
Sep 11 03:57:31.591267 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:57:31.591233    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ecac9ada-1da0-4ecd-8af5-07501bd2aeb9-istio-podinfo\") pod \"e2e-shared-b-w3-dd5b30-openshift-default-86dfc4597d-b7w6c\" (UID: \"ecac9ada-1da0-4ecd-8af5-07501bd2aeb9\") " pod="openshift-ingress/e2e-shared-b-w3-dd5b30-openshift-default-86dfc4597d-b7w6c"
Sep 11 03:57:31.591459 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:57:31.591271    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ecac9ada-1da0-4ecd-8af5-07501bd2aeb9-workload-certs\") pod \"e2e-shared-b-w3-dd5b30-openshift-default-86dfc4597d-b7w6c\" (UID: \"ecac9ada-1da0-4ecd-8af5-07501bd2aeb9\") " pod="openshift-ingress/e2e-shared-b-w3-dd5b30-openshift-default-86dfc4597d-b7w6c"
Sep 11 03:57:31.591459 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:57:31.591294    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ecac9ada-1da0-4ecd-8af5-07501bd2aeb9-istiod-ca-cert\") pod \"e2e-shared-b-w3-dd5b30-openshift-default-86dfc4597d-b7w6c\" (UID: \"ecac9ada-1da0-4ecd-8af5-07501bd2aeb9\") " pod="openshift-ingress/e2e-shared-b-w3-dd5b30-openshift-default-86dfc4597d-b7w6c"
Sep 11 03:57:31.591459 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:57:31.591354    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ecac9ada-1da0-4ecd-8af5-07501bd2aeb9-istio-token\") pod \"e2e-shared-b-w3-dd5b30-openshift-default-86dfc4597d-b7w6c\" (UID: \"ecac9ada-1da0-4ecd-8af5-07501bd2aeb9\") " pod="openshift-ingress/e2e-shared-b-w3-dd5b30-openshift-default-86dfc4597d-b7w6c"
Sep 11 03:57:31.591459 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:57:31.591400    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-45g8t\" (UniqueName: \"kubernetes.io/projected/ecac9ada-1da0-4ecd-8af5-07501bd2aeb9-kube-api-access-45g8t\") pod \"e2e-shared-b-w3-dd5b30-openshift-default-86dfc4597d-b7w6c\" (UID: \"ecac9ada-1da0-4ecd-8af5-07501bd2aeb9\") " pod="openshift-ingress/e2e-shared-b-w3-dd5b30-openshift-default-86dfc4597d-b7w6c"
Sep 11 03:57:31.591459 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:57:31.591442    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ecac9ada-1da0-4ecd-8af5-07501bd2aeb9-workload-socket\") pod \"e2e-shared-b-w3-dd5b30-openshift-default-86dfc4597d-b7w6c\" (UID: \"ecac9ada-1da0-4ecd-8af5-07501bd2aeb9\") " pod="openshift-ingress/e2e-shared-b-w3-dd5b30-openshift-default-86dfc4597d-b7w6c"
Sep 11 03:57:31.591644 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:57:31.591505    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ecac9ada-1da0-4ecd-8af5-07501bd2aeb9-istio-data\") pod \"e2e-shared-b-w3-dd5b30-openshift-default-86dfc4597d-b7w6c\" (UID: \"ecac9ada-1da0-4ecd-8af5-07501bd2aeb9\") " pod="openshift-ingress/e2e-shared-b-w3-dd5b30-openshift-default-86dfc4597d-b7w6c"
Sep 11 03:57:31.591644 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:57:31.591563    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ecac9ada-1da0-4ecd-8af5-07501bd2aeb9-credential-socket\") pod \"e2e-shared-b-w3-dd5b30-openshift-default-86dfc4597d-b7w6c\" (UID: \"ecac9ada-1da0-4ecd-8af5-07501bd2aeb9\") " pod="openshift-ingress/e2e-shared-b-w3-dd5b30-openshift-default-86dfc4597d-b7w6c"
Sep 11 03:57:31.591644 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:57:31.591600    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ecac9ada-1da0-4ecd-8af5-07501bd2aeb9-istio-envoy\") pod \"e2e-shared-b-w3-dd5b30-openshift-default-86dfc4597d-b7w6c\" (UID: \"ecac9ada-1da0-4ecd-8af5-07501bd2aeb9\") " pod="openshift-ingress/e2e-shared-b-w3-dd5b30-openshift-default-86dfc4597d-b7w6c"
Sep 11 03:57:31.692437 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:57:31.692396    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ecac9ada-1da0-4ecd-8af5-07501bd2aeb9-istio-podinfo\") pod \"e2e-shared-b-w3-dd5b30-openshift-default-86dfc4597d-b7w6c\" (UID: \"ecac9ada-1da0-4ecd-8af5-07501bd2aeb9\") " pod="openshift-ingress/e2e-shared-b-w3-dd5b30-openshift-default-86dfc4597d-b7w6c"
Sep 11 03:57:31.692437 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:57:31.692440    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ecac9ada-1da0-4ecd-8af5-07501bd2aeb9-workload-certs\") pod \"e2e-shared-b-w3-dd5b30-openshift-default-86dfc4597d-b7w6c\" (UID: \"ecac9ada-1da0-4ecd-8af5-07501bd2aeb9\") " pod="openshift-ingress/e2e-shared-b-w3-dd5b30-openshift-default-86dfc4597d-b7w6c"
Sep 11 03:57:31.692703 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:57:31.692471    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ecac9ada-1da0-4ecd-8af5-07501bd2aeb9-istiod-ca-cert\") pod \"e2e-shared-b-w3-dd5b30-openshift-default-86dfc4597d-b7w6c\" (UID: \"ecac9ada-1da0-4ecd-8af5-07501bd2aeb9\") " pod="openshift-ingress/e2e-shared-b-w3-dd5b30-openshift-default-86dfc4597d-b7w6c"
Sep 11 03:57:31.692703 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:57:31.692498    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ecac9ada-1da0-4ecd-8af5-07501bd2aeb9-istio-token\") pod \"e2e-shared-b-w3-dd5b30-openshift-default-86dfc4597d-b7w6c\" (UID: \"ecac9ada-1da0-4ecd-8af5-07501bd2aeb9\") " pod="openshift-ingress/e2e-shared-b-w3-dd5b30-openshift-default-86dfc4597d-b7w6c"
Sep 11 03:57:31.692703 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:57:31.692526    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-45g8t\" (UniqueName: \"kubernetes.io/projected/ecac9ada-1da0-4ecd-8af5-07501bd2aeb9-kube-api-access-45g8t\") pod \"e2e-shared-b-w3-dd5b30-openshift-default-86dfc4597d-b7w6c\" (UID: \"ecac9ada-1da0-4ecd-8af5-07501bd2aeb9\") " pod="openshift-ingress/e2e-shared-b-w3-dd5b30-openshift-default-86dfc4597d-b7w6c"
Sep 11 03:57:31.692703 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:57:31.692565    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ecac9ada-1da0-4ecd-8af5-07501bd2aeb9-workload-socket\") pod \"e2e-shared-b-w3-dd5b30-openshift-default-86dfc4597d-b7w6c\" (UID: \"ecac9ada-1da0-4ecd-8af5-07501bd2aeb9\") " pod="openshift-ingress/e2e-shared-b-w3-dd5b30-openshift-default-86dfc4597d-b7w6c"
Sep 11 03:57:31.692703 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:57:31.692606    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ecac9ada-1da0-4ecd-8af5-07501bd2aeb9-istio-data\") pod \"e2e-shared-b-w3-dd5b30-openshift-default-86dfc4597d-b7w6c\" (UID: \"ecac9ada-1da0-4ecd-8af5-07501bd2aeb9\") " pod="openshift-ingress/e2e-shared-b-w3-dd5b30-openshift-default-86dfc4597d-b7w6c"
Sep 11 03:57:31.692703 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:57:31.692653    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ecac9ada-1da0-4ecd-8af5-07501bd2aeb9-credential-socket\") pod \"e2e-shared-b-w3-dd5b30-openshift-default-86dfc4597d-b7w6c\" (UID: \"ecac9ada-1da0-4ecd-8af5-07501bd2aeb9\") " pod="openshift-ingress/e2e-shared-b-w3-dd5b30-openshift-default-86dfc4597d-b7w6c"
Sep 11 03:57:31.692703 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:57:31.692694    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ecac9ada-1da0-4ecd-8af5-07501bd2aeb9-istio-envoy\") pod \"e2e-shared-b-w3-dd5b30-openshift-default-86dfc4597d-b7w6c\" (UID: \"ecac9ada-1da0-4ecd-8af5-07501bd2aeb9\") " pod="openshift-ingress/e2e-shared-b-w3-dd5b30-openshift-default-86dfc4597d-b7w6c"
Sep 11 03:57:31.693136 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:57:31.692899    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ecac9ada-1da0-4ecd-8af5-07501bd2aeb9-workload-certs\") pod \"e2e-shared-b-w3-dd5b30-openshift-default-86dfc4597d-b7w6c\" (UID: \"ecac9ada-1da0-4ecd-8af5-07501bd2aeb9\") " pod="openshift-ingress/e2e-shared-b-w3-dd5b30-openshift-default-86dfc4597d-b7w6c"
Sep 11 03:57:31.693136 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:57:31.692966    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ecac9ada-1da0-4ecd-8af5-07501bd2aeb9-workload-socket\") pod \"e2e-shared-b-w3-dd5b30-openshift-default-86dfc4597d-b7w6c\" (UID: \"ecac9ada-1da0-4ecd-8af5-07501bd2aeb9\") " pod="openshift-ingress/e2e-shared-b-w3-dd5b30-openshift-default-86dfc4597d-b7w6c"
Sep 11 03:57:31.693136 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:57:31.693126    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ecac9ada-1da0-4ecd-8af5-07501bd2aeb9-istio-data\") pod \"e2e-shared-b-w3-dd5b30-openshift-default-86dfc4597d-b7w6c\" (UID: \"ecac9ada-1da0-4ecd-8af5-07501bd2aeb9\") " pod="openshift-ingress/e2e-shared-b-w3-dd5b30-openshift-default-86dfc4597d-b7w6c"
Sep 11 03:57:31.693312 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:57:31.693196    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ecac9ada-1da0-4ecd-8af5-07501bd2aeb9-credential-socket\") pod \"e2e-shared-b-w3-dd5b30-openshift-default-86dfc4597d-b7w6c\" (UID: \"ecac9ada-1da0-4ecd-8af5-07501bd2aeb9\") " pod="openshift-ingress/e2e-shared-b-w3-dd5b30-openshift-default-86dfc4597d-b7w6c"
Sep 11 03:57:31.693471 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:57:31.693449    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ecac9ada-1da0-4ecd-8af5-07501bd2aeb9-istiod-ca-cert\") pod \"e2e-shared-b-w3-dd5b30-openshift-default-86dfc4597d-b7w6c\" (UID: \"ecac9ada-1da0-4ecd-8af5-07501bd2aeb9\") " pod="openshift-ingress/e2e-shared-b-w3-dd5b30-openshift-default-86dfc4597d-b7w6c"
Sep 11 03:57:31.695041 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:57:31.695023    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ecac9ada-1da0-4ecd-8af5-07501bd2aeb9-istio-envoy\") pod \"e2e-shared-b-w3-dd5b30-openshift-default-86dfc4597d-b7w6c\" (UID: \"ecac9ada-1da0-4ecd-8af5-07501bd2aeb9\") " pod="openshift-ingress/e2e-shared-b-w3-dd5b30-openshift-default-86dfc4597d-b7w6c"
Sep 11 03:57:31.695318 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:57:31.695298    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ecac9ada-1da0-4ecd-8af5-07501bd2aeb9-istio-podinfo\") pod \"e2e-shared-b-w3-dd5b30-openshift-default-86dfc4597d-b7w6c\" (UID: \"ecac9ada-1da0-4ecd-8af5-07501bd2aeb9\") " pod="openshift-ingress/e2e-shared-b-w3-dd5b30-openshift-default-86dfc4597d-b7w6c"
Sep 11 03:57:31.700408 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:57:31.700382    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ecac9ada-1da0-4ecd-8af5-07501bd2aeb9-istio-token\") pod \"e2e-shared-b-w3-dd5b30-openshift-default-86dfc4597d-b7w6c\" (UID: \"ecac9ada-1da0-4ecd-8af5-07501bd2aeb9\") " pod="openshift-ingress/e2e-shared-b-w3-dd5b30-openshift-default-86dfc4597d-b7w6c"
Sep 11 03:57:31.700693 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:57:31.700676    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-45g8t\" (UniqueName: \"kubernetes.io/projected/ecac9ada-1da0-4ecd-8af5-07501bd2aeb9-kube-api-access-45g8t\") pod \"e2e-shared-b-w3-dd5b30-openshift-default-86dfc4597d-b7w6c\" (UID: \"ecac9ada-1da0-4ecd-8af5-07501bd2aeb9\") " pod="openshift-ingress/e2e-shared-b-w3-dd5b30-openshift-default-86dfc4597d-b7w6c"
Sep 11 03:57:31.795956 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:57:31.795867    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-dd5b30-openshift-default-86dfc4597d-b7w6c"
Sep 11 03:57:32.072497 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:57:32.072471    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-dd5b30-openshift-default-86dfc4597d-b7w6c"]
Sep 11 03:57:32.074691 ip-10-0-141-105 kubenswrapper[2579]: W0911 03:57:32.074655    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podecac9ada_1da0_4ecd_8af5_07501bd2aeb9.slice/crio-3535aee47fa72fd1cc1c34f3b90140b268394d818f1d2f5a9125a1915e0144f8 WatchSource:0}: Error finding container 3535aee47fa72fd1cc1c34f3b90140b268394d818f1d2f5a9125a1915e0144f8: Status 404 returned error can't find the container with id 3535aee47fa72fd1cc1c34f3b90140b268394d818f1d2f5a9125a1915e0144f8
Sep 11 03:57:32.076879 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:57:32.076837    2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 11 03:57:32.076974 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:57:32.076928    2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 11 03:57:32.076974 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:57:32.076965    2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 11 03:57:33.070702 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:57:33.070661    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-dd5b30-openshift-default-86dfc4597d-b7w6c" event={"ID":"ecac9ada-1da0-4ecd-8af5-07501bd2aeb9","Type":"ContainerStarted","Data":"546139f5e12a36002009e7a5aae8a01f1a1d969b0c8b5831054c704af544876e"}
Sep 11 03:57:33.071203 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:57:33.070717    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-dd5b30-openshift-default-86dfc4597d-b7w6c" event={"ID":"ecac9ada-1da0-4ecd-8af5-07501bd2aeb9","Type":"ContainerStarted","Data":"3535aee47fa72fd1cc1c34f3b90140b268394d818f1d2f5a9125a1915e0144f8"}
Sep 11 03:57:33.089751 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:57:33.089698    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w3-dd5b30-openshift-default-86dfc4597d-b7w6c" podStartSLOduration=2.089682017 podStartE2EDuration="2.089682017s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 03:57:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 03:57:33.088909755 +0000 UTC m=+1030.744115323" watchObservedRunningTime="2026-09-11 03:57:33.089682017 +0000 UTC m=+1030.744887583"
Sep 11 03:57:33.104243 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:57:33.104210    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-5z4cn"]
Sep 11 03:57:33.796236 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:57:33.796199    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w3-dd5b30-openshift-default-86dfc4597d-b7w6c"
Sep 11 03:57:33.801664 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:57:33.801633    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w3-dd5b30-openshift-default-86dfc4597d-b7w6c"
Sep 11 03:57:34.076872 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:57:34.076781    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w3-dd5b30-openshift-default-86dfc4597d-b7w6c"
Sep 11 03:57:34.077898 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:57:34.077880    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w3-dd5b30-openshift-default-86dfc4597d-b7w6c"
Sep 11 03:57:37.000634 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:57:37.000602    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-dd5b30-685f9c74fc-4dnsz"]
Sep 11 03:57:37.004265 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:57:37.004250    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-dd5b30-685f9c74fc-4dnsz"
Sep 11 03:57:37.006696 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:57:37.006671    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w3-dd5b30-dockercfg-c67g2\""
Sep 11 03:57:37.006815 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:57:37.006680    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w3-dd5b30\""
Sep 11 03:57:37.013649 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:57:37.013623    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-dd5b30-685f9c74fc-4dnsz"]
Sep 11 03:57:37.143043 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:57:37.143005    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cc2cq\" (UniqueName: \"kubernetes.io/projected/398172c7-42dc-417a-9d9e-ed3f5e04ecdc-kube-api-access-cc2cq\") pod \"payload-processing-e2e-shared-b-w3-dd5b30-685f9c74fc-4dnsz\" (UID: \"398172c7-42dc-417a-9d9e-ed3f5e04ecdc\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-dd5b30-685f9c74fc-4dnsz"
Sep 11 03:57:37.143255 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:57:37.143151    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/398172c7-42dc-417a-9d9e-ed3f5e04ecdc-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w3-dd5b30-685f9c74fc-4dnsz\" (UID: \"398172c7-42dc-417a-9d9e-ed3f5e04ecdc\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-dd5b30-685f9c74fc-4dnsz"
Sep 11 03:57:37.244053 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:57:37.244020    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cc2cq\" (UniqueName: \"kubernetes.io/projected/398172c7-42dc-417a-9d9e-ed3f5e04ecdc-kube-api-access-cc2cq\") pod \"payload-processing-e2e-shared-b-w3-dd5b30-685f9c74fc-4dnsz\" (UID: \"398172c7-42dc-417a-9d9e-ed3f5e04ecdc\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-dd5b30-685f9c74fc-4dnsz"
Sep 11 03:57:37.244230 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:57:37.244135    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/398172c7-42dc-417a-9d9e-ed3f5e04ecdc-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w3-dd5b30-685f9c74fc-4dnsz\" (UID: \"398172c7-42dc-417a-9d9e-ed3f5e04ecdc\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-dd5b30-685f9c74fc-4dnsz"
Sep 11 03:57:37.244790 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:57:37.244770    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/398172c7-42dc-417a-9d9e-ed3f5e04ecdc-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w3-dd5b30-685f9c74fc-4dnsz\" (UID: \"398172c7-42dc-417a-9d9e-ed3f5e04ecdc\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-dd5b30-685f9c74fc-4dnsz"
Sep 11 03:57:37.252100 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:57:37.252022    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cc2cq\" (UniqueName: \"kubernetes.io/projected/398172c7-42dc-417a-9d9e-ed3f5e04ecdc-kube-api-access-cc2cq\") pod \"payload-processing-e2e-shared-b-w3-dd5b30-685f9c74fc-4dnsz\" (UID: \"398172c7-42dc-417a-9d9e-ed3f5e04ecdc\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-dd5b30-685f9c74fc-4dnsz"
Sep 11 03:57:37.317641 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:57:37.317612    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-dd5b30-685f9c74fc-4dnsz"
Sep 11 03:57:37.588885 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:57:37.588861    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-dd5b30-685f9c74fc-4dnsz"]
Sep 11 03:57:37.591271 ip-10-0-141-105 kubenswrapper[2579]: W0911 03:57:37.591241    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod398172c7_42dc_417a_9d9e_ed3f5e04ecdc.slice/crio-708c8422ef5034fdd9392eabe0eb9247a12f72cd2a1705aba93d5531a79772b5 WatchSource:0}: Error finding container 708c8422ef5034fdd9392eabe0eb9247a12f72cd2a1705aba93d5531a79772b5: Status 404 returned error can't find the container with id 708c8422ef5034fdd9392eabe0eb9247a12f72cd2a1705aba93d5531a79772b5
Sep 11 03:57:38.100045 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:57:38.100011    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-dd5b30-685f9c74fc-4dnsz" event={"ID":"398172c7-42dc-417a-9d9e-ed3f5e04ecdc","Type":"ContainerStarted","Data":"80ad0471f297b6f88b10a16dff0b701bac36821c0b7b4768b4ad7217b07c9540"}
Sep 11 03:57:38.100440 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:57:38.100062    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-dd5b30-685f9c74fc-4dnsz" event={"ID":"398172c7-42dc-417a-9d9e-ed3f5e04ecdc","Type":"ContainerStarted","Data":"708c8422ef5034fdd9392eabe0eb9247a12f72cd2a1705aba93d5531a79772b5"}
Sep 11 03:57:38.100440 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:57:38.100153    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-dd5b30-685f9c74fc-4dnsz"
Sep 11 03:57:38.117155 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:57:38.117102    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-dd5b30-685f9c74fc-4dnsz" podStartSLOduration=2.117070041 podStartE2EDuration="2.117070041s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 03:57:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 03:57:38.115793049 +0000 UTC m=+1035.770998616" watchObservedRunningTime="2026-09-11 03:57:38.117070041 +0000 UTC m=+1035.772275608"
Sep 11 03:57:44.648142 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:57:44.645388    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-noten-96c7d4/noten-model-96c7d4-kserve-7bf94b96f5-gfvbq"]
Sep 11 03:57:46.399423 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:57:46.399387    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-5z4cn"]
Sep 11 03:57:49.107299 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:57:49.107265    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-dd5b30-685f9c74fc-4dnsz"
Sep 11 03:57:50.031978 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:57:50.031943    2579 patch_prober.go:28] interesting pod/e2e-shared-b-w3-dd5b30-openshift-default-86dfc4597d-b7w6c container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.64:15021/healthz/ready\": read tcp 10.134.0.2:56092->10.134.0.64:15021: read: connection reset by peer" start-of-body=
Sep 11 03:57:50.032422 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:57:50.032207    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-w3-dd5b30-openshift-default-86dfc4597d-b7w6c" podUID="ecac9ada-1da0-4ecd-8af5-07501bd2aeb9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.64:15021/healthz/ready\": read tcp 10.134.0.2:56092->10.134.0.64:15021: read: connection reset by peer"
Sep 11 03:57:50.173532 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:57:50.173503    2579 generic.go:356] "Generic (PLEG): container finished" podID="ecac9ada-1da0-4ecd-8af5-07501bd2aeb9" containerID="546139f5e12a36002009e7a5aae8a01f1a1d969b0c8b5831054c704af544876e" exitCode=0
Sep 11 03:57:50.173893 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:57:50.173578    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-dd5b30-openshift-default-86dfc4597d-b7w6c" event={"ID":"ecac9ada-1da0-4ecd-8af5-07501bd2aeb9","Type":"ContainerDied","Data":"546139f5e12a36002009e7a5aae8a01f1a1d969b0c8b5831054c704af544876e"}
Sep 11 03:57:50.173972 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:57:50.173957    2579 scope.go:117] "RemoveContainer" containerID="546139f5e12a36002009e7a5aae8a01f1a1d969b0c8b5831054c704af544876e"
Sep 11 03:57:50.174989 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:57:50.174964    2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 11 03:57:50.175064 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:57:50.175034    2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 11 03:57:50.175126 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:57:50.175065    2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 11 03:57:51.110337 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:57:51.110307    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-20ae9a-5ff4665549-ln6wx"]
Sep 11 03:57:51.112816 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:57:51.112796    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-20ae9a-5ff4665549-ln6wx"
Sep 11 03:57:51.115348 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:57:51.115316    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w2-20ae9a-dockercfg-pjqlm\""
Sep 11 03:57:51.115472 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:57:51.115362    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w2-20ae9a\""
Sep 11 03:57:51.121806 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:57:51.121784    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-20ae9a-5ff4665549-ln6wx"]
Sep 11 03:57:51.183046 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:57:51.183007    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-dd5b30-openshift-default-86dfc4597d-b7w6c" event={"ID":"ecac9ada-1da0-4ecd-8af5-07501bd2aeb9","Type":"ContainerStarted","Data":"930fc6e9f3aa5bf551dd5ad8dad00882fe941e3ed3eb39c5a9b919681afc894b"}
Sep 11 03:57:51.281952 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:57:51.281921    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xqhpb\" (UniqueName: \"kubernetes.io/projected/79b0235c-5f8a-4dd2-ab0b-e64a95a231aa-kube-api-access-xqhpb\") pod \"payload-processing-e2e-shared-a-w2-20ae9a-5ff4665549-ln6wx\" (UID: \"79b0235c-5f8a-4dd2-ab0b-e64a95a231aa\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-20ae9a-5ff4665549-ln6wx"
Sep 11 03:57:51.282152 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:57:51.282041    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/79b0235c-5f8a-4dd2-ab0b-e64a95a231aa-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w2-20ae9a-5ff4665549-ln6wx\" (UID: \"79b0235c-5f8a-4dd2-ab0b-e64a95a231aa\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-20ae9a-5ff4665549-ln6wx"
Sep 11 03:57:51.383404 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:57:51.383315    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/79b0235c-5f8a-4dd2-ab0b-e64a95a231aa-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w2-20ae9a-5ff4665549-ln6wx\" (UID: \"79b0235c-5f8a-4dd2-ab0b-e64a95a231aa\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-20ae9a-5ff4665549-ln6wx"
Sep 11 03:57:51.383557 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:57:51.383420    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xqhpb\" (UniqueName: \"kubernetes.io/projected/79b0235c-5f8a-4dd2-ab0b-e64a95a231aa-kube-api-access-xqhpb\") pod \"payload-processing-e2e-shared-a-w2-20ae9a-5ff4665549-ln6wx\" (UID: \"79b0235c-5f8a-4dd2-ab0b-e64a95a231aa\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-20ae9a-5ff4665549-ln6wx"
Sep 11 03:57:51.383996 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:57:51.383974    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/79b0235c-5f8a-4dd2-ab0b-e64a95a231aa-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w2-20ae9a-5ff4665549-ln6wx\" (UID: \"79b0235c-5f8a-4dd2-ab0b-e64a95a231aa\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-20ae9a-5ff4665549-ln6wx"
Sep 11 03:57:51.392898 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:57:51.392865    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xqhpb\" (UniqueName: \"kubernetes.io/projected/79b0235c-5f8a-4dd2-ab0b-e64a95a231aa-kube-api-access-xqhpb\") pod \"payload-processing-e2e-shared-a-w2-20ae9a-5ff4665549-ln6wx\" (UID: \"79b0235c-5f8a-4dd2-ab0b-e64a95a231aa\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-20ae9a-5ff4665549-ln6wx"
Sep 11 03:57:51.424657 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:57:51.424631    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-20ae9a-5ff4665549-ln6wx"
Sep 11 03:57:51.699576 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:57:51.697569    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-20ae9a-5ff4665549-ln6wx"]
Sep 11 03:57:51.796359 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:57:51.796336    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w3-dd5b30-openshift-default-86dfc4597d-b7w6c"
Sep 11 03:57:51.796525 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:57:51.796506    2579 patch_prober.go:28] interesting pod/e2e-shared-b-w3-dd5b30-openshift-default-86dfc4597d-b7w6c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" start-of-body=
Sep 11 03:57:51.796576 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:57:51.796552    2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-dd5b30-openshift-default-86dfc4597d-b7w6c" podUID="ecac9ada-1da0-4ecd-8af5-07501bd2aeb9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused"
Sep 11 03:57:52.190639 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:57:52.190602    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-20ae9a-5ff4665549-ln6wx" event={"ID":"79b0235c-5f8a-4dd2-ab0b-e64a95a231aa","Type":"ContainerStarted","Data":"96487f2505444c33875b17f0dc76ff2105f031439eac04eebc2f734b863ed505"}
Sep 11 03:57:52.191132 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:57:52.190659    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-20ae9a-5ff4665549-ln6wx" event={"ID":"79b0235c-5f8a-4dd2-ab0b-e64a95a231aa","Type":"ContainerStarted","Data":"86484cf4fd599ccac817633e673d55ada5ce77977af5cefa7b4e2df6c88215f1"}
Sep 11 03:57:52.207428 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:57:52.207378    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-20ae9a-5ff4665549-ln6wx" podStartSLOduration=1.207361952 podStartE2EDuration="1.207361952s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 03:57:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 03:57:52.206197363 +0000 UTC m=+1049.861402932" watchObservedRunningTime="2026-09-11 03:57:52.207361952 +0000 UTC m=+1049.862567518"
Sep 11 03:57:52.797159 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:57:52.797119    2579 patch_prober.go:28] interesting pod/e2e-shared-b-w3-dd5b30-openshift-default-86dfc4597d-b7w6c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" start-of-body=
Sep 11 03:57:52.797339 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:57:52.797185    2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-dd5b30-openshift-default-86dfc4597d-b7w6c" podUID="ecac9ada-1da0-4ecd-8af5-07501bd2aeb9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused"
Sep 11 03:57:53.195858 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:57:53.195827    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-20ae9a-5ff4665549-ln6wx"
Sep 11 03:57:53.797361 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:57:53.797321    2579 patch_prober.go:28] interesting pod/e2e-shared-b-w3-dd5b30-openshift-default-86dfc4597d-b7w6c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" start-of-body=
Sep 11 03:57:53.797559 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:57:53.797388    2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-dd5b30-openshift-default-86dfc4597d-b7w6c" podUID="ecac9ada-1da0-4ecd-8af5-07501bd2aeb9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused"
Sep 11 03:57:54.796689 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:57:54.796653    2579 patch_prober.go:28] interesting pod/e2e-shared-b-w3-dd5b30-openshift-default-86dfc4597d-b7w6c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" start-of-body=
Sep 11 03:57:54.797132 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:57:54.796718    2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-dd5b30-openshift-default-86dfc4597d-b7w6c" podUID="ecac9ada-1da0-4ecd-8af5-07501bd2aeb9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused"
Sep 11 03:57:55.797101 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:57:55.797052    2579 patch_prober.go:28] interesting pod/e2e-shared-b-w3-dd5b30-openshift-default-86dfc4597d-b7w6c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" start-of-body=
Sep 11 03:57:55.797572 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:57:55.797143    2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-dd5b30-openshift-default-86dfc4597d-b7w6c" podUID="ecac9ada-1da0-4ecd-8af5-07501bd2aeb9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused"
Sep 11 03:57:56.796346 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:57:56.796298    2579 patch_prober.go:28] interesting pod/e2e-shared-b-w3-dd5b30-openshift-default-86dfc4597d-b7w6c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" start-of-body=
Sep 11 03:57:56.796558 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:57:56.796374    2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-dd5b30-openshift-default-86dfc4597d-b7w6c" podUID="ecac9ada-1da0-4ecd-8af5-07501bd2aeb9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused"
Sep 11 03:57:56.917013 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:57:56.916970    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-5z4cn"]
Sep 11 03:57:57.800625 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:57:57.800596    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w3-dd5b30-openshift-default-86dfc4597d-b7w6c"
Sep 11 03:57:57.800899 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:57:57.800872    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w3-dd5b30-openshift-default-86dfc4597d-b7w6c"
Sep 11 03:57:57.801753 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:57:57.801735    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w3-dd5b30-openshift-default-86dfc4597d-b7w6c"
Sep 11 03:57:59.676262 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:57:59.676201    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-noten-96c7d4/noten-model-96c7d4-kserve-7bf94b96f5-gfvbq" podUID="21600213-6b07-4c47-a178-6fc211e22a1f" containerName="main" containerID="cri-o://72031f9e4544752e7127d372d9291b30efd8192f6daf5c5ec5cb20c6e9b0b98a" gracePeriod=45
Sep 11 03:57:59.924884 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:57:59.924859    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-noten-96c7d4/noten-model-96c7d4-kserve-7bf94b96f5-gfvbq"
Sep 11 03:57:59.958045 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:57:59.957972    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/21600213-6b07-4c47-a178-6fc211e22a1f-tmp-dir\") pod \"21600213-6b07-4c47-a178-6fc211e22a1f\" (UID: \"21600213-6b07-4c47-a178-6fc211e22a1f\") "
Sep 11 03:57:59.958045 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:57:59.958010    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/21600213-6b07-4c47-a178-6fc211e22a1f-dshm\") pod \"21600213-6b07-4c47-a178-6fc211e22a1f\" (UID: \"21600213-6b07-4c47-a178-6fc211e22a1f\") "
Sep 11 03:57:59.958045 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:57:59.958033    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/21600213-6b07-4c47-a178-6fc211e22a1f-home\") pod \"21600213-6b07-4c47-a178-6fc211e22a1f\" (UID: \"21600213-6b07-4c47-a178-6fc211e22a1f\") "
Sep 11 03:57:59.958386 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:57:59.958057    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/21600213-6b07-4c47-a178-6fc211e22a1f-tls-certs\") pod \"21600213-6b07-4c47-a178-6fc211e22a1f\" (UID: \"21600213-6b07-4c47-a178-6fc211e22a1f\") "
Sep 11 03:57:59.958386 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:57:59.958099    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6cqtt\" (UniqueName: \"kubernetes.io/projected/21600213-6b07-4c47-a178-6fc211e22a1f-kube-api-access-6cqtt\") pod \"21600213-6b07-4c47-a178-6fc211e22a1f\" (UID: \"21600213-6b07-4c47-a178-6fc211e22a1f\") "
Sep 11 03:57:59.958386 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:57:59.958232    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/21600213-6b07-4c47-a178-6fc211e22a1f-model-cache\") pod \"21600213-6b07-4c47-a178-6fc211e22a1f\" (UID: \"21600213-6b07-4c47-a178-6fc211e22a1f\") "
Sep 11 03:57:59.958386 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:57:59.958251    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/21600213-6b07-4c47-a178-6fc211e22a1f-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "21600213-6b07-4c47-a178-6fc211e22a1f" (UID: "21600213-6b07-4c47-a178-6fc211e22a1f"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 03:57:59.958627 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:57:59.958380    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/21600213-6b07-4c47-a178-6fc211e22a1f-home" (OuterVolumeSpecName: "home") pod "21600213-6b07-4c47-a178-6fc211e22a1f" (UID: "21600213-6b07-4c47-a178-6fc211e22a1f"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 03:57:59.958627 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:57:59.958526    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/21600213-6b07-4c47-a178-6fc211e22a1f-model-cache" (OuterVolumeSpecName: "model-cache") pod "21600213-6b07-4c47-a178-6fc211e22a1f" (UID: "21600213-6b07-4c47-a178-6fc211e22a1f"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 03:57:59.958627 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:57:59.958609    2579 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/21600213-6b07-4c47-a178-6fc211e22a1f-model-cache\") on node \"ip-10-0-141-105.ec2.internal\" DevicePath \"\""
Sep 11 03:57:59.958793 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:57:59.958628    2579 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/21600213-6b07-4c47-a178-6fc211e22a1f-tmp-dir\") on node \"ip-10-0-141-105.ec2.internal\" DevicePath \"\""
Sep 11 03:57:59.958793 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:57:59.958641    2579 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/21600213-6b07-4c47-a178-6fc211e22a1f-home\") on node \"ip-10-0-141-105.ec2.internal\" DevicePath \"\""
Sep 11 03:57:59.960568 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:57:59.960536    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/21600213-6b07-4c47-a178-6fc211e22a1f-kube-api-access-6cqtt" (OuterVolumeSpecName: "kube-api-access-6cqtt") pod "21600213-6b07-4c47-a178-6fc211e22a1f" (UID: "21600213-6b07-4c47-a178-6fc211e22a1f"). InnerVolumeSpecName "kube-api-access-6cqtt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 03:57:59.960568 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:57:59.960558    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/21600213-6b07-4c47-a178-6fc211e22a1f-dshm" (OuterVolumeSpecName: "dshm") pod "21600213-6b07-4c47-a178-6fc211e22a1f" (UID: "21600213-6b07-4c47-a178-6fc211e22a1f"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 03:57:59.960712 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:57:59.960615    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/21600213-6b07-4c47-a178-6fc211e22a1f-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "21600213-6b07-4c47-a178-6fc211e22a1f" (UID: "21600213-6b07-4c47-a178-6fc211e22a1f"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 03:58:00.059439 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:58:00.059400    2579 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/21600213-6b07-4c47-a178-6fc211e22a1f-dshm\") on node \"ip-10-0-141-105.ec2.internal\" DevicePath \"\""
Sep 11 03:58:00.059439 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:58:00.059435    2579 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/21600213-6b07-4c47-a178-6fc211e22a1f-tls-certs\") on node \"ip-10-0-141-105.ec2.internal\" DevicePath \"\""
Sep 11 03:58:00.059439 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:58:00.059447    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6cqtt\" (UniqueName: \"kubernetes.io/projected/21600213-6b07-4c47-a178-6fc211e22a1f-kube-api-access-6cqtt\") on node \"ip-10-0-141-105.ec2.internal\" DevicePath \"\""
Sep 11 03:58:00.240319 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:58:00.240218    2579 generic.go:356] "Generic (PLEG): container finished" podID="21600213-6b07-4c47-a178-6fc211e22a1f" containerID="72031f9e4544752e7127d372d9291b30efd8192f6daf5c5ec5cb20c6e9b0b98a" exitCode=0
Sep 11 03:58:00.240319 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:58:00.240264    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-noten-96c7d4/noten-model-96c7d4-kserve-7bf94b96f5-gfvbq"
Sep 11 03:58:00.240528 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:58:00.240301    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-noten-96c7d4/noten-model-96c7d4-kserve-7bf94b96f5-gfvbq" event={"ID":"21600213-6b07-4c47-a178-6fc211e22a1f","Type":"ContainerDied","Data":"72031f9e4544752e7127d372d9291b30efd8192f6daf5c5ec5cb20c6e9b0b98a"}
Sep 11 03:58:00.240528 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:58:00.240355    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-noten-96c7d4/noten-model-96c7d4-kserve-7bf94b96f5-gfvbq" event={"ID":"21600213-6b07-4c47-a178-6fc211e22a1f","Type":"ContainerDied","Data":"5cb1f011e98e41891b3f3d619d0abd01001daa26ea0f746ac060637ab152e5d1"}
Sep 11 03:58:00.240528 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:58:00.240372    2579 scope.go:117] "RemoveContainer" containerID="72031f9e4544752e7127d372d9291b30efd8192f6daf5c5ec5cb20c6e9b0b98a"
Sep 11 03:58:00.250277 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:58:00.250261    2579 scope.go:117] "RemoveContainer" containerID="72031f9e4544752e7127d372d9291b30efd8192f6daf5c5ec5cb20c6e9b0b98a"
Sep 11 03:58:00.250574 ip-10-0-141-105 kubenswrapper[2579]: E0911 03:58:00.250555    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"72031f9e4544752e7127d372d9291b30efd8192f6daf5c5ec5cb20c6e9b0b98a\": container with ID starting with 72031f9e4544752e7127d372d9291b30efd8192f6daf5c5ec5cb20c6e9b0b98a not found: ID does not exist" containerID="72031f9e4544752e7127d372d9291b30efd8192f6daf5c5ec5cb20c6e9b0b98a"
Sep 11 03:58:00.250662 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:58:00.250582    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"72031f9e4544752e7127d372d9291b30efd8192f6daf5c5ec5cb20c6e9b0b98a"} err="failed to get container status \"72031f9e4544752e7127d372d9291b30efd8192f6daf5c5ec5cb20c6e9b0b98a\": rpc error: code = NotFound desc = could not find container \"72031f9e4544752e7127d372d9291b30efd8192f6daf5c5ec5cb20c6e9b0b98a\": container with ID starting with 72031f9e4544752e7127d372d9291b30efd8192f6daf5c5ec5cb20c6e9b0b98a not found: ID does not exist"
Sep 11 03:58:00.260620 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:58:00.260579    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-noten-96c7d4/noten-model-96c7d4-kserve-7bf94b96f5-gfvbq"]
Sep 11 03:58:00.262845 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:58:00.262818    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-noten-96c7d4/noten-model-96c7d4-kserve-7bf94b96f5-gfvbq"]
Sep 11 03:58:00.851931 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:58:00.851898    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="21600213-6b07-4c47-a178-6fc211e22a1f" path="/var/lib/kubelet/pods/21600213-6b07-4c47-a178-6fc211e22a1f/volumes"
Sep 11 03:58:04.203223 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:58:04.203193    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-20ae9a-5ff4665549-ln6wx"
Sep 11 03:58:06.991040 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:58:06.991004    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w3-b4d3cf/auth-test-model-b4d3cf-kserve-7b594c5fff-dp5pp"]
Sep 11 03:58:06.991504 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:58:06.991380    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="21600213-6b07-4c47-a178-6fc211e22a1f" containerName="main"
Sep 11 03:58:06.991504 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:58:06.991391    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="21600213-6b07-4c47-a178-6fc211e22a1f" containerName="main"
Sep 11 03:58:06.991504 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:58:06.991459    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="21600213-6b07-4c47-a178-6fc211e22a1f" containerName="main"
Sep 11 03:58:06.995566 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:58:06.995543    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-b4d3cf/auth-test-model-b4d3cf-kserve-7b594c5fff-dp5pp"
Sep 11 03:58:06.998171 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:58:06.998146    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w3-b4d3cf\"/\"openshift-service-ca.crt\""
Sep 11 03:58:06.998356 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:58:06.998330    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-b4d3cf\"/\"default-dockercfg-5x4f6\""
Sep 11 03:58:06.999442 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:58:06.999424    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w3-b4d3cf\"/\"kube-root-ca.crt\""
Sep 11 03:58:06.999552 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:58:06.999498    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-b4d3cf\"/\"auth-test-model-b4d3cf-kserve-self-signed-certs\""
Sep 11 03:58:07.004721 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:58:07.004696    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-b4d3cf/auth-test-model-b4d3cf-kserve-7b594c5fff-dp5pp"]
Sep 11 03:58:07.024452 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:58:07.024424    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/37955b55-9d5a-4d4e-8883-399b2160d5b8-tmp-dir\") pod \"auth-test-model-b4d3cf-kserve-7b594c5fff-dp5pp\" (UID: \"37955b55-9d5a-4d4e-8883-399b2160d5b8\") " pod="ai-tenant-e2e-shared-a-w3-b4d3cf/auth-test-model-b4d3cf-kserve-7b594c5fff-dp5pp"
Sep 11 03:58:07.024617 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:58:07.024470    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/37955b55-9d5a-4d4e-8883-399b2160d5b8-tls-certs\") pod \"auth-test-model-b4d3cf-kserve-7b594c5fff-dp5pp\" (UID: \"37955b55-9d5a-4d4e-8883-399b2160d5b8\") " pod="ai-tenant-e2e-shared-a-w3-b4d3cf/auth-test-model-b4d3cf-kserve-7b594c5fff-dp5pp"
Sep 11 03:58:07.024617 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:58:07.024542    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/37955b55-9d5a-4d4e-8883-399b2160d5b8-model-cache\") pod \"auth-test-model-b4d3cf-kserve-7b594c5fff-dp5pp\" (UID: \"37955b55-9d5a-4d4e-8883-399b2160d5b8\") " pod="ai-tenant-e2e-shared-a-w3-b4d3cf/auth-test-model-b4d3cf-kserve-7b594c5fff-dp5pp"
Sep 11 03:58:07.024700 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:58:07.024627    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/37955b55-9d5a-4d4e-8883-399b2160d5b8-home\") pod \"auth-test-model-b4d3cf-kserve-7b594c5fff-dp5pp\" (UID: \"37955b55-9d5a-4d4e-8883-399b2160d5b8\") " pod="ai-tenant-e2e-shared-a-w3-b4d3cf/auth-test-model-b4d3cf-kserve-7b594c5fff-dp5pp"
Sep 11 03:58:07.024700 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:58:07.024666    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cz28k\" (UniqueName: \"kubernetes.io/projected/37955b55-9d5a-4d4e-8883-399b2160d5b8-kube-api-access-cz28k\") pod \"auth-test-model-b4d3cf-kserve-7b594c5fff-dp5pp\" (UID: \"37955b55-9d5a-4d4e-8883-399b2160d5b8\") " pod="ai-tenant-e2e-shared-a-w3-b4d3cf/auth-test-model-b4d3cf-kserve-7b594c5fff-dp5pp"
Sep 11 03:58:07.024774 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:58:07.024727    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/37955b55-9d5a-4d4e-8883-399b2160d5b8-dshm\") pod \"auth-test-model-b4d3cf-kserve-7b594c5fff-dp5pp\" (UID: \"37955b55-9d5a-4d4e-8883-399b2160d5b8\") " pod="ai-tenant-e2e-shared-a-w3-b4d3cf/auth-test-model-b4d3cf-kserve-7b594c5fff-dp5pp"
Sep 11 03:58:07.125766 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:58:07.125732    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/37955b55-9d5a-4d4e-8883-399b2160d5b8-home\") pod \"auth-test-model-b4d3cf-kserve-7b594c5fff-dp5pp\" (UID: \"37955b55-9d5a-4d4e-8883-399b2160d5b8\") " pod="ai-tenant-e2e-shared-a-w3-b4d3cf/auth-test-model-b4d3cf-kserve-7b594c5fff-dp5pp"
Sep 11 03:58:07.125766 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:58:07.125768    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cz28k\" (UniqueName: \"kubernetes.io/projected/37955b55-9d5a-4d4e-8883-399b2160d5b8-kube-api-access-cz28k\") pod \"auth-test-model-b4d3cf-kserve-7b594c5fff-dp5pp\" (UID: \"37955b55-9d5a-4d4e-8883-399b2160d5b8\") " pod="ai-tenant-e2e-shared-a-w3-b4d3cf/auth-test-model-b4d3cf-kserve-7b594c5fff-dp5pp"
Sep 11 03:58:07.126034 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:58:07.125809    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/37955b55-9d5a-4d4e-8883-399b2160d5b8-dshm\") pod \"auth-test-model-b4d3cf-kserve-7b594c5fff-dp5pp\" (UID: \"37955b55-9d5a-4d4e-8883-399b2160d5b8\") " pod="ai-tenant-e2e-shared-a-w3-b4d3cf/auth-test-model-b4d3cf-kserve-7b594c5fff-dp5pp"
Sep 11 03:58:07.126034 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:58:07.125848    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/37955b55-9d5a-4d4e-8883-399b2160d5b8-tmp-dir\") pod \"auth-test-model-b4d3cf-kserve-7b594c5fff-dp5pp\" (UID: \"37955b55-9d5a-4d4e-8883-399b2160d5b8\") " pod="ai-tenant-e2e-shared-a-w3-b4d3cf/auth-test-model-b4d3cf-kserve-7b594c5fff-dp5pp"
Sep 11 03:58:07.126034 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:58:07.125871    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/37955b55-9d5a-4d4e-8883-399b2160d5b8-tls-certs\") pod \"auth-test-model-b4d3cf-kserve-7b594c5fff-dp5pp\" (UID: \"37955b55-9d5a-4d4e-8883-399b2160d5b8\") " pod="ai-tenant-e2e-shared-a-w3-b4d3cf/auth-test-model-b4d3cf-kserve-7b594c5fff-dp5pp"
Sep 11 03:58:07.126034 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:58:07.125917    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/37955b55-9d5a-4d4e-8883-399b2160d5b8-model-cache\") pod \"auth-test-model-b4d3cf-kserve-7b594c5fff-dp5pp\" (UID: \"37955b55-9d5a-4d4e-8883-399b2160d5b8\") " pod="ai-tenant-e2e-shared-a-w3-b4d3cf/auth-test-model-b4d3cf-kserve-7b594c5fff-dp5pp"
Sep 11 03:58:07.126283 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:58:07.126176    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/37955b55-9d5a-4d4e-8883-399b2160d5b8-home\") pod \"auth-test-model-b4d3cf-kserve-7b594c5fff-dp5pp\" (UID: \"37955b55-9d5a-4d4e-8883-399b2160d5b8\") " pod="ai-tenant-e2e-shared-a-w3-b4d3cf/auth-test-model-b4d3cf-kserve-7b594c5fff-dp5pp"
Sep 11 03:58:07.126283 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:58:07.126245    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/37955b55-9d5a-4d4e-8883-399b2160d5b8-tmp-dir\") pod \"auth-test-model-b4d3cf-kserve-7b594c5fff-dp5pp\" (UID: \"37955b55-9d5a-4d4e-8883-399b2160d5b8\") " pod="ai-tenant-e2e-shared-a-w3-b4d3cf/auth-test-model-b4d3cf-kserve-7b594c5fff-dp5pp"
Sep 11 03:58:07.126388 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:58:07.126297    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/37955b55-9d5a-4d4e-8883-399b2160d5b8-model-cache\") pod \"auth-test-model-b4d3cf-kserve-7b594c5fff-dp5pp\" (UID: \"37955b55-9d5a-4d4e-8883-399b2160d5b8\") " pod="ai-tenant-e2e-shared-a-w3-b4d3cf/auth-test-model-b4d3cf-kserve-7b594c5fff-dp5pp"
Sep 11 03:58:07.128380 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:58:07.128351    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/37955b55-9d5a-4d4e-8883-399b2160d5b8-dshm\") pod \"auth-test-model-b4d3cf-kserve-7b594c5fff-dp5pp\" (UID: \"37955b55-9d5a-4d4e-8883-399b2160d5b8\") " pod="ai-tenant-e2e-shared-a-w3-b4d3cf/auth-test-model-b4d3cf-kserve-7b594c5fff-dp5pp"
Sep 11 03:58:07.128501 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:58:07.128426    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/37955b55-9d5a-4d4e-8883-399b2160d5b8-tls-certs\") pod \"auth-test-model-b4d3cf-kserve-7b594c5fff-dp5pp\" (UID: \"37955b55-9d5a-4d4e-8883-399b2160d5b8\") " pod="ai-tenant-e2e-shared-a-w3-b4d3cf/auth-test-model-b4d3cf-kserve-7b594c5fff-dp5pp"
Sep 11 03:58:07.135036 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:58:07.135013    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cz28k\" (UniqueName: \"kubernetes.io/projected/37955b55-9d5a-4d4e-8883-399b2160d5b8-kube-api-access-cz28k\") pod \"auth-test-model-b4d3cf-kserve-7b594c5fff-dp5pp\" (UID: \"37955b55-9d5a-4d4e-8883-399b2160d5b8\") " pod="ai-tenant-e2e-shared-a-w3-b4d3cf/auth-test-model-b4d3cf-kserve-7b594c5fff-dp5pp"
Sep 11 03:58:07.307074 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:58:07.306987    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-b4d3cf/auth-test-model-b4d3cf-kserve-7b594c5fff-dp5pp"
Sep 11 03:58:07.582054 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:58:07.582024    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-b4d3cf/auth-test-model-b4d3cf-kserve-7b594c5fff-dp5pp"]
Sep 11 03:58:07.584359 ip-10-0-141-105 kubenswrapper[2579]: W0911 03:58:07.584327    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod37955b55_9d5a_4d4e_8883_399b2160d5b8.slice/crio-b0d3e9ffd66b2e6abac6c59784bc2faa6a3fce32f39c484e1776700bbf41341c WatchSource:0}: Error finding container b0d3e9ffd66b2e6abac6c59784bc2faa6a3fce32f39c484e1776700bbf41341c: Status 404 returned error can't find the container with id b0d3e9ffd66b2e6abac6c59784bc2faa6a3fce32f39c484e1776700bbf41341c
Sep 11 03:58:08.285252 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:58:08.285218    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-b4d3cf/auth-test-model-b4d3cf-kserve-7b594c5fff-dp5pp" event={"ID":"37955b55-9d5a-4d4e-8883-399b2160d5b8","Type":"ContainerStarted","Data":"b865bb14d5b0b15fe27fc2f89f0eb871c8d7d37fb326b2a8be00e20e1e73ae75"}
Sep 11 03:58:08.285645 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:58:08.285268    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-b4d3cf/auth-test-model-b4d3cf-kserve-7b594c5fff-dp5pp" event={"ID":"37955b55-9d5a-4d4e-8883-399b2160d5b8","Type":"ContainerStarted","Data":"b0d3e9ffd66b2e6abac6c59784bc2faa6a3fce32f39c484e1776700bbf41341c"}
Sep 11 03:58:08.301073 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:58:08.301028    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w3-b4d3cf/auth-test-model-b4d3cf-kserve-7b594c5fff-dp5pp" podStartSLOduration=2.30100501 podStartE2EDuration="2.30100501s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 03:58:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 03:58:08.299935111 +0000 UTC m=+1065.955140680" watchObservedRunningTime="2026-09-11 03:58:08.30100501 +0000 UTC m=+1065.956210577"
Sep 11 03:58:10.508863 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:58:10.508823    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-5z4cn"]
Sep 11 03:58:17.307514 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:58:17.307429    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w3-b4d3cf/auth-test-model-b4d3cf-kserve-7b594c5fff-dp5pp"
Sep 11 03:58:17.307967 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:58:17.307696    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w3-b4d3cf/auth-test-model-b4d3cf-kserve-7b594c5fff-dp5pp"
Sep 11 03:58:17.321068 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:58:17.321035    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w3-b4d3cf/auth-test-model-b4d3cf-kserve-7b594c5fff-dp5pp"
Sep 11 03:58:17.348214 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:58:17.348187    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w3-b4d3cf/auth-test-model-b4d3cf-kserve-7b594c5fff-dp5pp"
Sep 11 03:58:19.581584 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:58:19.581549    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-7dc787-d9778787d-k5mkc"]
Sep 11 03:58:19.584758 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:58:19.584741    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-7dc787-d9778787d-k5mkc"
Sep 11 03:58:19.587116 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:58:19.587056    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w2-7dc787\""
Sep 11 03:58:19.587263 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:58:19.587153    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w2-7dc787-dockercfg-2rb6k\""
Sep 11 03:58:19.595816 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:58:19.595781    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-7dc787-d9778787d-k5mkc"]
Sep 11 03:58:19.608016 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:58:19.607979    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-7dc787-744d659f5-vv99b"]
Sep 11 03:58:19.611259 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:58:19.611240    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-7dc787-744d659f5-vv99b"
Sep 11 03:58:19.620343 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:58:19.620317    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-7dc787-744d659f5-vv99b"]
Sep 11 03:58:19.639658 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:58:19.639622    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ddaa3ca0-9cba-48a7-8e0b-3f7881aa7e81-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w2-7dc787-d9778787d-k5mkc\" (UID: \"ddaa3ca0-9cba-48a7-8e0b-3f7881aa7e81\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-7dc787-d9778787d-k5mkc"
Sep 11 03:58:19.639819 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:58:19.639688    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j2s2j\" (UniqueName: \"kubernetes.io/projected/ddaa3ca0-9cba-48a7-8e0b-3f7881aa7e81-kube-api-access-j2s2j\") pod \"payload-pre-processing-e2e-shared-b-w2-7dc787-d9778787d-k5mkc\" (UID: \"ddaa3ca0-9cba-48a7-8e0b-3f7881aa7e81\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-7dc787-d9778787d-k5mkc"
Sep 11 03:58:19.740403 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:58:19.740362    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ddaa3ca0-9cba-48a7-8e0b-3f7881aa7e81-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w2-7dc787-d9778787d-k5mkc\" (UID: \"ddaa3ca0-9cba-48a7-8e0b-3f7881aa7e81\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-7dc787-d9778787d-k5mkc"
Sep 11 03:58:19.740609 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:58:19.740416    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j2s2j\" (UniqueName: \"kubernetes.io/projected/ddaa3ca0-9cba-48a7-8e0b-3f7881aa7e81-kube-api-access-j2s2j\") pod \"payload-pre-processing-e2e-shared-b-w2-7dc787-d9778787d-k5mkc\" (UID: \"ddaa3ca0-9cba-48a7-8e0b-3f7881aa7e81\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-7dc787-d9778787d-k5mkc"
Sep 11 03:58:19.740609 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:58:19.740451    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dkgmg\" (UniqueName: \"kubernetes.io/projected/14ea3d95-462e-4f0e-aee3-d37f36f30204-kube-api-access-dkgmg\") pod \"payload-processing-e2e-shared-b-w2-7dc787-744d659f5-vv99b\" (UID: \"14ea3d95-462e-4f0e-aee3-d37f36f30204\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-7dc787-744d659f5-vv99b"
Sep 11 03:58:19.740609 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:58:19.740497    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/14ea3d95-462e-4f0e-aee3-d37f36f30204-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w2-7dc787-744d659f5-vv99b\" (UID: \"14ea3d95-462e-4f0e-aee3-d37f36f30204\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-7dc787-744d659f5-vv99b"
Sep 11 03:58:19.741228 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:58:19.741203    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ddaa3ca0-9cba-48a7-8e0b-3f7881aa7e81-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w2-7dc787-d9778787d-k5mkc\" (UID: \"ddaa3ca0-9cba-48a7-8e0b-3f7881aa7e81\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-7dc787-d9778787d-k5mkc"
Sep 11 03:58:19.747966 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:58:19.747935    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-j2s2j\" (UniqueName: \"kubernetes.io/projected/ddaa3ca0-9cba-48a7-8e0b-3f7881aa7e81-kube-api-access-j2s2j\") pod \"payload-pre-processing-e2e-shared-b-w2-7dc787-d9778787d-k5mkc\" (UID: \"ddaa3ca0-9cba-48a7-8e0b-3f7881aa7e81\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-7dc787-d9778787d-k5mkc"
Sep 11 03:58:19.841367 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:58:19.841280    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dkgmg\" (UniqueName: \"kubernetes.io/projected/14ea3d95-462e-4f0e-aee3-d37f36f30204-kube-api-access-dkgmg\") pod \"payload-processing-e2e-shared-b-w2-7dc787-744d659f5-vv99b\" (UID: \"14ea3d95-462e-4f0e-aee3-d37f36f30204\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-7dc787-744d659f5-vv99b"
Sep 11 03:58:19.841367 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:58:19.841348    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/14ea3d95-462e-4f0e-aee3-d37f36f30204-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w2-7dc787-744d659f5-vv99b\" (UID: \"14ea3d95-462e-4f0e-aee3-d37f36f30204\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-7dc787-744d659f5-vv99b"
Sep 11 03:58:19.842001 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:58:19.841982    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/14ea3d95-462e-4f0e-aee3-d37f36f30204-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w2-7dc787-744d659f5-vv99b\" (UID: \"14ea3d95-462e-4f0e-aee3-d37f36f30204\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-7dc787-744d659f5-vv99b"
Sep 11 03:58:19.850330 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:58:19.850301    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dkgmg\" (UniqueName: \"kubernetes.io/projected/14ea3d95-462e-4f0e-aee3-d37f36f30204-kube-api-access-dkgmg\") pod \"payload-processing-e2e-shared-b-w2-7dc787-744d659f5-vv99b\" (UID: \"14ea3d95-462e-4f0e-aee3-d37f36f30204\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-7dc787-744d659f5-vv99b"
Sep 11 03:58:19.897805 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:58:19.897762    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-7dc787-d9778787d-k5mkc"
Sep 11 03:58:19.923372 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:58:19.923347    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-7dc787-744d659f5-vv99b"
Sep 11 03:58:20.173755 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:58:20.173731    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-7dc787-d9778787d-k5mkc"]
Sep 11 03:58:20.175832 ip-10-0-141-105 kubenswrapper[2579]: W0911 03:58:20.175798    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podddaa3ca0_9cba_48a7_8e0b_3f7881aa7e81.slice/crio-002978a6c1e580de02ad200bea444d7ecff7917c731408b704735b70560638b9 WatchSource:0}: Error finding container 002978a6c1e580de02ad200bea444d7ecff7917c731408b704735b70560638b9: Status 404 returned error can't find the container with id 002978a6c1e580de02ad200bea444d7ecff7917c731408b704735b70560638b9
Sep 11 03:58:20.206563 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:58:20.206540    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-7dc787-744d659f5-vv99b"]
Sep 11 03:58:20.210233 ip-10-0-141-105 kubenswrapper[2579]: W0911 03:58:20.210210    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod14ea3d95_462e_4f0e_aee3_d37f36f30204.slice/crio-014fa3b164f7d3fabe0c621efb76d3c623a1ed5db403a7b6a06c78b6eb67e882 WatchSource:0}: Error finding container 014fa3b164f7d3fabe0c621efb76d3c623a1ed5db403a7b6a06c78b6eb67e882: Status 404 returned error can't find the container with id 014fa3b164f7d3fabe0c621efb76d3c623a1ed5db403a7b6a06c78b6eb67e882
Sep 11 03:58:20.355448 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:58:20.355399    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-7dc787-d9778787d-k5mkc" event={"ID":"ddaa3ca0-9cba-48a7-8e0b-3f7881aa7e81","Type":"ContainerStarted","Data":"1c89fe0c455a1c964747abf4745b866c6ada2250f98784b1e73cf26295591d93"}
Sep 11 03:58:20.355627 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:58:20.355516    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-7dc787-d9778787d-k5mkc"
Sep 11 03:58:20.355627 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:58:20.355546    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-7dc787-d9778787d-k5mkc" event={"ID":"ddaa3ca0-9cba-48a7-8e0b-3f7881aa7e81","Type":"ContainerStarted","Data":"002978a6c1e580de02ad200bea444d7ecff7917c731408b704735b70560638b9"}
Sep 11 03:58:20.357290 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:58:20.357266    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-7dc787-744d659f5-vv99b" event={"ID":"14ea3d95-462e-4f0e-aee3-d37f36f30204","Type":"ContainerStarted","Data":"3abf8ac2ef7a0f815e39dbfbae7548e989bd354f6ab94501abebe2c3d3475cda"}
Sep 11 03:58:20.357439 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:58:20.357304    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-7dc787-744d659f5-vv99b" event={"ID":"14ea3d95-462e-4f0e-aee3-d37f36f30204","Type":"ContainerStarted","Data":"014fa3b164f7d3fabe0c621efb76d3c623a1ed5db403a7b6a06c78b6eb67e882"}
Sep 11 03:58:20.357439 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:58:20.357418    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-7dc787-744d659f5-vv99b"
Sep 11 03:58:20.371911 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:58:20.371863    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-7dc787-d9778787d-k5mkc" podStartSLOduration=1.371846807 podStartE2EDuration="1.371846807s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 03:58:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 03:58:20.370656491 +0000 UTC m=+1078.025862063" watchObservedRunningTime="2026-09-11 03:58:20.371846807 +0000 UTC m=+1078.027052373"
Sep 11 03:58:20.388017 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:58:20.387963    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-7dc787-744d659f5-vv99b" podStartSLOduration=1.387946042 podStartE2EDuration="1.387946042s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 03:58:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 03:58:20.387297829 +0000 UTC m=+1078.042503399" watchObservedRunningTime="2026-09-11 03:58:20.387946042 +0000 UTC m=+1078.043151609"
Sep 11 03:58:24.004852 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:58:24.004811    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-5z4cn"]
Sep 11 03:58:31.366896 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:58:31.366865    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-7dc787-d9778787d-k5mkc"
Sep 11 03:58:31.367345 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:58:31.367031    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-7dc787-744d659f5-vv99b"
Sep 11 03:58:42.458104 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:58:42.458054    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7701-d21a6e"]
Sep 11 03:58:42.463341 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:58:42.463320    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7701-d21a6e"
Sep 11 03:58:42.465487 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:58:42.465460    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7701-d21a6e"]
Sep 11 03:58:42.465710 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:58:42.465683    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-fjqk5\""
Sep 11 03:58:42.554775 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:58:42.554743    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cjx86\" (UniqueName: \"kubernetes.io/projected/6bfdcb7e-0ff0-40ea-a456-026d88ac5d73-kube-api-access-cjx86\") pod \"test-curl-7701-d21a6e\" (UID: \"6bfdcb7e-0ff0-40ea-a456-026d88ac5d73\") " pod="opendatahub/test-curl-7701-d21a6e"
Sep 11 03:58:42.655965 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:58:42.655932    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cjx86\" (UniqueName: \"kubernetes.io/projected/6bfdcb7e-0ff0-40ea-a456-026d88ac5d73-kube-api-access-cjx86\") pod \"test-curl-7701-d21a6e\" (UID: \"6bfdcb7e-0ff0-40ea-a456-026d88ac5d73\") " pod="opendatahub/test-curl-7701-d21a6e"
Sep 11 03:58:42.668546 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:58:42.664855    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cjx86\" (UniqueName: \"kubernetes.io/projected/6bfdcb7e-0ff0-40ea-a456-026d88ac5d73-kube-api-access-cjx86\") pod \"test-curl-7701-d21a6e\" (UID: \"6bfdcb7e-0ff0-40ea-a456-026d88ac5d73\") " pod="opendatahub/test-curl-7701-d21a6e"
Sep 11 03:58:42.774417 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:58:42.774331    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7701-d21a6e"
Sep 11 03:58:43.046807 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:58:43.046783    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7701-d21a6e"]
Sep 11 03:58:43.049227 ip-10-0-141-105 kubenswrapper[2579]: W0911 03:58:43.049198    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod6bfdcb7e_0ff0_40ea_a456_026d88ac5d73.slice/crio-0a47bf004cd6ac8e42d28ba489ae4dec6672f639f34a90da1cfba6850902bf0f WatchSource:0}: Error finding container 0a47bf004cd6ac8e42d28ba489ae4dec6672f639f34a90da1cfba6850902bf0f: Status 404 returned error can't find the container with id 0a47bf004cd6ac8e42d28ba489ae4dec6672f639f34a90da1cfba6850902bf0f
Sep 11 03:58:43.485515 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:58:43.485476    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7701-d21a6e" event={"ID":"6bfdcb7e-0ff0-40ea-a456-026d88ac5d73","Type":"ContainerStarted","Data":"ebbce64f50dba648b5397bfbe3813c4fe6df6d2469ac5186f30628e9eb7a1381"}
Sep 11 03:58:43.485903 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:58:43.485526    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7701-d21a6e" event={"ID":"6bfdcb7e-0ff0-40ea-a456-026d88ac5d73","Type":"ContainerStarted","Data":"0a47bf004cd6ac8e42d28ba489ae4dec6672f639f34a90da1cfba6850902bf0f"}
Sep 11 03:58:43.499179 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:58:43.499134    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-7701-d21a6e" podStartSLOduration=1.499117998 podStartE2EDuration="1.499117998s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 03:58:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 03:58:43.49823089 +0000 UTC m=+1101.153436461" watchObservedRunningTime="2026-09-11 03:58:43.499117998 +0000 UTC m=+1101.154323564"
Sep 11 03:58:44.083643 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:58:44.083615    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7701-d21a6e"]
Sep 11 03:58:44.087000 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:58:44.086975    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7701-d21a6e"]
Sep 11 03:58:44.491321 ip-10-0-141-105 kubenswrapper[2579]: E0911 03:58:44.491289    2579 kuberuntime_manager.go:649] "Missing actuated resource record" pod="opendatahub/test-curl-7701-d21a6e" container="test-curl-7701-d21a6e"
Sep 11 03:58:45.496745 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:58:45.496697    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-7701-d21a6e" podUID="6bfdcb7e-0ff0-40ea-a456-026d88ac5d73" containerName="test-curl-7701-d21a6e" containerID="cri-o://ebbce64f50dba648b5397bfbe3813c4fe6df6d2469ac5186f30628e9eb7a1381" gracePeriod=2
Sep 11 03:58:46.495132 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:58:46.495100    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7701-a73f26"]
Sep 11 03:58:46.495503 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:58:46.495490    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6bfdcb7e-0ff0-40ea-a456-026d88ac5d73" containerName="test-curl-7701-d21a6e"
Sep 11 03:58:46.495554 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:58:46.495505    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="6bfdcb7e-0ff0-40ea-a456-026d88ac5d73" containerName="test-curl-7701-d21a6e"
Sep 11 03:58:46.495592 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:58:46.495569    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="6bfdcb7e-0ff0-40ea-a456-026d88ac5d73" containerName="test-curl-7701-d21a6e"
Sep 11 03:58:46.498622 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:58:46.498595    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7701-a73f26"
Sep 11 03:58:46.504610 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:58:46.504584    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7701-a73f26"]
Sep 11 03:58:46.586135 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:58:46.586101    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d4pnh\" (UniqueName: \"kubernetes.io/projected/96c7f8c8-e3fd-4b39-90b6-7ffd2342c9ef-kube-api-access-d4pnh\") pod \"test-curl-7701-a73f26\" (UID: \"96c7f8c8-e3fd-4b39-90b6-7ffd2342c9ef\") " pod="opendatahub/test-curl-7701-a73f26"
Sep 11 03:58:46.687164 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:58:46.687132    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d4pnh\" (UniqueName: \"kubernetes.io/projected/96c7f8c8-e3fd-4b39-90b6-7ffd2342c9ef-kube-api-access-d4pnh\") pod \"test-curl-7701-a73f26\" (UID: \"96c7f8c8-e3fd-4b39-90b6-7ffd2342c9ef\") " pod="opendatahub/test-curl-7701-a73f26"
Sep 11 03:58:46.694646 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:58:46.694619    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-d4pnh\" (UniqueName: \"kubernetes.io/projected/96c7f8c8-e3fd-4b39-90b6-7ffd2342c9ef-kube-api-access-d4pnh\") pod \"test-curl-7701-a73f26\" (UID: \"96c7f8c8-e3fd-4b39-90b6-7ffd2342c9ef\") " pod="opendatahub/test-curl-7701-a73f26"
Sep 11 03:58:46.809944 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:58:46.809847    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7701-a73f26"
Sep 11 03:58:47.087701 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:58:47.087676    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7701-a73f26"]
Sep 11 03:58:47.090390 ip-10-0-141-105 kubenswrapper[2579]: W0911 03:58:47.090363    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod96c7f8c8_e3fd_4b39_90b6_7ffd2342c9ef.slice/crio-3bb1dd89075408bcb9ceb1e1fea5cfd1218e92f0c7711daf29da52a80bd91f7b WatchSource:0}: Error finding container 3bb1dd89075408bcb9ceb1e1fea5cfd1218e92f0c7711daf29da52a80bd91f7b: Status 404 returned error can't find the container with id 3bb1dd89075408bcb9ceb1e1fea5cfd1218e92f0c7711daf29da52a80bd91f7b
Sep 11 03:58:47.510936 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:58:47.510904    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7701-a73f26" event={"ID":"96c7f8c8-e3fd-4b39-90b6-7ffd2342c9ef","Type":"ContainerStarted","Data":"2e81a827b4ddf6a0e272a10ebcbecac4900937a864d7aef506e198eae2b940af"}
Sep 11 03:58:47.511322 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:58:47.510944    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7701-a73f26" event={"ID":"96c7f8c8-e3fd-4b39-90b6-7ffd2342c9ef","Type":"ContainerStarted","Data":"3bb1dd89075408bcb9ceb1e1fea5cfd1218e92f0c7711daf29da52a80bd91f7b"}
Sep 11 03:58:47.524163 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:58:47.524111    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-7701-a73f26" podStartSLOduration=1.524072539 podStartE2EDuration="1.524072539s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 03:58:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 03:58:47.523163408 +0000 UTC m=+1105.178368975" watchObservedRunningTime="2026-09-11 03:58:47.524072539 +0000 UTC m=+1105.179278105"
Sep 11 03:58:47.683122 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:58:47.683077    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7701-d21a6e"
Sep 11 03:58:47.694712 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:58:47.694688    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cjx86\" (UniqueName: \"kubernetes.io/projected/6bfdcb7e-0ff0-40ea-a456-026d88ac5d73-kube-api-access-cjx86\") pod \"6bfdcb7e-0ff0-40ea-a456-026d88ac5d73\" (UID: \"6bfdcb7e-0ff0-40ea-a456-026d88ac5d73\") "
Sep 11 03:58:47.696848 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:58:47.696822    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6bfdcb7e-0ff0-40ea-a456-026d88ac5d73-kube-api-access-cjx86" (OuterVolumeSpecName: "kube-api-access-cjx86") pod "6bfdcb7e-0ff0-40ea-a456-026d88ac5d73" (UID: "6bfdcb7e-0ff0-40ea-a456-026d88ac5d73"). InnerVolumeSpecName "kube-api-access-cjx86". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 03:58:47.795978 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:58:47.795896    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cjx86\" (UniqueName: \"kubernetes.io/projected/6bfdcb7e-0ff0-40ea-a456-026d88ac5d73-kube-api-access-cjx86\") on node \"ip-10-0-141-105.ec2.internal\" DevicePath \"\""
Sep 11 03:58:48.011613 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:58:48.011580    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-5z4cn"]
Sep 11 03:58:48.099964 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:58:48.099881    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7701-a73f26"]
Sep 11 03:58:48.103473 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:58:48.103446    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7701-a73f26"]
Sep 11 03:58:48.517539 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:58:48.517503    2579 generic.go:356] "Generic (PLEG): container finished" podID="6bfdcb7e-0ff0-40ea-a456-026d88ac5d73" containerID="ebbce64f50dba648b5397bfbe3813c4fe6df6d2469ac5186f30628e9eb7a1381" exitCode=137
Sep 11 03:58:48.518120 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:58:48.517544    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7701-d21a6e"
Sep 11 03:58:48.518120 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:58:48.517601    2579 scope.go:117] "RemoveContainer" containerID="ebbce64f50dba648b5397bfbe3813c4fe6df6d2469ac5186f30628e9eb7a1381"
Sep 11 03:58:48.518120 ip-10-0-141-105 kubenswrapper[2579]: E0911 03:58:48.517916    2579 kuberuntime_manager.go:649] "Missing actuated resource record" pod="opendatahub/test-curl-7701-a73f26" container="test-curl-7701-a73f26"
Sep 11 03:58:48.528249 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:58:48.528234    2579 scope.go:117] "RemoveContainer" containerID="ebbce64f50dba648b5397bfbe3813c4fe6df6d2469ac5186f30628e9eb7a1381"
Sep 11 03:58:48.528570 ip-10-0-141-105 kubenswrapper[2579]: E0911 03:58:48.528550    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ebbce64f50dba648b5397bfbe3813c4fe6df6d2469ac5186f30628e9eb7a1381\": container with ID starting with ebbce64f50dba648b5397bfbe3813c4fe6df6d2469ac5186f30628e9eb7a1381 not found: ID does not exist" containerID="ebbce64f50dba648b5397bfbe3813c4fe6df6d2469ac5186f30628e9eb7a1381"
Sep 11 03:58:48.528612 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:58:48.528580    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ebbce64f50dba648b5397bfbe3813c4fe6df6d2469ac5186f30628e9eb7a1381"} err="failed to get container status \"ebbce64f50dba648b5397bfbe3813c4fe6df6d2469ac5186f30628e9eb7a1381\": rpc error: code = NotFound desc = could not find container \"ebbce64f50dba648b5397bfbe3813c4fe6df6d2469ac5186f30628e9eb7a1381\": container with ID starting with ebbce64f50dba648b5397bfbe3813c4fe6df6d2469ac5186f30628e9eb7a1381 not found: ID does not exist"
Sep 11 03:58:48.528852 ip-10-0-141-105 kubenswrapper[2579]: E0911 03:58:48.528835    2579 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7701-d21a6e\" is forbidden: User \"system:node:ip-10-0-141-105.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-141-105.ec2.internal' and this object" podUID="6bfdcb7e-0ff0-40ea-a456-026d88ac5d73" pod="opendatahub/test-curl-7701-d21a6e"
Sep 11 03:58:48.855212 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:58:48.855184    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6bfdcb7e-0ff0-40ea-a456-026d88ac5d73" path="/var/lib/kubelet/pods/6bfdcb7e-0ff0-40ea-a456-026d88ac5d73/volumes"
Sep 11 03:58:49.523665 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:58:49.523622    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-7701-a73f26" podUID="96c7f8c8-e3fd-4b39-90b6-7ffd2342c9ef" containerName="test-curl-7701-a73f26" containerID="cri-o://2e81a827b4ddf6a0e272a10ebcbecac4900937a864d7aef506e198eae2b940af" gracePeriod=2
Sep 11 03:58:51.695136 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:58:51.695103    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7701-a73f26"
Sep 11 03:58:51.730285 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:58:51.730255    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-d4pnh\" (UniqueName: \"kubernetes.io/projected/96c7f8c8-e3fd-4b39-90b6-7ffd2342c9ef-kube-api-access-d4pnh\") pod \"96c7f8c8-e3fd-4b39-90b6-7ffd2342c9ef\" (UID: \"96c7f8c8-e3fd-4b39-90b6-7ffd2342c9ef\") "
Sep 11 03:58:51.732384 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:58:51.732354    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/96c7f8c8-e3fd-4b39-90b6-7ffd2342c9ef-kube-api-access-d4pnh" (OuterVolumeSpecName: "kube-api-access-d4pnh") pod "96c7f8c8-e3fd-4b39-90b6-7ffd2342c9ef" (UID: "96c7f8c8-e3fd-4b39-90b6-7ffd2342c9ef"). InnerVolumeSpecName "kube-api-access-d4pnh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 03:58:51.831443 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:58:51.831357    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d4pnh\" (UniqueName: \"kubernetes.io/projected/96c7f8c8-e3fd-4b39-90b6-7ffd2342c9ef-kube-api-access-d4pnh\") on node \"ip-10-0-141-105.ec2.internal\" DevicePath \"\""
Sep 11 03:58:52.542616 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:58:52.542580    2579 generic.go:356] "Generic (PLEG): container finished" podID="96c7f8c8-e3fd-4b39-90b6-7ffd2342c9ef" containerID="2e81a827b4ddf6a0e272a10ebcbecac4900937a864d7aef506e198eae2b940af" exitCode=137
Sep 11 03:58:52.542857 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:58:52.542632    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7701-a73f26"
Sep 11 03:58:52.542857 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:58:52.542686    2579 scope.go:117] "RemoveContainer" containerID="2e81a827b4ddf6a0e272a10ebcbecac4900937a864d7aef506e198eae2b940af"
Sep 11 03:58:52.552524 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:58:52.552501    2579 scope.go:117] "RemoveContainer" containerID="2e81a827b4ddf6a0e272a10ebcbecac4900937a864d7aef506e198eae2b940af"
Sep 11 03:58:52.552786 ip-10-0-141-105 kubenswrapper[2579]: E0911 03:58:52.552768    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2e81a827b4ddf6a0e272a10ebcbecac4900937a864d7aef506e198eae2b940af\": container with ID starting with 2e81a827b4ddf6a0e272a10ebcbecac4900937a864d7aef506e198eae2b940af not found: ID does not exist" containerID="2e81a827b4ddf6a0e272a10ebcbecac4900937a864d7aef506e198eae2b940af"
Sep 11 03:58:52.552863 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:58:52.552794    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2e81a827b4ddf6a0e272a10ebcbecac4900937a864d7aef506e198eae2b940af"} err="failed to get container status \"2e81a827b4ddf6a0e272a10ebcbecac4900937a864d7aef506e198eae2b940af\": rpc error: code = NotFound desc = could not find container \"2e81a827b4ddf6a0e272a10ebcbecac4900937a864d7aef506e198eae2b940af\": container with ID starting with 2e81a827b4ddf6a0e272a10ebcbecac4900937a864d7aef506e198eae2b940af not found: ID does not exist"
Sep 11 03:58:52.552863 ip-10-0-141-105 kubenswrapper[2579]: E0911 03:58:52.552838    2579 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7701-a73f26\" is forbidden: User \"system:node:ip-10-0-141-105.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-141-105.ec2.internal' and this object" podUID="96c7f8c8-e3fd-4b39-90b6-7ffd2342c9ef" pod="opendatahub/test-curl-7701-a73f26"
Sep 11 03:58:52.850061 ip-10-0-141-105 kubenswrapper[2579]: E0911 03:58:52.849973    2579 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7701-a73f26\" is forbidden: User \"system:node:ip-10-0-141-105.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-141-105.ec2.internal' and this object" podUID="96c7f8c8-e3fd-4b39-90b6-7ffd2342c9ef" pod="opendatahub/test-curl-7701-a73f26"
Sep 11 03:58:52.852159 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:58:52.852135    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="96c7f8c8-e3fd-4b39-90b6-7ffd2342c9ef" path="/var/lib/kubelet/pods/96c7f8c8-e3fd-4b39-90b6-7ffd2342c9ef/volumes"
Sep 11 03:58:57.859653 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:58:57.859612    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-89230689-867f69445c-g7p8s"]
Sep 11 03:58:57.860028 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:58:57.860015    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="96c7f8c8-e3fd-4b39-90b6-7ffd2342c9ef" containerName="test-curl-7701-a73f26"
Sep 11 03:58:57.860028 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:58:57.860029    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="96c7f8c8-e3fd-4b39-90b6-7ffd2342c9ef" containerName="test-curl-7701-a73f26"
Sep 11 03:58:57.860134 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:58:57.860110    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="96c7f8c8-e3fd-4b39-90b6-7ffd2342c9ef" containerName="test-curl-7701-a73f26"
Sep 11 03:58:57.864470 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:58:57.864452    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-89230689-867f69445c-g7p8s"
Sep 11 03:58:57.866739 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:58:57.866717    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-89230689\""
Sep 11 03:58:57.866873 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:58:57.866840    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-89230689-dockercfg-s8pqs\""
Sep 11 03:58:57.871138 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:58:57.871114    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-89230689-867f69445c-g7p8s"]
Sep 11 03:58:57.887050 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:58:57.887023    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-75wl2\" (UniqueName: \"kubernetes.io/projected/2ef658ef-733b-4b20-9831-51c7fc7968c8-kube-api-access-75wl2\") pod \"payload-processing-e2e-mt-89230689-867f69445c-g7p8s\" (UID: \"2ef658ef-733b-4b20-9831-51c7fc7968c8\") " pod="openshift-ingress/payload-processing-e2e-mt-89230689-867f69445c-g7p8s"
Sep 11 03:58:57.887246 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:58:57.887159    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2ef658ef-733b-4b20-9831-51c7fc7968c8-plugins-config-volume\") pod \"payload-processing-e2e-mt-89230689-867f69445c-g7p8s\" (UID: \"2ef658ef-733b-4b20-9831-51c7fc7968c8\") " pod="openshift-ingress/payload-processing-e2e-mt-89230689-867f69445c-g7p8s"
Sep 11 03:58:57.988500 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:58:57.988442    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2ef658ef-733b-4b20-9831-51c7fc7968c8-plugins-config-volume\") pod \"payload-processing-e2e-mt-89230689-867f69445c-g7p8s\" (UID: \"2ef658ef-733b-4b20-9831-51c7fc7968c8\") " pod="openshift-ingress/payload-processing-e2e-mt-89230689-867f69445c-g7p8s"
Sep 11 03:58:57.988715 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:58:57.988555    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-75wl2\" (UniqueName: \"kubernetes.io/projected/2ef658ef-733b-4b20-9831-51c7fc7968c8-kube-api-access-75wl2\") pod \"payload-processing-e2e-mt-89230689-867f69445c-g7p8s\" (UID: \"2ef658ef-733b-4b20-9831-51c7fc7968c8\") " pod="openshift-ingress/payload-processing-e2e-mt-89230689-867f69445c-g7p8s"
Sep 11 03:58:57.989183 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:58:57.989164    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2ef658ef-733b-4b20-9831-51c7fc7968c8-plugins-config-volume\") pod \"payload-processing-e2e-mt-89230689-867f69445c-g7p8s\" (UID: \"2ef658ef-733b-4b20-9831-51c7fc7968c8\") " pod="openshift-ingress/payload-processing-e2e-mt-89230689-867f69445c-g7p8s"
Sep 11 03:58:57.997763 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:58:57.997730    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-75wl2\" (UniqueName: \"kubernetes.io/projected/2ef658ef-733b-4b20-9831-51c7fc7968c8-kube-api-access-75wl2\") pod \"payload-processing-e2e-mt-89230689-867f69445c-g7p8s\" (UID: \"2ef658ef-733b-4b20-9831-51c7fc7968c8\") " pod="openshift-ingress/payload-processing-e2e-mt-89230689-867f69445c-g7p8s"
Sep 11 03:58:58.175538 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:58:58.175437    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-89230689-867f69445c-g7p8s"
Sep 11 03:58:58.447393 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:58:58.447367    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-89230689-867f69445c-g7p8s"]
Sep 11 03:58:58.453635 ip-10-0-141-105 kubenswrapper[2579]: W0911 03:58:58.453579    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2ef658ef_733b_4b20_9831_51c7fc7968c8.slice/crio-dfc279bd35da4f3023dea4fe386d40179c8ed5f50d2c8521186a298f2b28b62a WatchSource:0}: Error finding container dfc279bd35da4f3023dea4fe386d40179c8ed5f50d2c8521186a298f2b28b62a: Status 404 returned error can't find the container with id dfc279bd35da4f3023dea4fe386d40179c8ed5f50d2c8521186a298f2b28b62a
Sep 11 03:58:58.579369 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:58:58.579324    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-89230689-867f69445c-g7p8s" event={"ID":"2ef658ef-733b-4b20-9831-51c7fc7968c8","Type":"ContainerStarted","Data":"3469fef190b3f47da96041f85f7096a2fe6c4430fc0cf1ac7c11bc5cf66a3a5d"}
Sep 11 03:58:58.579369 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:58:58.579372    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-89230689-867f69445c-g7p8s" event={"ID":"2ef658ef-733b-4b20-9831-51c7fc7968c8","Type":"ContainerStarted","Data":"dfc279bd35da4f3023dea4fe386d40179c8ed5f50d2c8521186a298f2b28b62a"}
Sep 11 03:58:58.579618 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:58:58.579402    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-mt-89230689-867f69445c-g7p8s"
Sep 11 03:58:58.593955 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:58:58.593894    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-mt-89230689-867f69445c-g7p8s" podStartSLOduration=1.59387526 podStartE2EDuration="1.59387526s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 03:58:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 03:58:58.593286129 +0000 UTC m=+1116.248491697" watchObservedRunningTime="2026-09-11 03:58:58.59387526 +0000 UTC m=+1116.249080826"
Sep 11 03:59:04.303272 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:59:04.303233    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-5z4cn"]
Sep 11 03:59:09.587891 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:59:09.587853    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-mt-89230689-867f69445c-g7p8s"
Sep 11 03:59:22.935489 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:59:22.935455    2579 scope.go:117] "RemoveContainer" containerID="7d2c23ded6ac69c5e9678d8a214cfc4d2b5c70fad549debed45dc24662239010"
Sep 11 03:59:32.498018 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:59:32.497983    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-5z4cn"]
Sep 11 03:59:44.099614 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:59:44.099493    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-mt-89230689/e2e-lifecycle-model-89230689-kserve-56d764c4f9-wlb5l"]
Sep 11 03:59:44.104637 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:59:44.104618    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-89230689/e2e-lifecycle-model-89230689-kserve-56d764c4f9-wlb5l"
Sep 11 03:59:44.107105 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:59:44.107052    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-89230689\"/\"kube-root-ca.crt\""
Sep 11 03:59:44.107253 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:59:44.107235    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-89230689\"/\"openshift-service-ca.crt\""
Sep 11 03:59:44.108439 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:59:44.108415    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-89230689\"/\"default-dockercfg-7mldc\""
Sep 11 03:59:44.108560 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:59:44.108460    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-89230689\"/\"e2e-lifecycle-model-89230689-kserve-self-signed-certs\""
Sep 11 03:59:44.111043 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:59:44.111024    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-89230689/e2e-lifecycle-model-89230689-kserve-56d764c4f9-wlb5l"]
Sep 11 03:59:44.225483 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:59:44.225448    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9f5b03c9-ae3d-4834-963b-618cdb267a00-dshm\") pod \"e2e-lifecycle-model-89230689-kserve-56d764c4f9-wlb5l\" (UID: \"9f5b03c9-ae3d-4834-963b-618cdb267a00\") " pod="ai-tenant-e2e-mt-89230689/e2e-lifecycle-model-89230689-kserve-56d764c4f9-wlb5l"
Sep 11 03:59:44.225655 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:59:44.225508    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9f5b03c9-ae3d-4834-963b-618cdb267a00-home\") pod \"e2e-lifecycle-model-89230689-kserve-56d764c4f9-wlb5l\" (UID: \"9f5b03c9-ae3d-4834-963b-618cdb267a00\") " pod="ai-tenant-e2e-mt-89230689/e2e-lifecycle-model-89230689-kserve-56d764c4f9-wlb5l"
Sep 11 03:59:44.225655 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:59:44.225611    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9f5b03c9-ae3d-4834-963b-618cdb267a00-tmp-dir\") pod \"e2e-lifecycle-model-89230689-kserve-56d764c4f9-wlb5l\" (UID: \"9f5b03c9-ae3d-4834-963b-618cdb267a00\") " pod="ai-tenant-e2e-mt-89230689/e2e-lifecycle-model-89230689-kserve-56d764c4f9-wlb5l"
Sep 11 03:59:44.225737 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:59:44.225654    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9f5b03c9-ae3d-4834-963b-618cdb267a00-tls-certs\") pod \"e2e-lifecycle-model-89230689-kserve-56d764c4f9-wlb5l\" (UID: \"9f5b03c9-ae3d-4834-963b-618cdb267a00\") " pod="ai-tenant-e2e-mt-89230689/e2e-lifecycle-model-89230689-kserve-56d764c4f9-wlb5l"
Sep 11 03:59:44.225737 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:59:44.225709    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9f5b03c9-ae3d-4834-963b-618cdb267a00-model-cache\") pod \"e2e-lifecycle-model-89230689-kserve-56d764c4f9-wlb5l\" (UID: \"9f5b03c9-ae3d-4834-963b-618cdb267a00\") " pod="ai-tenant-e2e-mt-89230689/e2e-lifecycle-model-89230689-kserve-56d764c4f9-wlb5l"
Sep 11 03:59:44.225803 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:59:44.225787    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hm24l\" (UniqueName: \"kubernetes.io/projected/9f5b03c9-ae3d-4834-963b-618cdb267a00-kube-api-access-hm24l\") pod \"e2e-lifecycle-model-89230689-kserve-56d764c4f9-wlb5l\" (UID: \"9f5b03c9-ae3d-4834-963b-618cdb267a00\") " pod="ai-tenant-e2e-mt-89230689/e2e-lifecycle-model-89230689-kserve-56d764c4f9-wlb5l"
Sep 11 03:59:44.326868 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:59:44.326831    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9f5b03c9-ae3d-4834-963b-618cdb267a00-model-cache\") pod \"e2e-lifecycle-model-89230689-kserve-56d764c4f9-wlb5l\" (UID: \"9f5b03c9-ae3d-4834-963b-618cdb267a00\") " pod="ai-tenant-e2e-mt-89230689/e2e-lifecycle-model-89230689-kserve-56d764c4f9-wlb5l"
Sep 11 03:59:44.327036 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:59:44.326884    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hm24l\" (UniqueName: \"kubernetes.io/projected/9f5b03c9-ae3d-4834-963b-618cdb267a00-kube-api-access-hm24l\") pod \"e2e-lifecycle-model-89230689-kserve-56d764c4f9-wlb5l\" (UID: \"9f5b03c9-ae3d-4834-963b-618cdb267a00\") " pod="ai-tenant-e2e-mt-89230689/e2e-lifecycle-model-89230689-kserve-56d764c4f9-wlb5l"
Sep 11 03:59:44.327122 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:59:44.327104    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9f5b03c9-ae3d-4834-963b-618cdb267a00-dshm\") pod \"e2e-lifecycle-model-89230689-kserve-56d764c4f9-wlb5l\" (UID: \"9f5b03c9-ae3d-4834-963b-618cdb267a00\") " pod="ai-tenant-e2e-mt-89230689/e2e-lifecycle-model-89230689-kserve-56d764c4f9-wlb5l"
Sep 11 03:59:44.327209 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:59:44.327191    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9f5b03c9-ae3d-4834-963b-618cdb267a00-home\") pod \"e2e-lifecycle-model-89230689-kserve-56d764c4f9-wlb5l\" (UID: \"9f5b03c9-ae3d-4834-963b-618cdb267a00\") " pod="ai-tenant-e2e-mt-89230689/e2e-lifecycle-model-89230689-kserve-56d764c4f9-wlb5l"
Sep 11 03:59:44.327283 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:59:44.327264    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9f5b03c9-ae3d-4834-963b-618cdb267a00-tmp-dir\") pod \"e2e-lifecycle-model-89230689-kserve-56d764c4f9-wlb5l\" (UID: \"9f5b03c9-ae3d-4834-963b-618cdb267a00\") " pod="ai-tenant-e2e-mt-89230689/e2e-lifecycle-model-89230689-kserve-56d764c4f9-wlb5l"
Sep 11 03:59:44.327342 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:59:44.327320    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9f5b03c9-ae3d-4834-963b-618cdb267a00-tls-certs\") pod \"e2e-lifecycle-model-89230689-kserve-56d764c4f9-wlb5l\" (UID: \"9f5b03c9-ae3d-4834-963b-618cdb267a00\") " pod="ai-tenant-e2e-mt-89230689/e2e-lifecycle-model-89230689-kserve-56d764c4f9-wlb5l"
Sep 11 03:59:44.327403 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:59:44.327334    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9f5b03c9-ae3d-4834-963b-618cdb267a00-model-cache\") pod \"e2e-lifecycle-model-89230689-kserve-56d764c4f9-wlb5l\" (UID: \"9f5b03c9-ae3d-4834-963b-618cdb267a00\") " pod="ai-tenant-e2e-mt-89230689/e2e-lifecycle-model-89230689-kserve-56d764c4f9-wlb5l"
Sep 11 03:59:44.327525 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:59:44.327503    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9f5b03c9-ae3d-4834-963b-618cdb267a00-home\") pod \"e2e-lifecycle-model-89230689-kserve-56d764c4f9-wlb5l\" (UID: \"9f5b03c9-ae3d-4834-963b-618cdb267a00\") " pod="ai-tenant-e2e-mt-89230689/e2e-lifecycle-model-89230689-kserve-56d764c4f9-wlb5l"
Sep 11 03:59:44.327577 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:59:44.327562    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9f5b03c9-ae3d-4834-963b-618cdb267a00-tmp-dir\") pod \"e2e-lifecycle-model-89230689-kserve-56d764c4f9-wlb5l\" (UID: \"9f5b03c9-ae3d-4834-963b-618cdb267a00\") " pod="ai-tenant-e2e-mt-89230689/e2e-lifecycle-model-89230689-kserve-56d764c4f9-wlb5l"
Sep 11 03:59:44.329441 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:59:44.329408    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9f5b03c9-ae3d-4834-963b-618cdb267a00-dshm\") pod \"e2e-lifecycle-model-89230689-kserve-56d764c4f9-wlb5l\" (UID: \"9f5b03c9-ae3d-4834-963b-618cdb267a00\") " pod="ai-tenant-e2e-mt-89230689/e2e-lifecycle-model-89230689-kserve-56d764c4f9-wlb5l"
Sep 11 03:59:44.329799 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:59:44.329769    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9f5b03c9-ae3d-4834-963b-618cdb267a00-tls-certs\") pod \"e2e-lifecycle-model-89230689-kserve-56d764c4f9-wlb5l\" (UID: \"9f5b03c9-ae3d-4834-963b-618cdb267a00\") " pod="ai-tenant-e2e-mt-89230689/e2e-lifecycle-model-89230689-kserve-56d764c4f9-wlb5l"
Sep 11 03:59:44.334863 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:59:44.334845    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hm24l\" (UniqueName: \"kubernetes.io/projected/9f5b03c9-ae3d-4834-963b-618cdb267a00-kube-api-access-hm24l\") pod \"e2e-lifecycle-model-89230689-kserve-56d764c4f9-wlb5l\" (UID: \"9f5b03c9-ae3d-4834-963b-618cdb267a00\") " pod="ai-tenant-e2e-mt-89230689/e2e-lifecycle-model-89230689-kserve-56d764c4f9-wlb5l"
Sep 11 03:59:44.415434 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:59:44.415401    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-89230689/e2e-lifecycle-model-89230689-kserve-56d764c4f9-wlb5l"
Sep 11 03:59:44.688422 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:59:44.688398    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-89230689/e2e-lifecycle-model-89230689-kserve-56d764c4f9-wlb5l"]
Sep 11 03:59:44.690290 ip-10-0-141-105 kubenswrapper[2579]: W0911 03:59:44.690261    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod9f5b03c9_ae3d_4834_963b_618cdb267a00.slice/crio-2660f4d810f3a9b84e2cab55a567f49068da4158696ab698c3e895b0323a82f6 WatchSource:0}: Error finding container 2660f4d810f3a9b84e2cab55a567f49068da4158696ab698c3e895b0323a82f6: Status 404 returned error can't find the container with id 2660f4d810f3a9b84e2cab55a567f49068da4158696ab698c3e895b0323a82f6
Sep 11 03:59:44.830764 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:59:44.830723    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-89230689/e2e-lifecycle-model-89230689-kserve-56d764c4f9-wlb5l" event={"ID":"9f5b03c9-ae3d-4834-963b-618cdb267a00","Type":"ContainerStarted","Data":"55d4c11447f9283ffed128455b0270a3e9d05285dcef8199e503f47ce318cfc5"}
Sep 11 03:59:44.830931 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:59:44.830783    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-89230689/e2e-lifecycle-model-89230689-kserve-56d764c4f9-wlb5l" event={"ID":"9f5b03c9-ae3d-4834-963b-618cdb267a00","Type":"ContainerStarted","Data":"2660f4d810f3a9b84e2cab55a567f49068da4158696ab698c3e895b0323a82f6"}
Sep 11 03:59:44.849128 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:59:44.847957    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-mt-89230689/e2e-lifecycle-model-89230689-kserve-56d764c4f9-wlb5l" podStartSLOduration=0.847937266 podStartE2EDuration="847.937266ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 03:59:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 03:59:44.845582161 +0000 UTC m=+1162.500787729" watchObservedRunningTime="2026-09-11 03:59:44.847937266 +0000 UTC m=+1162.503142833"
Sep 11 03:59:49.599812 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:59:49.599777    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-5z4cn"]
Sep 11 03:59:54.415939 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:59:54.415898    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-mt-89230689/e2e-lifecycle-model-89230689-kserve-56d764c4f9-wlb5l"
Sep 11 03:59:54.416450 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:59:54.415954    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-mt-89230689/e2e-lifecycle-model-89230689-kserve-56d764c4f9-wlb5l"
Sep 11 03:59:54.428950 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:59:54.428928    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-mt-89230689/e2e-lifecycle-model-89230689-kserve-56d764c4f9-wlb5l"
Sep 11 03:59:54.895486 ip-10-0-141-105 kubenswrapper[2579]: I0911 03:59:54.895458    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-mt-89230689/e2e-lifecycle-model-89230689-kserve-56d764c4f9-wlb5l"
Sep 11 04:00:00.137239 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:00:00.137186    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-89230689-29818320-zhdvc"]
Sep 11 04:00:00.141176 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:00:00.141155    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29818320-v5kl8"]
Sep 11 04:00:00.141322 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:00:00.141292    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-89230689-29818320-zhdvc"
Sep 11 04:00:00.144118 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:00:00.144096    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-95ckr\""
Sep 11 04:00:00.145787 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:00:00.145757    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-20ae9a-29818320-4gxnn"]
Sep 11 04:00:00.145891 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:00:00.145845    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29818320-v5kl8"
Sep 11 04:00:00.150716 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:00:00.150034    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-20ae9a-29818320-4gxnn"
Sep 11 04:00:00.151181 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:00:00.151141    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-89230689-29818320-zhdvc"]
Sep 11 04:00:00.155309 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:00:00.155030    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29818320-v5kl8"]
Sep 11 04:00:00.157490 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:00:00.157466    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-20ae9a-29818320-4gxnn"]
Sep 11 04:00:00.160209 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:00:00.160188    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-89230689-29818320-zhdvc"
Sep 11 04:00:00.160934 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:00:00.160914    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-dd5b30-29818320-9mj9h"]
Sep 11 04:00:00.165304 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:00:00.165191    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-dd5b30-29818320-9mj9h"
Sep 11 04:00:00.169220 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:00:00.169198    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29818320-v5kl8"
Sep 11 04:00:00.170767 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:00:00.170748    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-dd5b30-29818320-9mj9h"]
Sep 11 04:00:00.202838 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:00:00.202802    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-20ae9a-29818320-4gxnn"
Sep 11 04:00:00.211140 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:00:00.211122    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-dd5b30-29818320-9mj9h"
Sep 11 04:00:00.446064 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:00:00.446041    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-89230689-29818320-zhdvc"]
Sep 11 04:00:00.448422 ip-10-0-141-105 kubenswrapper[2579]: W0911 04:00:00.448388    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7da7249c_b29b_4ad6_816c_5d534a49d6e3.slice/crio-919d12a262c6bf1c19f55e5f51381ef7f3c66a0879b5547a3564e081168bdafe WatchSource:0}: Error finding container 919d12a262c6bf1c19f55e5f51381ef7f3c66a0879b5547a3564e081168bdafe: Status 404 returned error can't find the container with id 919d12a262c6bf1c19f55e5f51381ef7f3c66a0879b5547a3564e081168bdafe
Sep 11 04:00:00.454654 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:00:00.454632    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29818320-v5kl8"]
Sep 11 04:00:00.455709 ip-10-0-141-105 kubenswrapper[2579]: W0911 04:00:00.455688    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbc6d6c39_8d93_41c5_924b_96d8cc2693ce.slice/crio-425b080cac5cb4babcac04c3ccce719b652fa51f308a8d26866ae57af91a8a9d WatchSource:0}: Error finding container 425b080cac5cb4babcac04c3ccce719b652fa51f308a8d26866ae57af91a8a9d: Status 404 returned error can't find the container with id 425b080cac5cb4babcac04c3ccce719b652fa51f308a8d26866ae57af91a8a9d
Sep 11 04:00:00.518859 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:00:00.518835    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-20ae9a-29818320-4gxnn"]
Sep 11 04:00:00.520344 ip-10-0-141-105 kubenswrapper[2579]: W0911 04:00:00.520313    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podda6ced35_8fa1_4306_98a8_ceabd2dd9909.slice/crio-d059d05187f411cf4ace76d71e0be4c26af328e24583073d914788f8653e0cfc WatchSource:0}: Error finding container d059d05187f411cf4ace76d71e0be4c26af328e24583073d914788f8653e0cfc: Status 404 returned error can't find the container with id d059d05187f411cf4ace76d71e0be4c26af328e24583073d914788f8653e0cfc
Sep 11 04:00:00.528684 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:00:00.528659    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-dd5b30-29818320-9mj9h"]
Sep 11 04:00:00.534549 ip-10-0-141-105 kubenswrapper[2579]: W0911 04:00:00.534519    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3e6c01fc_ed58_425c_a590_acdd4aef2d16.slice/crio-ca72e9b9e3e09307184ea29bfecb3daaa4eb88d3d6946a243fcff37c3e0afd85 WatchSource:0}: Error finding container ca72e9b9e3e09307184ea29bfecb3daaa4eb88d3d6946a243fcff37c3e0afd85: Status 404 returned error can't find the container with id ca72e9b9e3e09307184ea29bfecb3daaa4eb88d3d6946a243fcff37c3e0afd85
Sep 11 04:00:00.915062 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:00:00.915026    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-dd5b30-29818320-9mj9h" event={"ID":"3e6c01fc-ed58-425c-a590-acdd4aef2d16","Type":"ContainerStarted","Data":"16968aab8d31197fd71d04687b3f8fa7c08dc671ff9d06aa60c72c7afb2e106d"}
Sep 11 04:00:00.915277 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:00:00.915115    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-dd5b30-29818320-9mj9h" event={"ID":"3e6c01fc-ed58-425c-a590-acdd4aef2d16","Type":"ContainerStarted","Data":"ca72e9b9e3e09307184ea29bfecb3daaa4eb88d3d6946a243fcff37c3e0afd85"}
Sep 11 04:00:00.916752 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:00:00.916724    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-20ae9a-29818320-4gxnn" event={"ID":"da6ced35-8fa1-4306-98a8-ceabd2dd9909","Type":"ContainerStarted","Data":"9afb490d1689454b9f73a00d05b27b276d3c9dc99e066ec4b9fd7de86a7f98e7"}
Sep 11 04:00:00.916893 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:00:00.916775    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-20ae9a-29818320-4gxnn" event={"ID":"da6ced35-8fa1-4306-98a8-ceabd2dd9909","Type":"ContainerStarted","Data":"d059d05187f411cf4ace76d71e0be4c26af328e24583073d914788f8653e0cfc"}
Sep 11 04:00:00.921597 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:00:00.921567    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-89230689-29818320-zhdvc" event={"ID":"7da7249c-b29b-4ad6-816c-5d534a49d6e3","Type":"ContainerStarted","Data":"e66acb358f77288bf6feda4378fbc9e5603f20ab02d54782624e05125dedebc7"}
Sep 11 04:00:00.921730 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:00:00.921613    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-89230689-29818320-zhdvc" event={"ID":"7da7249c-b29b-4ad6-816c-5d534a49d6e3","Type":"ContainerStarted","Data":"919d12a262c6bf1c19f55e5f51381ef7f3c66a0879b5547a3564e081168bdafe"}
Sep 11 04:00:00.923329 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:00:00.923303    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29818320-v5kl8" event={"ID":"bc6d6c39-8d93-41c5-924b-96d8cc2693ce","Type":"ContainerStarted","Data":"99e31d77c9e93779e4ba4420737f64fca8265d81e70f03a9ccc6e57ccb1c1f20"}
Sep 11 04:00:00.923454 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:00:00.923342    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29818320-v5kl8" event={"ID":"bc6d6c39-8d93-41c5-924b-96d8cc2693ce","Type":"ContainerStarted","Data":"425b080cac5cb4babcac04c3ccce719b652fa51f308a8d26866ae57af91a8a9d"}
Sep 11 04:00:00.931624 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:00:00.931588    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-dd5b30-29818320-9mj9h" podStartSLOduration=0.931574528 podStartE2EDuration="931.574528ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 04:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 04:00:00.930333163 +0000 UTC m=+1178.585538757" watchObservedRunningTime="2026-09-11 04:00:00.931574528 +0000 UTC m=+1178.586780086"
Sep 11 04:00:00.945527 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:00:00.945490    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-20ae9a-29818320-4gxnn" podStartSLOduration=0.945478571 podStartE2EDuration="945.478571ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 04:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 04:00:00.944785062 +0000 UTC m=+1178.599990631" watchObservedRunningTime="2026-09-11 04:00:00.945478571 +0000 UTC m=+1178.600684127"
Sep 11 04:00:00.965978 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:00:00.965929    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-89230689-29818320-zhdvc" podStartSLOduration=0.96591378 podStartE2EDuration="965.91378ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 04:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 04:00:00.964415193 +0000 UTC m=+1178.619620763" watchObservedRunningTime="2026-09-11 04:00:00.96591378 +0000 UTC m=+1178.621119346"
Sep 11 04:00:00.981831 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:00:00.981772    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29818320-v5kl8" podStartSLOduration=0.981757492 podStartE2EDuration="981.757492ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 04:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 04:00:00.979789175 +0000 UTC m=+1178.634994743" watchObservedRunningTime="2026-09-11 04:00:00.981757492 +0000 UTC m=+1178.636963059"
Sep 11 04:00:01.931460 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:00:01.931425    2579 generic.go:356] "Generic (PLEG): container finished" podID="3e6c01fc-ed58-425c-a590-acdd4aef2d16" containerID="16968aab8d31197fd71d04687b3f8fa7c08dc671ff9d06aa60c72c7afb2e106d" exitCode=0
Sep 11 04:00:01.931911 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:00:01.931498    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-dd5b30-29818320-9mj9h" event={"ID":"3e6c01fc-ed58-425c-a590-acdd4aef2d16","Type":"ContainerDied","Data":"16968aab8d31197fd71d04687b3f8fa7c08dc671ff9d06aa60c72c7afb2e106d"}
Sep 11 04:00:01.933066 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:00:01.933043    2579 generic.go:356] "Generic (PLEG): container finished" podID="da6ced35-8fa1-4306-98a8-ceabd2dd9909" containerID="9afb490d1689454b9f73a00d05b27b276d3c9dc99e066ec4b9fd7de86a7f98e7" exitCode=0
Sep 11 04:00:01.933178 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:00:01.933126    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-20ae9a-29818320-4gxnn" event={"ID":"da6ced35-8fa1-4306-98a8-ceabd2dd9909","Type":"ContainerDied","Data":"9afb490d1689454b9f73a00d05b27b276d3c9dc99e066ec4b9fd7de86a7f98e7"}
Sep 11 04:00:01.934710 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:00:01.934687    2579 generic.go:356] "Generic (PLEG): container finished" podID="7da7249c-b29b-4ad6-816c-5d534a49d6e3" containerID="e66acb358f77288bf6feda4378fbc9e5603f20ab02d54782624e05125dedebc7" exitCode=0
Sep 11 04:00:01.934827 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:00:01.934764    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-89230689-29818320-zhdvc" event={"ID":"7da7249c-b29b-4ad6-816c-5d534a49d6e3","Type":"ContainerDied","Data":"e66acb358f77288bf6feda4378fbc9e5603f20ab02d54782624e05125dedebc7"}
Sep 11 04:00:01.936204 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:00:01.936181    2579 generic.go:356] "Generic (PLEG): container finished" podID="bc6d6c39-8d93-41c5-924b-96d8cc2693ce" containerID="99e31d77c9e93779e4ba4420737f64fca8265d81e70f03a9ccc6e57ccb1c1f20" exitCode=0
Sep 11 04:00:01.936381 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:00:01.936253    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29818320-v5kl8" event={"ID":"bc6d6c39-8d93-41c5-924b-96d8cc2693ce","Type":"ContainerDied","Data":"99e31d77c9e93779e4ba4420737f64fca8265d81e70f03a9ccc6e57ccb1c1f20"}
Sep 11 04:00:03.086663 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:00:03.085925    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29818320-v5kl8"
Sep 11 04:00:03.147971 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:00:03.147939    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-89230689-29818320-zhdvc"
Sep 11 04:00:03.175436 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:00:03.175398    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-20ae9a-29818320-4gxnn"
Sep 11 04:00:03.178864 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:00:03.178843    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-dd5b30-29818320-9mj9h"
Sep 11 04:00:03.948963 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:00:03.948935    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-dd5b30-29818320-9mj9h"
Sep 11 04:00:03.948963 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:00:03.948944    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-dd5b30-29818320-9mj9h" event={"ID":"3e6c01fc-ed58-425c-a590-acdd4aef2d16","Type":"ContainerDied","Data":"ca72e9b9e3e09307184ea29bfecb3daaa4eb88d3d6946a243fcff37c3e0afd85"}
Sep 11 04:00:03.949266 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:00:03.948993    2579 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ca72e9b9e3e09307184ea29bfecb3daaa4eb88d3d6946a243fcff37c3e0afd85"
Sep 11 04:00:03.950472 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:00:03.950456    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-20ae9a-29818320-4gxnn"
Sep 11 04:00:03.950472 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:00:03.950459    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-20ae9a-29818320-4gxnn" event={"ID":"da6ced35-8fa1-4306-98a8-ceabd2dd9909","Type":"ContainerDied","Data":"d059d05187f411cf4ace76d71e0be4c26af328e24583073d914788f8653e0cfc"}
Sep 11 04:00:03.950752 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:00:03.950508    2579 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d059d05187f411cf4ace76d71e0be4c26af328e24583073d914788f8653e0cfc"
Sep 11 04:00:03.952002 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:00:03.951984    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-89230689-29818320-zhdvc"
Sep 11 04:00:03.952150 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:00:03.952033    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-89230689-29818320-zhdvc" event={"ID":"7da7249c-b29b-4ad6-816c-5d534a49d6e3","Type":"ContainerDied","Data":"919d12a262c6bf1c19f55e5f51381ef7f3c66a0879b5547a3564e081168bdafe"}
Sep 11 04:00:03.952150 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:00:03.952075    2579 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="919d12a262c6bf1c19f55e5f51381ef7f3c66a0879b5547a3564e081168bdafe"
Sep 11 04:00:03.953530 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:00:03.953513    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29818320-v5kl8"
Sep 11 04:00:03.953530 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:00:03.953527    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29818320-v5kl8" event={"ID":"bc6d6c39-8d93-41c5-924b-96d8cc2693ce","Type":"ContainerDied","Data":"425b080cac5cb4babcac04c3ccce719b652fa51f308a8d26866ae57af91a8a9d"}
Sep 11 04:00:03.953684 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:00:03.953570    2579 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="425b080cac5cb4babcac04c3ccce719b652fa51f308a8d26866ae57af91a8a9d"
Sep 11 04:00:09.552874 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:00:09.552835    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-89230689-5k4zx"]
Sep 11 04:00:09.553292 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:00:09.553278    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7da7249c-b29b-4ad6-816c-5d534a49d6e3" containerName="cleanup"
Sep 11 04:00:09.553292 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:00:09.553292    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="7da7249c-b29b-4ad6-816c-5d534a49d6e3" containerName="cleanup"
Sep 11 04:00:09.553368 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:00:09.553305    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="da6ced35-8fa1-4306-98a8-ceabd2dd9909" containerName="cleanup"
Sep 11 04:00:09.553368 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:00:09.553311    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="da6ced35-8fa1-4306-98a8-ceabd2dd9909" containerName="cleanup"
Sep 11 04:00:09.553368 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:00:09.553317    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3e6c01fc-ed58-425c-a590-acdd4aef2d16" containerName="cleanup"
Sep 11 04:00:09.553368 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:00:09.553323    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="3e6c01fc-ed58-425c-a590-acdd4aef2d16" containerName="cleanup"
Sep 11 04:00:09.553368 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:00:09.553335    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bc6d6c39-8d93-41c5-924b-96d8cc2693ce" containerName="cleanup"
Sep 11 04:00:09.553368 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:00:09.553340    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="bc6d6c39-8d93-41c5-924b-96d8cc2693ce" containerName="cleanup"
Sep 11 04:00:09.553555 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:00:09.553401    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="7da7249c-b29b-4ad6-816c-5d534a49d6e3" containerName="cleanup"
Sep 11 04:00:09.553555 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:00:09.553409    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="bc6d6c39-8d93-41c5-924b-96d8cc2693ce" containerName="cleanup"
Sep 11 04:00:09.553555 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:00:09.553415    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="3e6c01fc-ed58-425c-a590-acdd4aef2d16" containerName="cleanup"
Sep 11 04:00:09.553555 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:00:09.553422    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="da6ced35-8fa1-4306-98a8-ceabd2dd9909" containerName="cleanup"
Sep 11 04:00:09.556506 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:00:09.556489    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-89230689-5k4zx"
Sep 11 04:00:09.559183 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:00:09.559156    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 11 04:00:09.559355 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:00:09.559337    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-95ckr\""
Sep 11 04:00:09.563830 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:00:09.563787    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-89230689-5k4zx"]
Sep 11 04:00:09.668261 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:00:09.668227    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2e2ec89d-74fd-438d-b0be-94b92f0ad0d7-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-89230689-5k4zx\" (UID: \"2e2ec89d-74fd-438d-b0be-94b92f0ad0d7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-89230689-5k4zx"
Sep 11 04:00:09.768748 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:00:09.768711    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2e2ec89d-74fd-438d-b0be-94b92f0ad0d7-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-89230689-5k4zx\" (UID: \"2e2ec89d-74fd-438d-b0be-94b92f0ad0d7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-89230689-5k4zx"
Sep 11 04:00:09.769386 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:00:09.769365    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2e2ec89d-74fd-438d-b0be-94b92f0ad0d7-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-89230689-5k4zx\" (UID: \"2e2ec89d-74fd-438d-b0be-94b92f0ad0d7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-89230689-5k4zx"
Sep 11 04:00:09.869748 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:00:09.869716    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-89230689-5k4zx"
Sep 11 04:00:10.141483 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:00:10.141446    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-89230689-5k4zx"]
Sep 11 04:00:10.142796 ip-10-0-141-105 kubenswrapper[2579]: W0911 04:00:10.142763    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2e2ec89d_74fd_438d_b0be_94b92f0ad0d7.slice/crio-44f57339d87fc5c986bdb3ed0d78f4fa9cf011b0d24c6f59cc51f25386d24e86 WatchSource:0}: Error finding container 44f57339d87fc5c986bdb3ed0d78f4fa9cf011b0d24c6f59cc51f25386d24e86: Status 404 returned error can't find the container with id 44f57339d87fc5c986bdb3ed0d78f4fa9cf011b0d24c6f59cc51f25386d24e86
Sep 11 04:00:10.994262 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:00:10.994229    2579 generic.go:356] "Generic (PLEG): container finished" podID="2e2ec89d-74fd-438d-b0be-94b92f0ad0d7" containerID="b79babd22f75f3469ce951e15452bf1cce7204b9917c1650ff24f454d0357d0b" exitCode=0
Sep 11 04:00:10.994671 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:00:10.994304    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-89230689-5k4zx" event={"ID":"2e2ec89d-74fd-438d-b0be-94b92f0ad0d7","Type":"ContainerDied","Data":"b79babd22f75f3469ce951e15452bf1cce7204b9917c1650ff24f454d0357d0b"}
Sep 11 04:00:10.994671 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:00:10.994342    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-89230689-5k4zx" event={"ID":"2e2ec89d-74fd-438d-b0be-94b92f0ad0d7","Type":"ContainerStarted","Data":"44f57339d87fc5c986bdb3ed0d78f4fa9cf011b0d24c6f59cc51f25386d24e86"}
Sep 11 04:00:12.141407 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:00:12.141381    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-89230689-5k4zx"
Sep 11 04:00:12.293265 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:00:12.293167    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2e2ec89d-74fd-438d-b0be-94b92f0ad0d7-maas-api-service-ca\") pod \"2e2ec89d-74fd-438d-b0be-94b92f0ad0d7\" (UID: \"2e2ec89d-74fd-438d-b0be-94b92f0ad0d7\") "
Sep 11 04:00:12.293535 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:00:12.293509    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2e2ec89d-74fd-438d-b0be-94b92f0ad0d7-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "2e2ec89d-74fd-438d-b0be-94b92f0ad0d7" (UID: "2e2ec89d-74fd-438d-b0be-94b92f0ad0d7"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 04:00:12.394654 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:00:12.394620    2579 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2e2ec89d-74fd-438d-b0be-94b92f0ad0d7-maas-api-service-ca\") on node \"ip-10-0-141-105.ec2.internal\" DevicePath \"\""
Sep 11 04:00:12.405818 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:00:12.405784    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-5z4cn"]
Sep 11 04:00:13.008579 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:00:13.008552    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-89230689-5k4zx"
Sep 11 04:00:13.008778 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:00:13.008578    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-89230689-5k4zx" event={"ID":"2e2ec89d-74fd-438d-b0be-94b92f0ad0d7","Type":"ContainerDied","Data":"44f57339d87fc5c986bdb3ed0d78f4fa9cf011b0d24c6f59cc51f25386d24e86"}
Sep 11 04:00:13.008778 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:00:13.008631    2579 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="44f57339d87fc5c986bdb3ed0d78f4fa9cf011b0d24c6f59cc51f25386d24e86"
Sep 11 04:00:22.967196 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:00:22.967159    2579 scope.go:117] "RemoveContainer" containerID="34e9d149f56159ae10cc0751a5f95508b6506bb9bfb7c37a7dbea4adde383673"
Sep 11 04:00:25.095101 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:00:25.095062    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-89230689-29818320-zhdvc"]
Sep 11 04:00:25.097059 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:00:25.097034    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-89230689-29818320-zhdvc"]
Sep 11 04:00:25.117444 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:00:25.117408    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-89230689-867f69445c-g7p8s"]
Sep 11 04:00:25.117672 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:00:25.117619    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-mt-89230689-867f69445c-g7p8s" podUID="2ef658ef-733b-4b20-9831-51c7fc7968c8" containerName="payload-processing" containerID="cri-o://3469fef190b3f47da96041f85f7096a2fe6c4430fc0cf1ac7c11bc5cf66a3a5d" gracePeriod=30
Sep 11 04:00:25.288622 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:00:25.288561    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-89230689-5k4zx"]
Sep 11 04:00:25.290937 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:00:25.290914    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-89230689-5k4zx"]
Sep 11 04:00:25.384470 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:00:25.384430    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-89230689-867f69445c-g7p8s"
Sep 11 04:00:25.418161 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:00:25.418132    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2ef658ef-733b-4b20-9831-51c7fc7968c8-plugins-config-volume\") pod \"2ef658ef-733b-4b20-9831-51c7fc7968c8\" (UID: \"2ef658ef-733b-4b20-9831-51c7fc7968c8\") "
Sep 11 04:00:25.418343 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:00:25.418204    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-75wl2\" (UniqueName: \"kubernetes.io/projected/2ef658ef-733b-4b20-9831-51c7fc7968c8-kube-api-access-75wl2\") pod \"2ef658ef-733b-4b20-9831-51c7fc7968c8\" (UID: \"2ef658ef-733b-4b20-9831-51c7fc7968c8\") "
Sep 11 04:00:25.418554 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:00:25.418530    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2ef658ef-733b-4b20-9831-51c7fc7968c8-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "2ef658ef-733b-4b20-9831-51c7fc7968c8" (UID: "2ef658ef-733b-4b20-9831-51c7fc7968c8"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 04:00:25.420435 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:00:25.420408    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2ef658ef-733b-4b20-9831-51c7fc7968c8-kube-api-access-75wl2" (OuterVolumeSpecName: "kube-api-access-75wl2") pod "2ef658ef-733b-4b20-9831-51c7fc7968c8" (UID: "2ef658ef-733b-4b20-9831-51c7fc7968c8"). InnerVolumeSpecName "kube-api-access-75wl2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 04:00:25.519187 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:00:25.519155    2579 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2ef658ef-733b-4b20-9831-51c7fc7968c8-plugins-config-volume\") on node \"ip-10-0-141-105.ec2.internal\" DevicePath \"\""
Sep 11 04:00:25.519187 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:00:25.519182    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-75wl2\" (UniqueName: \"kubernetes.io/projected/2ef658ef-733b-4b20-9831-51c7fc7968c8-kube-api-access-75wl2\") on node \"ip-10-0-141-105.ec2.internal\" DevicePath \"\""
Sep 11 04:00:26.087927 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:00:26.087892    2579 generic.go:356] "Generic (PLEG): container finished" podID="2ef658ef-733b-4b20-9831-51c7fc7968c8" containerID="3469fef190b3f47da96041f85f7096a2fe6c4430fc0cf1ac7c11bc5cf66a3a5d" exitCode=0
Sep 11 04:00:26.088140 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:00:26.087934    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-89230689-867f69445c-g7p8s"
Sep 11 04:00:26.088140 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:00:26.087983    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-89230689-867f69445c-g7p8s" event={"ID":"2ef658ef-733b-4b20-9831-51c7fc7968c8","Type":"ContainerDied","Data":"3469fef190b3f47da96041f85f7096a2fe6c4430fc0cf1ac7c11bc5cf66a3a5d"}
Sep 11 04:00:26.088140 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:00:26.088037    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-89230689-867f69445c-g7p8s" event={"ID":"2ef658ef-733b-4b20-9831-51c7fc7968c8","Type":"ContainerDied","Data":"dfc279bd35da4f3023dea4fe386d40179c8ed5f50d2c8521186a298f2b28b62a"}
Sep 11 04:00:26.088140 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:00:26.088063    2579 scope.go:117] "RemoveContainer" containerID="3469fef190b3f47da96041f85f7096a2fe6c4430fc0cf1ac7c11bc5cf66a3a5d"
Sep 11 04:00:26.098781 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:00:26.098764    2579 scope.go:117] "RemoveContainer" containerID="3469fef190b3f47da96041f85f7096a2fe6c4430fc0cf1ac7c11bc5cf66a3a5d"
Sep 11 04:00:26.099076 ip-10-0-141-105 kubenswrapper[2579]: E0911 04:00:26.099042    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3469fef190b3f47da96041f85f7096a2fe6c4430fc0cf1ac7c11bc5cf66a3a5d\": container with ID starting with 3469fef190b3f47da96041f85f7096a2fe6c4430fc0cf1ac7c11bc5cf66a3a5d not found: ID does not exist" containerID="3469fef190b3f47da96041f85f7096a2fe6c4430fc0cf1ac7c11bc5cf66a3a5d"
Sep 11 04:00:26.099076 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:00:26.099067    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3469fef190b3f47da96041f85f7096a2fe6c4430fc0cf1ac7c11bc5cf66a3a5d"} err="failed to get container status \"3469fef190b3f47da96041f85f7096a2fe6c4430fc0cf1ac7c11bc5cf66a3a5d\": rpc error: code = NotFound desc = could not find container \"3469fef190b3f47da96041f85f7096a2fe6c4430fc0cf1ac7c11bc5cf66a3a5d\": container with ID starting with 3469fef190b3f47da96041f85f7096a2fe6c4430fc0cf1ac7c11bc5cf66a3a5d not found: ID does not exist"
Sep 11 04:00:26.109660 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:00:26.109636    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-89230689-867f69445c-g7p8s"]
Sep 11 04:00:26.112831 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:00:26.112802    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-89230689-867f69445c-g7p8s"]
Sep 11 04:00:26.852916 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:00:26.852884    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2e2ec89d-74fd-438d-b0be-94b92f0ad0d7" path="/var/lib/kubelet/pods/2e2ec89d-74fd-438d-b0be-94b92f0ad0d7/volumes"
Sep 11 04:00:26.853280 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:00:26.853265    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2ef658ef-733b-4b20-9831-51c7fc7968c8" path="/var/lib/kubelet/pods/2ef658ef-733b-4b20-9831-51c7fc7968c8/volumes"
Sep 11 04:00:26.853515 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:00:26.853503    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7da7249c-b29b-4ad6-816c-5d534a49d6e3" path="/var/lib/kubelet/pods/7da7249c-b29b-4ad6-816c-5d534a49d6e3/volumes"
Sep 11 04:00:28.807153 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:00:28.807120    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-5z4cn"]
Sep 11 04:00:33.476140 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:00:33.476097    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-89230689/e2e-lifecycle-model-89230689-kserve-56d764c4f9-wlb5l"]
Sep 11 04:00:44.600684 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:00:44.600649    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-5z4cn"]
Sep 11 04:00:48.496360 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:00:48.496311    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-mt-89230689/e2e-lifecycle-model-89230689-kserve-56d764c4f9-wlb5l" podUID="9f5b03c9-ae3d-4834-963b-618cdb267a00" containerName="main" containerID="cri-o://55d4c11447f9283ffed128455b0270a3e9d05285dcef8199e503f47ce318cfc5" gracePeriod=45
Sep 11 04:00:48.754060 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:00:48.753994    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-89230689/e2e-lifecycle-model-89230689-kserve-56d764c4f9-wlb5l"
Sep 11 04:00:48.837714 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:00:48.837686    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9f5b03c9-ae3d-4834-963b-618cdb267a00-home\") pod \"9f5b03c9-ae3d-4834-963b-618cdb267a00\" (UID: \"9f5b03c9-ae3d-4834-963b-618cdb267a00\") "
Sep 11 04:00:48.837883 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:00:48.837733    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9f5b03c9-ae3d-4834-963b-618cdb267a00-tls-certs\") pod \"9f5b03c9-ae3d-4834-963b-618cdb267a00\" (UID: \"9f5b03c9-ae3d-4834-963b-618cdb267a00\") "
Sep 11 04:00:48.837883 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:00:48.837796    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hm24l\" (UniqueName: \"kubernetes.io/projected/9f5b03c9-ae3d-4834-963b-618cdb267a00-kube-api-access-hm24l\") pod \"9f5b03c9-ae3d-4834-963b-618cdb267a00\" (UID: \"9f5b03c9-ae3d-4834-963b-618cdb267a00\") "
Sep 11 04:00:48.837883 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:00:48.837848    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9f5b03c9-ae3d-4834-963b-618cdb267a00-model-cache\") pod \"9f5b03c9-ae3d-4834-963b-618cdb267a00\" (UID: \"9f5b03c9-ae3d-4834-963b-618cdb267a00\") "
Sep 11 04:00:48.838056 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:00:48.837885    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9f5b03c9-ae3d-4834-963b-618cdb267a00-tmp-dir\") pod \"9f5b03c9-ae3d-4834-963b-618cdb267a00\" (UID: \"9f5b03c9-ae3d-4834-963b-618cdb267a00\") "
Sep 11 04:00:48.838056 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:00:48.837924    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9f5b03c9-ae3d-4834-963b-618cdb267a00-dshm\") pod \"9f5b03c9-ae3d-4834-963b-618cdb267a00\" (UID: \"9f5b03c9-ae3d-4834-963b-618cdb267a00\") "
Sep 11 04:00:48.838056 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:00:48.837975    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9f5b03c9-ae3d-4834-963b-618cdb267a00-home" (OuterVolumeSpecName: "home") pod "9f5b03c9-ae3d-4834-963b-618cdb267a00" (UID: "9f5b03c9-ae3d-4834-963b-618cdb267a00"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 04:00:48.838262 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:00:48.838235    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9f5b03c9-ae3d-4834-963b-618cdb267a00-model-cache" (OuterVolumeSpecName: "model-cache") pod "9f5b03c9-ae3d-4834-963b-618cdb267a00" (UID: "9f5b03c9-ae3d-4834-963b-618cdb267a00"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 04:00:48.838340 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:00:48.838239    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9f5b03c9-ae3d-4834-963b-618cdb267a00-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "9f5b03c9-ae3d-4834-963b-618cdb267a00" (UID: "9f5b03c9-ae3d-4834-963b-618cdb267a00"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 04:00:48.838340 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:00:48.838280    2579 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9f5b03c9-ae3d-4834-963b-618cdb267a00-home\") on node \"ip-10-0-141-105.ec2.internal\" DevicePath \"\""
Sep 11 04:00:48.840257 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:00:48.840231    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9f5b03c9-ae3d-4834-963b-618cdb267a00-dshm" (OuterVolumeSpecName: "dshm") pod "9f5b03c9-ae3d-4834-963b-618cdb267a00" (UID: "9f5b03c9-ae3d-4834-963b-618cdb267a00"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 04:00:48.840375 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:00:48.840291    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9f5b03c9-ae3d-4834-963b-618cdb267a00-kube-api-access-hm24l" (OuterVolumeSpecName: "kube-api-access-hm24l") pod "9f5b03c9-ae3d-4834-963b-618cdb267a00" (UID: "9f5b03c9-ae3d-4834-963b-618cdb267a00"). InnerVolumeSpecName "kube-api-access-hm24l". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 04:00:48.840375 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:00:48.840349    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9f5b03c9-ae3d-4834-963b-618cdb267a00-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "9f5b03c9-ae3d-4834-963b-618cdb267a00" (UID: "9f5b03c9-ae3d-4834-963b-618cdb267a00"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 04:00:48.939095 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:00:48.939048    2579 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9f5b03c9-ae3d-4834-963b-618cdb267a00-model-cache\") on node \"ip-10-0-141-105.ec2.internal\" DevicePath \"\""
Sep 11 04:00:48.939277 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:00:48.939111    2579 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9f5b03c9-ae3d-4834-963b-618cdb267a00-tmp-dir\") on node \"ip-10-0-141-105.ec2.internal\" DevicePath \"\""
Sep 11 04:00:48.939277 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:00:48.939125    2579 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9f5b03c9-ae3d-4834-963b-618cdb267a00-dshm\") on node \"ip-10-0-141-105.ec2.internal\" DevicePath \"\""
Sep 11 04:00:48.939277 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:00:48.939139    2579 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9f5b03c9-ae3d-4834-963b-618cdb267a00-tls-certs\") on node \"ip-10-0-141-105.ec2.internal\" DevicePath \"\""
Sep 11 04:00:48.939277 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:00:48.939154    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hm24l\" (UniqueName: \"kubernetes.io/projected/9f5b03c9-ae3d-4834-963b-618cdb267a00-kube-api-access-hm24l\") on node \"ip-10-0-141-105.ec2.internal\" DevicePath \"\""
Sep 11 04:00:49.220930 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:00:49.220891    2579 generic.go:356] "Generic (PLEG): container finished" podID="9f5b03c9-ae3d-4834-963b-618cdb267a00" containerID="55d4c11447f9283ffed128455b0270a3e9d05285dcef8199e503f47ce318cfc5" exitCode=0
Sep 11 04:00:49.221137 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:00:49.220953    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-89230689/e2e-lifecycle-model-89230689-kserve-56d764c4f9-wlb5l"
Sep 11 04:00:49.221137 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:00:49.220991    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-89230689/e2e-lifecycle-model-89230689-kserve-56d764c4f9-wlb5l" event={"ID":"9f5b03c9-ae3d-4834-963b-618cdb267a00","Type":"ContainerDied","Data":"55d4c11447f9283ffed128455b0270a3e9d05285dcef8199e503f47ce318cfc5"}
Sep 11 04:00:49.221137 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:00:49.221036    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-89230689/e2e-lifecycle-model-89230689-kserve-56d764c4f9-wlb5l" event={"ID":"9f5b03c9-ae3d-4834-963b-618cdb267a00","Type":"ContainerDied","Data":"2660f4d810f3a9b84e2cab55a567f49068da4158696ab698c3e895b0323a82f6"}
Sep 11 04:00:49.221137 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:00:49.221053    2579 scope.go:117] "RemoveContainer" containerID="55d4c11447f9283ffed128455b0270a3e9d05285dcef8199e503f47ce318cfc5"
Sep 11 04:00:49.230232 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:00:49.230201    2579 scope.go:117] "RemoveContainer" containerID="55d4c11447f9283ffed128455b0270a3e9d05285dcef8199e503f47ce318cfc5"
Sep 11 04:00:49.230504 ip-10-0-141-105 kubenswrapper[2579]: E0911 04:00:49.230483    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"55d4c11447f9283ffed128455b0270a3e9d05285dcef8199e503f47ce318cfc5\": container with ID starting with 55d4c11447f9283ffed128455b0270a3e9d05285dcef8199e503f47ce318cfc5 not found: ID does not exist" containerID="55d4c11447f9283ffed128455b0270a3e9d05285dcef8199e503f47ce318cfc5"
Sep 11 04:00:49.230580 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:00:49.230514    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"55d4c11447f9283ffed128455b0270a3e9d05285dcef8199e503f47ce318cfc5"} err="failed to get container status \"55d4c11447f9283ffed128455b0270a3e9d05285dcef8199e503f47ce318cfc5\": rpc error: code = NotFound desc = could not find container \"55d4c11447f9283ffed128455b0270a3e9d05285dcef8199e503f47ce318cfc5\": container with ID starting with 55d4c11447f9283ffed128455b0270a3e9d05285dcef8199e503f47ce318cfc5 not found: ID does not exist"
Sep 11 04:00:49.237156 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:00:49.237132    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-89230689/e2e-lifecycle-model-89230689-kserve-56d764c4f9-wlb5l"]
Sep 11 04:00:49.242789 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:00:49.242767    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-mt-89230689/e2e-lifecycle-model-89230689-kserve-56d764c4f9-wlb5l"]
Sep 11 04:00:50.852386 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:00:50.852332    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9f5b03c9-ae3d-4834-963b-618cdb267a00" path="/var/lib/kubelet/pods/9f5b03c9-ae3d-4834-963b-618cdb267a00/volumes"
Sep 11 04:01:00.205035 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:01:00.204998    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-5z4cn"]
Sep 11 04:01:10.514162 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:01:10.514120    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7704-722178"]
Sep 11 04:01:10.514707 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:01:10.514684    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2ef658ef-733b-4b20-9831-51c7fc7968c8" containerName="payload-processing"
Sep 11 04:01:10.514793 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:01:10.514710    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="2ef658ef-733b-4b20-9831-51c7fc7968c8" containerName="payload-processing"
Sep 11 04:01:10.514793 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:01:10.514726    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9f5b03c9-ae3d-4834-963b-618cdb267a00" containerName="main"
Sep 11 04:01:10.514793 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:01:10.514734    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="9f5b03c9-ae3d-4834-963b-618cdb267a00" containerName="main"
Sep 11 04:01:10.514793 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:01:10.514743    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2e2ec89d-74fd-438d-b0be-94b92f0ad0d7" containerName="revoke-keys"
Sep 11 04:01:10.514793 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:01:10.514751    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="2e2ec89d-74fd-438d-b0be-94b92f0ad0d7" containerName="revoke-keys"
Sep 11 04:01:10.515046 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:01:10.514867    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="2e2ec89d-74fd-438d-b0be-94b92f0ad0d7" containerName="revoke-keys"
Sep 11 04:01:10.515046 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:01:10.514880    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="9f5b03c9-ae3d-4834-963b-618cdb267a00" containerName="main"
Sep 11 04:01:10.515046 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:01:10.514892    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="2ef658ef-733b-4b20-9831-51c7fc7968c8" containerName="payload-processing"
Sep 11 04:01:10.518411 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:01:10.518372    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7704-722178"
Sep 11 04:01:10.521308 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:01:10.521283    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-fjqk5\""
Sep 11 04:01:10.523389 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:01:10.523366    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7704-722178"]
Sep 11 04:01:10.634530 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:01:10.634493    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xdbll\" (UniqueName: \"kubernetes.io/projected/a90270b2-725b-486e-b505-d03525e42c9a-kube-api-access-xdbll\") pod \"test-curl-7704-722178\" (UID: \"a90270b2-725b-486e-b505-d03525e42c9a\") " pod="opendatahub/test-curl-7704-722178"
Sep 11 04:01:10.735668 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:01:10.735631    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xdbll\" (UniqueName: \"kubernetes.io/projected/a90270b2-725b-486e-b505-d03525e42c9a-kube-api-access-xdbll\") pod \"test-curl-7704-722178\" (UID: \"a90270b2-725b-486e-b505-d03525e42c9a\") " pod="opendatahub/test-curl-7704-722178"
Sep 11 04:01:10.744564 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:01:10.744535    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xdbll\" (UniqueName: \"kubernetes.io/projected/a90270b2-725b-486e-b505-d03525e42c9a-kube-api-access-xdbll\") pod \"test-curl-7704-722178\" (UID: \"a90270b2-725b-486e-b505-d03525e42c9a\") " pod="opendatahub/test-curl-7704-722178"
Sep 11 04:01:10.829381 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:01:10.829301    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7704-722178"
Sep 11 04:01:11.103981 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:01:11.103953    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7704-722178"]
Sep 11 04:01:11.105929 ip-10-0-141-105 kubenswrapper[2579]: W0911 04:01:11.105899    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda90270b2_725b_486e_b505_d03525e42c9a.slice/crio-cfb6d5382e73d31e29dd6c61e4c5164ef63e4d9e9e86bd6d3517070e9c61a443 WatchSource:0}: Error finding container cfb6d5382e73d31e29dd6c61e4c5164ef63e4d9e9e86bd6d3517070e9c61a443: Status 404 returned error can't find the container with id cfb6d5382e73d31e29dd6c61e4c5164ef63e4d9e9e86bd6d3517070e9c61a443
Sep 11 04:01:11.346269 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:01:11.346234    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7704-722178" event={"ID":"a90270b2-725b-486e-b505-d03525e42c9a","Type":"ContainerStarted","Data":"9b5ae1d924d27308f2267d980b878064d8c3d3b8e456aad544d401aeee41473a"}
Sep 11 04:01:11.346479 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:01:11.346279    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7704-722178" event={"ID":"a90270b2-725b-486e-b505-d03525e42c9a","Type":"ContainerStarted","Data":"cfb6d5382e73d31e29dd6c61e4c5164ef63e4d9e9e86bd6d3517070e9c61a443"}
Sep 11 04:01:11.361990 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:01:11.361939    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-7704-722178" podStartSLOduration=1.361923854 podStartE2EDuration="1.361923854s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 04:01:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 04:01:11.359991095 +0000 UTC m=+1249.015196662" watchObservedRunningTime="2026-09-11 04:01:11.361923854 +0000 UTC m=+1249.017129418"
Sep 11 04:01:11.948318 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:01:11.948269    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7704-722178"]
Sep 11 04:01:11.951582 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:01:11.951558    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7704-722178"]
Sep 11 04:01:12.351853 ip-10-0-141-105 kubenswrapper[2579]: E0911 04:01:12.351821    2579 kuberuntime_manager.go:649] "Missing actuated resource record" pod="opendatahub/test-curl-7704-722178" container="test-curl-7704-722178"
Sep 11 04:01:13.357272 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:01:13.357224    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-7704-722178" podUID="a90270b2-725b-486e-b505-d03525e42c9a" containerName="test-curl-7704-722178" containerID="cri-o://9b5ae1d924d27308f2267d980b878064d8c3d3b8e456aad544d401aeee41473a" gracePeriod=2
Sep 11 04:01:15.542009 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:01:15.541983    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7704-722178"
Sep 11 04:01:15.574152 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:01:15.574126    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xdbll\" (UniqueName: \"kubernetes.io/projected/a90270b2-725b-486e-b505-d03525e42c9a-kube-api-access-xdbll\") pod \"a90270b2-725b-486e-b505-d03525e42c9a\" (UID: \"a90270b2-725b-486e-b505-d03525e42c9a\") "
Sep 11 04:01:15.576304 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:01:15.576271    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a90270b2-725b-486e-b505-d03525e42c9a-kube-api-access-xdbll" (OuterVolumeSpecName: "kube-api-access-xdbll") pod "a90270b2-725b-486e-b505-d03525e42c9a" (UID: "a90270b2-725b-486e-b505-d03525e42c9a"). InnerVolumeSpecName "kube-api-access-xdbll". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 04:01:15.675317 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:01:15.675214    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xdbll\" (UniqueName: \"kubernetes.io/projected/a90270b2-725b-486e-b505-d03525e42c9a-kube-api-access-xdbll\") on node \"ip-10-0-141-105.ec2.internal\" DevicePath \"\""
Sep 11 04:01:16.374487 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:01:16.374443    2579 generic.go:356] "Generic (PLEG): container finished" podID="a90270b2-725b-486e-b505-d03525e42c9a" containerID="9b5ae1d924d27308f2267d980b878064d8c3d3b8e456aad544d401aeee41473a" exitCode=137
Sep 11 04:01:16.374713 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:01:16.374500    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7704-722178"
Sep 11 04:01:16.374713 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:01:16.374502    2579 scope.go:117] "RemoveContainer" containerID="9b5ae1d924d27308f2267d980b878064d8c3d3b8e456aad544d401aeee41473a"
Sep 11 04:01:16.384916 ip-10-0-141-105 kubenswrapper[2579]: E0911 04:01:16.384888    2579 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7704-722178\" is forbidden: User \"system:node:ip-10-0-141-105.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-141-105.ec2.internal' and this object" podUID="a90270b2-725b-486e-b505-d03525e42c9a" pod="opendatahub/test-curl-7704-722178"
Sep 11 04:01:16.385188 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:01:16.385169    2579 scope.go:117] "RemoveContainer" containerID="9b5ae1d924d27308f2267d980b878064d8c3d3b8e456aad544d401aeee41473a"
Sep 11 04:01:16.385466 ip-10-0-141-105 kubenswrapper[2579]: E0911 04:01:16.385446    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9b5ae1d924d27308f2267d980b878064d8c3d3b8e456aad544d401aeee41473a\": container with ID starting with 9b5ae1d924d27308f2267d980b878064d8c3d3b8e456aad544d401aeee41473a not found: ID does not exist" containerID="9b5ae1d924d27308f2267d980b878064d8c3d3b8e456aad544d401aeee41473a"
Sep 11 04:01:16.385538 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:01:16.385474    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9b5ae1d924d27308f2267d980b878064d8c3d3b8e456aad544d401aeee41473a"} err="failed to get container status \"9b5ae1d924d27308f2267d980b878064d8c3d3b8e456aad544d401aeee41473a\": rpc error: code = NotFound desc = could not find container \"9b5ae1d924d27308f2267d980b878064d8c3d3b8e456aad544d401aeee41473a\": container with ID starting with 9b5ae1d924d27308f2267d980b878064d8c3d3b8e456aad544d401aeee41473a not found: ID does not exist"
Sep 11 04:01:16.601601 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:01:16.601563    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-5z4cn"]
Sep 11 04:01:16.853004 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:01:16.852979    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a90270b2-725b-486e-b505-d03525e42c9a" path="/var/lib/kubelet/pods/a90270b2-725b-486e-b505-d03525e42c9a/volumes"
Sep 11 04:01:22.990917 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:01:22.990883    2579 scope.go:117] "RemoveContainer" containerID="7787147ccf33733827ca6905796604c5d250a61c3ec6296e2009952cd50ae0cb"
Sep 11 04:01:23.000319 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:01:23.000288    2579 scope.go:117] "RemoveContainer" containerID="54b5f180ead5fed753e611b80819efc8a32b55193aa407ae50f723c608c507a5"
Sep 11 04:01:23.010765 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:01:23.010742    2579 scope.go:117] "RemoveContainer" containerID="7942a49e797c5d92e65afa7c00cfc58c1399e3abb8e5a1a77fb4e82ffb3fab46"
Sep 11 04:01:27.029986 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:01:27.027929    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-b4d3cf/auth-test-model-b4d3cf-kserve-7b594c5fff-dp5pp"]
Sep 11 04:01:34.099580 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:01:34.099547    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w3-b4d3cf/sub-test-model-b4d3cf-kserve-7f7bcdfb8c-tg8w9"]
Sep 11 04:01:34.100153 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:01:34.099934    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a90270b2-725b-486e-b505-d03525e42c9a" containerName="test-curl-7704-722178"
Sep 11 04:01:34.100153 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:01:34.099949    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="a90270b2-725b-486e-b505-d03525e42c9a" containerName="test-curl-7704-722178"
Sep 11 04:01:34.100153 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:01:34.100037    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="a90270b2-725b-486e-b505-d03525e42c9a" containerName="test-curl-7704-722178"
Sep 11 04:01:34.104892 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:01:34.104871    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-b4d3cf/sub-test-model-b4d3cf-kserve-7f7bcdfb8c-tg8w9"
Sep 11 04:01:34.107273 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:01:34.107252    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-b4d3cf\"/\"sub-test-model-b4d3cf-kserve-self-signed-certs\""
Sep 11 04:01:34.121828 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:01:34.121794    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-b4d3cf/sub-test-model-b4d3cf-kserve-7f7bcdfb8c-tg8w9"]
Sep 11 04:01:34.259471 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:01:34.259433    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3c9f0173-a36d-4fe4-b1ad-756a3d48be13-dshm\") pod \"sub-test-model-b4d3cf-kserve-7f7bcdfb8c-tg8w9\" (UID: \"3c9f0173-a36d-4fe4-b1ad-756a3d48be13\") " pod="ai-tenant-e2e-shared-a-w3-b4d3cf/sub-test-model-b4d3cf-kserve-7f7bcdfb8c-tg8w9"
Sep 11 04:01:34.259653 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:01:34.259490    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3c9f0173-a36d-4fe4-b1ad-756a3d48be13-tmp-dir\") pod \"sub-test-model-b4d3cf-kserve-7f7bcdfb8c-tg8w9\" (UID: \"3c9f0173-a36d-4fe4-b1ad-756a3d48be13\") " pod="ai-tenant-e2e-shared-a-w3-b4d3cf/sub-test-model-b4d3cf-kserve-7f7bcdfb8c-tg8w9"
Sep 11 04:01:34.259653 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:01:34.259546    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3c9f0173-a36d-4fe4-b1ad-756a3d48be13-model-cache\") pod \"sub-test-model-b4d3cf-kserve-7f7bcdfb8c-tg8w9\" (UID: \"3c9f0173-a36d-4fe4-b1ad-756a3d48be13\") " pod="ai-tenant-e2e-shared-a-w3-b4d3cf/sub-test-model-b4d3cf-kserve-7f7bcdfb8c-tg8w9"
Sep 11 04:01:34.259653 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:01:34.259577    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sfl2f\" (UniqueName: \"kubernetes.io/projected/3c9f0173-a36d-4fe4-b1ad-756a3d48be13-kube-api-access-sfl2f\") pod \"sub-test-model-b4d3cf-kserve-7f7bcdfb8c-tg8w9\" (UID: \"3c9f0173-a36d-4fe4-b1ad-756a3d48be13\") " pod="ai-tenant-e2e-shared-a-w3-b4d3cf/sub-test-model-b4d3cf-kserve-7f7bcdfb8c-tg8w9"
Sep 11 04:01:34.259653 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:01:34.259612    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3c9f0173-a36d-4fe4-b1ad-756a3d48be13-tls-certs\") pod \"sub-test-model-b4d3cf-kserve-7f7bcdfb8c-tg8w9\" (UID: \"3c9f0173-a36d-4fe4-b1ad-756a3d48be13\") " pod="ai-tenant-e2e-shared-a-w3-b4d3cf/sub-test-model-b4d3cf-kserve-7f7bcdfb8c-tg8w9"
Sep 11 04:01:34.259653 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:01:34.259632    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3c9f0173-a36d-4fe4-b1ad-756a3d48be13-home\") pod \"sub-test-model-b4d3cf-kserve-7f7bcdfb8c-tg8w9\" (UID: \"3c9f0173-a36d-4fe4-b1ad-756a3d48be13\") " pod="ai-tenant-e2e-shared-a-w3-b4d3cf/sub-test-model-b4d3cf-kserve-7f7bcdfb8c-tg8w9"
Sep 11 04:01:34.360797 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:01:34.360765    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3c9f0173-a36d-4fe4-b1ad-756a3d48be13-dshm\") pod \"sub-test-model-b4d3cf-kserve-7f7bcdfb8c-tg8w9\" (UID: \"3c9f0173-a36d-4fe4-b1ad-756a3d48be13\") " pod="ai-tenant-e2e-shared-a-w3-b4d3cf/sub-test-model-b4d3cf-kserve-7f7bcdfb8c-tg8w9"
Sep 11 04:01:34.360984 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:01:34.360803    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3c9f0173-a36d-4fe4-b1ad-756a3d48be13-tmp-dir\") pod \"sub-test-model-b4d3cf-kserve-7f7bcdfb8c-tg8w9\" (UID: \"3c9f0173-a36d-4fe4-b1ad-756a3d48be13\") " pod="ai-tenant-e2e-shared-a-w3-b4d3cf/sub-test-model-b4d3cf-kserve-7f7bcdfb8c-tg8w9"
Sep 11 04:01:34.360984 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:01:34.360820    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3c9f0173-a36d-4fe4-b1ad-756a3d48be13-model-cache\") pod \"sub-test-model-b4d3cf-kserve-7f7bcdfb8c-tg8w9\" (UID: \"3c9f0173-a36d-4fe4-b1ad-756a3d48be13\") " pod="ai-tenant-e2e-shared-a-w3-b4d3cf/sub-test-model-b4d3cf-kserve-7f7bcdfb8c-tg8w9"
Sep 11 04:01:34.360984 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:01:34.360836    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sfl2f\" (UniqueName: \"kubernetes.io/projected/3c9f0173-a36d-4fe4-b1ad-756a3d48be13-kube-api-access-sfl2f\") pod \"sub-test-model-b4d3cf-kserve-7f7bcdfb8c-tg8w9\" (UID: \"3c9f0173-a36d-4fe4-b1ad-756a3d48be13\") " pod="ai-tenant-e2e-shared-a-w3-b4d3cf/sub-test-model-b4d3cf-kserve-7f7bcdfb8c-tg8w9"
Sep 11 04:01:34.360984 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:01:34.360962    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3c9f0173-a36d-4fe4-b1ad-756a3d48be13-tls-certs\") pod \"sub-test-model-b4d3cf-kserve-7f7bcdfb8c-tg8w9\" (UID: \"3c9f0173-a36d-4fe4-b1ad-756a3d48be13\") " pod="ai-tenant-e2e-shared-a-w3-b4d3cf/sub-test-model-b4d3cf-kserve-7f7bcdfb8c-tg8w9"
Sep 11 04:01:34.361273 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:01:34.361006    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3c9f0173-a36d-4fe4-b1ad-756a3d48be13-home\") pod \"sub-test-model-b4d3cf-kserve-7f7bcdfb8c-tg8w9\" (UID: \"3c9f0173-a36d-4fe4-b1ad-756a3d48be13\") " pod="ai-tenant-e2e-shared-a-w3-b4d3cf/sub-test-model-b4d3cf-kserve-7f7bcdfb8c-tg8w9"
Sep 11 04:01:34.361349 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:01:34.361288    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3c9f0173-a36d-4fe4-b1ad-756a3d48be13-model-cache\") pod \"sub-test-model-b4d3cf-kserve-7f7bcdfb8c-tg8w9\" (UID: \"3c9f0173-a36d-4fe4-b1ad-756a3d48be13\") " pod="ai-tenant-e2e-shared-a-w3-b4d3cf/sub-test-model-b4d3cf-kserve-7f7bcdfb8c-tg8w9"
Sep 11 04:01:34.361349 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:01:34.361289    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3c9f0173-a36d-4fe4-b1ad-756a3d48be13-tmp-dir\") pod \"sub-test-model-b4d3cf-kserve-7f7bcdfb8c-tg8w9\" (UID: \"3c9f0173-a36d-4fe4-b1ad-756a3d48be13\") " pod="ai-tenant-e2e-shared-a-w3-b4d3cf/sub-test-model-b4d3cf-kserve-7f7bcdfb8c-tg8w9"
Sep 11 04:01:34.361349 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:01:34.361325    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3c9f0173-a36d-4fe4-b1ad-756a3d48be13-home\") pod \"sub-test-model-b4d3cf-kserve-7f7bcdfb8c-tg8w9\" (UID: \"3c9f0173-a36d-4fe4-b1ad-756a3d48be13\") " pod="ai-tenant-e2e-shared-a-w3-b4d3cf/sub-test-model-b4d3cf-kserve-7f7bcdfb8c-tg8w9"
Sep 11 04:01:34.363266 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:01:34.363226    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3c9f0173-a36d-4fe4-b1ad-756a3d48be13-dshm\") pod \"sub-test-model-b4d3cf-kserve-7f7bcdfb8c-tg8w9\" (UID: \"3c9f0173-a36d-4fe4-b1ad-756a3d48be13\") " pod="ai-tenant-e2e-shared-a-w3-b4d3cf/sub-test-model-b4d3cf-kserve-7f7bcdfb8c-tg8w9"
Sep 11 04:01:34.363497 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:01:34.363477    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3c9f0173-a36d-4fe4-b1ad-756a3d48be13-tls-certs\") pod \"sub-test-model-b4d3cf-kserve-7f7bcdfb8c-tg8w9\" (UID: \"3c9f0173-a36d-4fe4-b1ad-756a3d48be13\") " pod="ai-tenant-e2e-shared-a-w3-b4d3cf/sub-test-model-b4d3cf-kserve-7f7bcdfb8c-tg8w9"
Sep 11 04:01:34.368905 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:01:34.368884    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sfl2f\" (UniqueName: \"kubernetes.io/projected/3c9f0173-a36d-4fe4-b1ad-756a3d48be13-kube-api-access-sfl2f\") pod \"sub-test-model-b4d3cf-kserve-7f7bcdfb8c-tg8w9\" (UID: \"3c9f0173-a36d-4fe4-b1ad-756a3d48be13\") " pod="ai-tenant-e2e-shared-a-w3-b4d3cf/sub-test-model-b4d3cf-kserve-7f7bcdfb8c-tg8w9"
Sep 11 04:01:34.417952 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:01:34.417927    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-b4d3cf/sub-test-model-b4d3cf-kserve-7f7bcdfb8c-tg8w9"
Sep 11 04:01:34.695241 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:01:34.695216    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-b4d3cf/sub-test-model-b4d3cf-kserve-7f7bcdfb8c-tg8w9"]
Sep 11 04:01:34.697881 ip-10-0-141-105 kubenswrapper[2579]: W0911 04:01:34.697846    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod3c9f0173_a36d_4fe4_b1ad_756a3d48be13.slice/crio-ddee58cb366a1d33dce3a6a4b6aee4e5ea3877d8e00ef5126f9124a6db082b5a WatchSource:0}: Error finding container ddee58cb366a1d33dce3a6a4b6aee4e5ea3877d8e00ef5126f9124a6db082b5a: Status 404 returned error can't find the container with id ddee58cb366a1d33dce3a6a4b6aee4e5ea3877d8e00ef5126f9124a6db082b5a
Sep 11 04:01:35.213219 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:01:35.213186    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-5z4cn"]
Sep 11 04:01:35.485749 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:01:35.485651    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-b4d3cf/sub-test-model-b4d3cf-kserve-7f7bcdfb8c-tg8w9" event={"ID":"3c9f0173-a36d-4fe4-b1ad-756a3d48be13","Type":"ContainerStarted","Data":"2b9916761926c4c441eafb0864f67741217cb9a80aa82f3be4a0e398e543a9ca"}
Sep 11 04:01:35.485749 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:01:35.485696    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-b4d3cf/sub-test-model-b4d3cf-kserve-7f7bcdfb8c-tg8w9" event={"ID":"3c9f0173-a36d-4fe4-b1ad-756a3d48be13","Type":"ContainerStarted","Data":"ddee58cb366a1d33dce3a6a4b6aee4e5ea3877d8e00ef5126f9124a6db082b5a"}
Sep 11 04:01:35.502903 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:01:35.502855    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w3-b4d3cf/sub-test-model-b4d3cf-kserve-7f7bcdfb8c-tg8w9" podStartSLOduration=1.502839658 podStartE2EDuration="1.502839658s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 04:01:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 04:01:35.501014731 +0000 UTC m=+1273.156220309" watchObservedRunningTime="2026-09-11 04:01:35.502839658 +0000 UTC m=+1273.158045254"
Sep 11 04:01:42.050693 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:01:42.050643    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w3-b4d3cf/auth-test-model-b4d3cf-kserve-7b594c5fff-dp5pp" podUID="37955b55-9d5a-4d4e-8883-399b2160d5b8" containerName="main" containerID="cri-o://b865bb14d5b0b15fe27fc2f89f0eb871c8d7d37fb326b2a8be00e20e1e73ae75" gracePeriod=45
Sep 11 04:01:42.308278 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:01:42.308221    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-b4d3cf/auth-test-model-b4d3cf-kserve-7b594c5fff-dp5pp"
Sep 11 04:01:42.342749 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:01:42.342721    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/37955b55-9d5a-4d4e-8883-399b2160d5b8-dshm\") pod \"37955b55-9d5a-4d4e-8883-399b2160d5b8\" (UID: \"37955b55-9d5a-4d4e-8883-399b2160d5b8\") "
Sep 11 04:01:42.342898 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:01:42.342766    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/37955b55-9d5a-4d4e-8883-399b2160d5b8-tmp-dir\") pod \"37955b55-9d5a-4d4e-8883-399b2160d5b8\" (UID: \"37955b55-9d5a-4d4e-8883-399b2160d5b8\") "
Sep 11 04:01:42.342898 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:01:42.342800    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/37955b55-9d5a-4d4e-8883-399b2160d5b8-model-cache\") pod \"37955b55-9d5a-4d4e-8883-399b2160d5b8\" (UID: \"37955b55-9d5a-4d4e-8883-399b2160d5b8\") "
Sep 11 04:01:42.342898 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:01:42.342883    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/37955b55-9d5a-4d4e-8883-399b2160d5b8-home\") pod \"37955b55-9d5a-4d4e-8883-399b2160d5b8\" (UID: \"37955b55-9d5a-4d4e-8883-399b2160d5b8\") "
Sep 11 04:01:42.343036 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:01:42.342904    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cz28k\" (UniqueName: \"kubernetes.io/projected/37955b55-9d5a-4d4e-8883-399b2160d5b8-kube-api-access-cz28k\") pod \"37955b55-9d5a-4d4e-8883-399b2160d5b8\" (UID: \"37955b55-9d5a-4d4e-8883-399b2160d5b8\") "
Sep 11 04:01:42.343036 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:01:42.342951    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/37955b55-9d5a-4d4e-8883-399b2160d5b8-tls-certs\") pod \"37955b55-9d5a-4d4e-8883-399b2160d5b8\" (UID: \"37955b55-9d5a-4d4e-8883-399b2160d5b8\") "
Sep 11 04:01:42.343223 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:01:42.343107    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/37955b55-9d5a-4d4e-8883-399b2160d5b8-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "37955b55-9d5a-4d4e-8883-399b2160d5b8" (UID: "37955b55-9d5a-4d4e-8883-399b2160d5b8"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 04:01:42.343223 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:01:42.343196    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/37955b55-9d5a-4d4e-8883-399b2160d5b8-home" (OuterVolumeSpecName: "home") pod "37955b55-9d5a-4d4e-8883-399b2160d5b8" (UID: "37955b55-9d5a-4d4e-8883-399b2160d5b8"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 04:01:42.343339 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:01:42.343245    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/37955b55-9d5a-4d4e-8883-399b2160d5b8-model-cache" (OuterVolumeSpecName: "model-cache") pod "37955b55-9d5a-4d4e-8883-399b2160d5b8" (UID: "37955b55-9d5a-4d4e-8883-399b2160d5b8"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 04:01:42.343339 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:01:42.343310    2579 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/37955b55-9d5a-4d4e-8883-399b2160d5b8-home\") on node \"ip-10-0-141-105.ec2.internal\" DevicePath \"\""
Sep 11 04:01:42.343339 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:01:42.343329    2579 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/37955b55-9d5a-4d4e-8883-399b2160d5b8-tmp-dir\") on node \"ip-10-0-141-105.ec2.internal\" DevicePath \"\""
Sep 11 04:01:42.345204 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:01:42.345179    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/37955b55-9d5a-4d4e-8883-399b2160d5b8-dshm" (OuterVolumeSpecName: "dshm") pod "37955b55-9d5a-4d4e-8883-399b2160d5b8" (UID: "37955b55-9d5a-4d4e-8883-399b2160d5b8"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 04:01:42.345335 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:01:42.345314    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/37955b55-9d5a-4d4e-8883-399b2160d5b8-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "37955b55-9d5a-4d4e-8883-399b2160d5b8" (UID: "37955b55-9d5a-4d4e-8883-399b2160d5b8"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 04:01:42.345491 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:01:42.345467    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/37955b55-9d5a-4d4e-8883-399b2160d5b8-kube-api-access-cz28k" (OuterVolumeSpecName: "kube-api-access-cz28k") pod "37955b55-9d5a-4d4e-8883-399b2160d5b8" (UID: "37955b55-9d5a-4d4e-8883-399b2160d5b8"). InnerVolumeSpecName "kube-api-access-cz28k". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 04:01:42.444136 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:01:42.444109    2579 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/37955b55-9d5a-4d4e-8883-399b2160d5b8-tls-certs\") on node \"ip-10-0-141-105.ec2.internal\" DevicePath \"\""
Sep 11 04:01:42.444136 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:01:42.444134    2579 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/37955b55-9d5a-4d4e-8883-399b2160d5b8-dshm\") on node \"ip-10-0-141-105.ec2.internal\" DevicePath \"\""
Sep 11 04:01:42.444136 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:01:42.444144    2579 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/37955b55-9d5a-4d4e-8883-399b2160d5b8-model-cache\") on node \"ip-10-0-141-105.ec2.internal\" DevicePath \"\""
Sep 11 04:01:42.444361 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:01:42.444152    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cz28k\" (UniqueName: \"kubernetes.io/projected/37955b55-9d5a-4d4e-8883-399b2160d5b8-kube-api-access-cz28k\") on node \"ip-10-0-141-105.ec2.internal\" DevicePath \"\""
Sep 11 04:01:42.525122 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:01:42.525059    2579 generic.go:356] "Generic (PLEG): container finished" podID="37955b55-9d5a-4d4e-8883-399b2160d5b8" containerID="b865bb14d5b0b15fe27fc2f89f0eb871c8d7d37fb326b2a8be00e20e1e73ae75" exitCode=0
Sep 11 04:01:42.525122 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:01:42.525114    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-b4d3cf/auth-test-model-b4d3cf-kserve-7b594c5fff-dp5pp"
Sep 11 04:01:42.525331 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:01:42.525124    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-b4d3cf/auth-test-model-b4d3cf-kserve-7b594c5fff-dp5pp" event={"ID":"37955b55-9d5a-4d4e-8883-399b2160d5b8","Type":"ContainerDied","Data":"b865bb14d5b0b15fe27fc2f89f0eb871c8d7d37fb326b2a8be00e20e1e73ae75"}
Sep 11 04:01:42.525331 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:01:42.525183    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-b4d3cf/auth-test-model-b4d3cf-kserve-7b594c5fff-dp5pp" event={"ID":"37955b55-9d5a-4d4e-8883-399b2160d5b8","Type":"ContainerDied","Data":"b0d3e9ffd66b2e6abac6c59784bc2faa6a3fce32f39c484e1776700bbf41341c"}
Sep 11 04:01:42.525331 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:01:42.525203    2579 scope.go:117] "RemoveContainer" containerID="b865bb14d5b0b15fe27fc2f89f0eb871c8d7d37fb326b2a8be00e20e1e73ae75"
Sep 11 04:01:42.535240 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:01:42.535222    2579 scope.go:117] "RemoveContainer" containerID="b865bb14d5b0b15fe27fc2f89f0eb871c8d7d37fb326b2a8be00e20e1e73ae75"
Sep 11 04:01:42.535479 ip-10-0-141-105 kubenswrapper[2579]: E0911 04:01:42.535462    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b865bb14d5b0b15fe27fc2f89f0eb871c8d7d37fb326b2a8be00e20e1e73ae75\": container with ID starting with b865bb14d5b0b15fe27fc2f89f0eb871c8d7d37fb326b2a8be00e20e1e73ae75 not found: ID does not exist" containerID="b865bb14d5b0b15fe27fc2f89f0eb871c8d7d37fb326b2a8be00e20e1e73ae75"
Sep 11 04:01:42.535535 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:01:42.535489    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b865bb14d5b0b15fe27fc2f89f0eb871c8d7d37fb326b2a8be00e20e1e73ae75"} err="failed to get container status \"b865bb14d5b0b15fe27fc2f89f0eb871c8d7d37fb326b2a8be00e20e1e73ae75\": rpc error: code = NotFound desc = could not find container \"b865bb14d5b0b15fe27fc2f89f0eb871c8d7d37fb326b2a8be00e20e1e73ae75\": container with ID starting with b865bb14d5b0b15fe27fc2f89f0eb871c8d7d37fb326b2a8be00e20e1e73ae75 not found: ID does not exist"
Sep 11 04:01:42.547759 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:01:42.547733    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-b4d3cf/auth-test-model-b4d3cf-kserve-7b594c5fff-dp5pp"]
Sep 11 04:01:42.551387 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:01:42.551366    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w3-b4d3cf/auth-test-model-b4d3cf-kserve-7b594c5fff-dp5pp"]
Sep 11 04:01:42.853393 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:01:42.853357    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="37955b55-9d5a-4d4e-8883-399b2160d5b8" path="/var/lib/kubelet/pods/37955b55-9d5a-4d4e-8883-399b2160d5b8/volumes"
Sep 11 04:01:44.418531 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:01:44.418484    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w3-b4d3cf/sub-test-model-b4d3cf-kserve-7f7bcdfb8c-tg8w9"
Sep 11 04:01:44.419033 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:01:44.418565    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w3-b4d3cf/sub-test-model-b4d3cf-kserve-7f7bcdfb8c-tg8w9"
Sep 11 04:01:44.431811 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:01:44.431784    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w3-b4d3cf/sub-test-model-b4d3cf-kserve-7f7bcdfb8c-tg8w9"
Sep 11 04:01:44.548696 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:01:44.548654    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w3-b4d3cf/sub-test-model-b4d3cf-kserve-7f7bcdfb8c-tg8w9"
Sep 11 04:01:52.414377 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:01:52.414340    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-5z4cn"]
Sep 11 04:02:12.890869 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:02:12.890838    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w3-dd5b30/sub-test-model-dd5b30-kserve-685d779ff9-64fzf"]
Sep 11 04:02:12.891509 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:02:12.891485    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="37955b55-9d5a-4d4e-8883-399b2160d5b8" containerName="main"
Sep 11 04:02:12.891626 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:02:12.891571    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="37955b55-9d5a-4d4e-8883-399b2160d5b8" containerName="main"
Sep 11 04:02:12.891715 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:02:12.891701    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="37955b55-9d5a-4d4e-8883-399b2160d5b8" containerName="main"
Sep 11 04:02:12.893936 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:02:12.893902    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-dd5b30/sub-test-model-dd5b30-kserve-685d779ff9-64fzf"
Sep 11 04:02:12.897596 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:02:12.897567    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-dd5b30\"/\"sub-test-model-dd5b30-kserve-self-signed-certs\""
Sep 11 04:02:12.897723 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:02:12.897577    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-dd5b30\"/\"openshift-service-ca.crt\""
Sep 11 04:02:12.897723 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:02:12.897602    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-dd5b30\"/\"kube-root-ca.crt\""
Sep 11 04:02:12.897833 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:02:12.897647    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-dd5b30\"/\"default-dockercfg-lvpcb\""
Sep 11 04:02:12.906119 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:02:12.906071    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-dd5b30/sub-test-model-dd5b30-kserve-685d779ff9-64fzf"]
Sep 11 04:02:13.036547 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:02:13.036517    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5560d37f-bb4b-4284-b06b-97a1187c39ed-model-cache\") pod \"sub-test-model-dd5b30-kserve-685d779ff9-64fzf\" (UID: \"5560d37f-bb4b-4284-b06b-97a1187c39ed\") " pod="ai-tenant-e2e-shared-b-w3-dd5b30/sub-test-model-dd5b30-kserve-685d779ff9-64fzf"
Sep 11 04:02:13.036547 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:02:13.036549    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gbw45\" (UniqueName: \"kubernetes.io/projected/5560d37f-bb4b-4284-b06b-97a1187c39ed-kube-api-access-gbw45\") pod \"sub-test-model-dd5b30-kserve-685d779ff9-64fzf\" (UID: \"5560d37f-bb4b-4284-b06b-97a1187c39ed\") " pod="ai-tenant-e2e-shared-b-w3-dd5b30/sub-test-model-dd5b30-kserve-685d779ff9-64fzf"
Sep 11 04:02:13.036741 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:02:13.036585    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5560d37f-bb4b-4284-b06b-97a1187c39ed-home\") pod \"sub-test-model-dd5b30-kserve-685d779ff9-64fzf\" (UID: \"5560d37f-bb4b-4284-b06b-97a1187c39ed\") " pod="ai-tenant-e2e-shared-b-w3-dd5b30/sub-test-model-dd5b30-kserve-685d779ff9-64fzf"
Sep 11 04:02:13.036741 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:02:13.036647    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5560d37f-bb4b-4284-b06b-97a1187c39ed-dshm\") pod \"sub-test-model-dd5b30-kserve-685d779ff9-64fzf\" (UID: \"5560d37f-bb4b-4284-b06b-97a1187c39ed\") " pod="ai-tenant-e2e-shared-b-w3-dd5b30/sub-test-model-dd5b30-kserve-685d779ff9-64fzf"
Sep 11 04:02:13.036741 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:02:13.036668    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5560d37f-bb4b-4284-b06b-97a1187c39ed-tls-certs\") pod \"sub-test-model-dd5b30-kserve-685d779ff9-64fzf\" (UID: \"5560d37f-bb4b-4284-b06b-97a1187c39ed\") " pod="ai-tenant-e2e-shared-b-w3-dd5b30/sub-test-model-dd5b30-kserve-685d779ff9-64fzf"
Sep 11 04:02:13.036741 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:02:13.036712    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5560d37f-bb4b-4284-b06b-97a1187c39ed-tmp-dir\") pod \"sub-test-model-dd5b30-kserve-685d779ff9-64fzf\" (UID: \"5560d37f-bb4b-4284-b06b-97a1187c39ed\") " pod="ai-tenant-e2e-shared-b-w3-dd5b30/sub-test-model-dd5b30-kserve-685d779ff9-64fzf"
Sep 11 04:02:13.137226 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:02:13.137193    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5560d37f-bb4b-4284-b06b-97a1187c39ed-tmp-dir\") pod \"sub-test-model-dd5b30-kserve-685d779ff9-64fzf\" (UID: \"5560d37f-bb4b-4284-b06b-97a1187c39ed\") " pod="ai-tenant-e2e-shared-b-w3-dd5b30/sub-test-model-dd5b30-kserve-685d779ff9-64fzf"
Sep 11 04:02:13.137226 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:02:13.137234    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5560d37f-bb4b-4284-b06b-97a1187c39ed-model-cache\") pod \"sub-test-model-dd5b30-kserve-685d779ff9-64fzf\" (UID: \"5560d37f-bb4b-4284-b06b-97a1187c39ed\") " pod="ai-tenant-e2e-shared-b-w3-dd5b30/sub-test-model-dd5b30-kserve-685d779ff9-64fzf"
Sep 11 04:02:13.137516 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:02:13.137258    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gbw45\" (UniqueName: \"kubernetes.io/projected/5560d37f-bb4b-4284-b06b-97a1187c39ed-kube-api-access-gbw45\") pod \"sub-test-model-dd5b30-kserve-685d779ff9-64fzf\" (UID: \"5560d37f-bb4b-4284-b06b-97a1187c39ed\") " pod="ai-tenant-e2e-shared-b-w3-dd5b30/sub-test-model-dd5b30-kserve-685d779ff9-64fzf"
Sep 11 04:02:13.137516 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:02:13.137318    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5560d37f-bb4b-4284-b06b-97a1187c39ed-home\") pod \"sub-test-model-dd5b30-kserve-685d779ff9-64fzf\" (UID: \"5560d37f-bb4b-4284-b06b-97a1187c39ed\") " pod="ai-tenant-e2e-shared-b-w3-dd5b30/sub-test-model-dd5b30-kserve-685d779ff9-64fzf"
Sep 11 04:02:13.137516 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:02:13.137375    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5560d37f-bb4b-4284-b06b-97a1187c39ed-dshm\") pod \"sub-test-model-dd5b30-kserve-685d779ff9-64fzf\" (UID: \"5560d37f-bb4b-4284-b06b-97a1187c39ed\") " pod="ai-tenant-e2e-shared-b-w3-dd5b30/sub-test-model-dd5b30-kserve-685d779ff9-64fzf"
Sep 11 04:02:13.137516 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:02:13.137405    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5560d37f-bb4b-4284-b06b-97a1187c39ed-tls-certs\") pod \"sub-test-model-dd5b30-kserve-685d779ff9-64fzf\" (UID: \"5560d37f-bb4b-4284-b06b-97a1187c39ed\") " pod="ai-tenant-e2e-shared-b-w3-dd5b30/sub-test-model-dd5b30-kserve-685d779ff9-64fzf"
Sep 11 04:02:13.137692 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:02:13.137655    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5560d37f-bb4b-4284-b06b-97a1187c39ed-tmp-dir\") pod \"sub-test-model-dd5b30-kserve-685d779ff9-64fzf\" (UID: \"5560d37f-bb4b-4284-b06b-97a1187c39ed\") " pod="ai-tenant-e2e-shared-b-w3-dd5b30/sub-test-model-dd5b30-kserve-685d779ff9-64fzf"
Sep 11 04:02:13.137750 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:02:13.137731    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5560d37f-bb4b-4284-b06b-97a1187c39ed-home\") pod \"sub-test-model-dd5b30-kserve-685d779ff9-64fzf\" (UID: \"5560d37f-bb4b-4284-b06b-97a1187c39ed\") " pod="ai-tenant-e2e-shared-b-w3-dd5b30/sub-test-model-dd5b30-kserve-685d779ff9-64fzf"
Sep 11 04:02:13.137926 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:02:13.137907    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5560d37f-bb4b-4284-b06b-97a1187c39ed-model-cache\") pod \"sub-test-model-dd5b30-kserve-685d779ff9-64fzf\" (UID: \"5560d37f-bb4b-4284-b06b-97a1187c39ed\") " pod="ai-tenant-e2e-shared-b-w3-dd5b30/sub-test-model-dd5b30-kserve-685d779ff9-64fzf"
Sep 11 04:02:13.140193 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:02:13.140174    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5560d37f-bb4b-4284-b06b-97a1187c39ed-dshm\") pod \"sub-test-model-dd5b30-kserve-685d779ff9-64fzf\" (UID: \"5560d37f-bb4b-4284-b06b-97a1187c39ed\") " pod="ai-tenant-e2e-shared-b-w3-dd5b30/sub-test-model-dd5b30-kserve-685d779ff9-64fzf"
Sep 11 04:02:13.140335 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:02:13.140315    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5560d37f-bb4b-4284-b06b-97a1187c39ed-tls-certs\") pod \"sub-test-model-dd5b30-kserve-685d779ff9-64fzf\" (UID: \"5560d37f-bb4b-4284-b06b-97a1187c39ed\") " pod="ai-tenant-e2e-shared-b-w3-dd5b30/sub-test-model-dd5b30-kserve-685d779ff9-64fzf"
Sep 11 04:02:13.145482 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:02:13.145427    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gbw45\" (UniqueName: \"kubernetes.io/projected/5560d37f-bb4b-4284-b06b-97a1187c39ed-kube-api-access-gbw45\") pod \"sub-test-model-dd5b30-kserve-685d779ff9-64fzf\" (UID: \"5560d37f-bb4b-4284-b06b-97a1187c39ed\") " pod="ai-tenant-e2e-shared-b-w3-dd5b30/sub-test-model-dd5b30-kserve-685d779ff9-64fzf"
Sep 11 04:02:13.204280 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:02:13.204249    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-dd5b30/sub-test-model-dd5b30-kserve-685d779ff9-64fzf"
Sep 11 04:02:13.474455 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:02:13.474427    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-dd5b30/sub-test-model-dd5b30-kserve-685d779ff9-64fzf"]
Sep 11 04:02:13.476499 ip-10-0-141-105 kubenswrapper[2579]: W0911 04:02:13.476462    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod5560d37f_bb4b_4284_b06b_97a1187c39ed.slice/crio-c144e9f57463a79fb3ffb7c2ee881419c7335114bc29168bad8cf743e22f5825 WatchSource:0}: Error finding container c144e9f57463a79fb3ffb7c2ee881419c7335114bc29168bad8cf743e22f5825: Status 404 returned error can't find the container with id c144e9f57463a79fb3ffb7c2ee881419c7335114bc29168bad8cf743e22f5825
Sep 11 04:02:13.696127 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:02:13.696075    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-dd5b30/sub-test-model-dd5b30-kserve-685d779ff9-64fzf" event={"ID":"5560d37f-bb4b-4284-b06b-97a1187c39ed","Type":"ContainerStarted","Data":"106464b8cae69fc80a470930a1554c5cef4df6880cda335904f2276393fb7e7f"}
Sep 11 04:02:13.696127 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:02:13.696134    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-dd5b30/sub-test-model-dd5b30-kserve-685d779ff9-64fzf" event={"ID":"5560d37f-bb4b-4284-b06b-97a1187c39ed","Type":"ContainerStarted","Data":"c144e9f57463a79fb3ffb7c2ee881419c7335114bc29168bad8cf743e22f5825"}
Sep 11 04:02:13.717464 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:02:13.717418    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w3-dd5b30/sub-test-model-dd5b30-kserve-685d779ff9-64fzf" podStartSLOduration=1.717402579 podStartE2EDuration="1.717402579s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 04:02:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 04:02:13.715770045 +0000 UTC m=+1311.370975612" watchObservedRunningTime="2026-09-11 04:02:13.717402579 +0000 UTC m=+1311.372608145"
Sep 11 04:02:17.328454 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:02:17.328407    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-5z4cn"]
Sep 11 04:02:23.049766 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:02:23.049732    2579 scope.go:117] "RemoveContainer" containerID="008205caba7da40850508c34cda38e9b5706e1fb6fd163092ff281a5dbc71034"
Sep 11 04:02:23.204874 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:02:23.204847    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w3-dd5b30/sub-test-model-dd5b30-kserve-685d779ff9-64fzf"
Sep 11 04:02:23.205036 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:02:23.204892    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w3-dd5b30/sub-test-model-dd5b30-kserve-685d779ff9-64fzf"
Sep 11 04:02:23.217630 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:02:23.217604    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w3-dd5b30/sub-test-model-dd5b30-kserve-685d779ff9-64fzf"
Sep 11 04:02:23.765397 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:02:23.765366    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w3-dd5b30/sub-test-model-dd5b30-kserve-685d779ff9-64fzf"
Sep 11 04:02:38.519410 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:02:38.519371    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-5z4cn"]
Sep 11 04:03:03.709535 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:03:03.709489    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-5z4cn"]
Sep 11 04:03:04.497211 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:03:04.497175    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-b61c46-5c9c66b79d-l9qzp"]
Sep 11 04:03:04.500823 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:03:04.500798    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-b61c46-5c9c66b79d-l9qzp"
Sep 11 04:03:04.503721 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:03:04.503690    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-b-b61c46-dockercfg-x4dgk\""
Sep 11 04:03:04.504239 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:03:04.504222    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-b-b61c46\""
Sep 11 04:03:04.517297 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:03:04.517260    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-b61c46-d5d4b49bf-dpb5m"]
Sep 11 04:03:04.520755 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:03:04.520739    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-b61c46-d5d4b49bf-dpb5m"
Sep 11 04:03:04.527554 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:03:04.527532    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-b61c46-5c9c66b79d-l9qzp"]
Sep 11 04:03:04.539840 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:03:04.539818    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-b61c46-d5d4b49bf-dpb5m"]
Sep 11 04:03:04.602237 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:03:04.602204    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6snrz\" (UniqueName: \"kubernetes.io/projected/63febe17-6300-476c-b49e-74e1098cca67-kube-api-access-6snrz\") pod \"payload-pre-processing-e2e-api-b-b61c46-5c9c66b79d-l9qzp\" (UID: \"63febe17-6300-476c-b49e-74e1098cca67\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-b61c46-5c9c66b79d-l9qzp"
Sep 11 04:03:04.602399 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:03:04.602265    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qczsn\" (UniqueName: \"kubernetes.io/projected/174bdd05-1f30-4bd1-819f-8d534357f38c-kube-api-access-qczsn\") pod \"payload-processing-e2e-api-b-b61c46-d5d4b49bf-dpb5m\" (UID: \"174bdd05-1f30-4bd1-819f-8d534357f38c\") " pod="openshift-ingress/payload-processing-e2e-api-b-b61c46-d5d4b49bf-dpb5m"
Sep 11 04:03:04.602399 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:03:04.602287    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/63febe17-6300-476c-b49e-74e1098cca67-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-b61c46-5c9c66b79d-l9qzp\" (UID: \"63febe17-6300-476c-b49e-74e1098cca67\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-b61c46-5c9c66b79d-l9qzp"
Sep 11 04:03:04.602399 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:03:04.602312    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/174bdd05-1f30-4bd1-819f-8d534357f38c-plugins-config-volume\") pod \"payload-processing-e2e-api-b-b61c46-d5d4b49bf-dpb5m\" (UID: \"174bdd05-1f30-4bd1-819f-8d534357f38c\") " pod="openshift-ingress/payload-processing-e2e-api-b-b61c46-d5d4b49bf-dpb5m"
Sep 11 04:03:04.702954 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:03:04.702915    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qczsn\" (UniqueName: \"kubernetes.io/projected/174bdd05-1f30-4bd1-819f-8d534357f38c-kube-api-access-qczsn\") pod \"payload-processing-e2e-api-b-b61c46-d5d4b49bf-dpb5m\" (UID: \"174bdd05-1f30-4bd1-819f-8d534357f38c\") " pod="openshift-ingress/payload-processing-e2e-api-b-b61c46-d5d4b49bf-dpb5m"
Sep 11 04:03:04.703171 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:03:04.702960    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/63febe17-6300-476c-b49e-74e1098cca67-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-b61c46-5c9c66b79d-l9qzp\" (UID: \"63febe17-6300-476c-b49e-74e1098cca67\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-b61c46-5c9c66b79d-l9qzp"
Sep 11 04:03:04.703171 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:03:04.703000    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/174bdd05-1f30-4bd1-819f-8d534357f38c-plugins-config-volume\") pod \"payload-processing-e2e-api-b-b61c46-d5d4b49bf-dpb5m\" (UID: \"174bdd05-1f30-4bd1-819f-8d534357f38c\") " pod="openshift-ingress/payload-processing-e2e-api-b-b61c46-d5d4b49bf-dpb5m"
Sep 11 04:03:04.703171 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:03:04.703121    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6snrz\" (UniqueName: \"kubernetes.io/projected/63febe17-6300-476c-b49e-74e1098cca67-kube-api-access-6snrz\") pod \"payload-pre-processing-e2e-api-b-b61c46-5c9c66b79d-l9qzp\" (UID: \"63febe17-6300-476c-b49e-74e1098cca67\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-b61c46-5c9c66b79d-l9qzp"
Sep 11 04:03:04.703782 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:03:04.703762    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/63febe17-6300-476c-b49e-74e1098cca67-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-b61c46-5c9c66b79d-l9qzp\" (UID: \"63febe17-6300-476c-b49e-74e1098cca67\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-b61c46-5c9c66b79d-l9qzp"
Sep 11 04:03:04.703947 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:03:04.703925    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/174bdd05-1f30-4bd1-819f-8d534357f38c-plugins-config-volume\") pod \"payload-processing-e2e-api-b-b61c46-d5d4b49bf-dpb5m\" (UID: \"174bdd05-1f30-4bd1-819f-8d534357f38c\") " pod="openshift-ingress/payload-processing-e2e-api-b-b61c46-d5d4b49bf-dpb5m"
Sep 11 04:03:04.713832 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:03:04.713809    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6snrz\" (UniqueName: \"kubernetes.io/projected/63febe17-6300-476c-b49e-74e1098cca67-kube-api-access-6snrz\") pod \"payload-pre-processing-e2e-api-b-b61c46-5c9c66b79d-l9qzp\" (UID: \"63febe17-6300-476c-b49e-74e1098cca67\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-b61c46-5c9c66b79d-l9qzp"
Sep 11 04:03:04.714177 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:03:04.713886    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qczsn\" (UniqueName: \"kubernetes.io/projected/174bdd05-1f30-4bd1-819f-8d534357f38c-kube-api-access-qczsn\") pod \"payload-processing-e2e-api-b-b61c46-d5d4b49bf-dpb5m\" (UID: \"174bdd05-1f30-4bd1-819f-8d534357f38c\") " pod="openshift-ingress/payload-processing-e2e-api-b-b61c46-d5d4b49bf-dpb5m"
Sep 11 04:03:04.812073 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:03:04.811976    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-b61c46-5c9c66b79d-l9qzp"
Sep 11 04:03:04.831618 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:03:04.831587    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-b61c46-d5d4b49bf-dpb5m"
Sep 11 04:03:05.094968 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:03:05.094940    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-b61c46-5c9c66b79d-l9qzp"]
Sep 11 04:03:05.096354 ip-10-0-141-105 kubenswrapper[2579]: W0911 04:03:05.096327    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod63febe17_6300_476c_b49e_74e1098cca67.slice/crio-bdaa6edcb8babdb53a71dad690a23cbf4bc560f9dd74469c855103e1dd1730ec WatchSource:0}: Error finding container bdaa6edcb8babdb53a71dad690a23cbf4bc560f9dd74469c855103e1dd1730ec: Status 404 returned error can't find the container with id bdaa6edcb8babdb53a71dad690a23cbf4bc560f9dd74469c855103e1dd1730ec
Sep 11 04:03:05.126522 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:03:05.126440    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-b61c46-d5d4b49bf-dpb5m"]
Sep 11 04:03:05.132247 ip-10-0-141-105 kubenswrapper[2579]: W0911 04:03:05.132224    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod174bdd05_1f30_4bd1_819f_8d534357f38c.slice/crio-2ea2e168de4b16e907436870667b8e067c553b64096693cd897290c2d587ee4f WatchSource:0}: Error finding container 2ea2e168de4b16e907436870667b8e067c553b64096693cd897290c2d587ee4f: Status 404 returned error can't find the container with id 2ea2e168de4b16e907436870667b8e067c553b64096693cd897290c2d587ee4f
Sep 11 04:03:05.979911 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:03:05.979874    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-b61c46-d5d4b49bf-dpb5m" event={"ID":"174bdd05-1f30-4bd1-819f-8d534357f38c","Type":"ContainerStarted","Data":"d919bcdaf2ab93b06ca7d4f6d1c9fe9d09732fba59f831c3c29706bf6b3c97bd"}
Sep 11 04:03:05.980424 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:03:05.979948    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-b-b61c46-d5d4b49bf-dpb5m"
Sep 11 04:03:05.980424 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:03:05.979975    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-b61c46-d5d4b49bf-dpb5m" event={"ID":"174bdd05-1f30-4bd1-819f-8d534357f38c","Type":"ContainerStarted","Data":"2ea2e168de4b16e907436870667b8e067c553b64096693cd897290c2d587ee4f"}
Sep 11 04:03:05.981705 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:03:05.981681    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-b61c46-5c9c66b79d-l9qzp" event={"ID":"63febe17-6300-476c-b49e-74e1098cca67","Type":"ContainerStarted","Data":"1495ad6e83a59676cf21f937d064afb20aab080af17611609aa5495012dee9e1"}
Sep 11 04:03:05.981830 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:03:05.981722    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-b61c46-5c9c66b79d-l9qzp" event={"ID":"63febe17-6300-476c-b49e-74e1098cca67","Type":"ContainerStarted","Data":"bdaa6edcb8babdb53a71dad690a23cbf4bc560f9dd74469c855103e1dd1730ec"}
Sep 11 04:03:05.981830 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:03:05.981765    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-b61c46-5c9c66b79d-l9qzp"
Sep 11 04:03:05.999166 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:03:05.999105    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-b-b61c46-d5d4b49bf-dpb5m" podStartSLOduration=1.999066158 podStartE2EDuration="1.999066158s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 04:03:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 04:03:05.997631419 +0000 UTC m=+1363.652836985" watchObservedRunningTime="2026-09-11 04:03:05.999066158 +0000 UTC m=+1363.654271728"
Sep 11 04:03:08.184285 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:03:08.184229    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-b-b61c46-5c9c66b79d-l9qzp" podStartSLOduration=4.184211636 podStartE2EDuration="4.184211636s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 04:03:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 04:03:06.014168945 +0000 UTC m=+1363.669374509" watchObservedRunningTime="2026-09-11 04:03:08.184211636 +0000 UTC m=+1365.839417224"
Sep 11 04:03:08.185861 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:03:08.185841    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7704-14a445"]
Sep 11 04:03:08.189990 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:03:08.189974    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7704-14a445"
Sep 11 04:03:08.193077 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:03:08.193049    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-fjqk5\""
Sep 11 04:03:08.195001 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:03:08.194974    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7704-14a445"]
Sep 11 04:03:08.339312 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:03:08.339276    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b6t9g\" (UniqueName: \"kubernetes.io/projected/1206e065-fe76-486c-bc16-13ce3c452fcf-kube-api-access-b6t9g\") pod \"test-curl-7704-14a445\" (UID: \"1206e065-fe76-486c-bc16-13ce3c452fcf\") " pod="opendatahub/test-curl-7704-14a445"
Sep 11 04:03:08.440315 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:03:08.440223    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b6t9g\" (UniqueName: \"kubernetes.io/projected/1206e065-fe76-486c-bc16-13ce3c452fcf-kube-api-access-b6t9g\") pod \"test-curl-7704-14a445\" (UID: \"1206e065-fe76-486c-bc16-13ce3c452fcf\") " pod="opendatahub/test-curl-7704-14a445"
Sep 11 04:03:08.450502 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:03:08.450471    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-b6t9g\" (UniqueName: \"kubernetes.io/projected/1206e065-fe76-486c-bc16-13ce3c452fcf-kube-api-access-b6t9g\") pod \"test-curl-7704-14a445\" (UID: \"1206e065-fe76-486c-bc16-13ce3c452fcf\") " pod="opendatahub/test-curl-7704-14a445"
Sep 11 04:03:08.501845 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:03:08.501818    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7704-14a445"
Sep 11 04:03:08.777912 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:03:08.777883    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7704-14a445"]
Sep 11 04:03:08.780558 ip-10-0-141-105 kubenswrapper[2579]: W0911 04:03:08.780529    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod1206e065_fe76_486c_bc16_13ce3c452fcf.slice/crio-37d440e2f678f72ff0880c9d0d6a2dcf8187503d04deb0e2c92adf0f0684d2bd WatchSource:0}: Error finding container 37d440e2f678f72ff0880c9d0d6a2dcf8187503d04deb0e2c92adf0f0684d2bd: Status 404 returned error can't find the container with id 37d440e2f678f72ff0880c9d0d6a2dcf8187503d04deb0e2c92adf0f0684d2bd
Sep 11 04:03:08.998351 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:03:08.998312    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7704-14a445" event={"ID":"1206e065-fe76-486c-bc16-13ce3c452fcf","Type":"ContainerStarted","Data":"a98a24f84a2b3028aa044c05cbeb1555d7844c892592194996bdb457f96649b8"}
Sep 11 04:03:08.998530 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:03:08.998359    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7704-14a445" event={"ID":"1206e065-fe76-486c-bc16-13ce3c452fcf","Type":"ContainerStarted","Data":"37d440e2f678f72ff0880c9d0d6a2dcf8187503d04deb0e2c92adf0f0684d2bd"}
Sep 11 04:03:09.019234 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:03:09.019185    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-7704-14a445" podStartSLOduration=1.019170927 podStartE2EDuration="1.019170927s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 04:03:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 04:03:09.017554287 +0000 UTC m=+1366.672759868" watchObservedRunningTime="2026-09-11 04:03:09.019170927 +0000 UTC m=+1366.674376513"
Sep 11 04:03:09.620818 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:03:09.620786    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7704-14a445"]
Sep 11 04:03:09.624811 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:03:09.624783    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7704-14a445"]
Sep 11 04:03:10.003827 ip-10-0-141-105 kubenswrapper[2579]: E0911 04:03:10.003749    2579 kuberuntime_manager.go:649] "Missing actuated resource record" pod="opendatahub/test-curl-7704-14a445" container="test-curl-7704-14a445"
Sep 11 04:03:11.009214 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:03:11.009176    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-7704-14a445" podUID="1206e065-fe76-486c-bc16-13ce3c452fcf" containerName="test-curl-7704-14a445" containerID="cri-o://a98a24f84a2b3028aa044c05cbeb1555d7844c892592194996bdb457f96649b8" gracePeriod=2
Sep 11 04:03:11.709769 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:03:11.709737    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-b4d3cf/sub-test-model-b4d3cf-kserve-7f7bcdfb8c-tg8w9"]
Sep 11 04:03:12.688647 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:03:12.688603    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-dd5b30/sub-test-model-dd5b30-kserve-685d779ff9-64fzf"]
Sep 11 04:03:13.188628 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:03:13.188599    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7704-14a445"
Sep 11 04:03:13.286031 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:03:13.285937    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-b6t9g\" (UniqueName: \"kubernetes.io/projected/1206e065-fe76-486c-bc16-13ce3c452fcf-kube-api-access-b6t9g\") pod \"1206e065-fe76-486c-bc16-13ce3c452fcf\" (UID: \"1206e065-fe76-486c-bc16-13ce3c452fcf\") "
Sep 11 04:03:13.288243 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:03:13.288207    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1206e065-fe76-486c-bc16-13ce3c452fcf-kube-api-access-b6t9g" (OuterVolumeSpecName: "kube-api-access-b6t9g") pod "1206e065-fe76-486c-bc16-13ce3c452fcf" (UID: "1206e065-fe76-486c-bc16-13ce3c452fcf"). InnerVolumeSpecName "kube-api-access-b6t9g". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 04:03:13.386696 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:03:13.386666    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b6t9g\" (UniqueName: \"kubernetes.io/projected/1206e065-fe76-486c-bc16-13ce3c452fcf-kube-api-access-b6t9g\") on node \"ip-10-0-141-105.ec2.internal\" DevicePath \"\""
Sep 11 04:03:14.028208 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:03:14.028172    2579 generic.go:356] "Generic (PLEG): container finished" podID="1206e065-fe76-486c-bc16-13ce3c452fcf" containerID="a98a24f84a2b3028aa044c05cbeb1555d7844c892592194996bdb457f96649b8" exitCode=137
Sep 11 04:03:14.028636 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:03:14.028218    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7704-14a445"
Sep 11 04:03:14.028636 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:03:14.028278    2579 scope.go:117] "RemoveContainer" containerID="a98a24f84a2b3028aa044c05cbeb1555d7844c892592194996bdb457f96649b8"
Sep 11 04:03:14.037613 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:03:14.037594    2579 scope.go:117] "RemoveContainer" containerID="a98a24f84a2b3028aa044c05cbeb1555d7844c892592194996bdb457f96649b8"
Sep 11 04:03:14.037908 ip-10-0-141-105 kubenswrapper[2579]: E0911 04:03:14.037888    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a98a24f84a2b3028aa044c05cbeb1555d7844c892592194996bdb457f96649b8\": container with ID starting with a98a24f84a2b3028aa044c05cbeb1555d7844c892592194996bdb457f96649b8 not found: ID does not exist" containerID="a98a24f84a2b3028aa044c05cbeb1555d7844c892592194996bdb457f96649b8"
Sep 11 04:03:14.037970 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:03:14.037923    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a98a24f84a2b3028aa044c05cbeb1555d7844c892592194996bdb457f96649b8"} err="failed to get container status \"a98a24f84a2b3028aa044c05cbeb1555d7844c892592194996bdb457f96649b8\": rpc error: code = NotFound desc = could not find container \"a98a24f84a2b3028aa044c05cbeb1555d7844c892592194996bdb457f96649b8\": container with ID starting with a98a24f84a2b3028aa044c05cbeb1555d7844c892592194996bdb457f96649b8 not found: ID does not exist"
Sep 11 04:03:14.038811 ip-10-0-141-105 kubenswrapper[2579]: E0911 04:03:14.038790    2579 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7704-14a445\" is forbidden: User \"system:node:ip-10-0-141-105.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-141-105.ec2.internal' and this object" podUID="1206e065-fe76-486c-bc16-13ce3c452fcf" pod="opendatahub/test-curl-7704-14a445"
Sep 11 04:03:14.852457 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:03:14.852425    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1206e065-fe76-486c-bc16-13ce3c452fcf" path="/var/lib/kubelet/pods/1206e065-fe76-486c-bc16-13ce3c452fcf/volumes"
Sep 11 04:03:16.988597 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:03:16.988563    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-b-b61c46-d5d4b49bf-dpb5m"
Sep 11 04:03:16.989045 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:03:16.988760    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-b61c46-5c9c66b79d-l9qzp"
Sep 11 04:03:23.074237 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:03:23.074204    2579 scope.go:117] "RemoveContainer" containerID="1fa514b2f3408a13c8a5510758d14341d95a03f16b023a0cb282bed1d82c4cf5"
Sep 11 04:03:23.083959 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:03:23.083930    2579 scope.go:117] "RemoveContainer" containerID="432b0e24a23b2f81c75b86a0cf2d1df72a1d2b85e6d275d5a380fbea832c5ea9"
Sep 11 04:03:23.091697 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:03:23.091670    2579 scope.go:117] "RemoveContainer" containerID="277a1759d01df140ffe425c366919c8d1c59e3fd3229482a3ae71d9885b734e3"
Sep 11 04:03:26.008761 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:03:26.008726    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-5z4cn"]
Sep 11 04:03:26.729299 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:03:26.729264    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w3-b4d3cf/sub-test-model-b4d3cf-kserve-7f7bcdfb8c-tg8w9" podUID="3c9f0173-a36d-4fe4-b1ad-756a3d48be13" containerName="main" containerID="cri-o://2b9916761926c4c441eafb0864f67741217cb9a80aa82f3be4a0e398e543a9ca" gracePeriod=45
Sep 11 04:03:26.984903 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:03:26.984838    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-b4d3cf/sub-test-model-b4d3cf-kserve-7f7bcdfb8c-tg8w9"
Sep 11 04:03:27.017561 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:03:27.017535    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3c9f0173-a36d-4fe4-b1ad-756a3d48be13-tls-certs\") pod \"3c9f0173-a36d-4fe4-b1ad-756a3d48be13\" (UID: \"3c9f0173-a36d-4fe4-b1ad-756a3d48be13\") "
Sep 11 04:03:27.017985 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:03:27.017592    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-sfl2f\" (UniqueName: \"kubernetes.io/projected/3c9f0173-a36d-4fe4-b1ad-756a3d48be13-kube-api-access-sfl2f\") pod \"3c9f0173-a36d-4fe4-b1ad-756a3d48be13\" (UID: \"3c9f0173-a36d-4fe4-b1ad-756a3d48be13\") "
Sep 11 04:03:27.017985 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:03:27.017628    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3c9f0173-a36d-4fe4-b1ad-756a3d48be13-home\") pod \"3c9f0173-a36d-4fe4-b1ad-756a3d48be13\" (UID: \"3c9f0173-a36d-4fe4-b1ad-756a3d48be13\") "
Sep 11 04:03:27.017985 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:03:27.017672    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3c9f0173-a36d-4fe4-b1ad-756a3d48be13-dshm\") pod \"3c9f0173-a36d-4fe4-b1ad-756a3d48be13\" (UID: \"3c9f0173-a36d-4fe4-b1ad-756a3d48be13\") "
Sep 11 04:03:27.017985 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:03:27.017693    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3c9f0173-a36d-4fe4-b1ad-756a3d48be13-model-cache\") pod \"3c9f0173-a36d-4fe4-b1ad-756a3d48be13\" (UID: \"3c9f0173-a36d-4fe4-b1ad-756a3d48be13\") "
Sep 11 04:03:27.017985 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:03:27.017735    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3c9f0173-a36d-4fe4-b1ad-756a3d48be13-tmp-dir\") pod \"3c9f0173-a36d-4fe4-b1ad-756a3d48be13\" (UID: \"3c9f0173-a36d-4fe4-b1ad-756a3d48be13\") "
Sep 11 04:03:27.017985 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:03:27.017936    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3c9f0173-a36d-4fe4-b1ad-756a3d48be13-home" (OuterVolumeSpecName: "home") pod "3c9f0173-a36d-4fe4-b1ad-756a3d48be13" (UID: "3c9f0173-a36d-4fe4-b1ad-756a3d48be13"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 04:03:27.018277 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:03:27.017996    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3c9f0173-a36d-4fe4-b1ad-756a3d48be13-model-cache" (OuterVolumeSpecName: "model-cache") pod "3c9f0173-a36d-4fe4-b1ad-756a3d48be13" (UID: "3c9f0173-a36d-4fe4-b1ad-756a3d48be13"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 04:03:27.018277 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:03:27.018173    2579 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3c9f0173-a36d-4fe4-b1ad-756a3d48be13-home\") on node \"ip-10-0-141-105.ec2.internal\" DevicePath \"\""
Sep 11 04:03:27.018277 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:03:27.018195    2579 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3c9f0173-a36d-4fe4-b1ad-756a3d48be13-model-cache\") on node \"ip-10-0-141-105.ec2.internal\" DevicePath \"\""
Sep 11 04:03:27.018277 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:03:27.018171    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3c9f0173-a36d-4fe4-b1ad-756a3d48be13-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "3c9f0173-a36d-4fe4-b1ad-756a3d48be13" (UID: "3c9f0173-a36d-4fe4-b1ad-756a3d48be13"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 04:03:27.019850 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:03:27.019826    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3c9f0173-a36d-4fe4-b1ad-756a3d48be13-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "3c9f0173-a36d-4fe4-b1ad-756a3d48be13" (UID: "3c9f0173-a36d-4fe4-b1ad-756a3d48be13"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 04:03:27.020027 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:03:27.019996    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3c9f0173-a36d-4fe4-b1ad-756a3d48be13-dshm" (OuterVolumeSpecName: "dshm") pod "3c9f0173-a36d-4fe4-b1ad-756a3d48be13" (UID: "3c9f0173-a36d-4fe4-b1ad-756a3d48be13"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 04:03:27.020269 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:03:27.020244    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3c9f0173-a36d-4fe4-b1ad-756a3d48be13-kube-api-access-sfl2f" (OuterVolumeSpecName: "kube-api-access-sfl2f") pod "3c9f0173-a36d-4fe4-b1ad-756a3d48be13" (UID: "3c9f0173-a36d-4fe4-b1ad-756a3d48be13"). InnerVolumeSpecName "kube-api-access-sfl2f". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 04:03:27.104448 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:03:27.104409    2579 generic.go:356] "Generic (PLEG): container finished" podID="3c9f0173-a36d-4fe4-b1ad-756a3d48be13" containerID="2b9916761926c4c441eafb0864f67741217cb9a80aa82f3be4a0e398e543a9ca" exitCode=0
Sep 11 04:03:27.104629 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:03:27.104468    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-b4d3cf/sub-test-model-b4d3cf-kserve-7f7bcdfb8c-tg8w9"
Sep 11 04:03:27.104629 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:03:27.104472    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-b4d3cf/sub-test-model-b4d3cf-kserve-7f7bcdfb8c-tg8w9" event={"ID":"3c9f0173-a36d-4fe4-b1ad-756a3d48be13","Type":"ContainerDied","Data":"2b9916761926c4c441eafb0864f67741217cb9a80aa82f3be4a0e398e543a9ca"}
Sep 11 04:03:27.104629 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:03:27.104517    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-b4d3cf/sub-test-model-b4d3cf-kserve-7f7bcdfb8c-tg8w9" event={"ID":"3c9f0173-a36d-4fe4-b1ad-756a3d48be13","Type":"ContainerDied","Data":"ddee58cb366a1d33dce3a6a4b6aee4e5ea3877d8e00ef5126f9124a6db082b5a"}
Sep 11 04:03:27.104629 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:03:27.104541    2579 scope.go:117] "RemoveContainer" containerID="2b9916761926c4c441eafb0864f67741217cb9a80aa82f3be4a0e398e543a9ca"
Sep 11 04:03:27.114523 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:03:27.114504    2579 scope.go:117] "RemoveContainer" containerID="2b9916761926c4c441eafb0864f67741217cb9a80aa82f3be4a0e398e543a9ca"
Sep 11 04:03:27.114809 ip-10-0-141-105 kubenswrapper[2579]: E0911 04:03:27.114790    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2b9916761926c4c441eafb0864f67741217cb9a80aa82f3be4a0e398e543a9ca\": container with ID starting with 2b9916761926c4c441eafb0864f67741217cb9a80aa82f3be4a0e398e543a9ca not found: ID does not exist" containerID="2b9916761926c4c441eafb0864f67741217cb9a80aa82f3be4a0e398e543a9ca"
Sep 11 04:03:27.114862 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:03:27.114819    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2b9916761926c4c441eafb0864f67741217cb9a80aa82f3be4a0e398e543a9ca"} err="failed to get container status \"2b9916761926c4c441eafb0864f67741217cb9a80aa82f3be4a0e398e543a9ca\": rpc error: code = NotFound desc = could not find container \"2b9916761926c4c441eafb0864f67741217cb9a80aa82f3be4a0e398e543a9ca\": container with ID starting with 2b9916761926c4c441eafb0864f67741217cb9a80aa82f3be4a0e398e543a9ca not found: ID does not exist"
Sep 11 04:03:27.118877 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:03:27.118855    2579 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3c9f0173-a36d-4fe4-b1ad-756a3d48be13-tls-certs\") on node \"ip-10-0-141-105.ec2.internal\" DevicePath \"\""
Sep 11 04:03:27.118877 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:03:27.118874    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sfl2f\" (UniqueName: \"kubernetes.io/projected/3c9f0173-a36d-4fe4-b1ad-756a3d48be13-kube-api-access-sfl2f\") on node \"ip-10-0-141-105.ec2.internal\" DevicePath \"\""
Sep 11 04:03:27.119008 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:03:27.118884    2579 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3c9f0173-a36d-4fe4-b1ad-756a3d48be13-dshm\") on node \"ip-10-0-141-105.ec2.internal\" DevicePath \"\""
Sep 11 04:03:27.119008 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:03:27.118893    2579 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3c9f0173-a36d-4fe4-b1ad-756a3d48be13-tmp-dir\") on node \"ip-10-0-141-105.ec2.internal\" DevicePath \"\""
Sep 11 04:03:27.130068 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:03:27.130045    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-b4d3cf/sub-test-model-b4d3cf-kserve-7f7bcdfb8c-tg8w9"]
Sep 11 04:03:27.133159 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:03:27.133138    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w3-b4d3cf/sub-test-model-b4d3cf-kserve-7f7bcdfb8c-tg8w9"]
Sep 11 04:03:27.709174 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:03:27.709132    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w3-dd5b30/sub-test-model-dd5b30-kserve-685d779ff9-64fzf" podUID="5560d37f-bb4b-4284-b06b-97a1187c39ed" containerName="main" containerID="cri-o://106464b8cae69fc80a470930a1554c5cef4df6880cda335904f2276393fb7e7f" gracePeriod=45
Sep 11 04:03:27.957269 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:03:27.957247    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-dd5b30/sub-test-model-dd5b30-kserve-685d779ff9-64fzf"
Sep 11 04:03:28.025843 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:03:28.025754    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gbw45\" (UniqueName: \"kubernetes.io/projected/5560d37f-bb4b-4284-b06b-97a1187c39ed-kube-api-access-gbw45\") pod \"5560d37f-bb4b-4284-b06b-97a1187c39ed\" (UID: \"5560d37f-bb4b-4284-b06b-97a1187c39ed\") "
Sep 11 04:03:28.026341 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:03:28.025846    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5560d37f-bb4b-4284-b06b-97a1187c39ed-home\") pod \"5560d37f-bb4b-4284-b06b-97a1187c39ed\" (UID: \"5560d37f-bb4b-4284-b06b-97a1187c39ed\") "
Sep 11 04:03:28.026341 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:03:28.025949    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5560d37f-bb4b-4284-b06b-97a1187c39ed-dshm\") pod \"5560d37f-bb4b-4284-b06b-97a1187c39ed\" (UID: \"5560d37f-bb4b-4284-b06b-97a1187c39ed\") "
Sep 11 04:03:28.026341 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:03:28.025983    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5560d37f-bb4b-4284-b06b-97a1187c39ed-model-cache\") pod \"5560d37f-bb4b-4284-b06b-97a1187c39ed\" (UID: \"5560d37f-bb4b-4284-b06b-97a1187c39ed\") "
Sep 11 04:03:28.026341 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:03:28.025992    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5560d37f-bb4b-4284-b06b-97a1187c39ed-home" (OuterVolumeSpecName: "home") pod "5560d37f-bb4b-4284-b06b-97a1187c39ed" (UID: "5560d37f-bb4b-4284-b06b-97a1187c39ed"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 04:03:28.026341 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:03:28.026025    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5560d37f-bb4b-4284-b06b-97a1187c39ed-tmp-dir\") pod \"5560d37f-bb4b-4284-b06b-97a1187c39ed\" (UID: \"5560d37f-bb4b-4284-b06b-97a1187c39ed\") "
Sep 11 04:03:28.026341 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:03:28.026057    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5560d37f-bb4b-4284-b06b-97a1187c39ed-tls-certs\") pod \"5560d37f-bb4b-4284-b06b-97a1187c39ed\" (UID: \"5560d37f-bb4b-4284-b06b-97a1187c39ed\") "
Sep 11 04:03:28.026341 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:03:28.026319    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5560d37f-bb4b-4284-b06b-97a1187c39ed-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "5560d37f-bb4b-4284-b06b-97a1187c39ed" (UID: "5560d37f-bb4b-4284-b06b-97a1187c39ed"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 04:03:28.026341 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:03:28.026333    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5560d37f-bb4b-4284-b06b-97a1187c39ed-model-cache" (OuterVolumeSpecName: "model-cache") pod "5560d37f-bb4b-4284-b06b-97a1187c39ed" (UID: "5560d37f-bb4b-4284-b06b-97a1187c39ed"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 04:03:28.026710 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:03:28.026422    2579 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5560d37f-bb4b-4284-b06b-97a1187c39ed-tmp-dir\") on node \"ip-10-0-141-105.ec2.internal\" DevicePath \"\""
Sep 11 04:03:28.026710 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:03:28.026442    2579 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5560d37f-bb4b-4284-b06b-97a1187c39ed-home\") on node \"ip-10-0-141-105.ec2.internal\" DevicePath \"\""
Sep 11 04:03:28.026710 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:03:28.026454    2579 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5560d37f-bb4b-4284-b06b-97a1187c39ed-model-cache\") on node \"ip-10-0-141-105.ec2.internal\" DevicePath \"\""
Sep 11 04:03:28.028231 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:03:28.028200    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5560d37f-bb4b-4284-b06b-97a1187c39ed-dshm" (OuterVolumeSpecName: "dshm") pod "5560d37f-bb4b-4284-b06b-97a1187c39ed" (UID: "5560d37f-bb4b-4284-b06b-97a1187c39ed"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 04:03:28.028355 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:03:28.028248    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5560d37f-bb4b-4284-b06b-97a1187c39ed-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "5560d37f-bb4b-4284-b06b-97a1187c39ed" (UID: "5560d37f-bb4b-4284-b06b-97a1187c39ed"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 04:03:28.028355 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:03:28.028293    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5560d37f-bb4b-4284-b06b-97a1187c39ed-kube-api-access-gbw45" (OuterVolumeSpecName: "kube-api-access-gbw45") pod "5560d37f-bb4b-4284-b06b-97a1187c39ed" (UID: "5560d37f-bb4b-4284-b06b-97a1187c39ed"). InnerVolumeSpecName "kube-api-access-gbw45". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 04:03:28.110759 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:03:28.110728    2579 generic.go:356] "Generic (PLEG): container finished" podID="5560d37f-bb4b-4284-b06b-97a1187c39ed" containerID="106464b8cae69fc80a470930a1554c5cef4df6880cda335904f2276393fb7e7f" exitCode=0
Sep 11 04:03:28.110911 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:03:28.110771    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-dd5b30/sub-test-model-dd5b30-kserve-685d779ff9-64fzf"
Sep 11 04:03:28.110911 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:03:28.110808    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-dd5b30/sub-test-model-dd5b30-kserve-685d779ff9-64fzf" event={"ID":"5560d37f-bb4b-4284-b06b-97a1187c39ed","Type":"ContainerDied","Data":"106464b8cae69fc80a470930a1554c5cef4df6880cda335904f2276393fb7e7f"}
Sep 11 04:03:28.110911 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:03:28.110850    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-dd5b30/sub-test-model-dd5b30-kserve-685d779ff9-64fzf" event={"ID":"5560d37f-bb4b-4284-b06b-97a1187c39ed","Type":"ContainerDied","Data":"c144e9f57463a79fb3ffb7c2ee881419c7335114bc29168bad8cf743e22f5825"}
Sep 11 04:03:28.110911 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:03:28.110867    2579 scope.go:117] "RemoveContainer" containerID="106464b8cae69fc80a470930a1554c5cef4df6880cda335904f2276393fb7e7f"
Sep 11 04:03:28.120468 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:03:28.120445    2579 scope.go:117] "RemoveContainer" containerID="106464b8cae69fc80a470930a1554c5cef4df6880cda335904f2276393fb7e7f"
Sep 11 04:03:28.120781 ip-10-0-141-105 kubenswrapper[2579]: E0911 04:03:28.120762    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"106464b8cae69fc80a470930a1554c5cef4df6880cda335904f2276393fb7e7f\": container with ID starting with 106464b8cae69fc80a470930a1554c5cef4df6880cda335904f2276393fb7e7f not found: ID does not exist" containerID="106464b8cae69fc80a470930a1554c5cef4df6880cda335904f2276393fb7e7f"
Sep 11 04:03:28.120826 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:03:28.120789    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"106464b8cae69fc80a470930a1554c5cef4df6880cda335904f2276393fb7e7f"} err="failed to get container status \"106464b8cae69fc80a470930a1554c5cef4df6880cda335904f2276393fb7e7f\": rpc error: code = NotFound desc = could not find container \"106464b8cae69fc80a470930a1554c5cef4df6880cda335904f2276393fb7e7f\": container with ID starting with 106464b8cae69fc80a470930a1554c5cef4df6880cda335904f2276393fb7e7f not found: ID does not exist"
Sep 11 04:03:28.127207 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:03:28.127189    2579 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5560d37f-bb4b-4284-b06b-97a1187c39ed-dshm\") on node \"ip-10-0-141-105.ec2.internal\" DevicePath \"\""
Sep 11 04:03:28.127293 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:03:28.127208    2579 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5560d37f-bb4b-4284-b06b-97a1187c39ed-tls-certs\") on node \"ip-10-0-141-105.ec2.internal\" DevicePath \"\""
Sep 11 04:03:28.127293 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:03:28.127218    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gbw45\" (UniqueName: \"kubernetes.io/projected/5560d37f-bb4b-4284-b06b-97a1187c39ed-kube-api-access-gbw45\") on node \"ip-10-0-141-105.ec2.internal\" DevicePath \"\""
Sep 11 04:03:28.132314 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:03:28.132288    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-dd5b30/sub-test-model-dd5b30-kserve-685d779ff9-64fzf"]
Sep 11 04:03:28.137976 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:03:28.137953    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w3-dd5b30/sub-test-model-dd5b30-kserve-685d779ff9-64fzf"]
Sep 11 04:03:28.852035 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:03:28.851997    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3c9f0173-a36d-4fe4-b1ad-756a3d48be13" path="/var/lib/kubelet/pods/3c9f0173-a36d-4fe4-b1ad-756a3d48be13/volumes"
Sep 11 04:03:28.852534 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:03:28.852511    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5560d37f-bb4b-4284-b06b-97a1187c39ed" path="/var/lib/kubelet/pods/5560d37f-bb4b-4284-b06b-97a1187c39ed/volumes"
Sep 11 04:03:30.369767 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:03:30.369735    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-49518b-hzx9b"]
Sep 11 04:03:30.370345 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:03:30.370325    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5560d37f-bb4b-4284-b06b-97a1187c39ed" containerName="main"
Sep 11 04:03:30.370444 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:03:30.370348    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="5560d37f-bb4b-4284-b06b-97a1187c39ed" containerName="main"
Sep 11 04:03:30.370444 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:03:30.370358    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1206e065-fe76-486c-bc16-13ce3c452fcf" containerName="test-curl-7704-14a445"
Sep 11 04:03:30.370444 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:03:30.370366    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="1206e065-fe76-486c-bc16-13ce3c452fcf" containerName="test-curl-7704-14a445"
Sep 11 04:03:30.370444 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:03:30.370384    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3c9f0173-a36d-4fe4-b1ad-756a3d48be13" containerName="main"
Sep 11 04:03:30.370444 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:03:30.370391    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="3c9f0173-a36d-4fe4-b1ad-756a3d48be13" containerName="main"
Sep 11 04:03:30.370700 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:03:30.370512    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="5560d37f-bb4b-4284-b06b-97a1187c39ed" containerName="main"
Sep 11 04:03:30.370700 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:03:30.370527    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="3c9f0173-a36d-4fe4-b1ad-756a3d48be13" containerName="main"
Sep 11 04:03:30.370700 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:03:30.370536    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="1206e065-fe76-486c-bc16-13ce3c452fcf" containerName="test-curl-7704-14a445"
Sep 11 04:03:30.374911 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:03:30.374892    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-49518b-hzx9b"
Sep 11 04:03:30.378425 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:03:30.378359    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-95ckr\""
Sep 11 04:03:30.378425 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:03:30.378408    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 11 04:03:30.380346 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:03:30.380318    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-49518b-hzx9b"]
Sep 11 04:03:30.447691 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:03:30.447657    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4719291a-c30a-49f5-8509-8ff9e3f90efb-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-a-49518b-hzx9b\" (UID: \"4719291a-c30a-49f5-8509-8ff9e3f90efb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-49518b-hzx9b"
Sep 11 04:03:30.549046 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:03:30.549016    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4719291a-c30a-49f5-8509-8ff9e3f90efb-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-a-49518b-hzx9b\" (UID: \"4719291a-c30a-49f5-8509-8ff9e3f90efb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-49518b-hzx9b"
Sep 11 04:03:30.549604 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:03:30.549585    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4719291a-c30a-49f5-8509-8ff9e3f90efb-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-a-49518b-hzx9b\" (UID: \"4719291a-c30a-49f5-8509-8ff9e3f90efb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-49518b-hzx9b"
Sep 11 04:03:30.688348 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:03:30.688271    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-49518b-hzx9b"
Sep 11 04:03:30.957957 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:03:30.957932    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-49518b-hzx9b"]
Sep 11 04:03:30.959880 ip-10-0-141-105 kubenswrapper[2579]: W0911 04:03:30.959855    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4719291a_c30a_49f5_8509_8ff9e3f90efb.slice/crio-2e38dd172e5bf6d00c9bac7a9e276b15dd6308a35bd02dd1d281359d82f172e1 WatchSource:0}: Error finding container 2e38dd172e5bf6d00c9bac7a9e276b15dd6308a35bd02dd1d281359d82f172e1: Status 404 returned error can't find the container with id 2e38dd172e5bf6d00c9bac7a9e276b15dd6308a35bd02dd1d281359d82f172e1
Sep 11 04:03:31.130163 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:03:31.130119    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-49518b-hzx9b" event={"ID":"4719291a-c30a-49f5-8509-8ff9e3f90efb","Type":"ContainerStarted","Data":"2e38dd172e5bf6d00c9bac7a9e276b15dd6308a35bd02dd1d281359d82f172e1"}
Sep 11 04:03:32.140140 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:03:32.140111    2579 generic.go:356] "Generic (PLEG): container finished" podID="4719291a-c30a-49f5-8509-8ff9e3f90efb" containerID="72450b69e797fe358a60ad26f7153802f67996310ab8b8ac2842de55dc040bb1" exitCode=0
Sep 11 04:03:32.140521 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:03:32.140186    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-49518b-hzx9b" event={"ID":"4719291a-c30a-49f5-8509-8ff9e3f90efb","Type":"ContainerDied","Data":"72450b69e797fe358a60ad26f7153802f67996310ab8b8ac2842de55dc040bb1"}
Sep 11 04:03:33.281789 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:03:33.281759    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-49518b-hzx9b"
Sep 11 04:03:33.375499 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:03:33.375466    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4719291a-c30a-49f5-8509-8ff9e3f90efb-maas-api-service-ca\") pod \"4719291a-c30a-49f5-8509-8ff9e3f90efb\" (UID: \"4719291a-c30a-49f5-8509-8ff9e3f90efb\") "
Sep 11 04:03:33.375838 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:03:33.375815    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4719291a-c30a-49f5-8509-8ff9e3f90efb-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "4719291a-c30a-49f5-8509-8ff9e3f90efb" (UID: "4719291a-c30a-49f5-8509-8ff9e3f90efb"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 04:03:33.477144 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:03:33.477034    2579 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4719291a-c30a-49f5-8509-8ff9e3f90efb-maas-api-service-ca\") on node \"ip-10-0-141-105.ec2.internal\" DevicePath \"\""
Sep 11 04:03:34.151759 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:03:34.151728    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-49518b-hzx9b"
Sep 11 04:03:34.152062 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:03:34.151766    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-49518b-hzx9b" event={"ID":"4719291a-c30a-49f5-8509-8ff9e3f90efb","Type":"ContainerDied","Data":"2e38dd172e5bf6d00c9bac7a9e276b15dd6308a35bd02dd1d281359d82f172e1"}
Sep 11 04:03:34.152062 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:03:34.151814    2579 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2e38dd172e5bf6d00c9bac7a9e276b15dd6308a35bd02dd1d281359d82f172e1"
Sep 11 04:03:34.798401 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:03:34.798365    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w3-b4d3cf/rate-test-model-b4d3cf-kserve-6f654d7c9f-hxsxq"]
Sep 11 04:03:34.798925 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:03:34.798905    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4719291a-c30a-49f5-8509-8ff9e3f90efb" containerName="revoke-keys"
Sep 11 04:03:34.798925 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:03:34.798926    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="4719291a-c30a-49f5-8509-8ff9e3f90efb" containerName="revoke-keys"
Sep 11 04:03:34.799098 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:03:34.798991    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="4719291a-c30a-49f5-8509-8ff9e3f90efb" containerName="revoke-keys"
Sep 11 04:03:34.802188 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:03:34.802168    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-b4d3cf/rate-test-model-b4d3cf-kserve-6f654d7c9f-hxsxq"
Sep 11 04:03:34.806166 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:03:34.806133    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w3-b4d3cf\"/\"openshift-service-ca.crt\""
Sep 11 04:03:34.806166 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:03:34.806155    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w3-b4d3cf\"/\"kube-root-ca.crt\""
Sep 11 04:03:34.806341 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:03:34.806155    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-b4d3cf\"/\"rate-test-model-b4d3cf-kserve-self-signed-certs\""
Sep 11 04:03:34.806341 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:03:34.806133    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-b4d3cf\"/\"default-dockercfg-5x4f6\""
Sep 11 04:03:34.811271 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:03:34.811231    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-b4d3cf/rate-test-model-b4d3cf-kserve-6f654d7c9f-hxsxq"]
Sep 11 04:03:34.890032 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:03:34.889999    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d416cf69-55ae-4a8f-bd7e-e049dac3a73e-tls-certs\") pod \"rate-test-model-b4d3cf-kserve-6f654d7c9f-hxsxq\" (UID: \"d416cf69-55ae-4a8f-bd7e-e049dac3a73e\") " pod="ai-tenant-e2e-shared-a-w3-b4d3cf/rate-test-model-b4d3cf-kserve-6f654d7c9f-hxsxq"
Sep 11 04:03:34.890032 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:03:34.890034    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d416cf69-55ae-4a8f-bd7e-e049dac3a73e-model-cache\") pod \"rate-test-model-b4d3cf-kserve-6f654d7c9f-hxsxq\" (UID: \"d416cf69-55ae-4a8f-bd7e-e049dac3a73e\") " pod="ai-tenant-e2e-shared-a-w3-b4d3cf/rate-test-model-b4d3cf-kserve-6f654d7c9f-hxsxq"
Sep 11 04:03:34.890273 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:03:34.890060    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d416cf69-55ae-4a8f-bd7e-e049dac3a73e-dshm\") pod \"rate-test-model-b4d3cf-kserve-6f654d7c9f-hxsxq\" (UID: \"d416cf69-55ae-4a8f-bd7e-e049dac3a73e\") " pod="ai-tenant-e2e-shared-a-w3-b4d3cf/rate-test-model-b4d3cf-kserve-6f654d7c9f-hxsxq"
Sep 11 04:03:34.890273 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:03:34.890105    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6rqbk\" (UniqueName: \"kubernetes.io/projected/d416cf69-55ae-4a8f-bd7e-e049dac3a73e-kube-api-access-6rqbk\") pod \"rate-test-model-b4d3cf-kserve-6f654d7c9f-hxsxq\" (UID: \"d416cf69-55ae-4a8f-bd7e-e049dac3a73e\") " pod="ai-tenant-e2e-shared-a-w3-b4d3cf/rate-test-model-b4d3cf-kserve-6f654d7c9f-hxsxq"
Sep 11 04:03:34.890273 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:03:34.890183    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d416cf69-55ae-4a8f-bd7e-e049dac3a73e-home\") pod \"rate-test-model-b4d3cf-kserve-6f654d7c9f-hxsxq\" (UID: \"d416cf69-55ae-4a8f-bd7e-e049dac3a73e\") " pod="ai-tenant-e2e-shared-a-w3-b4d3cf/rate-test-model-b4d3cf-kserve-6f654d7c9f-hxsxq"
Sep 11 04:03:34.890273 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:03:34.890210    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d416cf69-55ae-4a8f-bd7e-e049dac3a73e-tmp-dir\") pod \"rate-test-model-b4d3cf-kserve-6f654d7c9f-hxsxq\" (UID: \"d416cf69-55ae-4a8f-bd7e-e049dac3a73e\") " pod="ai-tenant-e2e-shared-a-w3-b4d3cf/rate-test-model-b4d3cf-kserve-6f654d7c9f-hxsxq"
Sep 11 04:03:34.990911 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:03:34.990878    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d416cf69-55ae-4a8f-bd7e-e049dac3a73e-home\") pod \"rate-test-model-b4d3cf-kserve-6f654d7c9f-hxsxq\" (UID: \"d416cf69-55ae-4a8f-bd7e-e049dac3a73e\") " pod="ai-tenant-e2e-shared-a-w3-b4d3cf/rate-test-model-b4d3cf-kserve-6f654d7c9f-hxsxq"
Sep 11 04:03:34.991136 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:03:34.990927    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d416cf69-55ae-4a8f-bd7e-e049dac3a73e-tmp-dir\") pod \"rate-test-model-b4d3cf-kserve-6f654d7c9f-hxsxq\" (UID: \"d416cf69-55ae-4a8f-bd7e-e049dac3a73e\") " pod="ai-tenant-e2e-shared-a-w3-b4d3cf/rate-test-model-b4d3cf-kserve-6f654d7c9f-hxsxq"
Sep 11 04:03:34.991136 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:03:34.990963    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d416cf69-55ae-4a8f-bd7e-e049dac3a73e-tls-certs\") pod \"rate-test-model-b4d3cf-kserve-6f654d7c9f-hxsxq\" (UID: \"d416cf69-55ae-4a8f-bd7e-e049dac3a73e\") " pod="ai-tenant-e2e-shared-a-w3-b4d3cf/rate-test-model-b4d3cf-kserve-6f654d7c9f-hxsxq"
Sep 11 04:03:34.991136 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:03:34.990993    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d416cf69-55ae-4a8f-bd7e-e049dac3a73e-model-cache\") pod \"rate-test-model-b4d3cf-kserve-6f654d7c9f-hxsxq\" (UID: \"d416cf69-55ae-4a8f-bd7e-e049dac3a73e\") " pod="ai-tenant-e2e-shared-a-w3-b4d3cf/rate-test-model-b4d3cf-kserve-6f654d7c9f-hxsxq"
Sep 11 04:03:34.991339 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:03:34.991188    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d416cf69-55ae-4a8f-bd7e-e049dac3a73e-dshm\") pod \"rate-test-model-b4d3cf-kserve-6f654d7c9f-hxsxq\" (UID: \"d416cf69-55ae-4a8f-bd7e-e049dac3a73e\") " pod="ai-tenant-e2e-shared-a-w3-b4d3cf/rate-test-model-b4d3cf-kserve-6f654d7c9f-hxsxq"
Sep 11 04:03:34.991339 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:03:34.991225    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6rqbk\" (UniqueName: \"kubernetes.io/projected/d416cf69-55ae-4a8f-bd7e-e049dac3a73e-kube-api-access-6rqbk\") pod \"rate-test-model-b4d3cf-kserve-6f654d7c9f-hxsxq\" (UID: \"d416cf69-55ae-4a8f-bd7e-e049dac3a73e\") " pod="ai-tenant-e2e-shared-a-w3-b4d3cf/rate-test-model-b4d3cf-kserve-6f654d7c9f-hxsxq"
Sep 11 04:03:34.991450 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:03:34.991361    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d416cf69-55ae-4a8f-bd7e-e049dac3a73e-home\") pod \"rate-test-model-b4d3cf-kserve-6f654d7c9f-hxsxq\" (UID: \"d416cf69-55ae-4a8f-bd7e-e049dac3a73e\") " pod="ai-tenant-e2e-shared-a-w3-b4d3cf/rate-test-model-b4d3cf-kserve-6f654d7c9f-hxsxq"
Sep 11 04:03:34.991450 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:03:34.991425    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d416cf69-55ae-4a8f-bd7e-e049dac3a73e-model-cache\") pod \"rate-test-model-b4d3cf-kserve-6f654d7c9f-hxsxq\" (UID: \"d416cf69-55ae-4a8f-bd7e-e049dac3a73e\") " pod="ai-tenant-e2e-shared-a-w3-b4d3cf/rate-test-model-b4d3cf-kserve-6f654d7c9f-hxsxq"
Sep 11 04:03:34.991652 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:03:34.991636    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d416cf69-55ae-4a8f-bd7e-e049dac3a73e-tmp-dir\") pod \"rate-test-model-b4d3cf-kserve-6f654d7c9f-hxsxq\" (UID: \"d416cf69-55ae-4a8f-bd7e-e049dac3a73e\") " pod="ai-tenant-e2e-shared-a-w3-b4d3cf/rate-test-model-b4d3cf-kserve-6f654d7c9f-hxsxq"
Sep 11 04:03:34.993429 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:03:34.993411    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d416cf69-55ae-4a8f-bd7e-e049dac3a73e-dshm\") pod \"rate-test-model-b4d3cf-kserve-6f654d7c9f-hxsxq\" (UID: \"d416cf69-55ae-4a8f-bd7e-e049dac3a73e\") " pod="ai-tenant-e2e-shared-a-w3-b4d3cf/rate-test-model-b4d3cf-kserve-6f654d7c9f-hxsxq"
Sep 11 04:03:34.993836 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:03:34.993814    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d416cf69-55ae-4a8f-bd7e-e049dac3a73e-tls-certs\") pod \"rate-test-model-b4d3cf-kserve-6f654d7c9f-hxsxq\" (UID: \"d416cf69-55ae-4a8f-bd7e-e049dac3a73e\") " pod="ai-tenant-e2e-shared-a-w3-b4d3cf/rate-test-model-b4d3cf-kserve-6f654d7c9f-hxsxq"
Sep 11 04:03:34.999167 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:03:34.999143    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6rqbk\" (UniqueName: \"kubernetes.io/projected/d416cf69-55ae-4a8f-bd7e-e049dac3a73e-kube-api-access-6rqbk\") pod \"rate-test-model-b4d3cf-kserve-6f654d7c9f-hxsxq\" (UID: \"d416cf69-55ae-4a8f-bd7e-e049dac3a73e\") " pod="ai-tenant-e2e-shared-a-w3-b4d3cf/rate-test-model-b4d3cf-kserve-6f654d7c9f-hxsxq"
Sep 11 04:03:35.113209 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:03:35.113184    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-b4d3cf/rate-test-model-b4d3cf-kserve-6f654d7c9f-hxsxq"
Sep 11 04:03:35.387486 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:03:35.387459    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-b4d3cf/rate-test-model-b4d3cf-kserve-6f654d7c9f-hxsxq"]
Sep 11 04:03:35.389618 ip-10-0-141-105 kubenswrapper[2579]: W0911 04:03:35.389589    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd416cf69_55ae_4a8f_bd7e_e049dac3a73e.slice/crio-2601453be5c33ca786b1ed2d031a3e927931d7444dc873df733ee0a6cb0c2466 WatchSource:0}: Error finding container 2601453be5c33ca786b1ed2d031a3e927931d7444dc873df733ee0a6cb0c2466: Status 404 returned error can't find the container with id 2601453be5c33ca786b1ed2d031a3e927931d7444dc873df733ee0a6cb0c2466
Sep 11 04:03:36.166787 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:03:36.166750    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-b4d3cf/rate-test-model-b4d3cf-kserve-6f654d7c9f-hxsxq" event={"ID":"d416cf69-55ae-4a8f-bd7e-e049dac3a73e","Type":"ContainerStarted","Data":"dcac2bcbed732bc9d9a54cc677c781eb2aab8ea5013107d5eaff1ed676510f0d"}
Sep 11 04:03:36.167233 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:03:36.166797    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-b4d3cf/rate-test-model-b4d3cf-kserve-6f654d7c9f-hxsxq" event={"ID":"d416cf69-55ae-4a8f-bd7e-e049dac3a73e","Type":"ContainerStarted","Data":"2601453be5c33ca786b1ed2d031a3e927931d7444dc873df733ee0a6cb0c2466"}
Sep 11 04:03:36.186299 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:03:36.186248    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w3-b4d3cf/rate-test-model-b4d3cf-kserve-6f654d7c9f-hxsxq" podStartSLOduration=2.186233164 podStartE2EDuration="2.186233164s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 04:03:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 04:03:36.185744199 +0000 UTC m=+1393.840949776" watchObservedRunningTime="2026-09-11 04:03:36.186233164 +0000 UTC m=+1393.841438731"
Sep 11 04:03:38.308136 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:03:38.308096    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-5z4cn"]
Sep 11 04:03:45.113705 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:03:45.113648    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w3-b4d3cf/rate-test-model-b4d3cf-kserve-6f654d7c9f-hxsxq"
Sep 11 04:03:45.114168 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:03:45.113722    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w3-b4d3cf/rate-test-model-b4d3cf-kserve-6f654d7c9f-hxsxq"
Sep 11 04:03:45.126652 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:03:45.126629    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w3-b4d3cf/rate-test-model-b4d3cf-kserve-6f654d7c9f-hxsxq"
Sep 11 04:03:45.227263 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:03:45.227233    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w3-b4d3cf/rate-test-model-b4d3cf-kserve-6f654d7c9f-hxsxq"
Sep 11 04:03:47.038541 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:03:47.038512    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-49518b-hzx9b"]
Sep 11 04:03:47.042700 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:03:47.042673    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-49518b-hzx9b"]
Sep 11 04:03:48.852340 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:03:48.852307    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4719291a-c30a-49f5-8509-8ff9e3f90efb" path="/var/lib/kubelet/pods/4719291a-c30a-49f5-8509-8ff9e3f90efb/volumes"
Sep 11 04:03:53.793751 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:03:53.793718    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-b61c46-fhv4m"]
Sep 11 04:03:53.797261 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:03:53.797243    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-b61c46-fhv4m"
Sep 11 04:03:53.799877 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:03:53.799851    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 11 04:03:53.799991 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:03:53.799938    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-95ckr\""
Sep 11 04:03:53.804401 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:03:53.804377    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-b61c46-fhv4m"]
Sep 11 04:03:53.970114 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:03:53.970013    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fdc2f0d7-a540-4679-9329-b66ce8745edf-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-b-b61c46-fhv4m\" (UID: \"fdc2f0d7-a540-4679-9329-b66ce8745edf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-b61c46-fhv4m"
Sep 11 04:03:54.002491 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:03:54.002458    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-5z4cn"]
Sep 11 04:03:54.071733 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:03:54.071642    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fdc2f0d7-a540-4679-9329-b66ce8745edf-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-b-b61c46-fhv4m\" (UID: \"fdc2f0d7-a540-4679-9329-b66ce8745edf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-b61c46-fhv4m"
Sep 11 04:03:54.072300 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:03:54.072273    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fdc2f0d7-a540-4679-9329-b66ce8745edf-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-b-b61c46-fhv4m\" (UID: \"fdc2f0d7-a540-4679-9329-b66ce8745edf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-b61c46-fhv4m"
Sep 11 04:03:54.109817 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:03:54.109790    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-b61c46-fhv4m"
Sep 11 04:03:54.380134 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:03:54.380110    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-b61c46-fhv4m"]
Sep 11 04:03:54.382271 ip-10-0-141-105 kubenswrapper[2579]: W0911 04:03:54.382246    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfdc2f0d7_a540_4679_9329_b66ce8745edf.slice/crio-7b134d21e00daca200bb0bf2a3e549ebf6bee817a4bddf9ecd4f97dc34931626 WatchSource:0}: Error finding container 7b134d21e00daca200bb0bf2a3e549ebf6bee817a4bddf9ecd4f97dc34931626: Status 404 returned error can't find the container with id 7b134d21e00daca200bb0bf2a3e549ebf6bee817a4bddf9ecd4f97dc34931626
Sep 11 04:03:55.273847 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:03:55.273818    2579 generic.go:356] "Generic (PLEG): container finished" podID="fdc2f0d7-a540-4679-9329-b66ce8745edf" containerID="97533289e52ac689bc8465c5936fcce94107cfef608aeb8f94bd7b475e81ef6a" exitCode=0
Sep 11 04:03:55.274298 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:03:55.273903    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-b61c46-fhv4m" event={"ID":"fdc2f0d7-a540-4679-9329-b66ce8745edf","Type":"ContainerDied","Data":"97533289e52ac689bc8465c5936fcce94107cfef608aeb8f94bd7b475e81ef6a"}
Sep 11 04:03:55.274298 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:03:55.273950    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-b61c46-fhv4m" event={"ID":"fdc2f0d7-a540-4679-9329-b66ce8745edf","Type":"ContainerStarted","Data":"7b134d21e00daca200bb0bf2a3e549ebf6bee817a4bddf9ecd4f97dc34931626"}
Sep 11 04:03:56.418448 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:03:56.418423    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-b61c46-fhv4m"
Sep 11 04:03:56.496942 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:03:56.496913    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fdc2f0d7-a540-4679-9329-b66ce8745edf-maas-api-service-ca\") pod \"fdc2f0d7-a540-4679-9329-b66ce8745edf\" (UID: \"fdc2f0d7-a540-4679-9329-b66ce8745edf\") "
Sep 11 04:03:56.497304 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:03:56.497280    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fdc2f0d7-a540-4679-9329-b66ce8745edf-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "fdc2f0d7-a540-4679-9329-b66ce8745edf" (UID: "fdc2f0d7-a540-4679-9329-b66ce8745edf"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 04:03:56.597615 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:03:56.597549    2579 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fdc2f0d7-a540-4679-9329-b66ce8745edf-maas-api-service-ca\") on node \"ip-10-0-141-105.ec2.internal\" DevicePath \"\""
Sep 11 04:03:57.285920 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:03:57.285830    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-b61c46-fhv4m"
Sep 11 04:03:57.285920 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:03:57.285858    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-b61c46-fhv4m" event={"ID":"fdc2f0d7-a540-4679-9329-b66ce8745edf","Type":"ContainerDied","Data":"7b134d21e00daca200bb0bf2a3e549ebf6bee817a4bddf9ecd4f97dc34931626"}
Sep 11 04:03:57.285920 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:03:57.285907    2579 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7b134d21e00daca200bb0bf2a3e549ebf6bee817a4bddf9ecd4f97dc34931626"
Sep 11 04:04:07.592215 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:04:07.592178    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w3-dd5b30/rate-test-model-dd5b30-kserve-6cf4c47fb7-j4csz"]
Sep 11 04:04:07.592746 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:04:07.592574    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fdc2f0d7-a540-4679-9329-b66ce8745edf" containerName="revoke-keys"
Sep 11 04:04:07.592746 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:04:07.592585    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="fdc2f0d7-a540-4679-9329-b66ce8745edf" containerName="revoke-keys"
Sep 11 04:04:07.592746 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:04:07.592681    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="fdc2f0d7-a540-4679-9329-b66ce8745edf" containerName="revoke-keys"
Sep 11 04:04:07.596012 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:04:07.595995    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-dd5b30/rate-test-model-dd5b30-kserve-6cf4c47fb7-j4csz"
Sep 11 04:04:07.599627 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:04:07.599599    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-dd5b30\"/\"kube-root-ca.crt\""
Sep 11 04:04:07.599788 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:04:07.599675    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-dd5b30\"/\"openshift-service-ca.crt\""
Sep 11 04:04:07.599788 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:04:07.599688    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-dd5b30\"/\"default-dockercfg-lvpcb\""
Sep 11 04:04:07.599788 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:04:07.599751    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-dd5b30\"/\"rate-test-model-dd5b30-kserve-self-signed-certs\""
Sep 11 04:04:07.605757 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:04:07.605733    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-dd5b30/rate-test-model-dd5b30-kserve-6cf4c47fb7-j4csz"]
Sep 11 04:04:07.705128 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:04:07.705065    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/82bbb090-3ff4-4a87-a9e0-ef14c3b22b05-model-cache\") pod \"rate-test-model-dd5b30-kserve-6cf4c47fb7-j4csz\" (UID: \"82bbb090-3ff4-4a87-a9e0-ef14c3b22b05\") " pod="ai-tenant-e2e-shared-b-w3-dd5b30/rate-test-model-dd5b30-kserve-6cf4c47fb7-j4csz"
Sep 11 04:04:07.705299 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:04:07.705147    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/82bbb090-3ff4-4a87-a9e0-ef14c3b22b05-dshm\") pod \"rate-test-model-dd5b30-kserve-6cf4c47fb7-j4csz\" (UID: \"82bbb090-3ff4-4a87-a9e0-ef14c3b22b05\") " pod="ai-tenant-e2e-shared-b-w3-dd5b30/rate-test-model-dd5b30-kserve-6cf4c47fb7-j4csz"
Sep 11 04:04:07.705299 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:04:07.705177    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/82bbb090-3ff4-4a87-a9e0-ef14c3b22b05-tmp-dir\") pod \"rate-test-model-dd5b30-kserve-6cf4c47fb7-j4csz\" (UID: \"82bbb090-3ff4-4a87-a9e0-ef14c3b22b05\") " pod="ai-tenant-e2e-shared-b-w3-dd5b30/rate-test-model-dd5b30-kserve-6cf4c47fb7-j4csz"
Sep 11 04:04:07.705299 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:04:07.705212    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v986h\" (UniqueName: \"kubernetes.io/projected/82bbb090-3ff4-4a87-a9e0-ef14c3b22b05-kube-api-access-v986h\") pod \"rate-test-model-dd5b30-kserve-6cf4c47fb7-j4csz\" (UID: \"82bbb090-3ff4-4a87-a9e0-ef14c3b22b05\") " pod="ai-tenant-e2e-shared-b-w3-dd5b30/rate-test-model-dd5b30-kserve-6cf4c47fb7-j4csz"
Sep 11 04:04:07.705299 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:04:07.705269    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/82bbb090-3ff4-4a87-a9e0-ef14c3b22b05-tls-certs\") pod \"rate-test-model-dd5b30-kserve-6cf4c47fb7-j4csz\" (UID: \"82bbb090-3ff4-4a87-a9e0-ef14c3b22b05\") " pod="ai-tenant-e2e-shared-b-w3-dd5b30/rate-test-model-dd5b30-kserve-6cf4c47fb7-j4csz"
Sep 11 04:04:07.705468 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:04:07.705328    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/82bbb090-3ff4-4a87-a9e0-ef14c3b22b05-home\") pod \"rate-test-model-dd5b30-kserve-6cf4c47fb7-j4csz\" (UID: \"82bbb090-3ff4-4a87-a9e0-ef14c3b22b05\") " pod="ai-tenant-e2e-shared-b-w3-dd5b30/rate-test-model-dd5b30-kserve-6cf4c47fb7-j4csz"
Sep 11 04:04:07.806051 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:04:07.806014    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/82bbb090-3ff4-4a87-a9e0-ef14c3b22b05-home\") pod \"rate-test-model-dd5b30-kserve-6cf4c47fb7-j4csz\" (UID: \"82bbb090-3ff4-4a87-a9e0-ef14c3b22b05\") " pod="ai-tenant-e2e-shared-b-w3-dd5b30/rate-test-model-dd5b30-kserve-6cf4c47fb7-j4csz"
Sep 11 04:04:07.806269 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:04:07.806124    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/82bbb090-3ff4-4a87-a9e0-ef14c3b22b05-model-cache\") pod \"rate-test-model-dd5b30-kserve-6cf4c47fb7-j4csz\" (UID: \"82bbb090-3ff4-4a87-a9e0-ef14c3b22b05\") " pod="ai-tenant-e2e-shared-b-w3-dd5b30/rate-test-model-dd5b30-kserve-6cf4c47fb7-j4csz"
Sep 11 04:04:07.806269 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:04:07.806154    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/82bbb090-3ff4-4a87-a9e0-ef14c3b22b05-dshm\") pod \"rate-test-model-dd5b30-kserve-6cf4c47fb7-j4csz\" (UID: \"82bbb090-3ff4-4a87-a9e0-ef14c3b22b05\") " pod="ai-tenant-e2e-shared-b-w3-dd5b30/rate-test-model-dd5b30-kserve-6cf4c47fb7-j4csz"
Sep 11 04:04:07.806269 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:04:07.806190    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/82bbb090-3ff4-4a87-a9e0-ef14c3b22b05-tmp-dir\") pod \"rate-test-model-dd5b30-kserve-6cf4c47fb7-j4csz\" (UID: \"82bbb090-3ff4-4a87-a9e0-ef14c3b22b05\") " pod="ai-tenant-e2e-shared-b-w3-dd5b30/rate-test-model-dd5b30-kserve-6cf4c47fb7-j4csz"
Sep 11 04:04:07.806269 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:04:07.806242    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v986h\" (UniqueName: \"kubernetes.io/projected/82bbb090-3ff4-4a87-a9e0-ef14c3b22b05-kube-api-access-v986h\") pod \"rate-test-model-dd5b30-kserve-6cf4c47fb7-j4csz\" (UID: \"82bbb090-3ff4-4a87-a9e0-ef14c3b22b05\") " pod="ai-tenant-e2e-shared-b-w3-dd5b30/rate-test-model-dd5b30-kserve-6cf4c47fb7-j4csz"
Sep 11 04:04:07.806490 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:04:07.806278    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/82bbb090-3ff4-4a87-a9e0-ef14c3b22b05-tls-certs\") pod \"rate-test-model-dd5b30-kserve-6cf4c47fb7-j4csz\" (UID: \"82bbb090-3ff4-4a87-a9e0-ef14c3b22b05\") " pod="ai-tenant-e2e-shared-b-w3-dd5b30/rate-test-model-dd5b30-kserve-6cf4c47fb7-j4csz"
Sep 11 04:04:07.806490 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:04:07.806462    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/82bbb090-3ff4-4a87-a9e0-ef14c3b22b05-home\") pod \"rate-test-model-dd5b30-kserve-6cf4c47fb7-j4csz\" (UID: \"82bbb090-3ff4-4a87-a9e0-ef14c3b22b05\") " pod="ai-tenant-e2e-shared-b-w3-dd5b30/rate-test-model-dd5b30-kserve-6cf4c47fb7-j4csz"
Sep 11 04:04:07.806616 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:04:07.806527    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/82bbb090-3ff4-4a87-a9e0-ef14c3b22b05-model-cache\") pod \"rate-test-model-dd5b30-kserve-6cf4c47fb7-j4csz\" (UID: \"82bbb090-3ff4-4a87-a9e0-ef14c3b22b05\") " pod="ai-tenant-e2e-shared-b-w3-dd5b30/rate-test-model-dd5b30-kserve-6cf4c47fb7-j4csz"
Sep 11 04:04:07.806616 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:04:07.806595    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/82bbb090-3ff4-4a87-a9e0-ef14c3b22b05-tmp-dir\") pod \"rate-test-model-dd5b30-kserve-6cf4c47fb7-j4csz\" (UID: \"82bbb090-3ff4-4a87-a9e0-ef14c3b22b05\") " pod="ai-tenant-e2e-shared-b-w3-dd5b30/rate-test-model-dd5b30-kserve-6cf4c47fb7-j4csz"
Sep 11 04:04:07.808635 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:04:07.808615    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/82bbb090-3ff4-4a87-a9e0-ef14c3b22b05-dshm\") pod \"rate-test-model-dd5b30-kserve-6cf4c47fb7-j4csz\" (UID: \"82bbb090-3ff4-4a87-a9e0-ef14c3b22b05\") " pod="ai-tenant-e2e-shared-b-w3-dd5b30/rate-test-model-dd5b30-kserve-6cf4c47fb7-j4csz"
Sep 11 04:04:07.808879 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:04:07.808861    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/82bbb090-3ff4-4a87-a9e0-ef14c3b22b05-tls-certs\") pod \"rate-test-model-dd5b30-kserve-6cf4c47fb7-j4csz\" (UID: \"82bbb090-3ff4-4a87-a9e0-ef14c3b22b05\") " pod="ai-tenant-e2e-shared-b-w3-dd5b30/rate-test-model-dd5b30-kserve-6cf4c47fb7-j4csz"
Sep 11 04:04:07.815700 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:04:07.815678    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-v986h\" (UniqueName: \"kubernetes.io/projected/82bbb090-3ff4-4a87-a9e0-ef14c3b22b05-kube-api-access-v986h\") pod \"rate-test-model-dd5b30-kserve-6cf4c47fb7-j4csz\" (UID: \"82bbb090-3ff4-4a87-a9e0-ef14c3b22b05\") " pod="ai-tenant-e2e-shared-b-w3-dd5b30/rate-test-model-dd5b30-kserve-6cf4c47fb7-j4csz"
Sep 11 04:04:07.907037 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:04:07.907002    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-dd5b30/rate-test-model-dd5b30-kserve-6cf4c47fb7-j4csz"
Sep 11 04:04:08.185833 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:04:08.185805    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-dd5b30/rate-test-model-dd5b30-kserve-6cf4c47fb7-j4csz"]
Sep 11 04:04:08.187915 ip-10-0-141-105 kubenswrapper[2579]: W0911 04:04:08.187886    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod82bbb090_3ff4_4a87_a9e0_ef14c3b22b05.slice/crio-6f172d25714ab98e50935a90880588bcb6fb22f368155eeb5771ad30def550a1 WatchSource:0}: Error finding container 6f172d25714ab98e50935a90880588bcb6fb22f368155eeb5771ad30def550a1: Status 404 returned error can't find the container with id 6f172d25714ab98e50935a90880588bcb6fb22f368155eeb5771ad30def550a1
Sep 11 04:04:08.345728 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:04:08.345691    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-dd5b30/rate-test-model-dd5b30-kserve-6cf4c47fb7-j4csz" event={"ID":"82bbb090-3ff4-4a87-a9e0-ef14c3b22b05","Type":"ContainerStarted","Data":"ad3fe27bb5d60bd09975d849197a49bc89f17e1c1a395e10d7975592fc26d3d1"}
Sep 11 04:04:08.345932 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:04:08.345741    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-dd5b30/rate-test-model-dd5b30-kserve-6cf4c47fb7-j4csz" event={"ID":"82bbb090-3ff4-4a87-a9e0-ef14c3b22b05","Type":"ContainerStarted","Data":"6f172d25714ab98e50935a90880588bcb6fb22f368155eeb5771ad30def550a1"}
Sep 11 04:04:08.363409 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:04:08.363363    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w3-dd5b30/rate-test-model-dd5b30-kserve-6cf4c47fb7-j4csz" podStartSLOduration=1.363348254 podStartE2EDuration="1.363348254s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 04:04:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 04:04:08.360824243 +0000 UTC m=+1426.016029793" watchObservedRunningTime="2026-09-11 04:04:08.363348254 +0000 UTC m=+1426.018553821"
Sep 11 04:04:09.248956 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:04:09.248921    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-b61c46-d5d4b49bf-dpb5m"]
Sep 11 04:04:09.249469 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:04:09.249184    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-b-b61c46-d5d4b49bf-dpb5m" podUID="174bdd05-1f30-4bd1-819f-8d534357f38c" containerName="payload-processing" containerID="cri-o://d919bcdaf2ab93b06ca7d4f6d1c9fe9d09732fba59f831c3c29706bf6b3c97bd" gracePeriod=30
Sep 11 04:04:09.255288 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:04:09.255255    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-b61c46-5c9c66b79d-l9qzp"]
Sep 11 04:04:09.255573 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:04:09.255510    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-b-b61c46-5c9c66b79d-l9qzp" podUID="63febe17-6300-476c-b49e-74e1098cca67" containerName="payload-pre-processing" containerID="cri-o://1495ad6e83a59676cf21f937d064afb20aab080af17611609aa5495012dee9e1" gracePeriod=30
Sep 11 04:04:09.456246 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:04:09.456213    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-b61c46-fhv4m"]
Sep 11 04:04:09.461894 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:04:09.461871    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-b61c46-fhv4m"]
Sep 11 04:04:09.540226 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:04:09.540193    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-b61c46-d5d4b49bf-dpb5m"
Sep 11 04:04:09.550074 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:04:09.550051    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-b61c46-5c9c66b79d-l9qzp"
Sep 11 04:04:09.623248 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:04:09.623215    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6snrz\" (UniqueName: \"kubernetes.io/projected/63febe17-6300-476c-b49e-74e1098cca67-kube-api-access-6snrz\") pod \"63febe17-6300-476c-b49e-74e1098cca67\" (UID: \"63febe17-6300-476c-b49e-74e1098cca67\") "
Sep 11 04:04:09.623248 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:04:09.623249    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/174bdd05-1f30-4bd1-819f-8d534357f38c-plugins-config-volume\") pod \"174bdd05-1f30-4bd1-819f-8d534357f38c\" (UID: \"174bdd05-1f30-4bd1-819f-8d534357f38c\") "
Sep 11 04:04:09.623446 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:04:09.623305    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/63febe17-6300-476c-b49e-74e1098cca67-plugins-config-volume\") pod \"63febe17-6300-476c-b49e-74e1098cca67\" (UID: \"63febe17-6300-476c-b49e-74e1098cca67\") "
Sep 11 04:04:09.623446 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:04:09.623341    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qczsn\" (UniqueName: \"kubernetes.io/projected/174bdd05-1f30-4bd1-819f-8d534357f38c-kube-api-access-qczsn\") pod \"174bdd05-1f30-4bd1-819f-8d534357f38c\" (UID: \"174bdd05-1f30-4bd1-819f-8d534357f38c\") "
Sep 11 04:04:09.623756 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:04:09.623722    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/174bdd05-1f30-4bd1-819f-8d534357f38c-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "174bdd05-1f30-4bd1-819f-8d534357f38c" (UID: "174bdd05-1f30-4bd1-819f-8d534357f38c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 04:04:09.623836 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:04:09.623728    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/63febe17-6300-476c-b49e-74e1098cca67-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "63febe17-6300-476c-b49e-74e1098cca67" (UID: "63febe17-6300-476c-b49e-74e1098cca67"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 04:04:09.625566 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:04:09.625540    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/63febe17-6300-476c-b49e-74e1098cca67-kube-api-access-6snrz" (OuterVolumeSpecName: "kube-api-access-6snrz") pod "63febe17-6300-476c-b49e-74e1098cca67" (UID: "63febe17-6300-476c-b49e-74e1098cca67"). InnerVolumeSpecName "kube-api-access-6snrz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 04:04:09.625676 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:04:09.625624    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/174bdd05-1f30-4bd1-819f-8d534357f38c-kube-api-access-qczsn" (OuterVolumeSpecName: "kube-api-access-qczsn") pod "174bdd05-1f30-4bd1-819f-8d534357f38c" (UID: "174bdd05-1f30-4bd1-819f-8d534357f38c"). InnerVolumeSpecName "kube-api-access-qczsn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 04:04:09.724561 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:04:09.724527    2579 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/63febe17-6300-476c-b49e-74e1098cca67-plugins-config-volume\") on node \"ip-10-0-141-105.ec2.internal\" DevicePath \"\""
Sep 11 04:04:09.724561 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:04:09.724556    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qczsn\" (UniqueName: \"kubernetes.io/projected/174bdd05-1f30-4bd1-819f-8d534357f38c-kube-api-access-qczsn\") on node \"ip-10-0-141-105.ec2.internal\" DevicePath \"\""
Sep 11 04:04:09.724741 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:04:09.724569    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6snrz\" (UniqueName: \"kubernetes.io/projected/63febe17-6300-476c-b49e-74e1098cca67-kube-api-access-6snrz\") on node \"ip-10-0-141-105.ec2.internal\" DevicePath \"\""
Sep 11 04:04:09.724741 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:04:09.724579    2579 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/174bdd05-1f30-4bd1-819f-8d534357f38c-plugins-config-volume\") on node \"ip-10-0-141-105.ec2.internal\" DevicePath \"\""
Sep 11 04:04:10.359386 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:04:10.359350    2579 generic.go:356] "Generic (PLEG): container finished" podID="63febe17-6300-476c-b49e-74e1098cca67" containerID="1495ad6e83a59676cf21f937d064afb20aab080af17611609aa5495012dee9e1" exitCode=0
Sep 11 04:04:10.359891 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:04:10.359397    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-b61c46-5c9c66b79d-l9qzp"
Sep 11 04:04:10.359891 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:04:10.359444    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-b61c46-5c9c66b79d-l9qzp" event={"ID":"63febe17-6300-476c-b49e-74e1098cca67","Type":"ContainerDied","Data":"1495ad6e83a59676cf21f937d064afb20aab080af17611609aa5495012dee9e1"}
Sep 11 04:04:10.359891 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:04:10.359500    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-b61c46-5c9c66b79d-l9qzp" event={"ID":"63febe17-6300-476c-b49e-74e1098cca67","Type":"ContainerDied","Data":"bdaa6edcb8babdb53a71dad690a23cbf4bc560f9dd74469c855103e1dd1730ec"}
Sep 11 04:04:10.359891 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:04:10.359526    2579 scope.go:117] "RemoveContainer" containerID="1495ad6e83a59676cf21f937d064afb20aab080af17611609aa5495012dee9e1"
Sep 11 04:04:10.361002 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:04:10.360978    2579 generic.go:356] "Generic (PLEG): container finished" podID="174bdd05-1f30-4bd1-819f-8d534357f38c" containerID="d919bcdaf2ab93b06ca7d4f6d1c9fe9d09732fba59f831c3c29706bf6b3c97bd" exitCode=0
Sep 11 04:04:10.361101 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:04:10.361031    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-b61c46-d5d4b49bf-dpb5m"
Sep 11 04:04:10.361101 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:04:10.361041    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-b61c46-d5d4b49bf-dpb5m" event={"ID":"174bdd05-1f30-4bd1-819f-8d534357f38c","Type":"ContainerDied","Data":"d919bcdaf2ab93b06ca7d4f6d1c9fe9d09732fba59f831c3c29706bf6b3c97bd"}
Sep 11 04:04:10.361229 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:04:10.361104    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-b61c46-d5d4b49bf-dpb5m" event={"ID":"174bdd05-1f30-4bd1-819f-8d534357f38c","Type":"ContainerDied","Data":"2ea2e168de4b16e907436870667b8e067c553b64096693cd897290c2d587ee4f"}
Sep 11 04:04:10.369218 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:04:10.369137    2579 scope.go:117] "RemoveContainer" containerID="1495ad6e83a59676cf21f937d064afb20aab080af17611609aa5495012dee9e1"
Sep 11 04:04:10.369777 ip-10-0-141-105 kubenswrapper[2579]: E0911 04:04:10.369746    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1495ad6e83a59676cf21f937d064afb20aab080af17611609aa5495012dee9e1\": container with ID starting with 1495ad6e83a59676cf21f937d064afb20aab080af17611609aa5495012dee9e1 not found: ID does not exist" containerID="1495ad6e83a59676cf21f937d064afb20aab080af17611609aa5495012dee9e1"
Sep 11 04:04:10.369879 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:04:10.369788    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1495ad6e83a59676cf21f937d064afb20aab080af17611609aa5495012dee9e1"} err="failed to get container status \"1495ad6e83a59676cf21f937d064afb20aab080af17611609aa5495012dee9e1\": rpc error: code = NotFound desc = could not find container \"1495ad6e83a59676cf21f937d064afb20aab080af17611609aa5495012dee9e1\": container with ID starting with 1495ad6e83a59676cf21f937d064afb20aab080af17611609aa5495012dee9e1 not found: ID does not exist"
Sep 11 04:04:10.369879 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:04:10.369811    2579 scope.go:117] "RemoveContainer" containerID="d919bcdaf2ab93b06ca7d4f6d1c9fe9d09732fba59f831c3c29706bf6b3c97bd"
Sep 11 04:04:10.378831 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:04:10.378813    2579 scope.go:117] "RemoveContainer" containerID="d919bcdaf2ab93b06ca7d4f6d1c9fe9d09732fba59f831c3c29706bf6b3c97bd"
Sep 11 04:04:10.379129 ip-10-0-141-105 kubenswrapper[2579]: E0911 04:04:10.379108    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d919bcdaf2ab93b06ca7d4f6d1c9fe9d09732fba59f831c3c29706bf6b3c97bd\": container with ID starting with d919bcdaf2ab93b06ca7d4f6d1c9fe9d09732fba59f831c3c29706bf6b3c97bd not found: ID does not exist" containerID="d919bcdaf2ab93b06ca7d4f6d1c9fe9d09732fba59f831c3c29706bf6b3c97bd"
Sep 11 04:04:10.379200 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:04:10.379135    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d919bcdaf2ab93b06ca7d4f6d1c9fe9d09732fba59f831c3c29706bf6b3c97bd"} err="failed to get container status \"d919bcdaf2ab93b06ca7d4f6d1c9fe9d09732fba59f831c3c29706bf6b3c97bd\": rpc error: code = NotFound desc = could not find container \"d919bcdaf2ab93b06ca7d4f6d1c9fe9d09732fba59f831c3c29706bf6b3c97bd\": container with ID starting with d919bcdaf2ab93b06ca7d4f6d1c9fe9d09732fba59f831c3c29706bf6b3c97bd not found: ID does not exist"
Sep 11 04:04:10.387441 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:04:10.387421    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-b61c46-5c9c66b79d-l9qzp"]
Sep 11 04:04:10.391403 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:04:10.391383    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-b61c46-5c9c66b79d-l9qzp"]
Sep 11 04:04:10.403677 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:04:10.403630    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-b61c46-d5d4b49bf-dpb5m"]
Sep 11 04:04:10.405702 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:04:10.405682    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-b61c46-d5d4b49bf-dpb5m"]
Sep 11 04:04:10.853848 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:04:10.853824    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="174bdd05-1f30-4bd1-819f-8d534357f38c" path="/var/lib/kubelet/pods/174bdd05-1f30-4bd1-819f-8d534357f38c/volumes"
Sep 11 04:04:10.854209 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:04:10.854195    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="63febe17-6300-476c-b49e-74e1098cca67" path="/var/lib/kubelet/pods/63febe17-6300-476c-b49e-74e1098cca67/volumes"
Sep 11 04:04:10.854490 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:04:10.854478    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fdc2f0d7-a540-4679-9329-b66ce8745edf" path="/var/lib/kubelet/pods/fdc2f0d7-a540-4679-9329-b66ce8745edf/volumes"
Sep 11 04:04:13.612495 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:04:13.612402    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-5z4cn"]
Sep 11 04:04:16.201486 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:04:16.201448    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-20ae9a-5cwk7"]
Sep 11 04:04:16.201933 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:04:16.201916    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="63febe17-6300-476c-b49e-74e1098cca67" containerName="payload-pre-processing"
Sep 11 04:04:16.201979 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:04:16.201935    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="63febe17-6300-476c-b49e-74e1098cca67" containerName="payload-pre-processing"
Sep 11 04:04:16.201979 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:04:16.201943    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="174bdd05-1f30-4bd1-819f-8d534357f38c" containerName="payload-processing"
Sep 11 04:04:16.201979 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:04:16.201948    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="174bdd05-1f30-4bd1-819f-8d534357f38c" containerName="payload-processing"
Sep 11 04:04:16.202077 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:04:16.202017    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="174bdd05-1f30-4bd1-819f-8d534357f38c" containerName="payload-processing"
Sep 11 04:04:16.202077 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:04:16.202028    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="63febe17-6300-476c-b49e-74e1098cca67" containerName="payload-pre-processing"
Sep 11 04:04:16.206652 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:04:16.206636    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-20ae9a-5cwk7"
Sep 11 04:04:16.210113 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:04:16.210055    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-95ckr\""
Sep 11 04:04:16.210352 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:04:16.210323    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 11 04:04:16.211899 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:04:16.211880    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-20ae9a-5cwk7"]
Sep 11 04:04:16.289007 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:04:16.288974    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5041e0a2-f404-4331-8813-3709487e74c7-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-20ae9a-5cwk7\" (UID: \"5041e0a2-f404-4331-8813-3709487e74c7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-20ae9a-5cwk7"
Sep 11 04:04:16.390230 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:04:16.390194    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5041e0a2-f404-4331-8813-3709487e74c7-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-20ae9a-5cwk7\" (UID: \"5041e0a2-f404-4331-8813-3709487e74c7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-20ae9a-5cwk7"
Sep 11 04:04:16.390830 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:04:16.390808    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5041e0a2-f404-4331-8813-3709487e74c7-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-20ae9a-5cwk7\" (UID: \"5041e0a2-f404-4331-8813-3709487e74c7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-20ae9a-5cwk7"
Sep 11 04:04:16.520236 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:04:16.520149    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-20ae9a-5cwk7"
Sep 11 04:04:16.800577 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:04:16.800553    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-20ae9a-5cwk7"]
Sep 11 04:04:16.802470 ip-10-0-141-105 kubenswrapper[2579]: W0911 04:04:16.802445    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5041e0a2_f404_4331_8813_3709487e74c7.slice/crio-ced1d013d97d62cd1885ef65d6cc33626d58817cdcf0efac997f68a327426b8e WatchSource:0}: Error finding container ced1d013d97d62cd1885ef65d6cc33626d58817cdcf0efac997f68a327426b8e: Status 404 returned error can't find the container with id ced1d013d97d62cd1885ef65d6cc33626d58817cdcf0efac997f68a327426b8e
Sep 11 04:04:17.406852 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:04:17.406761    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-20ae9a-5cwk7" event={"ID":"5041e0a2-f404-4331-8813-3709487e74c7","Type":"ContainerStarted","Data":"6525f70442185cb69de2ee37d94d3e005138e6659b2ed9fdc218ee3d5ed977ee"}
Sep 11 04:04:17.406852 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:04:17.406848    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-20ae9a-5cwk7" event={"ID":"5041e0a2-f404-4331-8813-3709487e74c7","Type":"ContainerStarted","Data":"ced1d013d97d62cd1885ef65d6cc33626d58817cdcf0efac997f68a327426b8e"}
Sep 11 04:04:17.429610 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:04:17.429562    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-20ae9a-5cwk7" podStartSLOduration=1.429544121 podStartE2EDuration="1.429544121s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 04:04:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 04:04:17.428712188 +0000 UTC m=+1435.083917756" watchObservedRunningTime="2026-09-11 04:04:17.429544121 +0000 UTC m=+1435.084749689"
Sep 11 04:04:17.907196 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:04:17.907156    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w3-dd5b30/rate-test-model-dd5b30-kserve-6cf4c47fb7-j4csz"
Sep 11 04:04:17.907458 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:04:17.907433    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w3-dd5b30/rate-test-model-dd5b30-kserve-6cf4c47fb7-j4csz"
Sep 11 04:04:17.920202 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:04:17.920172    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w3-dd5b30/rate-test-model-dd5b30-kserve-6cf4c47fb7-j4csz"
Sep 11 04:04:18.414778 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:04:18.414743    2579 generic.go:356] "Generic (PLEG): container finished" podID="5041e0a2-f404-4331-8813-3709487e74c7" containerID="6525f70442185cb69de2ee37d94d3e005138e6659b2ed9fdc218ee3d5ed977ee" exitCode=0
Sep 11 04:04:18.415271 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:04:18.414823    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-20ae9a-5cwk7" event={"ID":"5041e0a2-f404-4331-8813-3709487e74c7","Type":"ContainerDied","Data":"6525f70442185cb69de2ee37d94d3e005138e6659b2ed9fdc218ee3d5ed977ee"}
Sep 11 04:04:18.426604 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:04:18.426584    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w3-dd5b30/rate-test-model-dd5b30-kserve-6cf4c47fb7-j4csz"
Sep 11 04:04:19.559339 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:04:19.559317    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-20ae9a-5cwk7"
Sep 11 04:04:19.621142 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:04:19.621112    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5041e0a2-f404-4331-8813-3709487e74c7-maas-api-service-ca\") pod \"5041e0a2-f404-4331-8813-3709487e74c7\" (UID: \"5041e0a2-f404-4331-8813-3709487e74c7\") "
Sep 11 04:04:19.621424 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:04:19.621403    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5041e0a2-f404-4331-8813-3709487e74c7-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "5041e0a2-f404-4331-8813-3709487e74c7" (UID: "5041e0a2-f404-4331-8813-3709487e74c7"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 04:04:19.621498 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:04:19.621475    2579 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5041e0a2-f404-4331-8813-3709487e74c7-maas-api-service-ca\") on node \"ip-10-0-141-105.ec2.internal\" DevicePath \"\""
Sep 11 04:04:20.427252 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:04:20.427220    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-20ae9a-5cwk7"
Sep 11 04:04:20.427417 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:04:20.427267    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-20ae9a-5cwk7" event={"ID":"5041e0a2-f404-4331-8813-3709487e74c7","Type":"ContainerDied","Data":"ced1d013d97d62cd1885ef65d6cc33626d58817cdcf0efac997f68a327426b8e"}
Sep 11 04:04:20.427417 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:04:20.427313    2579 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ced1d013d97d62cd1885ef65d6cc33626d58817cdcf0efac997f68a327426b8e"
Sep 11 04:04:26.625456 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:04:26.625425    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-20ae9a-29818320-4gxnn"]
Sep 11 04:04:26.628590 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:04:26.628565    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-20ae9a-29818320-4gxnn"]
Sep 11 04:04:26.654542 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:04:26.654507    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-20ae9a-5ff4665549-ln6wx"]
Sep 11 04:04:26.655639 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:04:26.655580    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-20ae9a-5ff4665549-ln6wx" podUID="79b0235c-5f8a-4dd2-ab0b-e64a95a231aa" containerName="payload-processing" containerID="cri-o://96487f2505444c33875b17f0dc76ff2105f031439eac04eebc2f734b863ed505" gracePeriod=30
Sep 11 04:04:26.807760 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:04:26.807726    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-20ae9a-5cwk7"]
Sep 11 04:04:26.810286 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:04:26.810261    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-20ae9a-5cwk7"]
Sep 11 04:04:26.853818 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:04:26.853789    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5041e0a2-f404-4331-8813-3709487e74c7" path="/var/lib/kubelet/pods/5041e0a2-f404-4331-8813-3709487e74c7/volumes"
Sep 11 04:04:26.854109 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:04:26.854077    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="da6ced35-8fa1-4306-98a8-ceabd2dd9909" path="/var/lib/kubelet/pods/da6ced35-8fa1-4306-98a8-ceabd2dd9909/volumes"
Sep 11 04:04:26.931909 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:04:26.931884    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-20ae9a-5ff4665549-ln6wx"
Sep 11 04:04:27.095955 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:04:27.095920    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/79b0235c-5f8a-4dd2-ab0b-e64a95a231aa-plugins-config-volume\") pod \"79b0235c-5f8a-4dd2-ab0b-e64a95a231aa\" (UID: \"79b0235c-5f8a-4dd2-ab0b-e64a95a231aa\") "
Sep 11 04:04:27.096156 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:04:27.096040    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xqhpb\" (UniqueName: \"kubernetes.io/projected/79b0235c-5f8a-4dd2-ab0b-e64a95a231aa-kube-api-access-xqhpb\") pod \"79b0235c-5f8a-4dd2-ab0b-e64a95a231aa\" (UID: \"79b0235c-5f8a-4dd2-ab0b-e64a95a231aa\") "
Sep 11 04:04:27.096407 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:04:27.096376    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/79b0235c-5f8a-4dd2-ab0b-e64a95a231aa-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "79b0235c-5f8a-4dd2-ab0b-e64a95a231aa" (UID: "79b0235c-5f8a-4dd2-ab0b-e64a95a231aa"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 04:04:27.098265 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:04:27.098242    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/79b0235c-5f8a-4dd2-ab0b-e64a95a231aa-kube-api-access-xqhpb" (OuterVolumeSpecName: "kube-api-access-xqhpb") pod "79b0235c-5f8a-4dd2-ab0b-e64a95a231aa" (UID: "79b0235c-5f8a-4dd2-ab0b-e64a95a231aa"). InnerVolumeSpecName "kube-api-access-xqhpb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 04:04:27.197772 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:04:27.197688    2579 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/79b0235c-5f8a-4dd2-ab0b-e64a95a231aa-plugins-config-volume\") on node \"ip-10-0-141-105.ec2.internal\" DevicePath \"\""
Sep 11 04:04:27.197772 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:04:27.197720    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xqhpb\" (UniqueName: \"kubernetes.io/projected/79b0235c-5f8a-4dd2-ab0b-e64a95a231aa-kube-api-access-xqhpb\") on node \"ip-10-0-141-105.ec2.internal\" DevicePath \"\""
Sep 11 04:04:27.470489 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:04:27.470403    2579 generic.go:356] "Generic (PLEG): container finished" podID="79b0235c-5f8a-4dd2-ab0b-e64a95a231aa" containerID="96487f2505444c33875b17f0dc76ff2105f031439eac04eebc2f734b863ed505" exitCode=0
Sep 11 04:04:27.470489 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:04:27.470449    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-20ae9a-5ff4665549-ln6wx"
Sep 11 04:04:27.470726 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:04:27.470485    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-20ae9a-5ff4665549-ln6wx" event={"ID":"79b0235c-5f8a-4dd2-ab0b-e64a95a231aa","Type":"ContainerDied","Data":"96487f2505444c33875b17f0dc76ff2105f031439eac04eebc2f734b863ed505"}
Sep 11 04:04:27.470726 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:04:27.470541    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-20ae9a-5ff4665549-ln6wx" event={"ID":"79b0235c-5f8a-4dd2-ab0b-e64a95a231aa","Type":"ContainerDied","Data":"86484cf4fd599ccac817633e673d55ada5ce77977af5cefa7b4e2df6c88215f1"}
Sep 11 04:04:27.470726 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:04:27.470563    2579 scope.go:117] "RemoveContainer" containerID="96487f2505444c33875b17f0dc76ff2105f031439eac04eebc2f734b863ed505"
Sep 11 04:04:27.480136 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:04:27.480118    2579 scope.go:117] "RemoveContainer" containerID="96487f2505444c33875b17f0dc76ff2105f031439eac04eebc2f734b863ed505"
Sep 11 04:04:27.480466 ip-10-0-141-105 kubenswrapper[2579]: E0911 04:04:27.480448    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"96487f2505444c33875b17f0dc76ff2105f031439eac04eebc2f734b863ed505\": container with ID starting with 96487f2505444c33875b17f0dc76ff2105f031439eac04eebc2f734b863ed505 not found: ID does not exist" containerID="96487f2505444c33875b17f0dc76ff2105f031439eac04eebc2f734b863ed505"
Sep 11 04:04:27.480528 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:04:27.480474    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"96487f2505444c33875b17f0dc76ff2105f031439eac04eebc2f734b863ed505"} err="failed to get container status \"96487f2505444c33875b17f0dc76ff2105f031439eac04eebc2f734b863ed505\": rpc error: code = NotFound desc = could not find container \"96487f2505444c33875b17f0dc76ff2105f031439eac04eebc2f734b863ed505\": container with ID starting with 96487f2505444c33875b17f0dc76ff2105f031439eac04eebc2f734b863ed505 not found: ID does not exist"
Sep 11 04:04:27.490914 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:04:27.490879    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-20ae9a-5ff4665549-ln6wx"]
Sep 11 04:04:27.494601 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:04:27.494574    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-20ae9a-5ff4665549-ln6wx"]
Sep 11 04:04:28.852145 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:04:28.852116    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="79b0235c-5f8a-4dd2-ab0b-e64a95a231aa" path="/var/lib/kubelet/pods/79b0235c-5f8a-4dd2-ab0b-e64a95a231aa/volumes"
Sep 11 04:04:31.700612 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:04:31.700576    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-5z4cn"]
Sep 11 04:04:33.595464 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:04:33.595432    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-7dc787-x758v"]
Sep 11 04:04:33.595855 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:04:33.595841    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5041e0a2-f404-4331-8813-3709487e74c7" containerName="revoke-keys"
Sep 11 04:04:33.595900 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:04:33.595857    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="5041e0a2-f404-4331-8813-3709487e74c7" containerName="revoke-keys"
Sep 11 04:04:33.595900 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:04:33.595871    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="79b0235c-5f8a-4dd2-ab0b-e64a95a231aa" containerName="payload-processing"
Sep 11 04:04:33.595900 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:04:33.595877    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="79b0235c-5f8a-4dd2-ab0b-e64a95a231aa" containerName="payload-processing"
Sep 11 04:04:33.595999 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:04:33.595953    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="5041e0a2-f404-4331-8813-3709487e74c7" containerName="revoke-keys"
Sep 11 04:04:33.595999 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:04:33.595966    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="79b0235c-5f8a-4dd2-ab0b-e64a95a231aa" containerName="payload-processing"
Sep 11 04:04:33.600303 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:04:33.600278    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-7dc787-x758v"
Sep 11 04:04:33.603869 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:04:33.603657    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-95ckr\""
Sep 11 04:04:33.603869 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:04:33.603707    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 11 04:04:33.605214 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:04:33.605191    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-7dc787-x758v"]
Sep 11 04:04:33.755604 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:04:33.755567    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/257a7b21-1e0a-4c5d-ade8-96407e2abe2e-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-7dc787-x758v\" (UID: \"257a7b21-1e0a-4c5d-ade8-96407e2abe2e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-7dc787-x758v"
Sep 11 04:04:33.856913 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:04:33.856880    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/257a7b21-1e0a-4c5d-ade8-96407e2abe2e-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-7dc787-x758v\" (UID: \"257a7b21-1e0a-4c5d-ade8-96407e2abe2e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-7dc787-x758v"
Sep 11 04:04:33.857553 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:04:33.857534    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/257a7b21-1e0a-4c5d-ade8-96407e2abe2e-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-7dc787-x758v\" (UID: \"257a7b21-1e0a-4c5d-ade8-96407e2abe2e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-7dc787-x758v"
Sep 11 04:04:33.913982 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:04:33.913940    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-7dc787-x758v"
Sep 11 04:04:34.194129 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:04:34.194076    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-7dc787-x758v"]
Sep 11 04:04:34.196341 ip-10-0-141-105 kubenswrapper[2579]: W0911 04:04:34.196308    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod257a7b21_1e0a_4c5d_ade8_96407e2abe2e.slice/crio-284157082fc3fadf34ded61a2a2d3679dbacaad7ce24d319463b74560b0d661a WatchSource:0}: Error finding container 284157082fc3fadf34ded61a2a2d3679dbacaad7ce24d319463b74560b0d661a: Status 404 returned error can't find the container with id 284157082fc3fadf34ded61a2a2d3679dbacaad7ce24d319463b74560b0d661a
Sep 11 04:04:34.512891 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:04:34.512858    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-7dc787-x758v" event={"ID":"257a7b21-1e0a-4c5d-ade8-96407e2abe2e","Type":"ContainerStarted","Data":"284157082fc3fadf34ded61a2a2d3679dbacaad7ce24d319463b74560b0d661a"}
Sep 11 04:04:35.523121 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:04:35.523075    2579 generic.go:356] "Generic (PLEG): container finished" podID="257a7b21-1e0a-4c5d-ade8-96407e2abe2e" containerID="b43ae14c450a071162d6da7db3450bea86bff97d29d337e9b50c4be9464020ed" exitCode=0
Sep 11 04:04:35.523510 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:04:35.523158    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-7dc787-x758v" event={"ID":"257a7b21-1e0a-4c5d-ade8-96407e2abe2e","Type":"ContainerDied","Data":"b43ae14c450a071162d6da7db3450bea86bff97d29d337e9b50c4be9464020ed"}
Sep 11 04:04:36.664238 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:04:36.664213    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-7dc787-x758v"
Sep 11 04:04:36.786159 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:04:36.786048    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/257a7b21-1e0a-4c5d-ade8-96407e2abe2e-maas-api-service-ca\") pod \"257a7b21-1e0a-4c5d-ade8-96407e2abe2e\" (UID: \"257a7b21-1e0a-4c5d-ade8-96407e2abe2e\") "
Sep 11 04:04:36.786454 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:04:36.786423    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/257a7b21-1e0a-4c5d-ade8-96407e2abe2e-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "257a7b21-1e0a-4c5d-ade8-96407e2abe2e" (UID: "257a7b21-1e0a-4c5d-ade8-96407e2abe2e"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 04:04:36.886933 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:04:36.886900    2579 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/257a7b21-1e0a-4c5d-ade8-96407e2abe2e-maas-api-service-ca\") on node \"ip-10-0-141-105.ec2.internal\" DevicePath \"\""
Sep 11 04:04:37.535456 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:04:37.535420    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-7dc787-x758v" event={"ID":"257a7b21-1e0a-4c5d-ade8-96407e2abe2e","Type":"ContainerDied","Data":"284157082fc3fadf34ded61a2a2d3679dbacaad7ce24d319463b74560b0d661a"}
Sep 11 04:04:37.535636 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:04:37.535475    2579 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="284157082fc3fadf34ded61a2a2d3679dbacaad7ce24d319463b74560b0d661a"
Sep 11 04:04:37.535636 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:04:37.535438    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-7dc787-x758v"
Sep 11 04:04:44.053026 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:04:44.052989    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-7dc787-744d659f5-vv99b"]
Sep 11 04:04:44.053607 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:04:44.053266    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-7dc787-744d659f5-vv99b" podUID="14ea3d95-462e-4f0e-aee3-d37f36f30204" containerName="payload-processing" containerID="cri-o://3abf8ac2ef7a0f815e39dbfbae7548e989bd354f6ab94501abebe2c3d3475cda" gracePeriod=30
Sep 11 04:04:44.060274 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:04:44.060241    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-7dc787-d9778787d-k5mkc"]
Sep 11 04:04:44.060509 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:04:44.060485    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-7dc787-d9778787d-k5mkc" podUID="ddaa3ca0-9cba-48a7-8e0b-3f7881aa7e81" containerName="payload-pre-processing" containerID="cri-o://1c89fe0c455a1c964747abf4745b866c6ada2250f98784b1e73cf26295591d93" gracePeriod=30
Sep 11 04:04:44.201805 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:04:44.201767    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-7dc787-x758v"]
Sep 11 04:04:44.205471 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:04:44.205441    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-7dc787-x758v"]
Sep 11 04:04:44.314592 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:04:44.314532    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-7dc787-744d659f5-vv99b"
Sep 11 04:04:44.346158 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:04:44.346135    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-7dc787-d9778787d-k5mkc"
Sep 11 04:04:44.357257 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:04:44.357234    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/14ea3d95-462e-4f0e-aee3-d37f36f30204-plugins-config-volume\") pod \"14ea3d95-462e-4f0e-aee3-d37f36f30204\" (UID: \"14ea3d95-462e-4f0e-aee3-d37f36f30204\") "
Sep 11 04:04:44.357372 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:04:44.357302    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dkgmg\" (UniqueName: \"kubernetes.io/projected/14ea3d95-462e-4f0e-aee3-d37f36f30204-kube-api-access-dkgmg\") pod \"14ea3d95-462e-4f0e-aee3-d37f36f30204\" (UID: \"14ea3d95-462e-4f0e-aee3-d37f36f30204\") "
Sep 11 04:04:44.357590 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:04:44.357564    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/14ea3d95-462e-4f0e-aee3-d37f36f30204-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "14ea3d95-462e-4f0e-aee3-d37f36f30204" (UID: "14ea3d95-462e-4f0e-aee3-d37f36f30204"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 04:04:44.359523 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:04:44.359497    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/14ea3d95-462e-4f0e-aee3-d37f36f30204-kube-api-access-dkgmg" (OuterVolumeSpecName: "kube-api-access-dkgmg") pod "14ea3d95-462e-4f0e-aee3-d37f36f30204" (UID: "14ea3d95-462e-4f0e-aee3-d37f36f30204"). InnerVolumeSpecName "kube-api-access-dkgmg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 04:04:44.458043 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:04:44.458014    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ddaa3ca0-9cba-48a7-8e0b-3f7881aa7e81-plugins-config-volume\") pod \"ddaa3ca0-9cba-48a7-8e0b-3f7881aa7e81\" (UID: \"ddaa3ca0-9cba-48a7-8e0b-3f7881aa7e81\") "
Sep 11 04:04:44.458043 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:04:44.458049    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-j2s2j\" (UniqueName: \"kubernetes.io/projected/ddaa3ca0-9cba-48a7-8e0b-3f7881aa7e81-kube-api-access-j2s2j\") pod \"ddaa3ca0-9cba-48a7-8e0b-3f7881aa7e81\" (UID: \"ddaa3ca0-9cba-48a7-8e0b-3f7881aa7e81\") "
Sep 11 04:04:44.458337 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:04:44.458322    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dkgmg\" (UniqueName: \"kubernetes.io/projected/14ea3d95-462e-4f0e-aee3-d37f36f30204-kube-api-access-dkgmg\") on node \"ip-10-0-141-105.ec2.internal\" DevicePath \"\""
Sep 11 04:04:44.458376 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:04:44.458340    2579 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/14ea3d95-462e-4f0e-aee3-d37f36f30204-plugins-config-volume\") on node \"ip-10-0-141-105.ec2.internal\" DevicePath \"\""
Sep 11 04:04:44.458465 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:04:44.458428    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ddaa3ca0-9cba-48a7-8e0b-3f7881aa7e81-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "ddaa3ca0-9cba-48a7-8e0b-3f7881aa7e81" (UID: "ddaa3ca0-9cba-48a7-8e0b-3f7881aa7e81"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 04:04:44.460248 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:04:44.460228    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ddaa3ca0-9cba-48a7-8e0b-3f7881aa7e81-kube-api-access-j2s2j" (OuterVolumeSpecName: "kube-api-access-j2s2j") pod "ddaa3ca0-9cba-48a7-8e0b-3f7881aa7e81" (UID: "ddaa3ca0-9cba-48a7-8e0b-3f7881aa7e81"). InnerVolumeSpecName "kube-api-access-j2s2j". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 04:04:44.559356 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:04:44.559320    2579 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ddaa3ca0-9cba-48a7-8e0b-3f7881aa7e81-plugins-config-volume\") on node \"ip-10-0-141-105.ec2.internal\" DevicePath \"\""
Sep 11 04:04:44.559356 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:04:44.559349    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j2s2j\" (UniqueName: \"kubernetes.io/projected/ddaa3ca0-9cba-48a7-8e0b-3f7881aa7e81-kube-api-access-j2s2j\") on node \"ip-10-0-141-105.ec2.internal\" DevicePath \"\""
Sep 11 04:04:44.576357 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:04:44.576257    2579 generic.go:356] "Generic (PLEG): container finished" podID="14ea3d95-462e-4f0e-aee3-d37f36f30204" containerID="3abf8ac2ef7a0f815e39dbfbae7548e989bd354f6ab94501abebe2c3d3475cda" exitCode=0
Sep 11 04:04:44.576357 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:04:44.576309    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-7dc787-744d659f5-vv99b"
Sep 11 04:04:44.576357 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:04:44.576334    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-7dc787-744d659f5-vv99b" event={"ID":"14ea3d95-462e-4f0e-aee3-d37f36f30204","Type":"ContainerDied","Data":"3abf8ac2ef7a0f815e39dbfbae7548e989bd354f6ab94501abebe2c3d3475cda"}
Sep 11 04:04:44.576577 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:04:44.576393    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-7dc787-744d659f5-vv99b" event={"ID":"14ea3d95-462e-4f0e-aee3-d37f36f30204","Type":"ContainerDied","Data":"014fa3b164f7d3fabe0c621efb76d3c623a1ed5db403a7b6a06c78b6eb67e882"}
Sep 11 04:04:44.576577 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:04:44.576415    2579 scope.go:117] "RemoveContainer" containerID="3abf8ac2ef7a0f815e39dbfbae7548e989bd354f6ab94501abebe2c3d3475cda"
Sep 11 04:04:44.577824 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:04:44.577796    2579 generic.go:356] "Generic (PLEG): container finished" podID="ddaa3ca0-9cba-48a7-8e0b-3f7881aa7e81" containerID="1c89fe0c455a1c964747abf4745b866c6ada2250f98784b1e73cf26295591d93" exitCode=0
Sep 11 04:04:44.577943 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:04:44.577831    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-7dc787-d9778787d-k5mkc"
Sep 11 04:04:44.577943 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:04:44.577881    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-7dc787-d9778787d-k5mkc" event={"ID":"ddaa3ca0-9cba-48a7-8e0b-3f7881aa7e81","Type":"ContainerDied","Data":"1c89fe0c455a1c964747abf4745b866c6ada2250f98784b1e73cf26295591d93"}
Sep 11 04:04:44.577943 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:04:44.577925    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-7dc787-d9778787d-k5mkc" event={"ID":"ddaa3ca0-9cba-48a7-8e0b-3f7881aa7e81","Type":"ContainerDied","Data":"002978a6c1e580de02ad200bea444d7ecff7917c731408b704735b70560638b9"}
Sep 11 04:04:44.587283 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:04:44.587251    2579 scope.go:117] "RemoveContainer" containerID="3abf8ac2ef7a0f815e39dbfbae7548e989bd354f6ab94501abebe2c3d3475cda"
Sep 11 04:04:44.587598 ip-10-0-141-105 kubenswrapper[2579]: E0911 04:04:44.587573    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3abf8ac2ef7a0f815e39dbfbae7548e989bd354f6ab94501abebe2c3d3475cda\": container with ID starting with 3abf8ac2ef7a0f815e39dbfbae7548e989bd354f6ab94501abebe2c3d3475cda not found: ID does not exist" containerID="3abf8ac2ef7a0f815e39dbfbae7548e989bd354f6ab94501abebe2c3d3475cda"
Sep 11 04:04:44.587674 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:04:44.587612    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3abf8ac2ef7a0f815e39dbfbae7548e989bd354f6ab94501abebe2c3d3475cda"} err="failed to get container status \"3abf8ac2ef7a0f815e39dbfbae7548e989bd354f6ab94501abebe2c3d3475cda\": rpc error: code = NotFound desc = could not find container \"3abf8ac2ef7a0f815e39dbfbae7548e989bd354f6ab94501abebe2c3d3475cda\": container with ID starting with 3abf8ac2ef7a0f815e39dbfbae7548e989bd354f6ab94501abebe2c3d3475cda not found: ID does not exist"
Sep 11 04:04:44.587674 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:04:44.587638    2579 scope.go:117] "RemoveContainer" containerID="1c89fe0c455a1c964747abf4745b866c6ada2250f98784b1e73cf26295591d93"
Sep 11 04:04:44.596982 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:04:44.596965    2579 scope.go:117] "RemoveContainer" containerID="1c89fe0c455a1c964747abf4745b866c6ada2250f98784b1e73cf26295591d93"
Sep 11 04:04:44.597399 ip-10-0-141-105 kubenswrapper[2579]: E0911 04:04:44.597370    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1c89fe0c455a1c964747abf4745b866c6ada2250f98784b1e73cf26295591d93\": container with ID starting with 1c89fe0c455a1c964747abf4745b866c6ada2250f98784b1e73cf26295591d93 not found: ID does not exist" containerID="1c89fe0c455a1c964747abf4745b866c6ada2250f98784b1e73cf26295591d93"
Sep 11 04:04:44.597528 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:04:44.597417    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1c89fe0c455a1c964747abf4745b866c6ada2250f98784b1e73cf26295591d93"} err="failed to get container status \"1c89fe0c455a1c964747abf4745b866c6ada2250f98784b1e73cf26295591d93\": rpc error: code = NotFound desc = could not find container \"1c89fe0c455a1c964747abf4745b866c6ada2250f98784b1e73cf26295591d93\": container with ID starting with 1c89fe0c455a1c964747abf4745b866c6ada2250f98784b1e73cf26295591d93 not found: ID does not exist"
Sep 11 04:04:44.605015 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:04:44.604995    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-7dc787-744d659f5-vv99b"]
Sep 11 04:04:44.610076 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:04:44.610054    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-7dc787-744d659f5-vv99b"]
Sep 11 04:04:44.614843 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:04:44.614823    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-5z4cn"]
Sep 11 04:04:44.621937 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:04:44.621918    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-7dc787-d9778787d-k5mkc"]
Sep 11 04:04:44.630009 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:04:44.629985    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-7dc787-d9778787d-k5mkc"]
Sep 11 04:04:44.852218 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:04:44.852140    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="14ea3d95-462e-4f0e-aee3-d37f36f30204" path="/var/lib/kubelet/pods/14ea3d95-462e-4f0e-aee3-d37f36f30204/volumes"
Sep 11 04:04:44.852469 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:04:44.852456    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="257a7b21-1e0a-4c5d-ade8-96407e2abe2e" path="/var/lib/kubelet/pods/257a7b21-1e0a-4c5d-ade8-96407e2abe2e/volumes"
Sep 11 04:04:44.852770 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:04:44.852759    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ddaa3ca0-9cba-48a7-8e0b-3f7881aa7e81" path="/var/lib/kubelet/pods/ddaa3ca0-9cba-48a7-8e0b-3f7881aa7e81/volumes"
Sep 11 04:04:51.381137 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:04:51.381104    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-b4d3cf/rate-test-model-b4d3cf-kserve-6f654d7c9f-hxsxq"]
Sep 11 04:04:52.290445 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:04:52.290400    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-dd5b30/rate-test-model-dd5b30-kserve-6cf4c47fb7-j4csz"]
Sep 11 04:04:55.701510 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:04:55.701473    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-5z4cn"]
Sep 11 04:05:06.300770 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:05:06.300733    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-5z4cn"]
Sep 11 04:05:06.401379 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:05:06.401317    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w3-b4d3cf/rate-test-model-b4d3cf-kserve-6f654d7c9f-hxsxq" podUID="d416cf69-55ae-4a8f-bd7e-e049dac3a73e" containerName="main" containerID="cri-o://dcac2bcbed732bc9d9a54cc677c781eb2aab8ea5013107d5eaff1ed676510f0d" gracePeriod=45
Sep 11 04:05:06.659888 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:05:06.659862    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-b4d3cf/rate-test-model-b4d3cf-kserve-6f654d7c9f-hxsxq"
Sep 11 04:05:06.705985 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:05:06.705951    2579 generic.go:356] "Generic (PLEG): container finished" podID="d416cf69-55ae-4a8f-bd7e-e049dac3a73e" containerID="dcac2bcbed732bc9d9a54cc677c781eb2aab8ea5013107d5eaff1ed676510f0d" exitCode=0
Sep 11 04:05:06.706172 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:05:06.706004    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-b4d3cf/rate-test-model-b4d3cf-kserve-6f654d7c9f-hxsxq"
Sep 11 04:05:06.706172 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:05:06.706042    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-b4d3cf/rate-test-model-b4d3cf-kserve-6f654d7c9f-hxsxq" event={"ID":"d416cf69-55ae-4a8f-bd7e-e049dac3a73e","Type":"ContainerDied","Data":"dcac2bcbed732bc9d9a54cc677c781eb2aab8ea5013107d5eaff1ed676510f0d"}
Sep 11 04:05:06.706172 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:05:06.706104    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-b4d3cf/rate-test-model-b4d3cf-kserve-6f654d7c9f-hxsxq" event={"ID":"d416cf69-55ae-4a8f-bd7e-e049dac3a73e","Type":"ContainerDied","Data":"2601453be5c33ca786b1ed2d031a3e927931d7444dc873df733ee0a6cb0c2466"}
Sep 11 04:05:06.706172 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:05:06.706126    2579 scope.go:117] "RemoveContainer" containerID="dcac2bcbed732bc9d9a54cc677c781eb2aab8ea5013107d5eaff1ed676510f0d"
Sep 11 04:05:06.715389 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:05:06.715370    2579 scope.go:117] "RemoveContainer" containerID="dcac2bcbed732bc9d9a54cc677c781eb2aab8ea5013107d5eaff1ed676510f0d"
Sep 11 04:05:06.715704 ip-10-0-141-105 kubenswrapper[2579]: E0911 04:05:06.715686    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dcac2bcbed732bc9d9a54cc677c781eb2aab8ea5013107d5eaff1ed676510f0d\": container with ID starting with dcac2bcbed732bc9d9a54cc677c781eb2aab8ea5013107d5eaff1ed676510f0d not found: ID does not exist" containerID="dcac2bcbed732bc9d9a54cc677c781eb2aab8ea5013107d5eaff1ed676510f0d"
Sep 11 04:05:06.715796 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:05:06.715712    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dcac2bcbed732bc9d9a54cc677c781eb2aab8ea5013107d5eaff1ed676510f0d"} err="failed to get container status \"dcac2bcbed732bc9d9a54cc677c781eb2aab8ea5013107d5eaff1ed676510f0d\": rpc error: code = NotFound desc = could not find container \"dcac2bcbed732bc9d9a54cc677c781eb2aab8ea5013107d5eaff1ed676510f0d\": container with ID starting with dcac2bcbed732bc9d9a54cc677c781eb2aab8ea5013107d5eaff1ed676510f0d not found: ID does not exist"
Sep 11 04:05:06.769257 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:05:06.769230    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d416cf69-55ae-4a8f-bd7e-e049dac3a73e-home\") pod \"d416cf69-55ae-4a8f-bd7e-e049dac3a73e\" (UID: \"d416cf69-55ae-4a8f-bd7e-e049dac3a73e\") "
Sep 11 04:05:06.769422 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:05:06.769274    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d416cf69-55ae-4a8f-bd7e-e049dac3a73e-model-cache\") pod \"d416cf69-55ae-4a8f-bd7e-e049dac3a73e\" (UID: \"d416cf69-55ae-4a8f-bd7e-e049dac3a73e\") "
Sep 11 04:05:06.769422 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:05:06.769314    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6rqbk\" (UniqueName: \"kubernetes.io/projected/d416cf69-55ae-4a8f-bd7e-e049dac3a73e-kube-api-access-6rqbk\") pod \"d416cf69-55ae-4a8f-bd7e-e049dac3a73e\" (UID: \"d416cf69-55ae-4a8f-bd7e-e049dac3a73e\") "
Sep 11 04:05:06.769422 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:05:06.769351    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d416cf69-55ae-4a8f-bd7e-e049dac3a73e-dshm\") pod \"d416cf69-55ae-4a8f-bd7e-e049dac3a73e\" (UID: \"d416cf69-55ae-4a8f-bd7e-e049dac3a73e\") "
Sep 11 04:05:06.769422 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:05:06.769384    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d416cf69-55ae-4a8f-bd7e-e049dac3a73e-tls-certs\") pod \"d416cf69-55ae-4a8f-bd7e-e049dac3a73e\" (UID: \"d416cf69-55ae-4a8f-bd7e-e049dac3a73e\") "
Sep 11 04:05:06.769422 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:05:06.769410    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d416cf69-55ae-4a8f-bd7e-e049dac3a73e-tmp-dir\") pod \"d416cf69-55ae-4a8f-bd7e-e049dac3a73e\" (UID: \"d416cf69-55ae-4a8f-bd7e-e049dac3a73e\") "
Sep 11 04:05:06.769743 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:05:06.769519    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d416cf69-55ae-4a8f-bd7e-e049dac3a73e-home" (OuterVolumeSpecName: "home") pod "d416cf69-55ae-4a8f-bd7e-e049dac3a73e" (UID: "d416cf69-55ae-4a8f-bd7e-e049dac3a73e"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 04:05:06.769743 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:05:06.769629    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d416cf69-55ae-4a8f-bd7e-e049dac3a73e-model-cache" (OuterVolumeSpecName: "model-cache") pod "d416cf69-55ae-4a8f-bd7e-e049dac3a73e" (UID: "d416cf69-55ae-4a8f-bd7e-e049dac3a73e"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 04:05:06.769743 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:05:06.769701    2579 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d416cf69-55ae-4a8f-bd7e-e049dac3a73e-home\") on node \"ip-10-0-141-105.ec2.internal\" DevicePath \"\""
Sep 11 04:05:06.769920 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:05:06.769791    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d416cf69-55ae-4a8f-bd7e-e049dac3a73e-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "d416cf69-55ae-4a8f-bd7e-e049dac3a73e" (UID: "d416cf69-55ae-4a8f-bd7e-e049dac3a73e"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 04:05:06.771651 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:05:06.771623    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d416cf69-55ae-4a8f-bd7e-e049dac3a73e-dshm" (OuterVolumeSpecName: "dshm") pod "d416cf69-55ae-4a8f-bd7e-e049dac3a73e" (UID: "d416cf69-55ae-4a8f-bd7e-e049dac3a73e"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 04:05:06.771782 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:05:06.771703    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d416cf69-55ae-4a8f-bd7e-e049dac3a73e-kube-api-access-6rqbk" (OuterVolumeSpecName: "kube-api-access-6rqbk") pod "d416cf69-55ae-4a8f-bd7e-e049dac3a73e" (UID: "d416cf69-55ae-4a8f-bd7e-e049dac3a73e"). InnerVolumeSpecName "kube-api-access-6rqbk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 04:05:06.771867 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:05:06.771842    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d416cf69-55ae-4a8f-bd7e-e049dac3a73e-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "d416cf69-55ae-4a8f-bd7e-e049dac3a73e" (UID: "d416cf69-55ae-4a8f-bd7e-e049dac3a73e"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 04:05:06.871108 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:05:06.871052    2579 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d416cf69-55ae-4a8f-bd7e-e049dac3a73e-model-cache\") on node \"ip-10-0-141-105.ec2.internal\" DevicePath \"\""
Sep 11 04:05:06.871108 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:05:06.871095    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6rqbk\" (UniqueName: \"kubernetes.io/projected/d416cf69-55ae-4a8f-bd7e-e049dac3a73e-kube-api-access-6rqbk\") on node \"ip-10-0-141-105.ec2.internal\" DevicePath \"\""
Sep 11 04:05:06.871108 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:05:06.871110    2579 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d416cf69-55ae-4a8f-bd7e-e049dac3a73e-dshm\") on node \"ip-10-0-141-105.ec2.internal\" DevicePath \"\""
Sep 11 04:05:06.871308 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:05:06.871123    2579 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d416cf69-55ae-4a8f-bd7e-e049dac3a73e-tls-certs\") on node \"ip-10-0-141-105.ec2.internal\" DevicePath \"\""
Sep 11 04:05:06.871308 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:05:06.871135    2579 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d416cf69-55ae-4a8f-bd7e-e049dac3a73e-tmp-dir\") on node \"ip-10-0-141-105.ec2.internal\" DevicePath \"\""
Sep 11 04:05:07.022592 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:05:07.022561    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-b4d3cf/rate-test-model-b4d3cf-kserve-6f654d7c9f-hxsxq"]
Sep 11 04:05:07.025683 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:05:07.025660    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w3-b4d3cf/rate-test-model-b4d3cf-kserve-6f654d7c9f-hxsxq"]
Sep 11 04:05:07.313492 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:05:07.313383    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w3-dd5b30/rate-test-model-dd5b30-kserve-6cf4c47fb7-j4csz" podUID="82bbb090-3ff4-4a87-a9e0-ef14c3b22b05" containerName="main" containerID="cri-o://ad3fe27bb5d60bd09975d849197a49bc89f17e1c1a395e10d7975592fc26d3d1" gracePeriod=45
Sep 11 04:05:07.576010 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:05:07.575953    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-dd5b30/rate-test-model-dd5b30-kserve-6cf4c47fb7-j4csz"
Sep 11 04:05:07.677376 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:05:07.677343    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/82bbb090-3ff4-4a87-a9e0-ef14c3b22b05-tls-certs\") pod \"82bbb090-3ff4-4a87-a9e0-ef14c3b22b05\" (UID: \"82bbb090-3ff4-4a87-a9e0-ef14c3b22b05\") "
Sep 11 04:05:07.677564 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:05:07.677390    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/82bbb090-3ff4-4a87-a9e0-ef14c3b22b05-dshm\") pod \"82bbb090-3ff4-4a87-a9e0-ef14c3b22b05\" (UID: \"82bbb090-3ff4-4a87-a9e0-ef14c3b22b05\") "
Sep 11 04:05:07.677564 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:05:07.677433    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/82bbb090-3ff4-4a87-a9e0-ef14c3b22b05-tmp-dir\") pod \"82bbb090-3ff4-4a87-a9e0-ef14c3b22b05\" (UID: \"82bbb090-3ff4-4a87-a9e0-ef14c3b22b05\") "
Sep 11 04:05:07.677564 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:05:07.677494    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/82bbb090-3ff4-4a87-a9e0-ef14c3b22b05-home\") pod \"82bbb090-3ff4-4a87-a9e0-ef14c3b22b05\" (UID: \"82bbb090-3ff4-4a87-a9e0-ef14c3b22b05\") "
Sep 11 04:05:07.677564 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:05:07.677520    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-v986h\" (UniqueName: \"kubernetes.io/projected/82bbb090-3ff4-4a87-a9e0-ef14c3b22b05-kube-api-access-v986h\") pod \"82bbb090-3ff4-4a87-a9e0-ef14c3b22b05\" (UID: \"82bbb090-3ff4-4a87-a9e0-ef14c3b22b05\") "
Sep 11 04:05:07.677564 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:05:07.677549    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/82bbb090-3ff4-4a87-a9e0-ef14c3b22b05-model-cache\") pod \"82bbb090-3ff4-4a87-a9e0-ef14c3b22b05\" (UID: \"82bbb090-3ff4-4a87-a9e0-ef14c3b22b05\") "
Sep 11 04:05:07.677828 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:05:07.677737    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/82bbb090-3ff4-4a87-a9e0-ef14c3b22b05-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "82bbb090-3ff4-4a87-a9e0-ef14c3b22b05" (UID: "82bbb090-3ff4-4a87-a9e0-ef14c3b22b05"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 04:05:07.677828 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:05:07.677755    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/82bbb090-3ff4-4a87-a9e0-ef14c3b22b05-home" (OuterVolumeSpecName: "home") pod "82bbb090-3ff4-4a87-a9e0-ef14c3b22b05" (UID: "82bbb090-3ff4-4a87-a9e0-ef14c3b22b05"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 04:05:07.677946 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:05:07.677894    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/82bbb090-3ff4-4a87-a9e0-ef14c3b22b05-model-cache" (OuterVolumeSpecName: "model-cache") pod "82bbb090-3ff4-4a87-a9e0-ef14c3b22b05" (UID: "82bbb090-3ff4-4a87-a9e0-ef14c3b22b05"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 04:05:07.679856 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:05:07.679824    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/82bbb090-3ff4-4a87-a9e0-ef14c3b22b05-dshm" (OuterVolumeSpecName: "dshm") pod "82bbb090-3ff4-4a87-a9e0-ef14c3b22b05" (UID: "82bbb090-3ff4-4a87-a9e0-ef14c3b22b05"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 04:05:07.679856 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:05:07.679839    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/82bbb090-3ff4-4a87-a9e0-ef14c3b22b05-kube-api-access-v986h" (OuterVolumeSpecName: "kube-api-access-v986h") pod "82bbb090-3ff4-4a87-a9e0-ef14c3b22b05" (UID: "82bbb090-3ff4-4a87-a9e0-ef14c3b22b05"). InnerVolumeSpecName "kube-api-access-v986h". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 04:05:07.680028 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:05:07.679896    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/82bbb090-3ff4-4a87-a9e0-ef14c3b22b05-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "82bbb090-3ff4-4a87-a9e0-ef14c3b22b05" (UID: "82bbb090-3ff4-4a87-a9e0-ef14c3b22b05"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 04:05:07.713419 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:05:07.713394    2579 generic.go:356] "Generic (PLEG): container finished" podID="82bbb090-3ff4-4a87-a9e0-ef14c3b22b05" containerID="ad3fe27bb5d60bd09975d849197a49bc89f17e1c1a395e10d7975592fc26d3d1" exitCode=0
Sep 11 04:05:07.713562 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:05:07.713445    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-dd5b30/rate-test-model-dd5b30-kserve-6cf4c47fb7-j4csz"
Sep 11 04:05:07.713562 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:05:07.713477    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-dd5b30/rate-test-model-dd5b30-kserve-6cf4c47fb7-j4csz" event={"ID":"82bbb090-3ff4-4a87-a9e0-ef14c3b22b05","Type":"ContainerDied","Data":"ad3fe27bb5d60bd09975d849197a49bc89f17e1c1a395e10d7975592fc26d3d1"}
Sep 11 04:05:07.713562 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:05:07.713517    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-dd5b30/rate-test-model-dd5b30-kserve-6cf4c47fb7-j4csz" event={"ID":"82bbb090-3ff4-4a87-a9e0-ef14c3b22b05","Type":"ContainerDied","Data":"6f172d25714ab98e50935a90880588bcb6fb22f368155eeb5771ad30def550a1"}
Sep 11 04:05:07.713562 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:05:07.713536    2579 scope.go:117] "RemoveContainer" containerID="ad3fe27bb5d60bd09975d849197a49bc89f17e1c1a395e10d7975592fc26d3d1"
Sep 11 04:05:07.725988 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:05:07.725967    2579 scope.go:117] "RemoveContainer" containerID="ad3fe27bb5d60bd09975d849197a49bc89f17e1c1a395e10d7975592fc26d3d1"
Sep 11 04:05:07.726280 ip-10-0-141-105 kubenswrapper[2579]: E0911 04:05:07.726262    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ad3fe27bb5d60bd09975d849197a49bc89f17e1c1a395e10d7975592fc26d3d1\": container with ID starting with ad3fe27bb5d60bd09975d849197a49bc89f17e1c1a395e10d7975592fc26d3d1 not found: ID does not exist" containerID="ad3fe27bb5d60bd09975d849197a49bc89f17e1c1a395e10d7975592fc26d3d1"
Sep 11 04:05:07.726359 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:05:07.726292    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ad3fe27bb5d60bd09975d849197a49bc89f17e1c1a395e10d7975592fc26d3d1"} err="failed to get container status \"ad3fe27bb5d60bd09975d849197a49bc89f17e1c1a395e10d7975592fc26d3d1\": rpc error: code = NotFound desc = could not find container \"ad3fe27bb5d60bd09975d849197a49bc89f17e1c1a395e10d7975592fc26d3d1\": container with ID starting with ad3fe27bb5d60bd09975d849197a49bc89f17e1c1a395e10d7975592fc26d3d1 not found: ID does not exist"
Sep 11 04:05:07.736705 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:05:07.736677    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-dd5b30/rate-test-model-dd5b30-kserve-6cf4c47fb7-j4csz"]
Sep 11 04:05:07.741590 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:05:07.741571    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w3-dd5b30/rate-test-model-dd5b30-kserve-6cf4c47fb7-j4csz"]
Sep 11 04:05:07.779144 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:05:07.779117    2579 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/82bbb090-3ff4-4a87-a9e0-ef14c3b22b05-home\") on node \"ip-10-0-141-105.ec2.internal\" DevicePath \"\""
Sep 11 04:05:07.779144 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:05:07.779143    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v986h\" (UniqueName: \"kubernetes.io/projected/82bbb090-3ff4-4a87-a9e0-ef14c3b22b05-kube-api-access-v986h\") on node \"ip-10-0-141-105.ec2.internal\" DevicePath \"\""
Sep 11 04:05:07.779296 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:05:07.779158    2579 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/82bbb090-3ff4-4a87-a9e0-ef14c3b22b05-model-cache\") on node \"ip-10-0-141-105.ec2.internal\" DevicePath \"\""
Sep 11 04:05:07.779296 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:05:07.779170    2579 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/82bbb090-3ff4-4a87-a9e0-ef14c3b22b05-tls-certs\") on node \"ip-10-0-141-105.ec2.internal\" DevicePath \"\""
Sep 11 04:05:07.779296 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:05:07.779181    2579 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/82bbb090-3ff4-4a87-a9e0-ef14c3b22b05-dshm\") on node \"ip-10-0-141-105.ec2.internal\" DevicePath \"\""
Sep 11 04:05:07.779296 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:05:07.779192    2579 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/82bbb090-3ff4-4a87-a9e0-ef14c3b22b05-tmp-dir\") on node \"ip-10-0-141-105.ec2.internal\" DevicePath \"\""
Sep 11 04:05:08.852327 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:05:08.852280    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="82bbb090-3ff4-4a87-a9e0-ef14c3b22b05" path="/var/lib/kubelet/pods/82bbb090-3ff4-4a87-a9e0-ef14c3b22b05/volumes"
Sep 11 04:05:08.852712 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:05:08.852702    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d416cf69-55ae-4a8f-bd7e-e049dac3a73e" path="/var/lib/kubelet/pods/d416cf69-55ae-4a8f-bd7e-e049dac3a73e/volumes"
Sep 11 04:05:21.918410 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:05:21.918375    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-b-0f85b8-openshift-default-7d957b55c4-pskrc"]
Sep 11 04:05:21.918928 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:05:21.918876    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="14ea3d95-462e-4f0e-aee3-d37f36f30204" containerName="payload-processing"
Sep 11 04:05:21.918928 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:05:21.918896    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="14ea3d95-462e-4f0e-aee3-d37f36f30204" containerName="payload-processing"
Sep 11 04:05:21.918928 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:05:21.918911    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d416cf69-55ae-4a8f-bd7e-e049dac3a73e" containerName="main"
Sep 11 04:05:21.918928 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:05:21.918919    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="d416cf69-55ae-4a8f-bd7e-e049dac3a73e" containerName="main"
Sep 11 04:05:21.919210 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:05:21.918940    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="82bbb090-3ff4-4a87-a9e0-ef14c3b22b05" containerName="main"
Sep 11 04:05:21.919210 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:05:21.918949    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="82bbb090-3ff4-4a87-a9e0-ef14c3b22b05" containerName="main"
Sep 11 04:05:21.919210 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:05:21.918978    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ddaa3ca0-9cba-48a7-8e0b-3f7881aa7e81" containerName="payload-pre-processing"
Sep 11 04:05:21.919210 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:05:21.918986    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="ddaa3ca0-9cba-48a7-8e0b-3f7881aa7e81" containerName="payload-pre-processing"
Sep 11 04:05:21.919210 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:05:21.918996    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="257a7b21-1e0a-4c5d-ade8-96407e2abe2e" containerName="revoke-keys"
Sep 11 04:05:21.919210 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:05:21.919004    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="257a7b21-1e0a-4c5d-ade8-96407e2abe2e" containerName="revoke-keys"
Sep 11 04:05:21.919210 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:05:21.919111    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="257a7b21-1e0a-4c5d-ade8-96407e2abe2e" containerName="revoke-keys"
Sep 11 04:05:21.919210 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:05:21.919121    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="82bbb090-3ff4-4a87-a9e0-ef14c3b22b05" containerName="main"
Sep 11 04:05:21.919210 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:05:21.919129    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="d416cf69-55ae-4a8f-bd7e-e049dac3a73e" containerName="main"
Sep 11 04:05:21.919210 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:05:21.919135    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="ddaa3ca0-9cba-48a7-8e0b-3f7881aa7e81" containerName="payload-pre-processing"
Sep 11 04:05:21.919210 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:05:21.919143    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="14ea3d95-462e-4f0e-aee3-d37f36f30204" containerName="payload-processing"
Sep 11 04:05:21.922222 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:05:21.922201    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-0f85b8-openshift-default-7d957b55c4-pskrc"
Sep 11 04:05:21.924740 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:05:21.924713    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-b-0f85b8-openshift-default-dockercfg-dd95b\""
Sep 11 04:05:21.934300 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:05:21.934275    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-0f85b8-openshift-default-7d957b55c4-pskrc"]
Sep 11 04:05:22.013328 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:05:22.013296    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ceb39acb-e2b8-4a34-8936-6c36677fa484-istio-envoy\") pod \"e2e-ipp-b-0f85b8-openshift-default-7d957b55c4-pskrc\" (UID: \"ceb39acb-e2b8-4a34-8936-6c36677fa484\") " pod="openshift-ingress/e2e-ipp-b-0f85b8-openshift-default-7d957b55c4-pskrc"
Sep 11 04:05:22.013538 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:05:22.013340    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4dnxf\" (UniqueName: \"kubernetes.io/projected/ceb39acb-e2b8-4a34-8936-6c36677fa484-kube-api-access-4dnxf\") pod \"e2e-ipp-b-0f85b8-openshift-default-7d957b55c4-pskrc\" (UID: \"ceb39acb-e2b8-4a34-8936-6c36677fa484\") " pod="openshift-ingress/e2e-ipp-b-0f85b8-openshift-default-7d957b55c4-pskrc"
Sep 11 04:05:22.013538 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:05:22.013403    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ceb39acb-e2b8-4a34-8936-6c36677fa484-istio-podinfo\") pod \"e2e-ipp-b-0f85b8-openshift-default-7d957b55c4-pskrc\" (UID: \"ceb39acb-e2b8-4a34-8936-6c36677fa484\") " pod="openshift-ingress/e2e-ipp-b-0f85b8-openshift-default-7d957b55c4-pskrc"
Sep 11 04:05:22.013538 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:05:22.013440    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ceb39acb-e2b8-4a34-8936-6c36677fa484-istio-token\") pod \"e2e-ipp-b-0f85b8-openshift-default-7d957b55c4-pskrc\" (UID: \"ceb39acb-e2b8-4a34-8936-6c36677fa484\") " pod="openshift-ingress/e2e-ipp-b-0f85b8-openshift-default-7d957b55c4-pskrc"
Sep 11 04:05:22.013538 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:05:22.013493    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ceb39acb-e2b8-4a34-8936-6c36677fa484-workload-socket\") pod \"e2e-ipp-b-0f85b8-openshift-default-7d957b55c4-pskrc\" (UID: \"ceb39acb-e2b8-4a34-8936-6c36677fa484\") " pod="openshift-ingress/e2e-ipp-b-0f85b8-openshift-default-7d957b55c4-pskrc"
Sep 11 04:05:22.013538 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:05:22.013516    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ceb39acb-e2b8-4a34-8936-6c36677fa484-workload-certs\") pod \"e2e-ipp-b-0f85b8-openshift-default-7d957b55c4-pskrc\" (UID: \"ceb39acb-e2b8-4a34-8936-6c36677fa484\") " pod="openshift-ingress/e2e-ipp-b-0f85b8-openshift-default-7d957b55c4-pskrc"
Sep 11 04:05:22.013814 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:05:22.013548    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ceb39acb-e2b8-4a34-8936-6c36677fa484-istio-data\") pod \"e2e-ipp-b-0f85b8-openshift-default-7d957b55c4-pskrc\" (UID: \"ceb39acb-e2b8-4a34-8936-6c36677fa484\") " pod="openshift-ingress/e2e-ipp-b-0f85b8-openshift-default-7d957b55c4-pskrc"
Sep 11 04:05:22.013814 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:05:22.013590    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ceb39acb-e2b8-4a34-8936-6c36677fa484-istiod-ca-cert\") pod \"e2e-ipp-b-0f85b8-openshift-default-7d957b55c4-pskrc\" (UID: \"ceb39acb-e2b8-4a34-8936-6c36677fa484\") " pod="openshift-ingress/e2e-ipp-b-0f85b8-openshift-default-7d957b55c4-pskrc"
Sep 11 04:05:22.013814 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:05:22.013627    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ceb39acb-e2b8-4a34-8936-6c36677fa484-credential-socket\") pod \"e2e-ipp-b-0f85b8-openshift-default-7d957b55c4-pskrc\" (UID: \"ceb39acb-e2b8-4a34-8936-6c36677fa484\") " pod="openshift-ingress/e2e-ipp-b-0f85b8-openshift-default-7d957b55c4-pskrc"
Sep 11 04:05:22.115131 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:05:22.115100    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ceb39acb-e2b8-4a34-8936-6c36677fa484-istio-envoy\") pod \"e2e-ipp-b-0f85b8-openshift-default-7d957b55c4-pskrc\" (UID: \"ceb39acb-e2b8-4a34-8936-6c36677fa484\") " pod="openshift-ingress/e2e-ipp-b-0f85b8-openshift-default-7d957b55c4-pskrc"
Sep 11 04:05:22.115312 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:05:22.115141    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4dnxf\" (UniqueName: \"kubernetes.io/projected/ceb39acb-e2b8-4a34-8936-6c36677fa484-kube-api-access-4dnxf\") pod \"e2e-ipp-b-0f85b8-openshift-default-7d957b55c4-pskrc\" (UID: \"ceb39acb-e2b8-4a34-8936-6c36677fa484\") " pod="openshift-ingress/e2e-ipp-b-0f85b8-openshift-default-7d957b55c4-pskrc"
Sep 11 04:05:22.115312 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:05:22.115174    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ceb39acb-e2b8-4a34-8936-6c36677fa484-istio-podinfo\") pod \"e2e-ipp-b-0f85b8-openshift-default-7d957b55c4-pskrc\" (UID: \"ceb39acb-e2b8-4a34-8936-6c36677fa484\") " pod="openshift-ingress/e2e-ipp-b-0f85b8-openshift-default-7d957b55c4-pskrc"
Sep 11 04:05:22.115312 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:05:22.115195    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ceb39acb-e2b8-4a34-8936-6c36677fa484-istio-token\") pod \"e2e-ipp-b-0f85b8-openshift-default-7d957b55c4-pskrc\" (UID: \"ceb39acb-e2b8-4a34-8936-6c36677fa484\") " pod="openshift-ingress/e2e-ipp-b-0f85b8-openshift-default-7d957b55c4-pskrc"
Sep 11 04:05:22.115312 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:05:22.115249    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ceb39acb-e2b8-4a34-8936-6c36677fa484-workload-socket\") pod \"e2e-ipp-b-0f85b8-openshift-default-7d957b55c4-pskrc\" (UID: \"ceb39acb-e2b8-4a34-8936-6c36677fa484\") " pod="openshift-ingress/e2e-ipp-b-0f85b8-openshift-default-7d957b55c4-pskrc"
Sep 11 04:05:22.115312 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:05:22.115275    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ceb39acb-e2b8-4a34-8936-6c36677fa484-workload-certs\") pod \"e2e-ipp-b-0f85b8-openshift-default-7d957b55c4-pskrc\" (UID: \"ceb39acb-e2b8-4a34-8936-6c36677fa484\") " pod="openshift-ingress/e2e-ipp-b-0f85b8-openshift-default-7d957b55c4-pskrc"
Sep 11 04:05:22.115589 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:05:22.115326    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ceb39acb-e2b8-4a34-8936-6c36677fa484-istio-data\") pod \"e2e-ipp-b-0f85b8-openshift-default-7d957b55c4-pskrc\" (UID: \"ceb39acb-e2b8-4a34-8936-6c36677fa484\") " pod="openshift-ingress/e2e-ipp-b-0f85b8-openshift-default-7d957b55c4-pskrc"
Sep 11 04:05:22.115589 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:05:22.115382    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ceb39acb-e2b8-4a34-8936-6c36677fa484-istiod-ca-cert\") pod \"e2e-ipp-b-0f85b8-openshift-default-7d957b55c4-pskrc\" (UID: \"ceb39acb-e2b8-4a34-8936-6c36677fa484\") " pod="openshift-ingress/e2e-ipp-b-0f85b8-openshift-default-7d957b55c4-pskrc"
Sep 11 04:05:22.115589 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:05:22.115420    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ceb39acb-e2b8-4a34-8936-6c36677fa484-credential-socket\") pod \"e2e-ipp-b-0f85b8-openshift-default-7d957b55c4-pskrc\" (UID: \"ceb39acb-e2b8-4a34-8936-6c36677fa484\") " pod="openshift-ingress/e2e-ipp-b-0f85b8-openshift-default-7d957b55c4-pskrc"
Sep 11 04:05:22.115755 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:05:22.115702    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ceb39acb-e2b8-4a34-8936-6c36677fa484-workload-certs\") pod \"e2e-ipp-b-0f85b8-openshift-default-7d957b55c4-pskrc\" (UID: \"ceb39acb-e2b8-4a34-8936-6c36677fa484\") " pod="openshift-ingress/e2e-ipp-b-0f85b8-openshift-default-7d957b55c4-pskrc"
Sep 11 04:05:22.115878 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:05:22.115860    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ceb39acb-e2b8-4a34-8936-6c36677fa484-credential-socket\") pod \"e2e-ipp-b-0f85b8-openshift-default-7d957b55c4-pskrc\" (UID: \"ceb39acb-e2b8-4a34-8936-6c36677fa484\") " pod="openshift-ingress/e2e-ipp-b-0f85b8-openshift-default-7d957b55c4-pskrc"
Sep 11 04:05:22.116112 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:05:22.115904    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ceb39acb-e2b8-4a34-8936-6c36677fa484-workload-socket\") pod \"e2e-ipp-b-0f85b8-openshift-default-7d957b55c4-pskrc\" (UID: \"ceb39acb-e2b8-4a34-8936-6c36677fa484\") " pod="openshift-ingress/e2e-ipp-b-0f85b8-openshift-default-7d957b55c4-pskrc"
Sep 11 04:05:22.116112 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:05:22.115949    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ceb39acb-e2b8-4a34-8936-6c36677fa484-istio-data\") pod \"e2e-ipp-b-0f85b8-openshift-default-7d957b55c4-pskrc\" (UID: \"ceb39acb-e2b8-4a34-8936-6c36677fa484\") " pod="openshift-ingress/e2e-ipp-b-0f85b8-openshift-default-7d957b55c4-pskrc"
Sep 11 04:05:22.116252 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:05:22.116136    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ceb39acb-e2b8-4a34-8936-6c36677fa484-istiod-ca-cert\") pod \"e2e-ipp-b-0f85b8-openshift-default-7d957b55c4-pskrc\" (UID: \"ceb39acb-e2b8-4a34-8936-6c36677fa484\") " pod="openshift-ingress/e2e-ipp-b-0f85b8-openshift-default-7d957b55c4-pskrc"
Sep 11 04:05:22.117457 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:05:22.117430    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ceb39acb-e2b8-4a34-8936-6c36677fa484-istio-envoy\") pod \"e2e-ipp-b-0f85b8-openshift-default-7d957b55c4-pskrc\" (UID: \"ceb39acb-e2b8-4a34-8936-6c36677fa484\") " pod="openshift-ingress/e2e-ipp-b-0f85b8-openshift-default-7d957b55c4-pskrc"
Sep 11 04:05:22.117740 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:05:22.117723    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ceb39acb-e2b8-4a34-8936-6c36677fa484-istio-podinfo\") pod \"e2e-ipp-b-0f85b8-openshift-default-7d957b55c4-pskrc\" (UID: \"ceb39acb-e2b8-4a34-8936-6c36677fa484\") " pod="openshift-ingress/e2e-ipp-b-0f85b8-openshift-default-7d957b55c4-pskrc"
Sep 11 04:05:22.123149 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:05:22.123128    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ceb39acb-e2b8-4a34-8936-6c36677fa484-istio-token\") pod \"e2e-ipp-b-0f85b8-openshift-default-7d957b55c4-pskrc\" (UID: \"ceb39acb-e2b8-4a34-8936-6c36677fa484\") " pod="openshift-ingress/e2e-ipp-b-0f85b8-openshift-default-7d957b55c4-pskrc"
Sep 11 04:05:22.123270 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:05:22.123247    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4dnxf\" (UniqueName: \"kubernetes.io/projected/ceb39acb-e2b8-4a34-8936-6c36677fa484-kube-api-access-4dnxf\") pod \"e2e-ipp-b-0f85b8-openshift-default-7d957b55c4-pskrc\" (UID: \"ceb39acb-e2b8-4a34-8936-6c36677fa484\") " pod="openshift-ingress/e2e-ipp-b-0f85b8-openshift-default-7d957b55c4-pskrc"
Sep 11 04:05:22.233550 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:05:22.233460    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-0f85b8-openshift-default-7d957b55c4-pskrc"
Sep 11 04:05:22.509570 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:05:22.509544    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-0f85b8-openshift-default-7d957b55c4-pskrc"]
Sep 11 04:05:22.511668 ip-10-0-141-105 kubenswrapper[2579]: W0911 04:05:22.511639    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podceb39acb_e2b8_4a34_8936_6c36677fa484.slice/crio-8fc66aec1b4a10cc9cc39279733787ff2a641fee10fbcc871c7b0ff1c4d8dc29 WatchSource:0}: Error finding container 8fc66aec1b4a10cc9cc39279733787ff2a641fee10fbcc871c7b0ff1c4d8dc29: Status 404 returned error can't find the container with id 8fc66aec1b4a10cc9cc39279733787ff2a641fee10fbcc871c7b0ff1c4d8dc29
Sep 11 04:05:22.514462 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:05:22.514423    2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 11 04:05:22.514602 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:05:22.514527    2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 11 04:05:22.514602 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:05:22.514587    2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 11 04:05:22.799752 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:05:22.799671    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-0f85b8-openshift-default-7d957b55c4-pskrc" event={"ID":"ceb39acb-e2b8-4a34-8936-6c36677fa484","Type":"ContainerStarted","Data":"21b2dac30f70bee611edf8cf906eadca5b16a887671fe815369af24dc205b989"}
Sep 11 04:05:22.799908 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:05:22.799763    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-0f85b8-openshift-default-7d957b55c4-pskrc" event={"ID":"ceb39acb-e2b8-4a34-8936-6c36677fa484","Type":"ContainerStarted","Data":"8fc66aec1b4a10cc9cc39279733787ff2a641fee10fbcc871c7b0ff1c4d8dc29"}
Sep 11 04:05:22.819953 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:05:22.819906    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-b-0f85b8-openshift-default-7d957b55c4-pskrc" podStartSLOduration=1.819888658 podStartE2EDuration="1.819888658s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 04:05:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 04:05:22.817559555 +0000 UTC m=+1500.472765123" watchObservedRunningTime="2026-09-11 04:05:22.819888658 +0000 UTC m=+1500.475094224"
Sep 11 04:05:23.233631 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:05:23.233602    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-0f85b8-openshift-default-7d957b55c4-pskrc"
Sep 11 04:05:23.239764 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:05:23.239739    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-0f85b8-openshift-default-7d957b55c4-pskrc"
Sep 11 04:05:23.805260 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:05:23.805225    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-0f85b8-openshift-default-7d957b55c4-pskrc"
Sep 11 04:05:23.806227 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:05:23.806207    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-0f85b8-openshift-default-7d957b55c4-pskrc"
Sep 11 04:05:27.853360 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:05:27.853322    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-0f85b8-6cd9479485-9nzzd"]
Sep 11 04:05:27.858274 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:05:27.858257    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-0f85b8-6cd9479485-9nzzd"
Sep 11 04:05:27.860952 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:05:27.860922    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-0f85b8-dockercfg-jzp7b\""
Sep 11 04:05:27.861129 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:05:27.860952    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-0f85b8\""
Sep 11 04:05:27.865624 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:05:27.865600    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-0f85b8-6cd9479485-9nzzd"]
Sep 11 04:05:27.898731 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:05:27.898696    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-0f85b8-88497b898-lq6rr"]
Sep 11 04:05:27.902098 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:05:27.902060    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-0f85b8-88497b898-lq6rr"
Sep 11 04:05:27.912354 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:05:27.912330    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-0f85b8-88497b898-lq6rr"]
Sep 11 04:05:27.974253 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:05:27.974206    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wm99n\" (UniqueName: \"kubernetes.io/projected/38ce28d0-5809-4f66-8772-c11d42660225-kube-api-access-wm99n\") pod \"payload-processing-e2e-ipp-b-0f85b8-6cd9479485-9nzzd\" (UID: \"38ce28d0-5809-4f66-8772-c11d42660225\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-0f85b8-6cd9479485-9nzzd"
Sep 11 04:05:27.974438 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:05:27.974264    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/38ce28d0-5809-4f66-8772-c11d42660225-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-0f85b8-6cd9479485-9nzzd\" (UID: \"38ce28d0-5809-4f66-8772-c11d42660225\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-0f85b8-6cd9479485-9nzzd"
Sep 11 04:05:27.974438 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:05:27.974369    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-48kcq\" (UniqueName: \"kubernetes.io/projected/a2497017-256e-4537-86f3-8bf8de45e4d3-kube-api-access-48kcq\") pod \"payload-pre-processing-e2e-ipp-b-0f85b8-88497b898-lq6rr\" (UID: \"a2497017-256e-4537-86f3-8bf8de45e4d3\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-0f85b8-88497b898-lq6rr"
Sep 11 04:05:27.974438 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:05:27.974398    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a2497017-256e-4537-86f3-8bf8de45e4d3-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-0f85b8-88497b898-lq6rr\" (UID: \"a2497017-256e-4537-86f3-8bf8de45e4d3\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-0f85b8-88497b898-lq6rr"
Sep 11 04:05:28.075311 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:05:28.075260    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wm99n\" (UniqueName: \"kubernetes.io/projected/38ce28d0-5809-4f66-8772-c11d42660225-kube-api-access-wm99n\") pod \"payload-processing-e2e-ipp-b-0f85b8-6cd9479485-9nzzd\" (UID: \"38ce28d0-5809-4f66-8772-c11d42660225\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-0f85b8-6cd9479485-9nzzd"
Sep 11 04:05:28.075311 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:05:28.075317    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/38ce28d0-5809-4f66-8772-c11d42660225-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-0f85b8-6cd9479485-9nzzd\" (UID: \"38ce28d0-5809-4f66-8772-c11d42660225\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-0f85b8-6cd9479485-9nzzd"
Sep 11 04:05:28.075577 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:05:28.075380    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-48kcq\" (UniqueName: \"kubernetes.io/projected/a2497017-256e-4537-86f3-8bf8de45e4d3-kube-api-access-48kcq\") pod \"payload-pre-processing-e2e-ipp-b-0f85b8-88497b898-lq6rr\" (UID: \"a2497017-256e-4537-86f3-8bf8de45e4d3\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-0f85b8-88497b898-lq6rr"
Sep 11 04:05:28.075577 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:05:28.075407    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a2497017-256e-4537-86f3-8bf8de45e4d3-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-0f85b8-88497b898-lq6rr\" (UID: \"a2497017-256e-4537-86f3-8bf8de45e4d3\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-0f85b8-88497b898-lq6rr"
Sep 11 04:05:28.076115 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:05:28.076057    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/38ce28d0-5809-4f66-8772-c11d42660225-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-0f85b8-6cd9479485-9nzzd\" (UID: \"38ce28d0-5809-4f66-8772-c11d42660225\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-0f85b8-6cd9479485-9nzzd"
Sep 11 04:05:28.076115 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:05:28.076067    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a2497017-256e-4537-86f3-8bf8de45e4d3-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-0f85b8-88497b898-lq6rr\" (UID: \"a2497017-256e-4537-86f3-8bf8de45e4d3\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-0f85b8-88497b898-lq6rr"
Sep 11 04:05:28.083361 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:05:28.083329    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wm99n\" (UniqueName: \"kubernetes.io/projected/38ce28d0-5809-4f66-8772-c11d42660225-kube-api-access-wm99n\") pod \"payload-processing-e2e-ipp-b-0f85b8-6cd9479485-9nzzd\" (UID: \"38ce28d0-5809-4f66-8772-c11d42660225\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-0f85b8-6cd9479485-9nzzd"
Sep 11 04:05:28.083361 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:05:28.083346    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-48kcq\" (UniqueName: \"kubernetes.io/projected/a2497017-256e-4537-86f3-8bf8de45e4d3-kube-api-access-48kcq\") pod \"payload-pre-processing-e2e-ipp-b-0f85b8-88497b898-lq6rr\" (UID: \"a2497017-256e-4537-86f3-8bf8de45e4d3\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-0f85b8-88497b898-lq6rr"
Sep 11 04:05:28.170145 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:05:28.170032    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-0f85b8-6cd9479485-9nzzd"
Sep 11 04:05:28.212613 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:05:28.212580    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-0f85b8-88497b898-lq6rr"
Sep 11 04:05:28.442282 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:05:28.442258    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-0f85b8-6cd9479485-9nzzd"]
Sep 11 04:05:28.444469 ip-10-0-141-105 kubenswrapper[2579]: W0911 04:05:28.444445    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod38ce28d0_5809_4f66_8772_c11d42660225.slice/crio-0582a99887dc67af10582c0f3cf05505f82705b20ead4d5fbd61fb8aa03bc691 WatchSource:0}: Error finding container 0582a99887dc67af10582c0f3cf05505f82705b20ead4d5fbd61fb8aa03bc691: Status 404 returned error can't find the container with id 0582a99887dc67af10582c0f3cf05505f82705b20ead4d5fbd61fb8aa03bc691
Sep 11 04:05:28.495956 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:05:28.495935    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-0f85b8-88497b898-lq6rr"]
Sep 11 04:05:28.497466 ip-10-0-141-105 kubenswrapper[2579]: W0911 04:05:28.497400    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda2497017_256e_4537_86f3_8bf8de45e4d3.slice/crio-f896185d6cf40992316eba3ea18d7c2dcab8388990c61aa5d7fd80f4a7523171 WatchSource:0}: Error finding container f896185d6cf40992316eba3ea18d7c2dcab8388990c61aa5d7fd80f4a7523171: Status 404 returned error can't find the container with id f896185d6cf40992316eba3ea18d7c2dcab8388990c61aa5d7fd80f4a7523171
Sep 11 04:05:28.833923 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:05:28.833821    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-0f85b8-88497b898-lq6rr" event={"ID":"a2497017-256e-4537-86f3-8bf8de45e4d3","Type":"ContainerStarted","Data":"40520880fa9826eb6dd3d335b673f10282d5bd9a0e64e86f18475aa214ec3170"}
Sep 11 04:05:28.833923 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:05:28.833888    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-0f85b8-88497b898-lq6rr" event={"ID":"a2497017-256e-4537-86f3-8bf8de45e4d3","Type":"ContainerStarted","Data":"f896185d6cf40992316eba3ea18d7c2dcab8388990c61aa5d7fd80f4a7523171"}
Sep 11 04:05:28.833923 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:05:28.833916    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-0f85b8-88497b898-lq6rr"
Sep 11 04:05:28.835602 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:05:28.835577    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-0f85b8-6cd9479485-9nzzd" event={"ID":"38ce28d0-5809-4f66-8772-c11d42660225","Type":"ContainerStarted","Data":"11ae01c1ef654a77b8cbdc811970a4bd7b75c30400378ed5667c5035eda9a411"}
Sep 11 04:05:28.835757 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:05:28.835625    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-0f85b8-6cd9479485-9nzzd" event={"ID":"38ce28d0-5809-4f66-8772-c11d42660225","Type":"ContainerStarted","Data":"0582a99887dc67af10582c0f3cf05505f82705b20ead4d5fbd61fb8aa03bc691"}
Sep 11 04:05:28.835757 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:05:28.835675    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-0f85b8-6cd9479485-9nzzd"
Sep 11 04:05:28.851045 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:05:28.850985    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-0f85b8-88497b898-lq6rr" podStartSLOduration=1.850969347 podStartE2EDuration="1.850969347s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 04:05:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 04:05:28.849115533 +0000 UTC m=+1506.504321100" watchObservedRunningTime="2026-09-11 04:05:28.850969347 +0000 UTC m=+1506.506175164"
Sep 11 04:05:28.866210 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:05:28.866164    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-0f85b8-6cd9479485-9nzzd" podStartSLOduration=1.8661491799999999 podStartE2EDuration="1.86614918s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 04:05:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 04:05:28.863928295 +0000 UTC m=+1506.519133862" watchObservedRunningTime="2026-09-11 04:05:28.86614918 +0000 UTC m=+1506.521354747"
Sep 11 04:05:31.108469 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:05:31.108431    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-5z4cn"]
Sep 11 04:05:39.844790 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:05:39.844753    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-0f85b8-6cd9479485-9nzzd"
Sep 11 04:05:39.845248 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:05:39.844852    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-0f85b8-88497b898-lq6rr"
Sep 11 04:05:42.915004 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:05:42.914966    2579 generic.go:356] "Generic (PLEG): container finished" podID="ceb39acb-e2b8-4a34-8936-6c36677fa484" containerID="21b2dac30f70bee611edf8cf906eadca5b16a887671fe815369af24dc205b989" exitCode=0
Sep 11 04:05:42.915502 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:05:42.915058    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-0f85b8-openshift-default-7d957b55c4-pskrc" event={"ID":"ceb39acb-e2b8-4a34-8936-6c36677fa484","Type":"ContainerDied","Data":"21b2dac30f70bee611edf8cf906eadca5b16a887671fe815369af24dc205b989"}
Sep 11 04:05:42.915502 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:05:42.915468    2579 scope.go:117] "RemoveContainer" containerID="21b2dac30f70bee611edf8cf906eadca5b16a887671fe815369af24dc205b989"
Sep 11 04:05:42.916555 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:05:42.916524    2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 11 04:05:42.916667 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:05:42.916592    2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 11 04:05:42.916667 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:05:42.916626    2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 11 04:05:43.923766 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:05:43.923688    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-0f85b8-openshift-default-7d957b55c4-pskrc" event={"ID":"ceb39acb-e2b8-4a34-8936-6c36677fa484","Type":"ContainerStarted","Data":"b52c194e64c5ae57cd4e7e0060f2b719f4f264911aabe5cb192d3e03531fecf5"}
Sep 11 04:05:44.234299 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:05:44.234139    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-0f85b8-openshift-default-7d957b55c4-pskrc"
Sep 11 04:05:44.234487 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:05:44.234407    2579 patch_prober.go:28] interesting pod/e2e-ipp-b-0f85b8-openshift-default-7d957b55c4-pskrc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.91:15021/healthz/ready\": dial tcp 10.134.0.91:15021: connect: connection refused" start-of-body=
Sep 11 04:05:44.234487 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:05:44.234470    2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-0f85b8-openshift-default-7d957b55c4-pskrc" podUID="ceb39acb-e2b8-4a34-8936-6c36677fa484" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.91:15021/healthz/ready\": dial tcp 10.134.0.91:15021: connect: connection refused"
Sep 11 04:05:45.234161 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:05:45.234122    2579 patch_prober.go:28] interesting pod/e2e-ipp-b-0f85b8-openshift-default-7d957b55c4-pskrc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.91:15021/healthz/ready\": dial tcp 10.134.0.91:15021: connect: connection refused" start-of-body=
Sep 11 04:05:45.234651 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:05:45.234196    2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-0f85b8-openshift-default-7d957b55c4-pskrc" podUID="ceb39acb-e2b8-4a34-8936-6c36677fa484" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.91:15021/healthz/ready\": dial tcp 10.134.0.91:15021: connect: connection refused"
Sep 11 04:05:46.234561 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:05:46.234514    2579 patch_prober.go:28] interesting pod/e2e-ipp-b-0f85b8-openshift-default-7d957b55c4-pskrc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.91:15021/healthz/ready\": dial tcp 10.134.0.91:15021: connect: connection refused" start-of-body=
Sep 11 04:05:46.235070 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:05:46.234586    2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-0f85b8-openshift-default-7d957b55c4-pskrc" podUID="ceb39acb-e2b8-4a34-8936-6c36677fa484" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.91:15021/healthz/ready\": dial tcp 10.134.0.91:15021: connect: connection refused"
Sep 11 04:05:47.234465 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:05:47.234423    2579 patch_prober.go:28] interesting pod/e2e-ipp-b-0f85b8-openshift-default-7d957b55c4-pskrc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.91:15021/healthz/ready\": dial tcp 10.134.0.91:15021: connect: connection refused" start-of-body=
Sep 11 04:05:47.234671 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:05:47.234489    2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-0f85b8-openshift-default-7d957b55c4-pskrc" podUID="ceb39acb-e2b8-4a34-8936-6c36677fa484" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.91:15021/healthz/ready\": dial tcp 10.134.0.91:15021: connect: connection refused"
Sep 11 04:05:48.234433 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:05:48.234385    2579 patch_prober.go:28] interesting pod/e2e-ipp-b-0f85b8-openshift-default-7d957b55c4-pskrc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.91:15021/healthz/ready\": dial tcp 10.134.0.91:15021: connect: connection refused" start-of-body=
Sep 11 04:05:48.234643 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:05:48.234459    2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-0f85b8-openshift-default-7d957b55c4-pskrc" podUID="ceb39acb-e2b8-4a34-8936-6c36677fa484" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.91:15021/healthz/ready\": dial tcp 10.134.0.91:15021: connect: connection refused"
Sep 11 04:05:49.238724 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:05:49.238696    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-0f85b8-openshift-default-7d957b55c4-pskrc"
Sep 11 04:05:49.239144 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:05:49.239120    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-0f85b8-openshift-default-7d957b55c4-pskrc"
Sep 11 04:05:49.239908 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:05:49.239886    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-0f85b8-openshift-default-7d957b55c4-pskrc"
Sep 11 04:05:54.921864 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:05:54.921836    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5b54b7fbb7-69ccr_598a451c-3387-4674-9c03-6ea4ca648208/payload-processing/0.log"
Sep 11 04:05:55.039642 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:05:55.039605    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-b-0f85b8-6cd9479485-9nzzd_38ce28d0-5809-4f66-8772-c11d42660225/payload-processing/0.log"
Sep 11 04:06:03.898357 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:06:03.898320    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-ipp-a-4cb558/ipp-route-4cb558-kserve-6c48d599c9-mhzjf"]
Sep 11 04:06:03.900979 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:06:03.900961    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-4cb558/ipp-route-4cb558-kserve-6c48d599c9-mhzjf"
Sep 11 04:06:03.904590 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:06:03.904566    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-ipp-a-4cb558\"/\"kube-root-ca.crt\""
Sep 11 04:06:03.904732 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:06:03.904565    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-ipp-a-4cb558\"/\"default-dockercfg-w4m7x\""
Sep 11 04:06:03.904732 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:06:03.904621    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-ipp-a-4cb558\"/\"ipp-route-4cb558-kserve-self-signed-certs\""
Sep 11 04:06:03.904732 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:06:03.904574    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-ipp-a-4cb558\"/\"openshift-service-ca.crt\""
Sep 11 04:06:03.909542 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:06:03.909515    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-ipp-a-4cb558/ipp-route-4cb558-kserve-6c48d599c9-mhzjf"]
Sep 11 04:06:04.031558 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:06:04.031523    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8bed8e52-a169-4466-bc3a-4bd5a34000f4-dshm\") pod \"ipp-route-4cb558-kserve-6c48d599c9-mhzjf\" (UID: \"8bed8e52-a169-4466-bc3a-4bd5a34000f4\") " pod="ai-tenant-e2e-ipp-a-4cb558/ipp-route-4cb558-kserve-6c48d599c9-mhzjf"
Sep 11 04:06:04.031740 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:06:04.031580    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j754p\" (UniqueName: \"kubernetes.io/projected/8bed8e52-a169-4466-bc3a-4bd5a34000f4-kube-api-access-j754p\") pod \"ipp-route-4cb558-kserve-6c48d599c9-mhzjf\" (UID: \"8bed8e52-a169-4466-bc3a-4bd5a34000f4\") " pod="ai-tenant-e2e-ipp-a-4cb558/ipp-route-4cb558-kserve-6c48d599c9-mhzjf"
Sep 11 04:06:04.031740 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:06:04.031694    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8bed8e52-a169-4466-bc3a-4bd5a34000f4-model-cache\") pod \"ipp-route-4cb558-kserve-6c48d599c9-mhzjf\" (UID: \"8bed8e52-a169-4466-bc3a-4bd5a34000f4\") " pod="ai-tenant-e2e-ipp-a-4cb558/ipp-route-4cb558-kserve-6c48d599c9-mhzjf"
Sep 11 04:06:04.031834 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:06:04.031777    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8bed8e52-a169-4466-bc3a-4bd5a34000f4-tmp-dir\") pod \"ipp-route-4cb558-kserve-6c48d599c9-mhzjf\" (UID: \"8bed8e52-a169-4466-bc3a-4bd5a34000f4\") " pod="ai-tenant-e2e-ipp-a-4cb558/ipp-route-4cb558-kserve-6c48d599c9-mhzjf"
Sep 11 04:06:04.031834 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:06:04.031801    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8bed8e52-a169-4466-bc3a-4bd5a34000f4-home\") pod \"ipp-route-4cb558-kserve-6c48d599c9-mhzjf\" (UID: \"8bed8e52-a169-4466-bc3a-4bd5a34000f4\") " pod="ai-tenant-e2e-ipp-a-4cb558/ipp-route-4cb558-kserve-6c48d599c9-mhzjf"
Sep 11 04:06:04.031923 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:06:04.031830    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8bed8e52-a169-4466-bc3a-4bd5a34000f4-tls-certs\") pod \"ipp-route-4cb558-kserve-6c48d599c9-mhzjf\" (UID: \"8bed8e52-a169-4466-bc3a-4bd5a34000f4\") " pod="ai-tenant-e2e-ipp-a-4cb558/ipp-route-4cb558-kserve-6c48d599c9-mhzjf"
Sep 11 04:06:04.132728 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:06:04.132688    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8bed8e52-a169-4466-bc3a-4bd5a34000f4-tls-certs\") pod \"ipp-route-4cb558-kserve-6c48d599c9-mhzjf\" (UID: \"8bed8e52-a169-4466-bc3a-4bd5a34000f4\") " pod="ai-tenant-e2e-ipp-a-4cb558/ipp-route-4cb558-kserve-6c48d599c9-mhzjf"
Sep 11 04:06:04.132926 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:06:04.132744    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8bed8e52-a169-4466-bc3a-4bd5a34000f4-dshm\") pod \"ipp-route-4cb558-kserve-6c48d599c9-mhzjf\" (UID: \"8bed8e52-a169-4466-bc3a-4bd5a34000f4\") " pod="ai-tenant-e2e-ipp-a-4cb558/ipp-route-4cb558-kserve-6c48d599c9-mhzjf"
Sep 11 04:06:04.132926 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:06:04.132773    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j754p\" (UniqueName: \"kubernetes.io/projected/8bed8e52-a169-4466-bc3a-4bd5a34000f4-kube-api-access-j754p\") pod \"ipp-route-4cb558-kserve-6c48d599c9-mhzjf\" (UID: \"8bed8e52-a169-4466-bc3a-4bd5a34000f4\") " pod="ai-tenant-e2e-ipp-a-4cb558/ipp-route-4cb558-kserve-6c48d599c9-mhzjf"
Sep 11 04:06:04.132926 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:06:04.132813    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8bed8e52-a169-4466-bc3a-4bd5a34000f4-model-cache\") pod \"ipp-route-4cb558-kserve-6c48d599c9-mhzjf\" (UID: \"8bed8e52-a169-4466-bc3a-4bd5a34000f4\") " pod="ai-tenant-e2e-ipp-a-4cb558/ipp-route-4cb558-kserve-6c48d599c9-mhzjf"
Sep 11 04:06:04.132926 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:06:04.132865    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8bed8e52-a169-4466-bc3a-4bd5a34000f4-tmp-dir\") pod \"ipp-route-4cb558-kserve-6c48d599c9-mhzjf\" (UID: \"8bed8e52-a169-4466-bc3a-4bd5a34000f4\") " pod="ai-tenant-e2e-ipp-a-4cb558/ipp-route-4cb558-kserve-6c48d599c9-mhzjf"
Sep 11 04:06:04.133201 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:06:04.133027    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8bed8e52-a169-4466-bc3a-4bd5a34000f4-home\") pod \"ipp-route-4cb558-kserve-6c48d599c9-mhzjf\" (UID: \"8bed8e52-a169-4466-bc3a-4bd5a34000f4\") " pod="ai-tenant-e2e-ipp-a-4cb558/ipp-route-4cb558-kserve-6c48d599c9-mhzjf"
Sep 11 04:06:04.133356 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:06:04.133337    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8bed8e52-a169-4466-bc3a-4bd5a34000f4-tmp-dir\") pod \"ipp-route-4cb558-kserve-6c48d599c9-mhzjf\" (UID: \"8bed8e52-a169-4466-bc3a-4bd5a34000f4\") " pod="ai-tenant-e2e-ipp-a-4cb558/ipp-route-4cb558-kserve-6c48d599c9-mhzjf"
Sep 11 04:06:04.133400 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:06:04.133382    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8bed8e52-a169-4466-bc3a-4bd5a34000f4-model-cache\") pod \"ipp-route-4cb558-kserve-6c48d599c9-mhzjf\" (UID: \"8bed8e52-a169-4466-bc3a-4bd5a34000f4\") " pod="ai-tenant-e2e-ipp-a-4cb558/ipp-route-4cb558-kserve-6c48d599c9-mhzjf"
Sep 11 04:06:04.133466 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:06:04.133450    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8bed8e52-a169-4466-bc3a-4bd5a34000f4-home\") pod \"ipp-route-4cb558-kserve-6c48d599c9-mhzjf\" (UID: \"8bed8e52-a169-4466-bc3a-4bd5a34000f4\") " pod="ai-tenant-e2e-ipp-a-4cb558/ipp-route-4cb558-kserve-6c48d599c9-mhzjf"
Sep 11 04:06:04.135368 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:06:04.135345    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8bed8e52-a169-4466-bc3a-4bd5a34000f4-dshm\") pod \"ipp-route-4cb558-kserve-6c48d599c9-mhzjf\" (UID: \"8bed8e52-a169-4466-bc3a-4bd5a34000f4\") " pod="ai-tenant-e2e-ipp-a-4cb558/ipp-route-4cb558-kserve-6c48d599c9-mhzjf"
Sep 11 04:06:04.135527 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:06:04.135510    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8bed8e52-a169-4466-bc3a-4bd5a34000f4-tls-certs\") pod \"ipp-route-4cb558-kserve-6c48d599c9-mhzjf\" (UID: \"8bed8e52-a169-4466-bc3a-4bd5a34000f4\") " pod="ai-tenant-e2e-ipp-a-4cb558/ipp-route-4cb558-kserve-6c48d599c9-mhzjf"
Sep 11 04:06:04.143626 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:06:04.143602    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-j754p\" (UniqueName: \"kubernetes.io/projected/8bed8e52-a169-4466-bc3a-4bd5a34000f4-kube-api-access-j754p\") pod \"ipp-route-4cb558-kserve-6c48d599c9-mhzjf\" (UID: \"8bed8e52-a169-4466-bc3a-4bd5a34000f4\") " pod="ai-tenant-e2e-ipp-a-4cb558/ipp-route-4cb558-kserve-6c48d599c9-mhzjf"
Sep 11 04:06:04.211799 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:06:04.211717    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-4cb558/ipp-route-4cb558-kserve-6c48d599c9-mhzjf"
Sep 11 04:06:04.488148 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:06:04.488118    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-ipp-a-4cb558/ipp-route-4cb558-kserve-6c48d599c9-mhzjf"]
Sep 11 04:06:04.489681 ip-10-0-141-105 kubenswrapper[2579]: W0911 04:06:04.489654    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod8bed8e52_a169_4466_bc3a_4bd5a34000f4.slice/crio-90595ad360610a370eebb5c9849d1ad87f9178aa73bc076a15824e50fa2d371a WatchSource:0}: Error finding container 90595ad360610a370eebb5c9849d1ad87f9178aa73bc076a15824e50fa2d371a: Status 404 returned error can't find the container with id 90595ad360610a370eebb5c9849d1ad87f9178aa73bc076a15824e50fa2d371a
Sep 11 04:06:05.048462 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:06:05.048420    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-4cb558/ipp-route-4cb558-kserve-6c48d599c9-mhzjf" event={"ID":"8bed8e52-a169-4466-bc3a-4bd5a34000f4","Type":"ContainerStarted","Data":"d1f52273b394956042fcf2ff05432307f9b8c1dc9b3ef01fe076ebfa7e02e7a3"}
Sep 11 04:06:05.048884 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:06:05.048469    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-4cb558/ipp-route-4cb558-kserve-6c48d599c9-mhzjf" event={"ID":"8bed8e52-a169-4466-bc3a-4bd5a34000f4","Type":"ContainerStarted","Data":"90595ad360610a370eebb5c9849d1ad87f9178aa73bc076a15824e50fa2d371a"}
Sep 11 04:06:05.066273 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:06:05.066224    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-ipp-a-4cb558/ipp-route-4cb558-kserve-6c48d599c9-mhzjf" podStartSLOduration=2.066210048 podStartE2EDuration="2.066210048s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 04:06:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 04:06:05.064508046 +0000 UTC m=+1542.719713614" watchObservedRunningTime="2026-09-11 04:06:05.066210048 +0000 UTC m=+1542.721415615"
Sep 11 04:06:07.606829 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:06:07.606791    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-5z4cn"]
Sep 11 04:06:14.212103 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:06:14.212035    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-ipp-a-4cb558/ipp-route-4cb558-kserve-6c48d599c9-mhzjf"
Sep 11 04:06:14.212103 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:06:14.212109    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-ipp-a-4cb558/ipp-route-4cb558-kserve-6c48d599c9-mhzjf"
Sep 11 04:06:14.225473 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:06:14.225443    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-ipp-a-4cb558/ipp-route-4cb558-kserve-6c48d599c9-mhzjf"
Sep 11 04:06:15.113269 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:06:15.113242    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-ipp-a-4cb558/ipp-route-4cb558-kserve-6c48d599c9-mhzjf"
Sep 11 04:06:23.177593 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:06:23.177560    2579 scope.go:117] "RemoveContainer" containerID="b79babd22f75f3469ce951e15452bf1cce7204b9917c1650ff24f454d0357d0b"
Sep 11 04:06:23.186634 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:06:23.186615    2579 scope.go:117] "RemoveContainer" containerID="9afb490d1689454b9f73a00d05b27b276d3c9dc99e066ec4b9fd7de86a7f98e7"
Sep 11 04:06:23.195037 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:06:23.195020    2579 scope.go:117] "RemoveContainer" containerID="e66acb358f77288bf6feda4378fbc9e5603f20ab02d54782624e05125dedebc7"
Sep 11 04:06:29.500614 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:06:29.500579    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-5z4cn"]
Sep 11 04:06:36.006965 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:06:36.006935    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5b54b7fbb7-69ccr_598a451c-3387-4674-9c03-6ea4ca648208/payload-processing/0.log"
Sep 11 04:06:36.133295 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:06:36.133264    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-b-0f85b8-6cd9479485-9nzzd_38ce28d0-5809-4f66-8772-c11d42660225/payload-processing/0.log"
Sep 11 04:06:43.139554 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:06:43.139512    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-ebb9b5-5cd7b7c997-6j6wt"]
Sep 11 04:06:43.143233 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:06:43.143210    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-ebb9b5-5cd7b7c997-6j6wt"
Sep 11 04:06:43.146304 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:06:43.146278    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-cleanup-ebb9b5-dockercfg-r62h5\""
Sep 11 04:06:43.146434 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:06:43.146304    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-cleanup-ebb9b5\""
Sep 11 04:06:43.154290 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:06:43.154262    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-ebb9b5-5cd7b7c997-6j6wt"]
Sep 11 04:06:43.221194 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:06:43.221160    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pfd4v\" (UniqueName: \"kubernetes.io/projected/9297c00e-c325-4db6-84e7-7ddc5a36dfe0-kube-api-access-pfd4v\") pod \"payload-processing-e2e-ipp-cleanup-ebb9b5-5cd7b7c997-6j6wt\" (UID: \"9297c00e-c325-4db6-84e7-7ddc5a36dfe0\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-ebb9b5-5cd7b7c997-6j6wt"
Sep 11 04:06:43.221403 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:06:43.221307    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9297c00e-c325-4db6-84e7-7ddc5a36dfe0-plugins-config-volume\") pod \"payload-processing-e2e-ipp-cleanup-ebb9b5-5cd7b7c997-6j6wt\" (UID: \"9297c00e-c325-4db6-84e7-7ddc5a36dfe0\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-ebb9b5-5cd7b7c997-6j6wt"
Sep 11 04:06:43.321856 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:06:43.321820    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9297c00e-c325-4db6-84e7-7ddc5a36dfe0-plugins-config-volume\") pod \"payload-processing-e2e-ipp-cleanup-ebb9b5-5cd7b7c997-6j6wt\" (UID: \"9297c00e-c325-4db6-84e7-7ddc5a36dfe0\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-ebb9b5-5cd7b7c997-6j6wt"
Sep 11 04:06:43.322048 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:06:43.321904    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pfd4v\" (UniqueName: \"kubernetes.io/projected/9297c00e-c325-4db6-84e7-7ddc5a36dfe0-kube-api-access-pfd4v\") pod \"payload-processing-e2e-ipp-cleanup-ebb9b5-5cd7b7c997-6j6wt\" (UID: \"9297c00e-c325-4db6-84e7-7ddc5a36dfe0\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-ebb9b5-5cd7b7c997-6j6wt"
Sep 11 04:06:43.322582 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:06:43.322561    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9297c00e-c325-4db6-84e7-7ddc5a36dfe0-plugins-config-volume\") pod \"payload-processing-e2e-ipp-cleanup-ebb9b5-5cd7b7c997-6j6wt\" (UID: \"9297c00e-c325-4db6-84e7-7ddc5a36dfe0\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-ebb9b5-5cd7b7c997-6j6wt"
Sep 11 04:06:43.331714 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:06:43.331694    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pfd4v\" (UniqueName: \"kubernetes.io/projected/9297c00e-c325-4db6-84e7-7ddc5a36dfe0-kube-api-access-pfd4v\") pod \"payload-processing-e2e-ipp-cleanup-ebb9b5-5cd7b7c997-6j6wt\" (UID: \"9297c00e-c325-4db6-84e7-7ddc5a36dfe0\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-ebb9b5-5cd7b7c997-6j6wt"
Sep 11 04:06:43.455381 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:06:43.455291    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-ebb9b5-5cd7b7c997-6j6wt"
Sep 11 04:06:43.725123 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:06:43.725094    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-ebb9b5-5cd7b7c997-6j6wt"]
Sep 11 04:06:43.727187 ip-10-0-141-105 kubenswrapper[2579]: W0911 04:06:43.727162    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9297c00e_c325_4db6_84e7_7ddc5a36dfe0.slice/crio-0eae428c22a1b2f5a9bdaa772f36b0d47ad96a3a0f23ae82a7b9009a4917561f WatchSource:0}: Error finding container 0eae428c22a1b2f5a9bdaa772f36b0d47ad96a3a0f23ae82a7b9009a4917561f: Status 404 returned error can't find the container with id 0eae428c22a1b2f5a9bdaa772f36b0d47ad96a3a0f23ae82a7b9009a4917561f
Sep 11 04:06:44.255333 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:06:44.255287    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-ebb9b5-5cd7b7c997-6j6wt" event={"ID":"9297c00e-c325-4db6-84e7-7ddc5a36dfe0","Type":"ContainerStarted","Data":"ae28e1f737c9cd43096e45772da0b9d641db9ec6d0f8b93e4c7b0fc622f7e9d2"}
Sep 11 04:06:44.255778 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:06:44.255342    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-ebb9b5-5cd7b7c997-6j6wt" event={"ID":"9297c00e-c325-4db6-84e7-7ddc5a36dfe0","Type":"ContainerStarted","Data":"0eae428c22a1b2f5a9bdaa772f36b0d47ad96a3a0f23ae82a7b9009a4917561f"}
Sep 11 04:06:44.255778 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:06:44.255485    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-ebb9b5-5cd7b7c997-6j6wt"
Sep 11 04:06:44.274318 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:06:44.274270    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-ebb9b5-5cd7b7c997-6j6wt" podStartSLOduration=1.274253621 podStartE2EDuration="1.274253621s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 04:06:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 04:06:44.272992107 +0000 UTC m=+1581.928197687" watchObservedRunningTime="2026-09-11 04:06:44.274253621 +0000 UTC m=+1581.929459188"
Sep 11 04:06:49.309072 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:06:49.309027    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-5z4cn"]
Sep 11 04:06:55.262893 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:06:55.262858    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-ebb9b5-5cd7b7c997-6j6wt"
Sep 11 04:07:10.413188 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:07:10.413148    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-ebb9b5-hflzg"]
Sep 11 04:07:10.415839 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:07:10.415816    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-ebb9b5-hflzg"
Sep 11 04:07:10.418320 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:07:10.418298    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 11 04:07:10.418449 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:07:10.418364    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-95ckr\""
Sep 11 04:07:10.424244 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:07:10.424212    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-ebb9b5-hflzg"]
Sep 11 04:07:10.479524 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:07:10.479492    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2bf28659-3f51-4544-adad-96ca3d69dd4e-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-cleanup-ebb9b5-hflzg\" (UID: \"2bf28659-3f51-4544-adad-96ca3d69dd4e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-ebb9b5-hflzg"
Sep 11 04:07:10.581138 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:07:10.581074    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2bf28659-3f51-4544-adad-96ca3d69dd4e-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-cleanup-ebb9b5-hflzg\" (UID: \"2bf28659-3f51-4544-adad-96ca3d69dd4e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-ebb9b5-hflzg"
Sep 11 04:07:10.581728 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:07:10.581708    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2bf28659-3f51-4544-adad-96ca3d69dd4e-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-cleanup-ebb9b5-hflzg\" (UID: \"2bf28659-3f51-4544-adad-96ca3d69dd4e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-ebb9b5-hflzg"
Sep 11 04:07:10.727713 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:07:10.727634    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-ebb9b5-hflzg"
Sep 11 04:07:10.999575 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:07:10.999546    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-ebb9b5-hflzg"]
Sep 11 04:07:11.001772 ip-10-0-141-105 kubenswrapper[2579]: W0911 04:07:11.001748    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2bf28659_3f51_4544_adad_96ca3d69dd4e.slice/crio-d326ac4fe7df3a3da28bf286cf9c77b5652bab5b5f09faed0f78065d8ab8a81c WatchSource:0}: Error finding container d326ac4fe7df3a3da28bf286cf9c77b5652bab5b5f09faed0f78065d8ab8a81c: Status 404 returned error can't find the container with id d326ac4fe7df3a3da28bf286cf9c77b5652bab5b5f09faed0f78065d8ab8a81c
Sep 11 04:07:11.400319 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:07:11.400285    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-ebb9b5-hflzg" event={"ID":"2bf28659-3f51-4544-adad-96ca3d69dd4e","Type":"ContainerStarted","Data":"15763093c33b4458ab5891a1649c3798b0108f1035ebf2a333af8ff32d7cd457"}
Sep 11 04:07:11.400505 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:07:11.400334    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-ebb9b5-hflzg" event={"ID":"2bf28659-3f51-4544-adad-96ca3d69dd4e","Type":"ContainerStarted","Data":"d326ac4fe7df3a3da28bf286cf9c77b5652bab5b5f09faed0f78065d8ab8a81c"}
Sep 11 04:07:11.417912 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:07:11.417864    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-ebb9b5-hflzg" podStartSLOduration=1.41784738 podStartE2EDuration="1.41784738s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 04:07:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 04:07:11.416676213 +0000 UTC m=+1609.071881780" watchObservedRunningTime="2026-09-11 04:07:11.41784738 +0000 UTC m=+1609.073052947"
Sep 11 04:07:12.406991 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:07:12.406958    2579 generic.go:356] "Generic (PLEG): container finished" podID="2bf28659-3f51-4544-adad-96ca3d69dd4e" containerID="15763093c33b4458ab5891a1649c3798b0108f1035ebf2a333af8ff32d7cd457" exitCode=0
Sep 11 04:07:12.407216 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:07:12.407052    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-ebb9b5-hflzg" event={"ID":"2bf28659-3f51-4544-adad-96ca3d69dd4e","Type":"ContainerDied","Data":"15763093c33b4458ab5891a1649c3798b0108f1035ebf2a333af8ff32d7cd457"}
Sep 11 04:07:13.545401 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:07:13.545368    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-ebb9b5-hflzg"
Sep 11 04:07:13.612431 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:07:13.612400    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2bf28659-3f51-4544-adad-96ca3d69dd4e-maas-api-service-ca\") pod \"2bf28659-3f51-4544-adad-96ca3d69dd4e\" (UID: \"2bf28659-3f51-4544-adad-96ca3d69dd4e\") "
Sep 11 04:07:13.612740 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:07:13.612716    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2bf28659-3f51-4544-adad-96ca3d69dd4e-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "2bf28659-3f51-4544-adad-96ca3d69dd4e" (UID: "2bf28659-3f51-4544-adad-96ca3d69dd4e"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 04:07:13.713450 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:07:13.713365    2579 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2bf28659-3f51-4544-adad-96ca3d69dd4e-maas-api-service-ca\") on node \"ip-10-0-141-105.ec2.internal\" DevicePath \"\""
Sep 11 04:07:14.419152 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:07:14.419121    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-ebb9b5-hflzg"
Sep 11 04:07:14.419152 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:07:14.419137    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-ebb9b5-hflzg" event={"ID":"2bf28659-3f51-4544-adad-96ca3d69dd4e","Type":"ContainerDied","Data":"d326ac4fe7df3a3da28bf286cf9c77b5652bab5b5f09faed0f78065d8ab8a81c"}
Sep 11 04:07:14.419406 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:07:14.419195    2579 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d326ac4fe7df3a3da28bf286cf9c77b5652bab5b5f09faed0f78065d8ab8a81c"
Sep 11 04:07:21.330536 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:07:21.330296    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-ebb9b5-5cd7b7c997-6j6wt"]
Sep 11 04:07:21.331146 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:07:21.330904    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-ebb9b5-5cd7b7c997-6j6wt" podUID="9297c00e-c325-4db6-84e7-7ddc5a36dfe0" containerName="payload-processing" containerID="cri-o://ae28e1f737c9cd43096e45772da0b9d641db9ec6d0f8b93e4c7b0fc622f7e9d2" gracePeriod=30
Sep 11 04:07:21.456100 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:07:21.456057    2579 generic.go:356] "Generic (PLEG): container finished" podID="9297c00e-c325-4db6-84e7-7ddc5a36dfe0" containerID="ae28e1f737c9cd43096e45772da0b9d641db9ec6d0f8b93e4c7b0fc622f7e9d2" exitCode=0
Sep 11 04:07:21.456268 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:07:21.456122    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-ebb9b5-5cd7b7c997-6j6wt" event={"ID":"9297c00e-c325-4db6-84e7-7ddc5a36dfe0","Type":"ContainerDied","Data":"ae28e1f737c9cd43096e45772da0b9d641db9ec6d0f8b93e4c7b0fc622f7e9d2"}
Sep 11 04:07:21.491045 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:07:21.491009    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-ebb9b5-hflzg"]
Sep 11 04:07:21.494420 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:07:21.494396    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-ebb9b5-hflzg"]
Sep 11 04:07:21.586812 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:07:21.586757    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-ebb9b5-5cd7b7c997-6j6wt"
Sep 11 04:07:21.689753 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:07:21.689719    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9297c00e-c325-4db6-84e7-7ddc5a36dfe0-plugins-config-volume\") pod \"9297c00e-c325-4db6-84e7-7ddc5a36dfe0\" (UID: \"9297c00e-c325-4db6-84e7-7ddc5a36dfe0\") "
Sep 11 04:07:21.689970 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:07:21.689834    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pfd4v\" (UniqueName: \"kubernetes.io/projected/9297c00e-c325-4db6-84e7-7ddc5a36dfe0-kube-api-access-pfd4v\") pod \"9297c00e-c325-4db6-84e7-7ddc5a36dfe0\" (UID: \"9297c00e-c325-4db6-84e7-7ddc5a36dfe0\") "
Sep 11 04:07:21.690210 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:07:21.690189    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9297c00e-c325-4db6-84e7-7ddc5a36dfe0-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "9297c00e-c325-4db6-84e7-7ddc5a36dfe0" (UID: "9297c00e-c325-4db6-84e7-7ddc5a36dfe0"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 04:07:21.692193 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:07:21.692171    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9297c00e-c325-4db6-84e7-7ddc5a36dfe0-kube-api-access-pfd4v" (OuterVolumeSpecName: "kube-api-access-pfd4v") pod "9297c00e-c325-4db6-84e7-7ddc5a36dfe0" (UID: "9297c00e-c325-4db6-84e7-7ddc5a36dfe0"). InnerVolumeSpecName "kube-api-access-pfd4v". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 04:07:21.791521 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:07:21.791478    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pfd4v\" (UniqueName: \"kubernetes.io/projected/9297c00e-c325-4db6-84e7-7ddc5a36dfe0-kube-api-access-pfd4v\") on node \"ip-10-0-141-105.ec2.internal\" DevicePath \"\""
Sep 11 04:07:21.791521 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:07:21.791509    2579 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9297c00e-c325-4db6-84e7-7ddc5a36dfe0-plugins-config-volume\") on node \"ip-10-0-141-105.ec2.internal\" DevicePath \"\""
Sep 11 04:07:22.462425 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:07:22.462394    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-ebb9b5-5cd7b7c997-6j6wt"
Sep 11 04:07:22.462869 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:07:22.462436    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-ebb9b5-5cd7b7c997-6j6wt" event={"ID":"9297c00e-c325-4db6-84e7-7ddc5a36dfe0","Type":"ContainerDied","Data":"0eae428c22a1b2f5a9bdaa772f36b0d47ad96a3a0f23ae82a7b9009a4917561f"}
Sep 11 04:07:22.462869 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:07:22.462502    2579 scope.go:117] "RemoveContainer" containerID="ae28e1f737c9cd43096e45772da0b9d641db9ec6d0f8b93e4c7b0fc622f7e9d2"
Sep 11 04:07:22.485414 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:07:22.485386    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-ebb9b5-5cd7b7c997-6j6wt"]
Sep 11 04:07:22.488589 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:07:22.488563    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-ebb9b5-5cd7b7c997-6j6wt"]
Sep 11 04:07:22.853294 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:07:22.853269    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2bf28659-3f51-4544-adad-96ca3d69dd4e" path="/var/lib/kubelet/pods/2bf28659-3f51-4544-adad-96ca3d69dd4e/volumes"
Sep 11 04:07:22.853627 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:07:22.853614    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9297c00e-c325-4db6-84e7-7ddc5a36dfe0" path="/var/lib/kubelet/pods/9297c00e-c325-4db6-84e7-7ddc5a36dfe0/volumes"
Sep 11 04:07:26.001571 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:07:26.001537    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-5z4cn"]
Sep 11 04:07:31.853918 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:07:31.853876    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-4cb558-7986j"]
Sep 11 04:07:31.854531 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:07:31.854510    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9297c00e-c325-4db6-84e7-7ddc5a36dfe0" containerName="payload-processing"
Sep 11 04:07:31.854621 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:07:31.854535    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="9297c00e-c325-4db6-84e7-7ddc5a36dfe0" containerName="payload-processing"
Sep 11 04:07:31.854621 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:07:31.854548    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2bf28659-3f51-4544-adad-96ca3d69dd4e" containerName="revoke-keys"
Sep 11 04:07:31.854621 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:07:31.854557    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="2bf28659-3f51-4544-adad-96ca3d69dd4e" containerName="revoke-keys"
Sep 11 04:07:31.854781 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:07:31.854649    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="2bf28659-3f51-4544-adad-96ca3d69dd4e" containerName="revoke-keys"
Sep 11 04:07:31.854781 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:07:31.854668    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="9297c00e-c325-4db6-84e7-7ddc5a36dfe0" containerName="payload-processing"
Sep 11 04:07:31.857822 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:07:31.857801    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-4cb558-7986j"
Sep 11 04:07:31.860350 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:07:31.860330    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 11 04:07:31.860467 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:07:31.860354    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-95ckr\""
Sep 11 04:07:31.865414 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:07:31.865390    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-4cb558-7986j"]
Sep 11 04:07:31.988378 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:07:31.988341    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7749da9f-814e-4f17-b974-34f0de2e4179-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-4cb558-7986j\" (UID: \"7749da9f-814e-4f17-b974-34f0de2e4179\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-4cb558-7986j"
Sep 11 04:07:32.089273 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:07:32.089234    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7749da9f-814e-4f17-b974-34f0de2e4179-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-4cb558-7986j\" (UID: \"7749da9f-814e-4f17-b974-34f0de2e4179\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-4cb558-7986j"
Sep 11 04:07:32.089830 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:07:32.089809    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7749da9f-814e-4f17-b974-34f0de2e4179-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-4cb558-7986j\" (UID: \"7749da9f-814e-4f17-b974-34f0de2e4179\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-4cb558-7986j"
Sep 11 04:07:32.170316 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:07:32.170229    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-4cb558-7986j"
Sep 11 04:07:32.446830 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:07:32.446803    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-4cb558-7986j"]
Sep 11 04:07:32.448342 ip-10-0-141-105 kubenswrapper[2579]: W0911 04:07:32.448317    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7749da9f_814e_4f17_b974_34f0de2e4179.slice/crio-a3a9b11846f165054ae7211a5658df081f24aa8353fa720e570233a1ade8b24a WatchSource:0}: Error finding container a3a9b11846f165054ae7211a5658df081f24aa8353fa720e570233a1ade8b24a: Status 404 returned error can't find the container with id a3a9b11846f165054ae7211a5658df081f24aa8353fa720e570233a1ade8b24a
Sep 11 04:07:32.522707 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:07:32.522670    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-4cb558-7986j" event={"ID":"7749da9f-814e-4f17-b974-34f0de2e4179","Type":"ContainerStarted","Data":"a3a9b11846f165054ae7211a5658df081f24aa8353fa720e570233a1ade8b24a"}
Sep 11 04:07:33.529633 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:07:33.529597    2579 generic.go:356] "Generic (PLEG): container finished" podID="7749da9f-814e-4f17-b974-34f0de2e4179" containerID="aa331242eaa4c2190eb1e32bf809448b767ca1097ac3ff5deb6cadceafc95436" exitCode=0
Sep 11 04:07:33.530056 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:07:33.529681    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-4cb558-7986j" event={"ID":"7749da9f-814e-4f17-b974-34f0de2e4179","Type":"ContainerDied","Data":"aa331242eaa4c2190eb1e32bf809448b767ca1097ac3ff5deb6cadceafc95436"}
Sep 11 04:07:34.674324 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:07:34.674301    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-4cb558-7986j"
Sep 11 04:07:34.816366 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:07:34.816283    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7749da9f-814e-4f17-b974-34f0de2e4179-maas-api-service-ca\") pod \"7749da9f-814e-4f17-b974-34f0de2e4179\" (UID: \"7749da9f-814e-4f17-b974-34f0de2e4179\") "
Sep 11 04:07:34.816630 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:07:34.816607    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7749da9f-814e-4f17-b974-34f0de2e4179-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "7749da9f-814e-4f17-b974-34f0de2e4179" (UID: "7749da9f-814e-4f17-b974-34f0de2e4179"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 04:07:34.917841 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:07:34.917818    2579 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7749da9f-814e-4f17-b974-34f0de2e4179-maas-api-service-ca\") on node \"ip-10-0-141-105.ec2.internal\" DevicePath \"\""
Sep 11 04:07:35.541252 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:07:35.541222    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-4cb558-7986j"
Sep 11 04:07:35.541423 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:07:35.541218    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-4cb558-7986j" event={"ID":"7749da9f-814e-4f17-b974-34f0de2e4179","Type":"ContainerDied","Data":"a3a9b11846f165054ae7211a5658df081f24aa8353fa720e570233a1ade8b24a"}
Sep 11 04:07:35.541423 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:07:35.541346    2579 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a3a9b11846f165054ae7211a5658df081f24aa8353fa720e570233a1ade8b24a"
Sep 11 04:07:45.115680 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:07:45.115646    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-5z4cn"]
Sep 11 04:07:47.763176 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:07:47.763145    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-4cb558-7986j"]
Sep 11 04:07:47.764920 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:07:47.764897    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-4cb558-7986j"]
Sep 11 04:07:48.852775 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:07:48.852731    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7749da9f-814e-4f17-b974-34f0de2e4179" path="/var/lib/kubelet/pods/7749da9f-814e-4f17-b974-34f0de2e4179/volumes"
Sep 11 04:07:53.128936 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:07:53.128901    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-ipp-a-4cb558/ipp-route-4cb558-kserve-6c48d599c9-mhzjf"]
Sep 11 04:07:57.806511 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:07:57.806477    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-5z4cn"]
Sep 11 04:08:08.149042 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:08:08.149001    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-ipp-a-4cb558/ipp-route-4cb558-kserve-6c48d599c9-mhzjf" podUID="8bed8e52-a169-4466-bc3a-4bd5a34000f4" containerName="main" containerID="cri-o://d1f52273b394956042fcf2ff05432307f9b8c1dc9b3ef01fe076ebfa7e02e7a3" gracePeriod=45
Sep 11 04:08:08.401380 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:08:08.401316    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-4cb558/ipp-route-4cb558-kserve-6c48d599c9-mhzjf"
Sep 11 04:08:08.536202 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:08:08.536165    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8bed8e52-a169-4466-bc3a-4bd5a34000f4-dshm\") pod \"8bed8e52-a169-4466-bc3a-4bd5a34000f4\" (UID: \"8bed8e52-a169-4466-bc3a-4bd5a34000f4\") "
Sep 11 04:08:08.536377 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:08:08.536217    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-j754p\" (UniqueName: \"kubernetes.io/projected/8bed8e52-a169-4466-bc3a-4bd5a34000f4-kube-api-access-j754p\") pod \"8bed8e52-a169-4466-bc3a-4bd5a34000f4\" (UID: \"8bed8e52-a169-4466-bc3a-4bd5a34000f4\") "
Sep 11 04:08:08.536377 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:08:08.536287    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8bed8e52-a169-4466-bc3a-4bd5a34000f4-model-cache\") pod \"8bed8e52-a169-4466-bc3a-4bd5a34000f4\" (UID: \"8bed8e52-a169-4466-bc3a-4bd5a34000f4\") "
Sep 11 04:08:08.536377 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:08:08.536328    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8bed8e52-a169-4466-bc3a-4bd5a34000f4-tls-certs\") pod \"8bed8e52-a169-4466-bc3a-4bd5a34000f4\" (UID: \"8bed8e52-a169-4466-bc3a-4bd5a34000f4\") "
Sep 11 04:08:08.536377 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:08:08.536357    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8bed8e52-a169-4466-bc3a-4bd5a34000f4-home\") pod \"8bed8e52-a169-4466-bc3a-4bd5a34000f4\" (UID: \"8bed8e52-a169-4466-bc3a-4bd5a34000f4\") "
Sep 11 04:08:08.536552 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:08:08.536391    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8bed8e52-a169-4466-bc3a-4bd5a34000f4-tmp-dir\") pod \"8bed8e52-a169-4466-bc3a-4bd5a34000f4\" (UID: \"8bed8e52-a169-4466-bc3a-4bd5a34000f4\") "
Sep 11 04:08:08.536661 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:08:08.536615    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8bed8e52-a169-4466-bc3a-4bd5a34000f4-model-cache" (OuterVolumeSpecName: "model-cache") pod "8bed8e52-a169-4466-bc3a-4bd5a34000f4" (UID: "8bed8e52-a169-4466-bc3a-4bd5a34000f4"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 04:08:08.536752 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:08:08.536675    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8bed8e52-a169-4466-bc3a-4bd5a34000f4-home" (OuterVolumeSpecName: "home") pod "8bed8e52-a169-4466-bc3a-4bd5a34000f4" (UID: "8bed8e52-a169-4466-bc3a-4bd5a34000f4"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 04:08:08.536752 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:08:08.536720    2579 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8bed8e52-a169-4466-bc3a-4bd5a34000f4-model-cache\") on node \"ip-10-0-141-105.ec2.internal\" DevicePath \"\""
Sep 11 04:08:08.536845 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:08:08.536747    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8bed8e52-a169-4466-bc3a-4bd5a34000f4-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "8bed8e52-a169-4466-bc3a-4bd5a34000f4" (UID: "8bed8e52-a169-4466-bc3a-4bd5a34000f4"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 04:08:08.538604 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:08:08.538580    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8bed8e52-a169-4466-bc3a-4bd5a34000f4-dshm" (OuterVolumeSpecName: "dshm") pod "8bed8e52-a169-4466-bc3a-4bd5a34000f4" (UID: "8bed8e52-a169-4466-bc3a-4bd5a34000f4"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 04:08:08.538709 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:08:08.538635    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8bed8e52-a169-4466-bc3a-4bd5a34000f4-kube-api-access-j754p" (OuterVolumeSpecName: "kube-api-access-j754p") pod "8bed8e52-a169-4466-bc3a-4bd5a34000f4" (UID: "8bed8e52-a169-4466-bc3a-4bd5a34000f4"). InnerVolumeSpecName "kube-api-access-j754p". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 04:08:08.538753 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:08:08.538731    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8bed8e52-a169-4466-bc3a-4bd5a34000f4-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "8bed8e52-a169-4466-bc3a-4bd5a34000f4" (UID: "8bed8e52-a169-4466-bc3a-4bd5a34000f4"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 04:08:08.637521 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:08:08.637494    2579 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8bed8e52-a169-4466-bc3a-4bd5a34000f4-tls-certs\") on node \"ip-10-0-141-105.ec2.internal\" DevicePath \"\""
Sep 11 04:08:08.637521 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:08:08.637517    2579 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8bed8e52-a169-4466-bc3a-4bd5a34000f4-home\") on node \"ip-10-0-141-105.ec2.internal\" DevicePath \"\""
Sep 11 04:08:08.637521 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:08:08.637526    2579 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8bed8e52-a169-4466-bc3a-4bd5a34000f4-tmp-dir\") on node \"ip-10-0-141-105.ec2.internal\" DevicePath \"\""
Sep 11 04:08:08.637756 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:08:08.637533    2579 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8bed8e52-a169-4466-bc3a-4bd5a34000f4-dshm\") on node \"ip-10-0-141-105.ec2.internal\" DevicePath \"\""
Sep 11 04:08:08.637756 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:08:08.637541    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j754p\" (UniqueName: \"kubernetes.io/projected/8bed8e52-a169-4466-bc3a-4bd5a34000f4-kube-api-access-j754p\") on node \"ip-10-0-141-105.ec2.internal\" DevicePath \"\""
Sep 11 04:08:08.728270 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:08:08.728174    2579 generic.go:356] "Generic (PLEG): container finished" podID="8bed8e52-a169-4466-bc3a-4bd5a34000f4" containerID="d1f52273b394956042fcf2ff05432307f9b8c1dc9b3ef01fe076ebfa7e02e7a3" exitCode=0
Sep 11 04:08:08.728270 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:08:08.728215    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-4cb558/ipp-route-4cb558-kserve-6c48d599c9-mhzjf"
Sep 11 04:08:08.728461 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:08:08.728267    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-4cb558/ipp-route-4cb558-kserve-6c48d599c9-mhzjf" event={"ID":"8bed8e52-a169-4466-bc3a-4bd5a34000f4","Type":"ContainerDied","Data":"d1f52273b394956042fcf2ff05432307f9b8c1dc9b3ef01fe076ebfa7e02e7a3"}
Sep 11 04:08:08.728461 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:08:08.728313    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-4cb558/ipp-route-4cb558-kserve-6c48d599c9-mhzjf" event={"ID":"8bed8e52-a169-4466-bc3a-4bd5a34000f4","Type":"ContainerDied","Data":"90595ad360610a370eebb5c9849d1ad87f9178aa73bc076a15824e50fa2d371a"}
Sep 11 04:08:08.728461 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:08:08.728331    2579 scope.go:117] "RemoveContainer" containerID="d1f52273b394956042fcf2ff05432307f9b8c1dc9b3ef01fe076ebfa7e02e7a3"
Sep 11 04:08:08.738425 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:08:08.738409    2579 scope.go:117] "RemoveContainer" containerID="d1f52273b394956042fcf2ff05432307f9b8c1dc9b3ef01fe076ebfa7e02e7a3"
Sep 11 04:08:08.738735 ip-10-0-141-105 kubenswrapper[2579]: E0911 04:08:08.738719    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d1f52273b394956042fcf2ff05432307f9b8c1dc9b3ef01fe076ebfa7e02e7a3\": container with ID starting with d1f52273b394956042fcf2ff05432307f9b8c1dc9b3ef01fe076ebfa7e02e7a3 not found: ID does not exist" containerID="d1f52273b394956042fcf2ff05432307f9b8c1dc9b3ef01fe076ebfa7e02e7a3"
Sep 11 04:08:08.738799 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:08:08.738747    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d1f52273b394956042fcf2ff05432307f9b8c1dc9b3ef01fe076ebfa7e02e7a3"} err="failed to get container status \"d1f52273b394956042fcf2ff05432307f9b8c1dc9b3ef01fe076ebfa7e02e7a3\": rpc error: code = NotFound desc = could not find container \"d1f52273b394956042fcf2ff05432307f9b8c1dc9b3ef01fe076ebfa7e02e7a3\": container with ID starting with d1f52273b394956042fcf2ff05432307f9b8c1dc9b3ef01fe076ebfa7e02e7a3 not found: ID does not exist"
Sep 11 04:08:08.750352 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:08:08.750327    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-ipp-a-4cb558/ipp-route-4cb558-kserve-6c48d599c9-mhzjf"]
Sep 11 04:08:08.754023 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:08:08.753992    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-ipp-a-4cb558/ipp-route-4cb558-kserve-6c48d599c9-mhzjf"]
Sep 11 04:08:08.851858 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:08:08.851826    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8bed8e52-a169-4466-bc3a-4bd5a34000f4" path="/var/lib/kubelet/pods/8bed8e52-a169-4466-bc3a-4bd5a34000f4/volumes"
Sep 11 04:08:36.513617 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:08:36.513579    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-0f85b8-vfsfz"]
Sep 11 04:08:36.514097 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:08:36.514005    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7749da9f-814e-4f17-b974-34f0de2e4179" containerName="revoke-keys"
Sep 11 04:08:36.514097 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:08:36.514016    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="7749da9f-814e-4f17-b974-34f0de2e4179" containerName="revoke-keys"
Sep 11 04:08:36.514097 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:08:36.514029    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8bed8e52-a169-4466-bc3a-4bd5a34000f4" containerName="main"
Sep 11 04:08:36.514097 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:08:36.514034    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="8bed8e52-a169-4466-bc3a-4bd5a34000f4" containerName="main"
Sep 11 04:08:36.514248 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:08:36.514109    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="7749da9f-814e-4f17-b974-34f0de2e4179" containerName="revoke-keys"
Sep 11 04:08:36.514248 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:08:36.514119    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="8bed8e52-a169-4466-bc3a-4bd5a34000f4" containerName="main"
Sep 11 04:08:36.518550 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:08:36.518534    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-0f85b8-vfsfz"
Sep 11 04:08:36.521378 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:08:36.521347    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 11 04:08:36.521512 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:08:36.521396    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-95ckr\""
Sep 11 04:08:36.524552 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:08:36.524531    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-0f85b8-vfsfz"]
Sep 11 04:08:36.604769 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:08:36.604733    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3da98911-d020-4050-9ed2-719bd1258430-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-0f85b8-vfsfz\" (UID: \"3da98911-d020-4050-9ed2-719bd1258430\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-0f85b8-vfsfz"
Sep 11 04:08:36.705505 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:08:36.705468    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3da98911-d020-4050-9ed2-719bd1258430-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-0f85b8-vfsfz\" (UID: \"3da98911-d020-4050-9ed2-719bd1258430\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-0f85b8-vfsfz"
Sep 11 04:08:36.706161 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:08:36.706142    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3da98911-d020-4050-9ed2-719bd1258430-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-0f85b8-vfsfz\" (UID: \"3da98911-d020-4050-9ed2-719bd1258430\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-0f85b8-vfsfz"
Sep 11 04:08:36.832366 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:08:36.832273    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-0f85b8-vfsfz"
Sep 11 04:08:37.110721 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:08:37.110698    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-0f85b8-vfsfz"]
Sep 11 04:08:37.113023 ip-10-0-141-105 kubenswrapper[2579]: W0911 04:08:37.112990    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3da98911_d020_4050_9ed2_719bd1258430.slice/crio-b2a16ba1a2455da44ce4f7d0ae68bb5f5c8550e7e301995dbf2a4d22292d1682 WatchSource:0}: Error finding container b2a16ba1a2455da44ce4f7d0ae68bb5f5c8550e7e301995dbf2a4d22292d1682: Status 404 returned error can't find the container with id b2a16ba1a2455da44ce4f7d0ae68bb5f5c8550e7e301995dbf2a4d22292d1682
Sep 11 04:08:37.909718 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:08:37.909685    2579 generic.go:356] "Generic (PLEG): container finished" podID="3da98911-d020-4050-9ed2-719bd1258430" containerID="74b8d2ab86001072da3ad1f91255fc2ca559c6d6d15a8a84714ce8bdd061f70b" exitCode=0
Sep 11 04:08:37.910048 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:08:37.909736    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-0f85b8-vfsfz" event={"ID":"3da98911-d020-4050-9ed2-719bd1258430","Type":"ContainerDied","Data":"74b8d2ab86001072da3ad1f91255fc2ca559c6d6d15a8a84714ce8bdd061f70b"}
Sep 11 04:08:37.910048 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:08:37.909780    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-0f85b8-vfsfz" event={"ID":"3da98911-d020-4050-9ed2-719bd1258430","Type":"ContainerStarted","Data":"b2a16ba1a2455da44ce4f7d0ae68bb5f5c8550e7e301995dbf2a4d22292d1682"}
Sep 11 04:08:39.053846 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:08:39.053822    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-0f85b8-vfsfz"
Sep 11 04:08:39.126931 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:08:39.126900    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3da98911-d020-4050-9ed2-719bd1258430-maas-api-service-ca\") pod \"3da98911-d020-4050-9ed2-719bd1258430\" (UID: \"3da98911-d020-4050-9ed2-719bd1258430\") "
Sep 11 04:08:39.127296 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:08:39.127275    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3da98911-d020-4050-9ed2-719bd1258430-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "3da98911-d020-4050-9ed2-719bd1258430" (UID: "3da98911-d020-4050-9ed2-719bd1258430"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 04:08:39.228503 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:08:39.228438    2579 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3da98911-d020-4050-9ed2-719bd1258430-maas-api-service-ca\") on node \"ip-10-0-141-105.ec2.internal\" DevicePath \"\""
Sep 11 04:08:39.922035 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:08:39.921997    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-0f85b8-vfsfz"
Sep 11 04:08:39.922035 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:08:39.922006    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-0f85b8-vfsfz" event={"ID":"3da98911-d020-4050-9ed2-719bd1258430","Type":"ContainerDied","Data":"b2a16ba1a2455da44ce4f7d0ae68bb5f5c8550e7e301995dbf2a4d22292d1682"}
Sep 11 04:08:39.922264 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:08:39.922046    2579 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b2a16ba1a2455da44ce4f7d0ae68bb5f5c8550e7e301995dbf2a4d22292d1682"
Sep 11 04:08:47.165232 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:08:47.165149    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-0f85b8-6cd9479485-9nzzd"]
Sep 11 04:08:47.166197 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:08:47.165405    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-0f85b8-6cd9479485-9nzzd" podUID="38ce28d0-5809-4f66-8772-c11d42660225" containerName="payload-processing" containerID="cri-o://11ae01c1ef654a77b8cbdc811970a4bd7b75c30400378ed5667c5035eda9a411" gracePeriod=30
Sep 11 04:08:47.177401 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:08:47.177359    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-0f85b8-88497b898-lq6rr"]
Sep 11 04:08:47.177675 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:08:47.177599    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-0f85b8-88497b898-lq6rr" podUID="a2497017-256e-4537-86f3-8bf8de45e4d3" containerName="payload-pre-processing" containerID="cri-o://40520880fa9826eb6dd3d335b673f10282d5bd9a0e64e86f18475aa214ec3170" gracePeriod=30
Sep 11 04:08:47.430665 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:08:47.430599    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-0f85b8-6cd9479485-9nzzd"
Sep 11 04:08:47.439708 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:08:47.439665    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-0f85b8-vfsfz"]
Sep 11 04:08:47.448358 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:08:47.448326    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-0f85b8-vfsfz"]
Sep 11 04:08:47.466531 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:08:47.466506    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-0f85b8-88497b898-lq6rr"
Sep 11 04:08:47.608013 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:08:47.607982    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/38ce28d0-5809-4f66-8772-c11d42660225-plugins-config-volume\") pod \"38ce28d0-5809-4f66-8772-c11d42660225\" (UID: \"38ce28d0-5809-4f66-8772-c11d42660225\") "
Sep 11 04:08:47.608204 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:08:47.608117    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-48kcq\" (UniqueName: \"kubernetes.io/projected/a2497017-256e-4537-86f3-8bf8de45e4d3-kube-api-access-48kcq\") pod \"a2497017-256e-4537-86f3-8bf8de45e4d3\" (UID: \"a2497017-256e-4537-86f3-8bf8de45e4d3\") "
Sep 11 04:08:47.608204 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:08:47.608151    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a2497017-256e-4537-86f3-8bf8de45e4d3-plugins-config-volume\") pod \"a2497017-256e-4537-86f3-8bf8de45e4d3\" (UID: \"a2497017-256e-4537-86f3-8bf8de45e4d3\") "
Sep 11 04:08:47.608315 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:08:47.608206    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wm99n\" (UniqueName: \"kubernetes.io/projected/38ce28d0-5809-4f66-8772-c11d42660225-kube-api-access-wm99n\") pod \"38ce28d0-5809-4f66-8772-c11d42660225\" (UID: \"38ce28d0-5809-4f66-8772-c11d42660225\") "
Sep 11 04:08:47.608533 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:08:47.608507    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/38ce28d0-5809-4f66-8772-c11d42660225-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "38ce28d0-5809-4f66-8772-c11d42660225" (UID: "38ce28d0-5809-4f66-8772-c11d42660225"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 04:08:47.608588 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:08:47.608542    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a2497017-256e-4537-86f3-8bf8de45e4d3-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "a2497017-256e-4537-86f3-8bf8de45e4d3" (UID: "a2497017-256e-4537-86f3-8bf8de45e4d3"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 04:08:47.610448 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:08:47.610413    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a2497017-256e-4537-86f3-8bf8de45e4d3-kube-api-access-48kcq" (OuterVolumeSpecName: "kube-api-access-48kcq") pod "a2497017-256e-4537-86f3-8bf8de45e4d3" (UID: "a2497017-256e-4537-86f3-8bf8de45e4d3"). InnerVolumeSpecName "kube-api-access-48kcq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 04:08:47.610566 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:08:47.610520    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/38ce28d0-5809-4f66-8772-c11d42660225-kube-api-access-wm99n" (OuterVolumeSpecName: "kube-api-access-wm99n") pod "38ce28d0-5809-4f66-8772-c11d42660225" (UID: "38ce28d0-5809-4f66-8772-c11d42660225"). InnerVolumeSpecName "kube-api-access-wm99n". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 04:08:47.709799 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:08:47.709711    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-48kcq\" (UniqueName: \"kubernetes.io/projected/a2497017-256e-4537-86f3-8bf8de45e4d3-kube-api-access-48kcq\") on node \"ip-10-0-141-105.ec2.internal\" DevicePath \"\""
Sep 11 04:08:47.709799 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:08:47.709743    2579 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a2497017-256e-4537-86f3-8bf8de45e4d3-plugins-config-volume\") on node \"ip-10-0-141-105.ec2.internal\" DevicePath \"\""
Sep 11 04:08:47.709799 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:08:47.709754    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wm99n\" (UniqueName: \"kubernetes.io/projected/38ce28d0-5809-4f66-8772-c11d42660225-kube-api-access-wm99n\") on node \"ip-10-0-141-105.ec2.internal\" DevicePath \"\""
Sep 11 04:08:47.709799 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:08:47.709763    2579 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/38ce28d0-5809-4f66-8772-c11d42660225-plugins-config-volume\") on node \"ip-10-0-141-105.ec2.internal\" DevicePath \"\""
Sep 11 04:08:47.966761 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:08:47.966671    2579 generic.go:356] "Generic (PLEG): container finished" podID="a2497017-256e-4537-86f3-8bf8de45e4d3" containerID="40520880fa9826eb6dd3d335b673f10282d5bd9a0e64e86f18475aa214ec3170" exitCode=0
Sep 11 04:08:47.966761 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:08:47.966722    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-0f85b8-88497b898-lq6rr"
Sep 11 04:08:47.966998 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:08:47.966760    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-0f85b8-88497b898-lq6rr" event={"ID":"a2497017-256e-4537-86f3-8bf8de45e4d3","Type":"ContainerDied","Data":"40520880fa9826eb6dd3d335b673f10282d5bd9a0e64e86f18475aa214ec3170"}
Sep 11 04:08:47.966998 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:08:47.966818    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-0f85b8-88497b898-lq6rr" event={"ID":"a2497017-256e-4537-86f3-8bf8de45e4d3","Type":"ContainerDied","Data":"f896185d6cf40992316eba3ea18d7c2dcab8388990c61aa5d7fd80f4a7523171"}
Sep 11 04:08:47.966998 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:08:47.966848    2579 scope.go:117] "RemoveContainer" containerID="40520880fa9826eb6dd3d335b673f10282d5bd9a0e64e86f18475aa214ec3170"
Sep 11 04:08:47.968211 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:08:47.968190    2579 generic.go:356] "Generic (PLEG): container finished" podID="38ce28d0-5809-4f66-8772-c11d42660225" containerID="11ae01c1ef654a77b8cbdc811970a4bd7b75c30400378ed5667c5035eda9a411" exitCode=0
Sep 11 04:08:47.968328 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:08:47.968240    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-0f85b8-6cd9479485-9nzzd"
Sep 11 04:08:47.968328 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:08:47.968271    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-0f85b8-6cd9479485-9nzzd" event={"ID":"38ce28d0-5809-4f66-8772-c11d42660225","Type":"ContainerDied","Data":"11ae01c1ef654a77b8cbdc811970a4bd7b75c30400378ed5667c5035eda9a411"}
Sep 11 04:08:47.968328 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:08:47.968314    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-0f85b8-6cd9479485-9nzzd" event={"ID":"38ce28d0-5809-4f66-8772-c11d42660225","Type":"ContainerDied","Data":"0582a99887dc67af10582c0f3cf05505f82705b20ead4d5fbd61fb8aa03bc691"}
Sep 11 04:08:47.978034 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:08:47.978004    2579 scope.go:117] "RemoveContainer" containerID="40520880fa9826eb6dd3d335b673f10282d5bd9a0e64e86f18475aa214ec3170"
Sep 11 04:08:47.978510 ip-10-0-141-105 kubenswrapper[2579]: E0911 04:08:47.978406    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"40520880fa9826eb6dd3d335b673f10282d5bd9a0e64e86f18475aa214ec3170\": container with ID starting with 40520880fa9826eb6dd3d335b673f10282d5bd9a0e64e86f18475aa214ec3170 not found: ID does not exist" containerID="40520880fa9826eb6dd3d335b673f10282d5bd9a0e64e86f18475aa214ec3170"
Sep 11 04:08:47.978510 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:08:47.978435    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"40520880fa9826eb6dd3d335b673f10282d5bd9a0e64e86f18475aa214ec3170"} err="failed to get container status \"40520880fa9826eb6dd3d335b673f10282d5bd9a0e64e86f18475aa214ec3170\": rpc error: code = NotFound desc = could not find container \"40520880fa9826eb6dd3d335b673f10282d5bd9a0e64e86f18475aa214ec3170\": container with ID starting with 40520880fa9826eb6dd3d335b673f10282d5bd9a0e64e86f18475aa214ec3170 not found: ID does not exist"
Sep 11 04:08:47.978510 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:08:47.978453    2579 scope.go:117] "RemoveContainer" containerID="11ae01c1ef654a77b8cbdc811970a4bd7b75c30400378ed5667c5035eda9a411"
Sep 11 04:08:47.986812 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:08:47.986796    2579 scope.go:117] "RemoveContainer" containerID="11ae01c1ef654a77b8cbdc811970a4bd7b75c30400378ed5667c5035eda9a411"
Sep 11 04:08:47.987117 ip-10-0-141-105 kubenswrapper[2579]: E0911 04:08:47.987094    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"11ae01c1ef654a77b8cbdc811970a4bd7b75c30400378ed5667c5035eda9a411\": container with ID starting with 11ae01c1ef654a77b8cbdc811970a4bd7b75c30400378ed5667c5035eda9a411 not found: ID does not exist" containerID="11ae01c1ef654a77b8cbdc811970a4bd7b75c30400378ed5667c5035eda9a411"
Sep 11 04:08:47.987204 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:08:47.987125    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"11ae01c1ef654a77b8cbdc811970a4bd7b75c30400378ed5667c5035eda9a411"} err="failed to get container status \"11ae01c1ef654a77b8cbdc811970a4bd7b75c30400378ed5667c5035eda9a411\": rpc error: code = NotFound desc = could not find container \"11ae01c1ef654a77b8cbdc811970a4bd7b75c30400378ed5667c5035eda9a411\": container with ID starting with 11ae01c1ef654a77b8cbdc811970a4bd7b75c30400378ed5667c5035eda9a411 not found: ID does not exist"
Sep 11 04:08:48.005930 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:08:48.005905    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-0f85b8-6cd9479485-9nzzd"]
Sep 11 04:08:48.018767 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:08:48.018744    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-0f85b8-6cd9479485-9nzzd"]
Sep 11 04:08:48.041526 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:08:48.041371    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-0f85b8-88497b898-lq6rr"]
Sep 11 04:08:48.045155 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:08:48.045132    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-0f85b8-88497b898-lq6rr"]
Sep 11 04:08:48.852613 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:08:48.852579    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="38ce28d0-5809-4f66-8772-c11d42660225" path="/var/lib/kubelet/pods/38ce28d0-5809-4f66-8772-c11d42660225/volumes"
Sep 11 04:08:48.852998 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:08:48.852916    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3da98911-d020-4050-9ed2-719bd1258430" path="/var/lib/kubelet/pods/3da98911-d020-4050-9ed2-719bd1258430/volumes"
Sep 11 04:08:48.853248 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:08:48.853234    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a2497017-256e-4537-86f3-8bf8de45e4d3" path="/var/lib/kubelet/pods/a2497017-256e-4537-86f3-8bf8de45e4d3/volumes"
Sep 11 04:08:50.227953 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:08:50.227921    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-5z4cn"]
Sep 11 04:08:54.067057 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:08:54.067022    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-0f85b8-openshift-default-7d957b55c4-pskrc"]
Sep 11 04:08:54.067589 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:08:54.067277    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-b-0f85b8-openshift-default-7d957b55c4-pskrc" podUID="ceb39acb-e2b8-4a34-8936-6c36677fa484" containerName="istio-proxy" containerID="cri-o://b52c194e64c5ae57cd4e7e0060f2b719f4f264911aabe5cb192d3e03531fecf5" gracePeriod=30
Sep 11 04:08:54.412046 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:08:54.412011    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-b4d3cf-xk7b8"]
Sep 11 04:08:54.412539 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:08:54.412518    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3da98911-d020-4050-9ed2-719bd1258430" containerName="revoke-keys"
Sep 11 04:08:54.412649 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:08:54.412541    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="3da98911-d020-4050-9ed2-719bd1258430" containerName="revoke-keys"
Sep 11 04:08:54.412649 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:08:54.412557    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a2497017-256e-4537-86f3-8bf8de45e4d3" containerName="payload-pre-processing"
Sep 11 04:08:54.412649 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:08:54.412565    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="a2497017-256e-4537-86f3-8bf8de45e4d3" containerName="payload-pre-processing"
Sep 11 04:08:54.412649 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:08:54.412595    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="38ce28d0-5809-4f66-8772-c11d42660225" containerName="payload-processing"
Sep 11 04:08:54.412649 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:08:54.412604    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="38ce28d0-5809-4f66-8772-c11d42660225" containerName="payload-processing"
Sep 11 04:08:54.412923 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:08:54.412692    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="38ce28d0-5809-4f66-8772-c11d42660225" containerName="payload-processing"
Sep 11 04:08:54.412923 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:08:54.412709    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="a2497017-256e-4537-86f3-8bf8de45e4d3" containerName="payload-pre-processing"
Sep 11 04:08:54.412923 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:08:54.412720    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="3da98911-d020-4050-9ed2-719bd1258430" containerName="revoke-keys"
Sep 11 04:08:54.415919 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:08:54.415898    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-b4d3cf-xk7b8"
Sep 11 04:08:54.418528 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:08:54.418510    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-95ckr\""
Sep 11 04:08:54.418815 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:08:54.418790    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 11 04:08:54.424273 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:08:54.424251    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-b4d3cf-xk7b8"]
Sep 11 04:08:54.476740 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:08:54.476708    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/08dfbebb-3864-4f96-b25e-404bd66c3fd0-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-b4d3cf-xk7b8\" (UID: \"08dfbebb-3864-4f96-b25e-404bd66c3fd0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-b4d3cf-xk7b8"
Sep 11 04:08:54.577787 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:08:54.577750    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/08dfbebb-3864-4f96-b25e-404bd66c3fd0-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-b4d3cf-xk7b8\" (UID: \"08dfbebb-3864-4f96-b25e-404bd66c3fd0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-b4d3cf-xk7b8"
Sep 11 04:08:54.578389 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:08:54.578370    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/08dfbebb-3864-4f96-b25e-404bd66c3fd0-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-b4d3cf-xk7b8\" (UID: \"08dfbebb-3864-4f96-b25e-404bd66c3fd0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-b4d3cf-xk7b8"
Sep 11 04:08:54.728335 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:08:54.728255    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-b4d3cf-xk7b8"
Sep 11 04:08:55.006873 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:08:55.006850    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-b4d3cf-xk7b8"]
Sep 11 04:08:55.008938 ip-10-0-141-105 kubenswrapper[2579]: W0911 04:08:55.008915    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod08dfbebb_3864_4f96_b25e_404bd66c3fd0.slice/crio-d410787c7422d03e13b759bc0d620a9d7a66a5cb8d55a23e294f0e0223166a13 WatchSource:0}: Error finding container d410787c7422d03e13b759bc0d620a9d7a66a5cb8d55a23e294f0e0223166a13: Status 404 returned error can't find the container with id d410787c7422d03e13b759bc0d620a9d7a66a5cb8d55a23e294f0e0223166a13
Sep 11 04:08:56.015456 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:08:56.015364    2579 generic.go:356] "Generic (PLEG): container finished" podID="08dfbebb-3864-4f96-b25e-404bd66c3fd0" containerID="b8eee1f55d04c0fbb7a4c99d52a7151b1c49152ffac9c040c03c7c68621a6cc6" exitCode=0
Sep 11 04:08:56.015845 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:08:56.015452    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-b4d3cf-xk7b8" event={"ID":"08dfbebb-3864-4f96-b25e-404bd66c3fd0","Type":"ContainerDied","Data":"b8eee1f55d04c0fbb7a4c99d52a7151b1c49152ffac9c040c03c7c68621a6cc6"}
Sep 11 04:08:56.015845 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:08:56.015503    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-b4d3cf-xk7b8" event={"ID":"08dfbebb-3864-4f96-b25e-404bd66c3fd0","Type":"ContainerStarted","Data":"d410787c7422d03e13b759bc0d620a9d7a66a5cb8d55a23e294f0e0223166a13"}
Sep 11 04:08:57.160963 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:08:57.160935    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-b4d3cf-xk7b8"
Sep 11 04:08:57.201729 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:08:57.201701    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/08dfbebb-3864-4f96-b25e-404bd66c3fd0-maas-api-service-ca\") pod \"08dfbebb-3864-4f96-b25e-404bd66c3fd0\" (UID: \"08dfbebb-3864-4f96-b25e-404bd66c3fd0\") "
Sep 11 04:08:57.202165 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:08:57.202141    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/08dfbebb-3864-4f96-b25e-404bd66c3fd0-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "08dfbebb-3864-4f96-b25e-404bd66c3fd0" (UID: "08dfbebb-3864-4f96-b25e-404bd66c3fd0"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 04:08:57.302889 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:08:57.302807    2579 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/08dfbebb-3864-4f96-b25e-404bd66c3fd0-maas-api-service-ca\") on node \"ip-10-0-141-105.ec2.internal\" DevicePath \"\""
Sep 11 04:08:58.027568 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:08:58.027540    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-b4d3cf-xk7b8"
Sep 11 04:08:58.027747 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:08:58.027539    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-b4d3cf-xk7b8" event={"ID":"08dfbebb-3864-4f96-b25e-404bd66c3fd0","Type":"ContainerDied","Data":"d410787c7422d03e13b759bc0d620a9d7a66a5cb8d55a23e294f0e0223166a13"}
Sep 11 04:08:58.027747 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:08:58.027657    2579 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d410787c7422d03e13b759bc0d620a9d7a66a5cb8d55a23e294f0e0223166a13"
Sep 11 04:08:59.324000 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:08:59.323974    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-0f85b8-openshift-default-7d957b55c4-pskrc"
Sep 11 04:08:59.422379 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:08:59.422345    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ceb39acb-e2b8-4a34-8936-6c36677fa484-istiod-ca-cert\") pod \"ceb39acb-e2b8-4a34-8936-6c36677fa484\" (UID: \"ceb39acb-e2b8-4a34-8936-6c36677fa484\") "
Sep 11 04:08:59.422567 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:08:59.422397    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ceb39acb-e2b8-4a34-8936-6c36677fa484-istio-token\") pod \"ceb39acb-e2b8-4a34-8936-6c36677fa484\" (UID: \"ceb39acb-e2b8-4a34-8936-6c36677fa484\") "
Sep 11 04:08:59.422567 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:08:59.422422    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ceb39acb-e2b8-4a34-8936-6c36677fa484-istio-podinfo\") pod \"ceb39acb-e2b8-4a34-8936-6c36677fa484\" (UID: \"ceb39acb-e2b8-4a34-8936-6c36677fa484\") "
Sep 11 04:08:59.422567 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:08:59.422439    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4dnxf\" (UniqueName: \"kubernetes.io/projected/ceb39acb-e2b8-4a34-8936-6c36677fa484-kube-api-access-4dnxf\") pod \"ceb39acb-e2b8-4a34-8936-6c36677fa484\" (UID: \"ceb39acb-e2b8-4a34-8936-6c36677fa484\") "
Sep 11 04:08:59.422567 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:08:59.422467    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ceb39acb-e2b8-4a34-8936-6c36677fa484-credential-socket\") pod \"ceb39acb-e2b8-4a34-8936-6c36677fa484\" (UID: \"ceb39acb-e2b8-4a34-8936-6c36677fa484\") "
Sep 11 04:08:59.422567 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:08:59.422506    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ceb39acb-e2b8-4a34-8936-6c36677fa484-istio-envoy\") pod \"ceb39acb-e2b8-4a34-8936-6c36677fa484\" (UID: \"ceb39acb-e2b8-4a34-8936-6c36677fa484\") "
Sep 11 04:08:59.422834 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:08:59.422583    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ceb39acb-e2b8-4a34-8936-6c36677fa484-workload-certs\") pod \"ceb39acb-e2b8-4a34-8936-6c36677fa484\" (UID: \"ceb39acb-e2b8-4a34-8936-6c36677fa484\") "
Sep 11 04:08:59.422834 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:08:59.422627    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ceb39acb-e2b8-4a34-8936-6c36677fa484-workload-socket\") pod \"ceb39acb-e2b8-4a34-8936-6c36677fa484\" (UID: \"ceb39acb-e2b8-4a34-8936-6c36677fa484\") "
Sep 11 04:08:59.422834 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:08:59.422654    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ceb39acb-e2b8-4a34-8936-6c36677fa484-istio-data\") pod \"ceb39acb-e2b8-4a34-8936-6c36677fa484\" (UID: \"ceb39acb-e2b8-4a34-8936-6c36677fa484\") "
Sep 11 04:08:59.422994 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:08:59.422901    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ceb39acb-e2b8-4a34-8936-6c36677fa484-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "ceb39acb-e2b8-4a34-8936-6c36677fa484" (UID: "ceb39acb-e2b8-4a34-8936-6c36677fa484"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 04:08:59.422994 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:08:59.422806    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ceb39acb-e2b8-4a34-8936-6c36677fa484-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "ceb39acb-e2b8-4a34-8936-6c36677fa484" (UID: "ceb39acb-e2b8-4a34-8936-6c36677fa484"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 04:08:59.423223 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:08:59.423189    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ceb39acb-e2b8-4a34-8936-6c36677fa484-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "ceb39acb-e2b8-4a34-8936-6c36677fa484" (UID: "ceb39acb-e2b8-4a34-8936-6c36677fa484"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 04:08:59.423223 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:08:59.423202    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ceb39acb-e2b8-4a34-8936-6c36677fa484-istio-data" (OuterVolumeSpecName: "istio-data") pod "ceb39acb-e2b8-4a34-8936-6c36677fa484" (UID: "ceb39acb-e2b8-4a34-8936-6c36677fa484"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 04:08:59.423416 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:08:59.423303    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ceb39acb-e2b8-4a34-8936-6c36677fa484-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "ceb39acb-e2b8-4a34-8936-6c36677fa484" (UID: "ceb39acb-e2b8-4a34-8936-6c36677fa484"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 04:08:59.425508 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:08:59.425469    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ceb39acb-e2b8-4a34-8936-6c36677fa484-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "ceb39acb-e2b8-4a34-8936-6c36677fa484" (UID: "ceb39acb-e2b8-4a34-8936-6c36677fa484"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 04:08:59.425508 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:08:59.425481    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ceb39acb-e2b8-4a34-8936-6c36677fa484-istio-token" (OuterVolumeSpecName: "istio-token") pod "ceb39acb-e2b8-4a34-8936-6c36677fa484" (UID: "ceb39acb-e2b8-4a34-8936-6c36677fa484"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 04:08:59.425672 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:08:59.425548    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ceb39acb-e2b8-4a34-8936-6c36677fa484-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "ceb39acb-e2b8-4a34-8936-6c36677fa484" (UID: "ceb39acb-e2b8-4a34-8936-6c36677fa484"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 11 04:08:59.425672 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:08:59.425548    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ceb39acb-e2b8-4a34-8936-6c36677fa484-kube-api-access-4dnxf" (OuterVolumeSpecName: "kube-api-access-4dnxf") pod "ceb39acb-e2b8-4a34-8936-6c36677fa484" (UID: "ceb39acb-e2b8-4a34-8936-6c36677fa484"). InnerVolumeSpecName "kube-api-access-4dnxf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 04:08:59.523442 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:08:59.523396    2579 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ceb39acb-e2b8-4a34-8936-6c36677fa484-workload-certs\") on node \"ip-10-0-141-105.ec2.internal\" DevicePath \"\""
Sep 11 04:08:59.523442 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:08:59.523436    2579 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ceb39acb-e2b8-4a34-8936-6c36677fa484-workload-socket\") on node \"ip-10-0-141-105.ec2.internal\" DevicePath \"\""
Sep 11 04:08:59.523442 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:08:59.523447    2579 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ceb39acb-e2b8-4a34-8936-6c36677fa484-istio-data\") on node \"ip-10-0-141-105.ec2.internal\" DevicePath \"\""
Sep 11 04:08:59.523686 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:08:59.523472    2579 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ceb39acb-e2b8-4a34-8936-6c36677fa484-istiod-ca-cert\") on node \"ip-10-0-141-105.ec2.internal\" DevicePath \"\""
Sep 11 04:08:59.523686 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:08:59.523481    2579 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ceb39acb-e2b8-4a34-8936-6c36677fa484-istio-token\") on node \"ip-10-0-141-105.ec2.internal\" DevicePath \"\""
Sep 11 04:08:59.523686 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:08:59.523488    2579 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ceb39acb-e2b8-4a34-8936-6c36677fa484-istio-podinfo\") on node \"ip-10-0-141-105.ec2.internal\" DevicePath \"\""
Sep 11 04:08:59.523686 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:08:59.523496    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4dnxf\" (UniqueName: \"kubernetes.io/projected/ceb39acb-e2b8-4a34-8936-6c36677fa484-kube-api-access-4dnxf\") on node \"ip-10-0-141-105.ec2.internal\" DevicePath \"\""
Sep 11 04:08:59.523686 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:08:59.523508    2579 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ceb39acb-e2b8-4a34-8936-6c36677fa484-credential-socket\") on node \"ip-10-0-141-105.ec2.internal\" DevicePath \"\""
Sep 11 04:08:59.523686 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:08:59.523532    2579 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ceb39acb-e2b8-4a34-8936-6c36677fa484-istio-envoy\") on node \"ip-10-0-141-105.ec2.internal\" DevicePath \"\""
Sep 11 04:09:00.039991 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:09:00.039958    2579 generic.go:356] "Generic (PLEG): container finished" podID="ceb39acb-e2b8-4a34-8936-6c36677fa484" containerID="b52c194e64c5ae57cd4e7e0060f2b719f4f264911aabe5cb192d3e03531fecf5" exitCode=0
Sep 11 04:09:00.040225 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:09:00.040018    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-0f85b8-openshift-default-7d957b55c4-pskrc"
Sep 11 04:09:00.040225 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:09:00.040044    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-0f85b8-openshift-default-7d957b55c4-pskrc" event={"ID":"ceb39acb-e2b8-4a34-8936-6c36677fa484","Type":"ContainerDied","Data":"b52c194e64c5ae57cd4e7e0060f2b719f4f264911aabe5cb192d3e03531fecf5"}
Sep 11 04:09:00.040225 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:09:00.040123    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-0f85b8-openshift-default-7d957b55c4-pskrc" event={"ID":"ceb39acb-e2b8-4a34-8936-6c36677fa484","Type":"ContainerDied","Data":"8fc66aec1b4a10cc9cc39279733787ff2a641fee10fbcc871c7b0ff1c4d8dc29"}
Sep 11 04:09:00.040225 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:09:00.040148    2579 scope.go:117] "RemoveContainer" containerID="b52c194e64c5ae57cd4e7e0060f2b719f4f264911aabe5cb192d3e03531fecf5"
Sep 11 04:09:00.050050 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:09:00.050033    2579 scope.go:117] "RemoveContainer" containerID="21b2dac30f70bee611edf8cf906eadca5b16a887671fe815369af24dc205b989"
Sep 11 04:09:00.058501 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:09:00.058487    2579 scope.go:117] "RemoveContainer" containerID="b52c194e64c5ae57cd4e7e0060f2b719f4f264911aabe5cb192d3e03531fecf5"
Sep 11 04:09:00.058903 ip-10-0-141-105 kubenswrapper[2579]: E0911 04:09:00.058864    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b52c194e64c5ae57cd4e7e0060f2b719f4f264911aabe5cb192d3e03531fecf5\": container with ID starting with b52c194e64c5ae57cd4e7e0060f2b719f4f264911aabe5cb192d3e03531fecf5 not found: ID does not exist" containerID="b52c194e64c5ae57cd4e7e0060f2b719f4f264911aabe5cb192d3e03531fecf5"
Sep 11 04:09:00.059033 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:09:00.058898    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b52c194e64c5ae57cd4e7e0060f2b719f4f264911aabe5cb192d3e03531fecf5"} err="failed to get container status \"b52c194e64c5ae57cd4e7e0060f2b719f4f264911aabe5cb192d3e03531fecf5\": rpc error: code = NotFound desc = could not find container \"b52c194e64c5ae57cd4e7e0060f2b719f4f264911aabe5cb192d3e03531fecf5\": container with ID starting with b52c194e64c5ae57cd4e7e0060f2b719f4f264911aabe5cb192d3e03531fecf5 not found: ID does not exist"
Sep 11 04:09:00.059033 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:09:00.058922    2579 scope.go:117] "RemoveContainer" containerID="21b2dac30f70bee611edf8cf906eadca5b16a887671fe815369af24dc205b989"
Sep 11 04:09:00.059661 ip-10-0-141-105 kubenswrapper[2579]: E0911 04:09:00.059633    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"21b2dac30f70bee611edf8cf906eadca5b16a887671fe815369af24dc205b989\": container with ID starting with 21b2dac30f70bee611edf8cf906eadca5b16a887671fe815369af24dc205b989 not found: ID does not exist" containerID="21b2dac30f70bee611edf8cf906eadca5b16a887671fe815369af24dc205b989"
Sep 11 04:09:00.059822 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:09:00.059781    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"21b2dac30f70bee611edf8cf906eadca5b16a887671fe815369af24dc205b989"} err="failed to get container status \"21b2dac30f70bee611edf8cf906eadca5b16a887671fe815369af24dc205b989\": rpc error: code = NotFound desc = could not find container \"21b2dac30f70bee611edf8cf906eadca5b16a887671fe815369af24dc205b989\": container with ID starting with 21b2dac30f70bee611edf8cf906eadca5b16a887671fe815369af24dc205b989 not found: ID does not exist"
Sep 11 04:09:00.061579 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:09:00.061562    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-0f85b8-openshift-default-7d957b55c4-pskrc"]
Sep 11 04:09:00.067675 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:09:00.067632    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-b-0f85b8-openshift-default-7d957b55c4-pskrc"]
Sep 11 04:09:00.852036 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:09:00.852002    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ceb39acb-e2b8-4a34-8936-6c36677fa484" path="/var/lib/kubelet/pods/ceb39acb-e2b8-4a34-8936-6c36677fa484/volumes"
Sep 11 04:09:05.113139 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:09:05.113105    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-b4d3cf-xk7b8"]
Sep 11 04:09:05.116715 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:09:05.116672    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-b4d3cf-xk7b8"]
Sep 11 04:09:06.852584 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:09:06.852550    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="08dfbebb-3864-4f96-b25e-404bd66c3fd0" path="/var/lib/kubelet/pods/08dfbebb-3864-4f96-b25e-404bd66c3fd0/volumes"
Sep 11 04:09:07.908010 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:09:07.907980    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-5z4cn"]
Sep 11 04:09:12.424159 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:09:12.424127    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-dd5b30-999n8"]
Sep 11 04:09:12.424575 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:09:12.424530    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ceb39acb-e2b8-4a34-8936-6c36677fa484" containerName="istio-proxy"
Sep 11 04:09:12.424575 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:09:12.424542    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="ceb39acb-e2b8-4a34-8936-6c36677fa484" containerName="istio-proxy"
Sep 11 04:09:12.424575 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:09:12.424549    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ceb39acb-e2b8-4a34-8936-6c36677fa484" containerName="istio-proxy"
Sep 11 04:09:12.424575 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:09:12.424555    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="ceb39acb-e2b8-4a34-8936-6c36677fa484" containerName="istio-proxy"
Sep 11 04:09:12.424575 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:09:12.424565    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="08dfbebb-3864-4f96-b25e-404bd66c3fd0" containerName="revoke-keys"
Sep 11 04:09:12.424575 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:09:12.424571    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="08dfbebb-3864-4f96-b25e-404bd66c3fd0" containerName="revoke-keys"
Sep 11 04:09:12.424768 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:09:12.424629    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="ceb39acb-e2b8-4a34-8936-6c36677fa484" containerName="istio-proxy"
Sep 11 04:09:12.424768 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:09:12.424637    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="ceb39acb-e2b8-4a34-8936-6c36677fa484" containerName="istio-proxy"
Sep 11 04:09:12.424768 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:09:12.424644    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="08dfbebb-3864-4f96-b25e-404bd66c3fd0" containerName="revoke-keys"
Sep 11 04:09:12.429341 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:09:12.429324    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-dd5b30-999n8"
Sep 11 04:09:12.432892 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:09:12.432868    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 11 04:09:12.433120 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:09:12.432886    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-95ckr\""
Sep 11 04:09:12.434332 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:09:12.434292    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-dd5b30-999n8"]
Sep 11 04:09:12.539350 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:09:12.539318    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/dbe6b3ae-8a54-49ad-b86e-e76bc438e410-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-dd5b30-999n8\" (UID: \"dbe6b3ae-8a54-49ad-b86e-e76bc438e410\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-dd5b30-999n8"
Sep 11 04:09:12.639785 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:09:12.639754    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/dbe6b3ae-8a54-49ad-b86e-e76bc438e410-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-dd5b30-999n8\" (UID: \"dbe6b3ae-8a54-49ad-b86e-e76bc438e410\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-dd5b30-999n8"
Sep 11 04:09:12.640469 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:09:12.640450    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/dbe6b3ae-8a54-49ad-b86e-e76bc438e410-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-dd5b30-999n8\" (UID: \"dbe6b3ae-8a54-49ad-b86e-e76bc438e410\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-dd5b30-999n8"
Sep 11 04:09:12.741439 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:09:12.741361    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-dd5b30-999n8"
Sep 11 04:09:13.015163 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:09:13.015134    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-dd5b30-999n8"]
Sep 11 04:09:13.017157 ip-10-0-141-105 kubenswrapper[2579]: W0911 04:09:13.017108    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddbe6b3ae_8a54_49ad_b86e_e76bc438e410.slice/crio-0b8590ad357c69f7288550d848a99ef751041f5e28833107342aaf239dae99f3 WatchSource:0}: Error finding container 0b8590ad357c69f7288550d848a99ef751041f5e28833107342aaf239dae99f3: Status 404 returned error can't find the container with id 0b8590ad357c69f7288550d848a99ef751041f5e28833107342aaf239dae99f3
Sep 11 04:09:13.110381 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:09:13.110347    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-dd5b30-999n8" event={"ID":"dbe6b3ae-8a54-49ad-b86e-e76bc438e410","Type":"ContainerStarted","Data":"0b8590ad357c69f7288550d848a99ef751041f5e28833107342aaf239dae99f3"}
Sep 11 04:09:14.117138 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:09:14.117104    2579 generic.go:356] "Generic (PLEG): container finished" podID="dbe6b3ae-8a54-49ad-b86e-e76bc438e410" containerID="7766a1e04dc43d17122eff9fefee413eae163da493b8698b02a6eea45fb9ee54" exitCode=0
Sep 11 04:09:14.117590 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:09:14.117157    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-dd5b30-999n8" event={"ID":"dbe6b3ae-8a54-49ad-b86e-e76bc438e410","Type":"ContainerDied","Data":"7766a1e04dc43d17122eff9fefee413eae163da493b8698b02a6eea45fb9ee54"}
Sep 11 04:09:15.256476 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:09:15.256451    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-dd5b30-999n8"
Sep 11 04:09:15.364028 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:09:15.363996    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/dbe6b3ae-8a54-49ad-b86e-e76bc438e410-maas-api-service-ca\") pod \"dbe6b3ae-8a54-49ad-b86e-e76bc438e410\" (UID: \"dbe6b3ae-8a54-49ad-b86e-e76bc438e410\") "
Sep 11 04:09:15.364407 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:09:15.364384    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dbe6b3ae-8a54-49ad-b86e-e76bc438e410-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "dbe6b3ae-8a54-49ad-b86e-e76bc438e410" (UID: "dbe6b3ae-8a54-49ad-b86e-e76bc438e410"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 04:09:15.465054 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:09:15.464972    2579 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/dbe6b3ae-8a54-49ad-b86e-e76bc438e410-maas-api-service-ca\") on node \"ip-10-0-141-105.ec2.internal\" DevicePath \"\""
Sep 11 04:09:16.136176 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:09:16.136143    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-dd5b30-999n8"
Sep 11 04:09:16.136176 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:09:16.136156    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-dd5b30-999n8" event={"ID":"dbe6b3ae-8a54-49ad-b86e-e76bc438e410","Type":"ContainerDied","Data":"0b8590ad357c69f7288550d848a99ef751041f5e28833107342aaf239dae99f3"}
Sep 11 04:09:16.136397 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:09:16.136242    2579 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0b8590ad357c69f7288550d848a99ef751041f5e28833107342aaf239dae99f3"
Sep 11 04:09:22.993948 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:09:22.993901    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-dd5b30-29818320-9mj9h"]
Sep 11 04:09:22.996651 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:09:22.996619    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-dd5b30-29818320-9mj9h"]
Sep 11 04:09:23.025753 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:09:23.025721    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-dd5b30-685f9c74fc-4dnsz"]
Sep 11 04:09:23.026032 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:09:23.025982    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-dd5b30-685f9c74fc-4dnsz" podUID="398172c7-42dc-417a-9d9e-ed3f5e04ecdc" containerName="payload-processing" containerID="cri-o://80ad0471f297b6f88b10a16dff0b701bac36821c0b7b4768b4ad7217b07c9540" gracePeriod=30
Sep 11 04:09:23.174739 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:09:23.174707    2579 generic.go:356] "Generic (PLEG): container finished" podID="398172c7-42dc-417a-9d9e-ed3f5e04ecdc" containerID="80ad0471f297b6f88b10a16dff0b701bac36821c0b7b4768b4ad7217b07c9540" exitCode=0
Sep 11 04:09:23.174922 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:09:23.174783    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-dd5b30-685f9c74fc-4dnsz" event={"ID":"398172c7-42dc-417a-9d9e-ed3f5e04ecdc","Type":"ContainerDied","Data":"80ad0471f297b6f88b10a16dff0b701bac36821c0b7b4768b4ad7217b07c9540"}
Sep 11 04:09:23.193006 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:09:23.192737    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-dd5b30-999n8"]
Sep 11 04:09:23.197075 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:09:23.197045    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-dd5b30-999n8"]
Sep 11 04:09:23.289259 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:09:23.289233    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-dd5b30-685f9c74fc-4dnsz"
Sep 11 04:09:23.438660 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:09:23.438613    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cc2cq\" (UniqueName: \"kubernetes.io/projected/398172c7-42dc-417a-9d9e-ed3f5e04ecdc-kube-api-access-cc2cq\") pod \"398172c7-42dc-417a-9d9e-ed3f5e04ecdc\" (UID: \"398172c7-42dc-417a-9d9e-ed3f5e04ecdc\") "
Sep 11 04:09:23.438870 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:09:23.438684    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/398172c7-42dc-417a-9d9e-ed3f5e04ecdc-plugins-config-volume\") pod \"398172c7-42dc-417a-9d9e-ed3f5e04ecdc\" (UID: \"398172c7-42dc-417a-9d9e-ed3f5e04ecdc\") "
Sep 11 04:09:23.439140 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:09:23.439109    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/398172c7-42dc-417a-9d9e-ed3f5e04ecdc-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "398172c7-42dc-417a-9d9e-ed3f5e04ecdc" (UID: "398172c7-42dc-417a-9d9e-ed3f5e04ecdc"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 04:09:23.440974 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:09:23.440953    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/398172c7-42dc-417a-9d9e-ed3f5e04ecdc-kube-api-access-cc2cq" (OuterVolumeSpecName: "kube-api-access-cc2cq") pod "398172c7-42dc-417a-9d9e-ed3f5e04ecdc" (UID: "398172c7-42dc-417a-9d9e-ed3f5e04ecdc"). InnerVolumeSpecName "kube-api-access-cc2cq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 04:09:23.540178 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:09:23.540074    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cc2cq\" (UniqueName: \"kubernetes.io/projected/398172c7-42dc-417a-9d9e-ed3f5e04ecdc-kube-api-access-cc2cq\") on node \"ip-10-0-141-105.ec2.internal\" DevicePath \"\""
Sep 11 04:09:23.540178 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:09:23.540126    2579 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/398172c7-42dc-417a-9d9e-ed3f5e04ecdc-plugins-config-volume\") on node \"ip-10-0-141-105.ec2.internal\" DevicePath \"\""
Sep 11 04:09:24.181520 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:09:24.181489    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-dd5b30-685f9c74fc-4dnsz" event={"ID":"398172c7-42dc-417a-9d9e-ed3f5e04ecdc","Type":"ContainerDied","Data":"708c8422ef5034fdd9392eabe0eb9247a12f72cd2a1705aba93d5531a79772b5"}
Sep 11 04:09:24.181951 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:09:24.181548    2579 scope.go:117] "RemoveContainer" containerID="80ad0471f297b6f88b10a16dff0b701bac36821c0b7b4768b4ad7217b07c9540"
Sep 11 04:09:24.181951 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:09:24.181505    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-dd5b30-685f9c74fc-4dnsz"
Sep 11 04:09:24.202923 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:09:24.202892    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-dd5b30-685f9c74fc-4dnsz"]
Sep 11 04:09:24.208804 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:09:24.208777    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-dd5b30-685f9c74fc-4dnsz"]
Sep 11 04:09:24.851712 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:09:24.851671    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="398172c7-42dc-417a-9d9e-ed3f5e04ecdc" path="/var/lib/kubelet/pods/398172c7-42dc-417a-9d9e-ed3f5e04ecdc/volumes"
Sep 11 04:09:24.852061 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:09:24.852041    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3e6c01fc-ed58-425c-a590-acdd4aef2d16" path="/var/lib/kubelet/pods/3e6c01fc-ed58-425c-a590-acdd4aef2d16/volumes"
Sep 11 04:09:24.852428 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:09:24.852409    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="dbe6b3ae-8a54-49ad-b86e-e76bc438e410" path="/var/lib/kubelet/pods/dbe6b3ae-8a54-49ad-b86e-e76bc438e410/volumes"
Sep 11 04:09:25.716506 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:09:25.716470    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-5z4cn"]
Sep 11 04:09:29.930991 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:09:29.930958    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-dd5b30-openshift-default-86dfc4597d-b7w6c"]
Sep 11 04:09:29.931517 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:09:29.931315    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w3-dd5b30-openshift-default-86dfc4597d-b7w6c" podUID="ecac9ada-1da0-4ecd-8af5-07501bd2aeb9" containerName="istio-proxy" containerID="cri-o://930fc6e9f3aa5bf551dd5ad8dad00882fe941e3ed3eb39c5a9b919681afc894b" gracePeriod=30
Sep 11 04:09:34.813181 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:09:34.813146    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-5z4cn"]
Sep 11 04:09:35.174212 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:09:35.174189    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-dd5b30-openshift-default-86dfc4597d-b7w6c"
Sep 11 04:09:35.240174 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:09:35.240140    2579 generic.go:356] "Generic (PLEG): container finished" podID="ecac9ada-1da0-4ecd-8af5-07501bd2aeb9" containerID="930fc6e9f3aa5bf551dd5ad8dad00882fe941e3ed3eb39c5a9b919681afc894b" exitCode=0
Sep 11 04:09:35.240367 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:09:35.240185    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-dd5b30-openshift-default-86dfc4597d-b7w6c"
Sep 11 04:09:35.240367 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:09:35.240229    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-dd5b30-openshift-default-86dfc4597d-b7w6c" event={"ID":"ecac9ada-1da0-4ecd-8af5-07501bd2aeb9","Type":"ContainerDied","Data":"930fc6e9f3aa5bf551dd5ad8dad00882fe941e3ed3eb39c5a9b919681afc894b"}
Sep 11 04:09:35.240367 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:09:35.240296    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-dd5b30-openshift-default-86dfc4597d-b7w6c" event={"ID":"ecac9ada-1da0-4ecd-8af5-07501bd2aeb9","Type":"ContainerDied","Data":"3535aee47fa72fd1cc1c34f3b90140b268394d818f1d2f5a9125a1915e0144f8"}
Sep 11 04:09:35.240367 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:09:35.240327    2579 scope.go:117] "RemoveContainer" containerID="930fc6e9f3aa5bf551dd5ad8dad00882fe941e3ed3eb39c5a9b919681afc894b"
Sep 11 04:09:35.243688 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:09:35.243670    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ecac9ada-1da0-4ecd-8af5-07501bd2aeb9-istiod-ca-cert\") pod \"ecac9ada-1da0-4ecd-8af5-07501bd2aeb9\" (UID: \"ecac9ada-1da0-4ecd-8af5-07501bd2aeb9\") "
Sep 11 04:09:35.243779 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:09:35.243718    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ecac9ada-1da0-4ecd-8af5-07501bd2aeb9-credential-socket\") pod \"ecac9ada-1da0-4ecd-8af5-07501bd2aeb9\" (UID: \"ecac9ada-1da0-4ecd-8af5-07501bd2aeb9\") "
Sep 11 04:09:35.243779 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:09:35.243744    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-45g8t\" (UniqueName: \"kubernetes.io/projected/ecac9ada-1da0-4ecd-8af5-07501bd2aeb9-kube-api-access-45g8t\") pod \"ecac9ada-1da0-4ecd-8af5-07501bd2aeb9\" (UID: \"ecac9ada-1da0-4ecd-8af5-07501bd2aeb9\") "
Sep 11 04:09:35.244024 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:09:35.244000    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ecac9ada-1da0-4ecd-8af5-07501bd2aeb9-istio-podinfo\") pod \"ecac9ada-1da0-4ecd-8af5-07501bd2aeb9\" (UID: \"ecac9ada-1da0-4ecd-8af5-07501bd2aeb9\") "
Sep 11 04:09:35.244178 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:09:35.244035    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ecac9ada-1da0-4ecd-8af5-07501bd2aeb9-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "ecac9ada-1da0-4ecd-8af5-07501bd2aeb9" (UID: "ecac9ada-1da0-4ecd-8af5-07501bd2aeb9"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 04:09:35.244178 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:09:35.244076    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ecac9ada-1da0-4ecd-8af5-07501bd2aeb9-workload-certs\") pod \"ecac9ada-1da0-4ecd-8af5-07501bd2aeb9\" (UID: \"ecac9ada-1da0-4ecd-8af5-07501bd2aeb9\") "
Sep 11 04:09:35.244178 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:09:35.244114    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ecac9ada-1da0-4ecd-8af5-07501bd2aeb9-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "ecac9ada-1da0-4ecd-8af5-07501bd2aeb9" (UID: "ecac9ada-1da0-4ecd-8af5-07501bd2aeb9"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 04:09:35.244178 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:09:35.244123    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ecac9ada-1da0-4ecd-8af5-07501bd2aeb9-istio-envoy\") pod \"ecac9ada-1da0-4ecd-8af5-07501bd2aeb9\" (UID: \"ecac9ada-1da0-4ecd-8af5-07501bd2aeb9\") "
Sep 11 04:09:35.244422 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:09:35.244186    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ecac9ada-1da0-4ecd-8af5-07501bd2aeb9-workload-socket\") pod \"ecac9ada-1da0-4ecd-8af5-07501bd2aeb9\" (UID: \"ecac9ada-1da0-4ecd-8af5-07501bd2aeb9\") "
Sep 11 04:09:35.244422 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:09:35.244223    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ecac9ada-1da0-4ecd-8af5-07501bd2aeb9-istio-data\") pod \"ecac9ada-1da0-4ecd-8af5-07501bd2aeb9\" (UID: \"ecac9ada-1da0-4ecd-8af5-07501bd2aeb9\") "
Sep 11 04:09:35.244422 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:09:35.244298    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ecac9ada-1da0-4ecd-8af5-07501bd2aeb9-istio-token\") pod \"ecac9ada-1da0-4ecd-8af5-07501bd2aeb9\" (UID: \"ecac9ada-1da0-4ecd-8af5-07501bd2aeb9\") "
Sep 11 04:09:35.244422 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:09:35.244394    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ecac9ada-1da0-4ecd-8af5-07501bd2aeb9-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "ecac9ada-1da0-4ecd-8af5-07501bd2aeb9" (UID: "ecac9ada-1da0-4ecd-8af5-07501bd2aeb9"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 04:09:35.244618 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:09:35.244564    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ecac9ada-1da0-4ecd-8af5-07501bd2aeb9-istio-data" (OuterVolumeSpecName: "istio-data") pod "ecac9ada-1da0-4ecd-8af5-07501bd2aeb9" (UID: "ecac9ada-1da0-4ecd-8af5-07501bd2aeb9"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 04:09:35.244737 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:09:35.244689    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ecac9ada-1da0-4ecd-8af5-07501bd2aeb9-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "ecac9ada-1da0-4ecd-8af5-07501bd2aeb9" (UID: "ecac9ada-1da0-4ecd-8af5-07501bd2aeb9"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 04:09:35.244806 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:09:35.244736    2579 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ecac9ada-1da0-4ecd-8af5-07501bd2aeb9-istio-data\") on node \"ip-10-0-141-105.ec2.internal\" DevicePath \"\""
Sep 11 04:09:35.244806 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:09:35.244754    2579 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ecac9ada-1da0-4ecd-8af5-07501bd2aeb9-istiod-ca-cert\") on node \"ip-10-0-141-105.ec2.internal\" DevicePath \"\""
Sep 11 04:09:35.244806 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:09:35.244767    2579 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ecac9ada-1da0-4ecd-8af5-07501bd2aeb9-credential-socket\") on node \"ip-10-0-141-105.ec2.internal\" DevicePath \"\""
Sep 11 04:09:35.244806 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:09:35.244782    2579 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ecac9ada-1da0-4ecd-8af5-07501bd2aeb9-workload-certs\") on node \"ip-10-0-141-105.ec2.internal\" DevicePath \"\""
Sep 11 04:09:35.246544 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:09:35.246521    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ecac9ada-1da0-4ecd-8af5-07501bd2aeb9-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "ecac9ada-1da0-4ecd-8af5-07501bd2aeb9" (UID: "ecac9ada-1da0-4ecd-8af5-07501bd2aeb9"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 04:09:35.246544 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:09:35.246534    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ecac9ada-1da0-4ecd-8af5-07501bd2aeb9-kube-api-access-45g8t" (OuterVolumeSpecName: "kube-api-access-45g8t") pod "ecac9ada-1da0-4ecd-8af5-07501bd2aeb9" (UID: "ecac9ada-1da0-4ecd-8af5-07501bd2aeb9"). InnerVolumeSpecName "kube-api-access-45g8t". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 04:09:35.246870 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:09:35.246846    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ecac9ada-1da0-4ecd-8af5-07501bd2aeb9-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "ecac9ada-1da0-4ecd-8af5-07501bd2aeb9" (UID: "ecac9ada-1da0-4ecd-8af5-07501bd2aeb9"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 11 04:09:35.246925 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:09:35.246909    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ecac9ada-1da0-4ecd-8af5-07501bd2aeb9-istio-token" (OuterVolumeSpecName: "istio-token") pod "ecac9ada-1da0-4ecd-8af5-07501bd2aeb9" (UID: "ecac9ada-1da0-4ecd-8af5-07501bd2aeb9"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 04:09:35.250285 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:09:35.250266    2579 scope.go:117] "RemoveContainer" containerID="546139f5e12a36002009e7a5aae8a01f1a1d969b0c8b5831054c704af544876e"
Sep 11 04:09:35.263686 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:09:35.263668    2579 scope.go:117] "RemoveContainer" containerID="930fc6e9f3aa5bf551dd5ad8dad00882fe941e3ed3eb39c5a9b919681afc894b"
Sep 11 04:09:35.263976 ip-10-0-141-105 kubenswrapper[2579]: E0911 04:09:35.263956    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"930fc6e9f3aa5bf551dd5ad8dad00882fe941e3ed3eb39c5a9b919681afc894b\": container with ID starting with 930fc6e9f3aa5bf551dd5ad8dad00882fe941e3ed3eb39c5a9b919681afc894b not found: ID does not exist" containerID="930fc6e9f3aa5bf551dd5ad8dad00882fe941e3ed3eb39c5a9b919681afc894b"
Sep 11 04:09:35.264053 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:09:35.263988    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"930fc6e9f3aa5bf551dd5ad8dad00882fe941e3ed3eb39c5a9b919681afc894b"} err="failed to get container status \"930fc6e9f3aa5bf551dd5ad8dad00882fe941e3ed3eb39c5a9b919681afc894b\": rpc error: code = NotFound desc = could not find container \"930fc6e9f3aa5bf551dd5ad8dad00882fe941e3ed3eb39c5a9b919681afc894b\": container with ID starting with 930fc6e9f3aa5bf551dd5ad8dad00882fe941e3ed3eb39c5a9b919681afc894b not found: ID does not exist"
Sep 11 04:09:35.264053 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:09:35.264010    2579 scope.go:117] "RemoveContainer" containerID="546139f5e12a36002009e7a5aae8a01f1a1d969b0c8b5831054c704af544876e"
Sep 11 04:09:35.264315 ip-10-0-141-105 kubenswrapper[2579]: E0911 04:09:35.264296    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"546139f5e12a36002009e7a5aae8a01f1a1d969b0c8b5831054c704af544876e\": container with ID starting with 546139f5e12a36002009e7a5aae8a01f1a1d969b0c8b5831054c704af544876e not found: ID does not exist" containerID="546139f5e12a36002009e7a5aae8a01f1a1d969b0c8b5831054c704af544876e"
Sep 11 04:09:35.264358 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:09:35.264322    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"546139f5e12a36002009e7a5aae8a01f1a1d969b0c8b5831054c704af544876e"} err="failed to get container status \"546139f5e12a36002009e7a5aae8a01f1a1d969b0c8b5831054c704af544876e\": rpc error: code = NotFound desc = could not find container \"546139f5e12a36002009e7a5aae8a01f1a1d969b0c8b5831054c704af544876e\": container with ID starting with 546139f5e12a36002009e7a5aae8a01f1a1d969b0c8b5831054c704af544876e not found: ID does not exist"
Sep 11 04:09:35.345524 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:09:35.345471    2579 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ecac9ada-1da0-4ecd-8af5-07501bd2aeb9-istio-token\") on node \"ip-10-0-141-105.ec2.internal\" DevicePath \"\""
Sep 11 04:09:35.345524 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:09:35.345491    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-45g8t\" (UniqueName: \"kubernetes.io/projected/ecac9ada-1da0-4ecd-8af5-07501bd2aeb9-kube-api-access-45g8t\") on node \"ip-10-0-141-105.ec2.internal\" DevicePath \"\""
Sep 11 04:09:35.345524 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:09:35.345502    2579 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ecac9ada-1da0-4ecd-8af5-07501bd2aeb9-istio-podinfo\") on node \"ip-10-0-141-105.ec2.internal\" DevicePath \"\""
Sep 11 04:09:35.345524 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:09:35.345510    2579 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ecac9ada-1da0-4ecd-8af5-07501bd2aeb9-istio-envoy\") on node \"ip-10-0-141-105.ec2.internal\" DevicePath \"\""
Sep 11 04:09:35.345524 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:09:35.345518    2579 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ecac9ada-1da0-4ecd-8af5-07501bd2aeb9-workload-socket\") on node \"ip-10-0-141-105.ec2.internal\" DevicePath \"\""
Sep 11 04:09:35.562648 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:09:35.562619    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-dd5b30-openshift-default-86dfc4597d-b7w6c"]
Sep 11 04:09:35.566713 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:09:35.566687    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-dd5b30-openshift-default-86dfc4597d-b7w6c"]
Sep 11 04:09:36.851923 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:09:36.851889    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ecac9ada-1da0-4ecd-8af5-07501bd2aeb9" path="/var/lib/kubelet/pods/ecac9ada-1da0-4ecd-8af5-07501bd2aeb9/volumes"
Sep 11 04:10:23.287425 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:10:23.287392    2579 scope.go:117] "RemoveContainer" containerID="97533289e52ac689bc8465c5936fcce94107cfef608aeb8f94bd7b475e81ef6a"
Sep 11 04:10:23.296064 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:10:23.296043    2579 scope.go:117] "RemoveContainer" containerID="6525f70442185cb69de2ee37d94d3e005138e6659b2ed9fdc218ee3d5ed977ee"
Sep 11 04:10:23.303980 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:10:23.303944    2579 scope.go:117] "RemoveContainer" containerID="72450b69e797fe358a60ad26f7153802f67996310ab8b8ac2842de55dc040bb1"
Sep 11 04:10:23.312295 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:10:23.312275    2579 scope.go:117] "RemoveContainer" containerID="16968aab8d31197fd71d04687b3f8fa7c08dc671ff9d06aa60c72c7afb2e106d"
Sep 11 04:10:57.205974 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:10:57.205938    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-5z4cn"]
Sep 11 04:11:20.197634 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:11:20.197600    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-5z4cn"]
Sep 11 04:11:23.349776 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:11:23.349741    2579 scope.go:117] "RemoveContainer" containerID="b43ae14c450a071162d6da7db3450bea86bff97d29d337e9b50c4be9464020ed"
Sep 11 04:11:27.709108 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:11:27.709057    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-5z4cn"]
Sep 11 04:11:34.620487 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:11:34.620447    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-5z4cn"]
Sep 11 04:11:42.307105 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:11:42.307052    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-5z4cn"]
Sep 11 04:11:51.435836 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:11:51.435756    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-5z4cn"]
Sep 11 04:11:58.517484 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:11:58.517440    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-5z4cn"]
Sep 11 04:12:06.411736 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:12:06.411690    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-5z4cn"]
Sep 11 04:12:15.205267 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:12:15.205224    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-5z4cn"]
Sep 11 04:12:22.151881 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:12:22.151840    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-5z4cn"]
Sep 11 04:13:23.375977 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:13:23.375889    2579 scope.go:117] "RemoveContainer" containerID="15763093c33b4458ab5891a1649c3798b0108f1035ebf2a333af8ff32d7cd457"
Sep 11 04:13:52.728912 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:13:52.728868    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-5z4cn"]
Sep 11 04:13:58.429181 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:13:58.429145    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-5z4cn"]
Sep 11 04:14:08.692283 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:14:08.692245    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-same-id-a-fa15b45f-kserve-6dc9dc589f-jnkmt"]
Sep 11 04:14:08.692644 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:14:08.692579    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ecac9ada-1da0-4ecd-8af5-07501bd2aeb9" containerName="istio-proxy"
Sep 11 04:14:08.692644 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:14:08.692591    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="ecac9ada-1da0-4ecd-8af5-07501bd2aeb9" containerName="istio-proxy"
Sep 11 04:14:08.692644 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:14:08.692600    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="398172c7-42dc-417a-9d9e-ed3f5e04ecdc" containerName="payload-processing"
Sep 11 04:14:08.692644 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:14:08.692606    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="398172c7-42dc-417a-9d9e-ed3f5e04ecdc" containerName="payload-processing"
Sep 11 04:14:08.692644 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:14:08.692625    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ecac9ada-1da0-4ecd-8af5-07501bd2aeb9" containerName="istio-proxy"
Sep 11 04:14:08.692644 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:14:08.692631    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="ecac9ada-1da0-4ecd-8af5-07501bd2aeb9" containerName="istio-proxy"
Sep 11 04:14:08.692644 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:14:08.692638    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dbe6b3ae-8a54-49ad-b86e-e76bc438e410" containerName="revoke-keys"
Sep 11 04:14:08.692644 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:14:08.692644    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="dbe6b3ae-8a54-49ad-b86e-e76bc438e410" containerName="revoke-keys"
Sep 11 04:14:08.692904 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:14:08.692699    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="ecac9ada-1da0-4ecd-8af5-07501bd2aeb9" containerName="istio-proxy"
Sep 11 04:14:08.692904 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:14:08.692707    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="ecac9ada-1da0-4ecd-8af5-07501bd2aeb9" containerName="istio-proxy"
Sep 11 04:14:08.692904 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:14:08.692715    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="398172c7-42dc-417a-9d9e-ed3f5e04ecdc" containerName="payload-processing"
Sep 11 04:14:08.692904 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:14:08.692722    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="dbe6b3ae-8a54-49ad-b86e-e76bc438e410" containerName="revoke-keys"
Sep 11 04:14:08.695652 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:14:08.695628    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-fa15b45f-kserve-6dc9dc589f-jnkmt"
Sep 11 04:14:08.698259 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:14:08.698233    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-same-id-a-fa15b45f-kserve-self-signed-certs\""
Sep 11 04:14:08.709705 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:14:08.709677    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-a-fa15b45f-kserve-6dc9dc589f-jnkmt"]
Sep 11 04:14:08.774630 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:14:08.774595    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hv5wh\" (UniqueName: \"kubernetes.io/projected/347b8352-a483-4682-8799-c9caef08bb74-kube-api-access-hv5wh\") pod \"e2e-same-id-a-fa15b45f-kserve-6dc9dc589f-jnkmt\" (UID: \"347b8352-a483-4682-8799-c9caef08bb74\") " pod="llm/e2e-same-id-a-fa15b45f-kserve-6dc9dc589f-jnkmt"
Sep 11 04:14:08.774630 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:14:08.774632    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/347b8352-a483-4682-8799-c9caef08bb74-model-cache\") pod \"e2e-same-id-a-fa15b45f-kserve-6dc9dc589f-jnkmt\" (UID: \"347b8352-a483-4682-8799-c9caef08bb74\") " pod="llm/e2e-same-id-a-fa15b45f-kserve-6dc9dc589f-jnkmt"
Sep 11 04:14:08.774852 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:14:08.774658    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/347b8352-a483-4682-8799-c9caef08bb74-home\") pod \"e2e-same-id-a-fa15b45f-kserve-6dc9dc589f-jnkmt\" (UID: \"347b8352-a483-4682-8799-c9caef08bb74\") " pod="llm/e2e-same-id-a-fa15b45f-kserve-6dc9dc589f-jnkmt"
Sep 11 04:14:08.774852 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:14:08.774684    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/347b8352-a483-4682-8799-c9caef08bb74-dshm\") pod \"e2e-same-id-a-fa15b45f-kserve-6dc9dc589f-jnkmt\" (UID: \"347b8352-a483-4682-8799-c9caef08bb74\") " pod="llm/e2e-same-id-a-fa15b45f-kserve-6dc9dc589f-jnkmt"
Sep 11 04:14:08.774852 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:14:08.774717    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/347b8352-a483-4682-8799-c9caef08bb74-tls-certs\") pod \"e2e-same-id-a-fa15b45f-kserve-6dc9dc589f-jnkmt\" (UID: \"347b8352-a483-4682-8799-c9caef08bb74\") " pod="llm/e2e-same-id-a-fa15b45f-kserve-6dc9dc589f-jnkmt"
Sep 11 04:14:08.774852 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:14:08.774748    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/347b8352-a483-4682-8799-c9caef08bb74-tmp-dir\") pod \"e2e-same-id-a-fa15b45f-kserve-6dc9dc589f-jnkmt\" (UID: \"347b8352-a483-4682-8799-c9caef08bb74\") " pod="llm/e2e-same-id-a-fa15b45f-kserve-6dc9dc589f-jnkmt"
Sep 11 04:14:08.875384 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:14:08.875338    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hv5wh\" (UniqueName: \"kubernetes.io/projected/347b8352-a483-4682-8799-c9caef08bb74-kube-api-access-hv5wh\") pod \"e2e-same-id-a-fa15b45f-kserve-6dc9dc589f-jnkmt\" (UID: \"347b8352-a483-4682-8799-c9caef08bb74\") " pod="llm/e2e-same-id-a-fa15b45f-kserve-6dc9dc589f-jnkmt"
Sep 11 04:14:08.875591 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:14:08.875393    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/347b8352-a483-4682-8799-c9caef08bb74-model-cache\") pod \"e2e-same-id-a-fa15b45f-kserve-6dc9dc589f-jnkmt\" (UID: \"347b8352-a483-4682-8799-c9caef08bb74\") " pod="llm/e2e-same-id-a-fa15b45f-kserve-6dc9dc589f-jnkmt"
Sep 11 04:14:08.875591 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:14:08.875416    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/347b8352-a483-4682-8799-c9caef08bb74-home\") pod \"e2e-same-id-a-fa15b45f-kserve-6dc9dc589f-jnkmt\" (UID: \"347b8352-a483-4682-8799-c9caef08bb74\") " pod="llm/e2e-same-id-a-fa15b45f-kserve-6dc9dc589f-jnkmt"
Sep 11 04:14:08.875591 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:14:08.875450    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/347b8352-a483-4682-8799-c9caef08bb74-dshm\") pod \"e2e-same-id-a-fa15b45f-kserve-6dc9dc589f-jnkmt\" (UID: \"347b8352-a483-4682-8799-c9caef08bb74\") " pod="llm/e2e-same-id-a-fa15b45f-kserve-6dc9dc589f-jnkmt"
Sep 11 04:14:08.875591 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:14:08.875474    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/347b8352-a483-4682-8799-c9caef08bb74-tls-certs\") pod \"e2e-same-id-a-fa15b45f-kserve-6dc9dc589f-jnkmt\" (UID: \"347b8352-a483-4682-8799-c9caef08bb74\") " pod="llm/e2e-same-id-a-fa15b45f-kserve-6dc9dc589f-jnkmt"
Sep 11 04:14:08.875754 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:14:08.875607    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/347b8352-a483-4682-8799-c9caef08bb74-tmp-dir\") pod \"e2e-same-id-a-fa15b45f-kserve-6dc9dc589f-jnkmt\" (UID: \"347b8352-a483-4682-8799-c9caef08bb74\") " pod="llm/e2e-same-id-a-fa15b45f-kserve-6dc9dc589f-jnkmt"
Sep 11 04:14:08.875921 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:14:08.875885    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/347b8352-a483-4682-8799-c9caef08bb74-home\") pod \"e2e-same-id-a-fa15b45f-kserve-6dc9dc589f-jnkmt\" (UID: \"347b8352-a483-4682-8799-c9caef08bb74\") " pod="llm/e2e-same-id-a-fa15b45f-kserve-6dc9dc589f-jnkmt"
Sep 11 04:14:08.876054 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:14:08.875939    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/347b8352-a483-4682-8799-c9caef08bb74-model-cache\") pod \"e2e-same-id-a-fa15b45f-kserve-6dc9dc589f-jnkmt\" (UID: \"347b8352-a483-4682-8799-c9caef08bb74\") " pod="llm/e2e-same-id-a-fa15b45f-kserve-6dc9dc589f-jnkmt"
Sep 11 04:14:08.876054 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:14:08.875995    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/347b8352-a483-4682-8799-c9caef08bb74-tmp-dir\") pod \"e2e-same-id-a-fa15b45f-kserve-6dc9dc589f-jnkmt\" (UID: \"347b8352-a483-4682-8799-c9caef08bb74\") " pod="llm/e2e-same-id-a-fa15b45f-kserve-6dc9dc589f-jnkmt"
Sep 11 04:14:08.878101 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:14:08.878060    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/347b8352-a483-4682-8799-c9caef08bb74-dshm\") pod \"e2e-same-id-a-fa15b45f-kserve-6dc9dc589f-jnkmt\" (UID: \"347b8352-a483-4682-8799-c9caef08bb74\") " pod="llm/e2e-same-id-a-fa15b45f-kserve-6dc9dc589f-jnkmt"
Sep 11 04:14:08.878264 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:14:08.878248    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/347b8352-a483-4682-8799-c9caef08bb74-tls-certs\") pod \"e2e-same-id-a-fa15b45f-kserve-6dc9dc589f-jnkmt\" (UID: \"347b8352-a483-4682-8799-c9caef08bb74\") " pod="llm/e2e-same-id-a-fa15b45f-kserve-6dc9dc589f-jnkmt"
Sep 11 04:14:08.883336 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:14:08.883317    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hv5wh\" (UniqueName: \"kubernetes.io/projected/347b8352-a483-4682-8799-c9caef08bb74-kube-api-access-hv5wh\") pod \"e2e-same-id-a-fa15b45f-kserve-6dc9dc589f-jnkmt\" (UID: \"347b8352-a483-4682-8799-c9caef08bb74\") " pod="llm/e2e-same-id-a-fa15b45f-kserve-6dc9dc589f-jnkmt"
Sep 11 04:14:09.006998 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:14:09.006904    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-fa15b45f-kserve-6dc9dc589f-jnkmt"
Sep 11 04:14:09.308281 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:14:09.308258    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-a-fa15b45f-kserve-6dc9dc589f-jnkmt"]
Sep 11 04:14:09.310757 ip-10-0-141-105 kubenswrapper[2579]: W0911 04:14:09.310729    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod347b8352_a483_4682_8799_c9caef08bb74.slice/crio-20bb844e7c33f1c29327896c1ed010e5cf7cd2a223a2e60af9643ad0b05a46af WatchSource:0}: Error finding container 20bb844e7c33f1c29327896c1ed010e5cf7cd2a223a2e60af9643ad0b05a46af: Status 404 returned error can't find the container with id 20bb844e7c33f1c29327896c1ed010e5cf7cd2a223a2e60af9643ad0b05a46af
Sep 11 04:14:09.622807 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:14:09.622769    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-fa15b45f-kserve-6dc9dc589f-jnkmt" event={"ID":"347b8352-a483-4682-8799-c9caef08bb74","Type":"ContainerStarted","Data":"678364dc4c04a38a6c5093b2275bd0bc1221c2bff28ea32bc4baaec19a4cef90"}
Sep 11 04:14:09.622807 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:14:09.622812    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-fa15b45f-kserve-6dc9dc589f-jnkmt" event={"ID":"347b8352-a483-4682-8799-c9caef08bb74","Type":"ContainerStarted","Data":"20bb844e7c33f1c29327896c1ed010e5cf7cd2a223a2e60af9643ad0b05a46af"}
Sep 11 04:14:09.641704 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:14:09.641657    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-same-id-a-fa15b45f-kserve-6dc9dc589f-jnkmt" podStartSLOduration=1.6416432090000002 podStartE2EDuration="1.641643209s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 04:14:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 04:14:09.640963812 +0000 UTC m=+2027.296169381" watchObservedRunningTime="2026-09-11 04:14:09.641643209 +0000 UTC m=+2027.296848775"
Sep 11 04:14:11.229909 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:14:11.229877    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-5z4cn"]
Sep 11 04:14:19.007612 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:14:19.007578    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-same-id-a-fa15b45f-kserve-6dc9dc589f-jnkmt"
Sep 11 04:14:19.008106 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:14:19.007635    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-same-id-a-fa15b45f-kserve-6dc9dc589f-jnkmt"
Sep 11 04:14:19.021326 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:14:19.021294    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-same-id-a-fa15b45f-kserve-6dc9dc589f-jnkmt"
Sep 11 04:14:19.683614 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:14:19.683579    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-same-id-a-fa15b45f-kserve-6dc9dc589f-jnkmt"
Sep 11 04:14:23.395719 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:14:23.395688    2579 scope.go:117] "RemoveContainer" containerID="aa331242eaa4c2190eb1e32bf809448b767ca1097ac3ff5deb6cadceafc95436"
Sep 11 04:14:29.237815 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:14:29.237777    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-5z4cn"]
Sep 11 04:15:13.165606 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:15:13.165570    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-a-fa15b45f-kserve-6dc9dc589f-jnkmt"]
Sep 11 04:15:15.964074 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:15:15.964035    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-5z4cn"]
Sep 11 04:15:23.417741 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:15:23.417708    2579 scope.go:117] "RemoveContainer" containerID="7766a1e04dc43d17122eff9fefee413eae163da493b8698b02a6eea45fb9ee54"
Sep 11 04:15:23.426519 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:15:23.426495    2579 scope.go:117] "RemoveContainer" containerID="b8eee1f55d04c0fbb7a4c99d52a7151b1c49152ffac9c040c03c7c68621a6cc6"
Sep 11 04:15:23.434001 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:15:23.433970    2579 scope.go:117] "RemoveContainer" containerID="74b8d2ab86001072da3ad1f91255fc2ca559c6d6d15a8a84714ce8bdd061f70b"
Sep 11 04:15:23.939835 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:15:23.939792    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-5z4cn"]
Sep 11 04:15:28.187963 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:15:28.187906    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="llm/e2e-same-id-a-fa15b45f-kserve-6dc9dc589f-jnkmt" podUID="347b8352-a483-4682-8799-c9caef08bb74" containerName="main" containerID="cri-o://678364dc4c04a38a6c5093b2275bd0bc1221c2bff28ea32bc4baaec19a4cef90" gracePeriod=45
Sep 11 04:15:28.431644 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:15:28.431621    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-fa15b45f-kserve-6dc9dc589f-jnkmt"
Sep 11 04:15:28.599664 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:15:28.599571    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/347b8352-a483-4682-8799-c9caef08bb74-dshm\") pod \"347b8352-a483-4682-8799-c9caef08bb74\" (UID: \"347b8352-a483-4682-8799-c9caef08bb74\") "
Sep 11 04:15:28.599664 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:15:28.599639    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/347b8352-a483-4682-8799-c9caef08bb74-home\") pod \"347b8352-a483-4682-8799-c9caef08bb74\" (UID: \"347b8352-a483-4682-8799-c9caef08bb74\") "
Sep 11 04:15:28.599934 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:15:28.599681    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/347b8352-a483-4682-8799-c9caef08bb74-tmp-dir\") pod \"347b8352-a483-4682-8799-c9caef08bb74\" (UID: \"347b8352-a483-4682-8799-c9caef08bb74\") "
Sep 11 04:15:28.599934 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:15:28.599705    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/347b8352-a483-4682-8799-c9caef08bb74-tls-certs\") pod \"347b8352-a483-4682-8799-c9caef08bb74\" (UID: \"347b8352-a483-4682-8799-c9caef08bb74\") "
Sep 11 04:15:28.599934 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:15:28.599726    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hv5wh\" (UniqueName: \"kubernetes.io/projected/347b8352-a483-4682-8799-c9caef08bb74-kube-api-access-hv5wh\") pod \"347b8352-a483-4682-8799-c9caef08bb74\" (UID: \"347b8352-a483-4682-8799-c9caef08bb74\") "
Sep 11 04:15:28.599934 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:15:28.599757    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/347b8352-a483-4682-8799-c9caef08bb74-model-cache\") pod \"347b8352-a483-4682-8799-c9caef08bb74\" (UID: \"347b8352-a483-4682-8799-c9caef08bb74\") "
Sep 11 04:15:28.600176 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:15:28.600002    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/347b8352-a483-4682-8799-c9caef08bb74-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "347b8352-a483-4682-8799-c9caef08bb74" (UID: "347b8352-a483-4682-8799-c9caef08bb74"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 04:15:28.600176 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:15:28.600016    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/347b8352-a483-4682-8799-c9caef08bb74-home" (OuterVolumeSpecName: "home") pod "347b8352-a483-4682-8799-c9caef08bb74" (UID: "347b8352-a483-4682-8799-c9caef08bb74"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 04:15:28.600176 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:15:28.600045    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/347b8352-a483-4682-8799-c9caef08bb74-model-cache" (OuterVolumeSpecName: "model-cache") pod "347b8352-a483-4682-8799-c9caef08bb74" (UID: "347b8352-a483-4682-8799-c9caef08bb74"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 04:15:28.601974 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:15:28.601944    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/347b8352-a483-4682-8799-c9caef08bb74-dshm" (OuterVolumeSpecName: "dshm") pod "347b8352-a483-4682-8799-c9caef08bb74" (UID: "347b8352-a483-4682-8799-c9caef08bb74"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 04:15:28.602112 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:15:28.601997    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/347b8352-a483-4682-8799-c9caef08bb74-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "347b8352-a483-4682-8799-c9caef08bb74" (UID: "347b8352-a483-4682-8799-c9caef08bb74"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 04:15:28.602112 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:15:28.602033    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/347b8352-a483-4682-8799-c9caef08bb74-kube-api-access-hv5wh" (OuterVolumeSpecName: "kube-api-access-hv5wh") pod "347b8352-a483-4682-8799-c9caef08bb74" (UID: "347b8352-a483-4682-8799-c9caef08bb74"). InnerVolumeSpecName "kube-api-access-hv5wh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 04:15:28.701101 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:15:28.701035    2579 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/347b8352-a483-4682-8799-c9caef08bb74-dshm\") on node \"ip-10-0-141-105.ec2.internal\" DevicePath \"\""
Sep 11 04:15:28.701101 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:15:28.701077    2579 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/347b8352-a483-4682-8799-c9caef08bb74-home\") on node \"ip-10-0-141-105.ec2.internal\" DevicePath \"\""
Sep 11 04:15:28.701312 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:15:28.701129    2579 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/347b8352-a483-4682-8799-c9caef08bb74-tmp-dir\") on node \"ip-10-0-141-105.ec2.internal\" DevicePath \"\""
Sep 11 04:15:28.701312 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:15:28.701139    2579 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/347b8352-a483-4682-8799-c9caef08bb74-tls-certs\") on node \"ip-10-0-141-105.ec2.internal\" DevicePath \"\""
Sep 11 04:15:28.701312 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:15:28.701149    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hv5wh\" (UniqueName: \"kubernetes.io/projected/347b8352-a483-4682-8799-c9caef08bb74-kube-api-access-hv5wh\") on node \"ip-10-0-141-105.ec2.internal\" DevicePath \"\""
Sep 11 04:15:28.701312 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:15:28.701159    2579 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/347b8352-a483-4682-8799-c9caef08bb74-model-cache\") on node \"ip-10-0-141-105.ec2.internal\" DevicePath \"\""
Sep 11 04:15:29.014529 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:15:29.014497    2579 generic.go:356] "Generic (PLEG): container finished" podID="347b8352-a483-4682-8799-c9caef08bb74" containerID="678364dc4c04a38a6c5093b2275bd0bc1221c2bff28ea32bc4baaec19a4cef90" exitCode=0
Sep 11 04:15:29.014705 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:15:29.014542    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-fa15b45f-kserve-6dc9dc589f-jnkmt"
Sep 11 04:15:29.014705 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:15:29.014590    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-fa15b45f-kserve-6dc9dc589f-jnkmt" event={"ID":"347b8352-a483-4682-8799-c9caef08bb74","Type":"ContainerDied","Data":"678364dc4c04a38a6c5093b2275bd0bc1221c2bff28ea32bc4baaec19a4cef90"}
Sep 11 04:15:29.014705 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:15:29.014640    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-fa15b45f-kserve-6dc9dc589f-jnkmt" event={"ID":"347b8352-a483-4682-8799-c9caef08bb74","Type":"ContainerDied","Data":"20bb844e7c33f1c29327896c1ed010e5cf7cd2a223a2e60af9643ad0b05a46af"}
Sep 11 04:15:29.014705 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:15:29.014663    2579 scope.go:117] "RemoveContainer" containerID="678364dc4c04a38a6c5093b2275bd0bc1221c2bff28ea32bc4baaec19a4cef90"
Sep 11 04:15:29.023238 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:15:29.023213    2579 scope.go:117] "RemoveContainer" containerID="678364dc4c04a38a6c5093b2275bd0bc1221c2bff28ea32bc4baaec19a4cef90"
Sep 11 04:15:29.023575 ip-10-0-141-105 kubenswrapper[2579]: E0911 04:15:29.023552    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"678364dc4c04a38a6c5093b2275bd0bc1221c2bff28ea32bc4baaec19a4cef90\": container with ID starting with 678364dc4c04a38a6c5093b2275bd0bc1221c2bff28ea32bc4baaec19a4cef90 not found: ID does not exist" containerID="678364dc4c04a38a6c5093b2275bd0bc1221c2bff28ea32bc4baaec19a4cef90"
Sep 11 04:15:29.023657 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:15:29.023585    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"678364dc4c04a38a6c5093b2275bd0bc1221c2bff28ea32bc4baaec19a4cef90"} err="failed to get container status \"678364dc4c04a38a6c5093b2275bd0bc1221c2bff28ea32bc4baaec19a4cef90\": rpc error: code = NotFound desc = could not find container \"678364dc4c04a38a6c5093b2275bd0bc1221c2bff28ea32bc4baaec19a4cef90\": container with ID starting with 678364dc4c04a38a6c5093b2275bd0bc1221c2bff28ea32bc4baaec19a4cef90 not found: ID does not exist"
Sep 11 04:15:29.031562 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:15:29.031538    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-a-fa15b45f-kserve-6dc9dc589f-jnkmt"]
Sep 11 04:15:29.035679 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:15:29.035657    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["llm/e2e-same-id-a-fa15b45f-kserve-6dc9dc589f-jnkmt"]
Sep 11 04:15:29.706579 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:15:29.706544    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-5z4cn"]
Sep 11 04:15:30.852204 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:15:30.852171    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="347b8352-a483-4682-8799-c9caef08bb74" path="/var/lib/kubelet/pods/347b8352-a483-4682-8799-c9caef08bb74/volumes"
Sep 11 04:15:37.209874 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:15:37.209840    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-5z4cn"]
Sep 11 04:15:45.429832 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:15:45.429797    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-5z4cn"]
Sep 11 04:15:52.433145 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:15:52.433097    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-5z4cn"]
Sep 11 04:16:00.311751 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:16:00.311716    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-5z4cn"]
Sep 11 04:16:05.609575 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:16:05.609539    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-5z4cn"]
Sep 11 04:16:08.720706 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:16:08.720673    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-fb4df7b9f-2mznf_0d12c92e-34d3-4af7-9d43-5de9ad14efe2/authorino/0.log"
Sep 11 04:16:14.797236 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:16:14.797154    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-6cf4d46bbf-jv5nx_584a1759-30fa-436c-b0d0-1aff194707e5/manager/0.log"
Sep 11 04:16:14.802390 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:16:14.802369    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-6cf4d46bbf-jv5nx_584a1759-30fa-436c-b0d0-1aff194707e5/copy-manifests/0.log"
Sep 11 04:16:15.660000 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:16:15.659973    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-key-cleanup-29818320-v5kl8_bc6d6c39-8d93-41c5-924b-96d8cc2693ce/cleanup/0.log"
Sep 11 04:16:16.733347 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:16:16.733306    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-fb4df7b9f-2mznf_0d12c92e-34d3-4af7-9d43-5de9ad14efe2/authorino/0.log"
Sep 11 04:16:16.974472 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:16:16.974440    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-7b84cd7c9c-grsb6_bee3cb9f-2f86-4ea2-ae3e-a3a0481dc5d3/manager/0.log"
Sep 11 04:16:17.374119 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:16:17.374073    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-84994f7ffc-5z4cn_cb7d6979-e4f6-469b-8be1-fcf4c05bd737/limitador/0.log"
Sep 11 04:16:18.219577 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:16:18.219552    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-86748cc5bf-kcpw8_68ec6bcd-85f8-46e2-bcc6-c70d0ec6e80f/kube-auth-proxy/0.log"
Sep 11 04:16:18.617399 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:16:18.617367    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5b54b7fbb7-69ccr_598a451c-3387-4674-9c03-6ea4ca648208/payload-processing/0.log"
Sep 11 04:16:18.745266 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:16:18.745234    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-7ff5fb48d-7h989_2695af89-0e52-4bd4-9bc5-da707f50efc9/router/0.log"
Sep 11 04:16:19.713477 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:16:19.713438    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_facebook-opt-125m-simulated-kserve-74b68447df-9cg5z_d922ae85-71de-4dbe-ac65-75b8ae0d20c9/main/0.log"
Sep 11 04:16:19.833030 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:16:19.833001    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_premium-simulated-simulated-premium-kserve-57fc66f697-m7krz_0938a98d-0acd-4959-80b0-da59ad7f756c/main/0.log"
Sep 11 04:16:25.735765 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:16:25.735729    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-85rjf/must-gather-tbdkp"]
Sep 11 04:16:25.736173 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:16:25.736127    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="347b8352-a483-4682-8799-c9caef08bb74" containerName="main"
Sep 11 04:16:25.736173 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:16:25.736139    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="347b8352-a483-4682-8799-c9caef08bb74" containerName="main"
Sep 11 04:16:25.736260 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:16:25.736189    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="347b8352-a483-4682-8799-c9caef08bb74" containerName="main"
Sep 11 04:16:25.747604 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:16:25.747576    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-85rjf/must-gather-tbdkp"
Sep 11 04:16:25.751216 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:16:25.751192    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-85rjf\"/\"default-dockercfg-j767r\""
Sep 11 04:16:25.751365 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:16:25.751193    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-85rjf\"/\"kube-root-ca.crt\""
Sep 11 04:16:25.751607 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:16:25.751583    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-85rjf\"/\"openshift-service-ca.crt\""
Sep 11 04:16:25.753934 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:16:25.753908    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-85rjf/must-gather-tbdkp"]
Sep 11 04:16:25.912419 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:16:25.912377    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-phkw6\" (UniqueName: \"kubernetes.io/projected/2be16f6b-bc36-4806-8097-4b89b329bd83-kube-api-access-phkw6\") pod \"must-gather-tbdkp\" (UID: \"2be16f6b-bc36-4806-8097-4b89b329bd83\") " pod="openshift-must-gather-85rjf/must-gather-tbdkp"
Sep 11 04:16:25.912610 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:16:25.912442    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/2be16f6b-bc36-4806-8097-4b89b329bd83-must-gather-output\") pod \"must-gather-tbdkp\" (UID: \"2be16f6b-bc36-4806-8097-4b89b329bd83\") " pod="openshift-must-gather-85rjf/must-gather-tbdkp"
Sep 11 04:16:26.013829 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:16:26.013744    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/2be16f6b-bc36-4806-8097-4b89b329bd83-must-gather-output\") pod \"must-gather-tbdkp\" (UID: \"2be16f6b-bc36-4806-8097-4b89b329bd83\") " pod="openshift-must-gather-85rjf/must-gather-tbdkp"
Sep 11 04:16:26.013973 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:16:26.013843    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-phkw6\" (UniqueName: \"kubernetes.io/projected/2be16f6b-bc36-4806-8097-4b89b329bd83-kube-api-access-phkw6\") pod \"must-gather-tbdkp\" (UID: \"2be16f6b-bc36-4806-8097-4b89b329bd83\") " pod="openshift-must-gather-85rjf/must-gather-tbdkp"
Sep 11 04:16:26.014187 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:16:26.014164    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/2be16f6b-bc36-4806-8097-4b89b329bd83-must-gather-output\") pod \"must-gather-tbdkp\" (UID: \"2be16f6b-bc36-4806-8097-4b89b329bd83\") " pod="openshift-must-gather-85rjf/must-gather-tbdkp"
Sep 11 04:16:26.024959 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:16:26.024925    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-phkw6\" (UniqueName: \"kubernetes.io/projected/2be16f6b-bc36-4806-8097-4b89b329bd83-kube-api-access-phkw6\") pod \"must-gather-tbdkp\" (UID: \"2be16f6b-bc36-4806-8097-4b89b329bd83\") " pod="openshift-must-gather-85rjf/must-gather-tbdkp"
Sep 11 04:16:26.057126 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:16:26.057099    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-85rjf/must-gather-tbdkp"
Sep 11 04:16:26.333997 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:16:26.333971    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-85rjf/must-gather-tbdkp"]
Sep 11 04:16:26.336739 ip-10-0-141-105 kubenswrapper[2579]: W0911 04:16:26.336711    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod2be16f6b_bc36_4806_8097_4b89b329bd83.slice/crio-2cb1b1adcaa5ceba45cb6cb1b3883b6e28499a2aec6d01a75b010ef3fd092992 WatchSource:0}: Error finding container 2cb1b1adcaa5ceba45cb6cb1b3883b6e28499a2aec6d01a75b010ef3fd092992: Status 404 returned error can't find the container with id 2cb1b1adcaa5ceba45cb6cb1b3883b6e28499a2aec6d01a75b010ef3fd092992
Sep 11 04:16:26.338637 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:16:26.338617    2579 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 11 04:16:27.292022 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:16:27.291982    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-85rjf/must-gather-tbdkp" event={"ID":"2be16f6b-bc36-4806-8097-4b89b329bd83","Type":"ContainerStarted","Data":"2cb1b1adcaa5ceba45cb6cb1b3883b6e28499a2aec6d01a75b010ef3fd092992"}
Sep 11 04:16:28.302806 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:16:28.302748    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-85rjf/must-gather-tbdkp" event={"ID":"2be16f6b-bc36-4806-8097-4b89b329bd83","Type":"ContainerStarted","Data":"a6f0dd412442f7a62ad5074a0d5ba9a93ba6a0d300c2b1b8e334e58f56733b29"}
Sep 11 04:16:28.303491 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:16:28.303424    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-85rjf/must-gather-tbdkp" event={"ID":"2be16f6b-bc36-4806-8097-4b89b329bd83","Type":"ContainerStarted","Data":"b9cc8a46a3b7347cc063539e5a96756daf7cbf63e90b76251370e2677c29a7a0"}
Sep 11 04:16:28.321407 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:16:28.321356    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-85rjf/must-gather-tbdkp" podStartSLOduration=2.461766232 podStartE2EDuration="3.321340159s" totalImagesPullingTime="859.573927ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 04:16:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 04:16:28.31973671 +0000 UTC m=+2165.974942288" watchObservedRunningTime="2026-09-11 04:16:28.321340159 +0000 UTC m=+2165.976545764"
Sep 11 04:16:28.940847 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:16:28.940815    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-jwm97_1a7a4b58-e08a-45ef-ab7e-692d3a4f5314/global-pull-secret-syncer/0.log"
Sep 11 04:16:29.087955 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:16:29.087925    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-s9xqv_ff572c4a-7a16-4c2b-b0a8-160b3b274834/konnectivity-agent/0.log"
Sep 11 04:16:29.238176 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:16:29.238053    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-141-105.ec2.internal_c9ea5fe55135c7b1e5541fdd12fac4be/haproxy/0.log"
Sep 11 04:16:34.291581 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:16:34.291505    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-fb4df7b9f-2mznf_0d12c92e-34d3-4af7-9d43-5de9ad14efe2/authorino/0.log"
Sep 11 04:16:34.357624 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:16:34.357593    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-7b84cd7c9c-grsb6_bee3cb9f-2f86-4ea2-ae3e-a3a0481dc5d3/manager/0.log"
Sep 11 04:16:34.464796 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:16:34.464768    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-84994f7ffc-5z4cn_cb7d6979-e4f6-469b-8be1-fcf4c05bd737/limitador/0.log"
Sep 11 04:16:36.247067 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:16:36.247000    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-59fbdd77d-b5qx4_addfafec-32a6-4f08-95a8-80909d0e6d73/kube-state-metrics/0.log"
Sep 11 04:16:36.280924 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:16:36.280797    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-59fbdd77d-b5qx4_addfafec-32a6-4f08-95a8-80909d0e6d73/kube-rbac-proxy-main/0.log"
Sep 11 04:16:36.304674 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:16:36.304641    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-59fbdd77d-b5qx4_addfafec-32a6-4f08-95a8-80909d0e6d73/kube-rbac-proxy-self/0.log"
Sep 11 04:16:36.360238 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:16:36.360211    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_metrics-server-7ff6c96894-qdqpx_4721c4ee-ba92-4cec-98e1-00040dc00315/metrics-server/0.log"
Sep 11 04:16:36.607457 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:16:36.607415    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-ph5tp_a3b10037-14c1-484f-9733-d0b6913c73a9/node-exporter/0.log"
Sep 11 04:16:36.671752 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:16:36.671679    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-ph5tp_a3b10037-14c1-484f-9733-d0b6913c73a9/kube-rbac-proxy/0.log"
Sep 11 04:16:36.719702 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:16:36.719677    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-ph5tp_a3b10037-14c1-484f-9733-d0b6913c73a9/init-textfile/0.log"
Sep 11 04:16:37.158931 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:16:37.158889    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_eb03d7bc-abea-4d73-8833-1829d3d7f2d4/prometheus/0.log"
Sep 11 04:16:37.233021 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:16:37.232995    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_eb03d7bc-abea-4d73-8833-1829d3d7f2d4/config-reloader/0.log"
Sep 11 04:16:37.286242 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:16:37.286173    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_eb03d7bc-abea-4d73-8833-1829d3d7f2d4/thanos-sidecar/0.log"
Sep 11 04:16:37.334205 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:16:37.334146    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_eb03d7bc-abea-4d73-8833-1829d3d7f2d4/kube-rbac-proxy-web/0.log"
Sep 11 04:16:37.380490 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:16:37.380458    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_eb03d7bc-abea-4d73-8833-1829d3d7f2d4/kube-rbac-proxy/0.log"
Sep 11 04:16:37.419803 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:16:37.419714    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_eb03d7bc-abea-4d73-8833-1829d3d7f2d4/kube-rbac-proxy-thanos/0.log"
Sep 11 04:16:37.474470 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:16:37.474405    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_eb03d7bc-abea-4d73-8833-1829d3d7f2d4/init-config-reloader/0.log"
Sep 11 04:16:37.739332 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:16:37.739239    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-9f6946f8f-5lcb2_0fdb316e-3b02-4e85-856d-56526c0c1e48/thanos-query/0.log"
Sep 11 04:16:37.768297 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:16:37.768204    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-9f6946f8f-5lcb2_0fdb316e-3b02-4e85-856d-56526c0c1e48/kube-rbac-proxy-web/0.log"
Sep 11 04:16:37.797036 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:16:37.797007    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-9f6946f8f-5lcb2_0fdb316e-3b02-4e85-856d-56526c0c1e48/kube-rbac-proxy/0.log"
Sep 11 04:16:37.831460 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:16:37.831432    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-9f6946f8f-5lcb2_0fdb316e-3b02-4e85-856d-56526c0c1e48/prom-label-proxy/0.log"
Sep 11 04:16:37.865418 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:16:37.865389    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-9f6946f8f-5lcb2_0fdb316e-3b02-4e85-856d-56526c0c1e48/kube-rbac-proxy-rules/0.log"
Sep 11 04:16:37.891611 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:16:37.891578    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-9f6946f8f-5lcb2_0fdb316e-3b02-4e85-856d-56526c0c1e48/kube-rbac-proxy-metrics/0.log"
Sep 11 04:16:39.710946 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:16:39.710900    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-85rjf/perf-node-gather-daemonset-wwrhg"]
Sep 11 04:16:39.718556 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:16:39.718520    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-85rjf/perf-node-gather-daemonset-wwrhg"
Sep 11 04:16:39.729573 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:16:39.728074    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-85rjf/perf-node-gather-daemonset-wwrhg"]
Sep 11 04:16:39.771540 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:16:39.771500    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/caf28a82-0763-44f3-a1a8-e348ed364010-sys\") pod \"perf-node-gather-daemonset-wwrhg\" (UID: \"caf28a82-0763-44f3-a1a8-e348ed364010\") " pod="openshift-must-gather-85rjf/perf-node-gather-daemonset-wwrhg"
Sep 11 04:16:39.771732 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:16:39.771570    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fjj6p\" (UniqueName: \"kubernetes.io/projected/caf28a82-0763-44f3-a1a8-e348ed364010-kube-api-access-fjj6p\") pod \"perf-node-gather-daemonset-wwrhg\" (UID: \"caf28a82-0763-44f3-a1a8-e348ed364010\") " pod="openshift-must-gather-85rjf/perf-node-gather-daemonset-wwrhg"
Sep 11 04:16:39.771732 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:16:39.771617    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/caf28a82-0763-44f3-a1a8-e348ed364010-podres\") pod \"perf-node-gather-daemonset-wwrhg\" (UID: \"caf28a82-0763-44f3-a1a8-e348ed364010\") " pod="openshift-must-gather-85rjf/perf-node-gather-daemonset-wwrhg"
Sep 11 04:16:39.771732 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:16:39.771652    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/caf28a82-0763-44f3-a1a8-e348ed364010-proc\") pod \"perf-node-gather-daemonset-wwrhg\" (UID: \"caf28a82-0763-44f3-a1a8-e348ed364010\") " pod="openshift-must-gather-85rjf/perf-node-gather-daemonset-wwrhg"
Sep 11 04:16:39.771732 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:16:39.771682    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/caf28a82-0763-44f3-a1a8-e348ed364010-lib-modules\") pod \"perf-node-gather-daemonset-wwrhg\" (UID: \"caf28a82-0763-44f3-a1a8-e348ed364010\") " pod="openshift-must-gather-85rjf/perf-node-gather-daemonset-wwrhg"
Sep 11 04:16:39.873069 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:16:39.873029    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/caf28a82-0763-44f3-a1a8-e348ed364010-sys\") pod \"perf-node-gather-daemonset-wwrhg\" (UID: \"caf28a82-0763-44f3-a1a8-e348ed364010\") " pod="openshift-must-gather-85rjf/perf-node-gather-daemonset-wwrhg"
Sep 11 04:16:39.873372 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:16:39.873351    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fjj6p\" (UniqueName: \"kubernetes.io/projected/caf28a82-0763-44f3-a1a8-e348ed364010-kube-api-access-fjj6p\") pod \"perf-node-gather-daemonset-wwrhg\" (UID: \"caf28a82-0763-44f3-a1a8-e348ed364010\") " pod="openshift-must-gather-85rjf/perf-node-gather-daemonset-wwrhg"
Sep 11 04:16:39.873521 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:16:39.873506    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/caf28a82-0763-44f3-a1a8-e348ed364010-podres\") pod \"perf-node-gather-daemonset-wwrhg\" (UID: \"caf28a82-0763-44f3-a1a8-e348ed364010\") " pod="openshift-must-gather-85rjf/perf-node-gather-daemonset-wwrhg"
Sep 11 04:16:39.873649 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:16:39.873624    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/caf28a82-0763-44f3-a1a8-e348ed364010-proc\") pod \"perf-node-gather-daemonset-wwrhg\" (UID: \"caf28a82-0763-44f3-a1a8-e348ed364010\") " pod="openshift-must-gather-85rjf/perf-node-gather-daemonset-wwrhg"
Sep 11 04:16:39.873782 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:16:39.873768    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/caf28a82-0763-44f3-a1a8-e348ed364010-lib-modules\") pod \"perf-node-gather-daemonset-wwrhg\" (UID: \"caf28a82-0763-44f3-a1a8-e348ed364010\") " pod="openshift-must-gather-85rjf/perf-node-gather-daemonset-wwrhg"
Sep 11 04:16:39.873922 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:16:39.873893    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/caf28a82-0763-44f3-a1a8-e348ed364010-lib-modules\") pod \"perf-node-gather-daemonset-wwrhg\" (UID: \"caf28a82-0763-44f3-a1a8-e348ed364010\") " pod="openshift-must-gather-85rjf/perf-node-gather-daemonset-wwrhg"
Sep 11 04:16:39.874990 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:16:39.873702    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/caf28a82-0763-44f3-a1a8-e348ed364010-proc\") pod \"perf-node-gather-daemonset-wwrhg\" (UID: \"caf28a82-0763-44f3-a1a8-e348ed364010\") " pod="openshift-must-gather-85rjf/perf-node-gather-daemonset-wwrhg"
Sep 11 04:16:39.875154 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:16:39.873656    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/caf28a82-0763-44f3-a1a8-e348ed364010-podres\") pod \"perf-node-gather-daemonset-wwrhg\" (UID: \"caf28a82-0763-44f3-a1a8-e348ed364010\") " pod="openshift-must-gather-85rjf/perf-node-gather-daemonset-wwrhg"
Sep 11 04:16:39.875262 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:16:39.873190    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/caf28a82-0763-44f3-a1a8-e348ed364010-sys\") pod \"perf-node-gather-daemonset-wwrhg\" (UID: \"caf28a82-0763-44f3-a1a8-e348ed364010\") " pod="openshift-must-gather-85rjf/perf-node-gather-daemonset-wwrhg"
Sep 11 04:16:39.884220 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:16:39.884196    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fjj6p\" (UniqueName: \"kubernetes.io/projected/caf28a82-0763-44f3-a1a8-e348ed364010-kube-api-access-fjj6p\") pod \"perf-node-gather-daemonset-wwrhg\" (UID: \"caf28a82-0763-44f3-a1a8-e348ed364010\") " pod="openshift-must-gather-85rjf/perf-node-gather-daemonset-wwrhg"
Sep 11 04:16:40.033569 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:16:40.033483    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-85rjf/perf-node-gather-daemonset-wwrhg"
Sep 11 04:16:40.326655 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:16:40.326617    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-85rjf/perf-node-gather-daemonset-wwrhg"]
Sep 11 04:16:40.387065 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:16:40.387024    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-85rjf/perf-node-gather-daemonset-wwrhg" event={"ID":"caf28a82-0763-44f3-a1a8-e348ed364010","Type":"ContainerStarted","Data":"59ceb2c205f61711d42fefd7d5ceb916feedb1bd79c3d1c3fcab37acb2f9edda"}
Sep 11 04:16:41.150138 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:16:41.150108    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-storage-operator_volume-data-source-validator-86695b7cc7-82sjt_5a0a9ca4-faf7-4b53-8f35-7f5138ab0895/volume-data-source-validator/0.log"
Sep 11 04:16:41.397177 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:16:41.396851    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-85rjf/perf-node-gather-daemonset-wwrhg" event={"ID":"caf28a82-0763-44f3-a1a8-e348ed364010","Type":"ContainerStarted","Data":"9ee8164b9fa92e3d6d26ae4f02e7c40863c001c883bc8b075e5874e5cb9307cc"}
Sep 11 04:16:41.397177 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:16:41.397132    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-85rjf/perf-node-gather-daemonset-wwrhg"
Sep 11 04:16:41.417914 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:16:41.417780    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-85rjf/perf-node-gather-daemonset-wwrhg" podStartSLOduration=2.417757403 podStartE2EDuration="2.417757403s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 04:16:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 04:16:41.414481224 +0000 UTC m=+2179.069686792" watchObservedRunningTime="2026-09-11 04:16:41.417757403 +0000 UTC m=+2179.072962971"
Sep 11 04:16:42.319805 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:16:42.319767    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-rkpx7_82475b03-f46c-4157-a864-0e92015820af/dns/0.log"
Sep 11 04:16:42.344228 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:16:42.344194    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-rkpx7_82475b03-f46c-4157-a864-0e92015820af/kube-rbac-proxy/0.log"
Sep 11 04:16:42.395421 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:16:42.395387    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-mhjgr_c35abd6b-ad57-4b78-b072-56e6cd032381/dns-node-resolver/0.log"
Sep 11 04:16:43.040383 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:16:43.040354    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-zg5qq_9fc36641-9051-4f34-a897-bb4c92977307/node-ca/0.log"
Sep 11 04:16:44.358470 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:16:44.358440    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-86748cc5bf-kcpw8_68ec6bcd-85f8-46e2-bcc6-c70d0ec6e80f/kube-auth-proxy/0.log"
Sep 11 04:16:44.496807 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:16:44.496773    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5b54b7fbb7-69ccr_598a451c-3387-4674-9c03-6ea4ca648208/payload-processing/0.log"
Sep 11 04:16:44.524022 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:16:44.523992    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-7ff5fb48d-7h989_2695af89-0e52-4bd4-9bc5-da707f50efc9/router/0.log"
Sep 11 04:16:45.541937 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:16:45.541905    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-wttz6_7eae32af-5f15-4e41-993e-ae4397cdd634/serve-healthcheck-canary/0.log"
Sep 11 04:16:46.211162 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:16:46.211131    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-8zk4s_6bc2f317-5df1-4b84-9cc6-496fd598ee89/kube-rbac-proxy/0.log"
Sep 11 04:16:46.254149 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:16:46.254120    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-8zk4s_6bc2f317-5df1-4b84-9cc6-496fd598ee89/exporter/0.log"
Sep 11 04:16:46.291967 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:16:46.291929    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-8zk4s_6bc2f317-5df1-4b84-9cc6-496fd598ee89/extractor/0.log"
Sep 11 04:16:47.412951 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:16:47.412922    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-85rjf/perf-node-gather-daemonset-wwrhg"
Sep 11 04:16:49.863830 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:16:49.863796    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-6cf4d46bbf-jv5nx_584a1759-30fa-436c-b0d0-1aff194707e5/manager/0.log"
Sep 11 04:16:49.897984 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:16:49.897960    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-6cf4d46bbf-jv5nx_584a1759-30fa-436c-b0d0-1aff194707e5/copy-manifests/0.log"
Sep 11 04:16:50.507511 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:16:50.507427    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-86748cc5bf-kcpw8_68ec6bcd-85f8-46e2-bcc6-c70d0ec6e80f/kube-auth-proxy/0.log"
Sep 11 04:16:50.640773 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:16:50.640737    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5b54b7fbb7-69ccr_598a451c-3387-4674-9c03-6ea4ca648208/payload-processing/0.log"
Sep 11 04:16:50.661826 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:16:50.661794    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-7ff5fb48d-7h989_2695af89-0e52-4bd4-9bc5-da707f50efc9/router/0.log"
Sep 11 04:16:51.579576 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:16:51.579497    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_lws-controller-manager-67c8664749-z6zpb_17e941b0-9899-4103-ba1a-2566a6c2f42d/manager/0.log"
Sep 11 04:16:52.659519 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:16:52.659491    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-86748cc5bf-kcpw8_68ec6bcd-85f8-46e2-bcc6-c70d0ec6e80f/kube-auth-proxy/0.log"
Sep 11 04:16:52.818610 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:16:52.818580    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5b54b7fbb7-69ccr_598a451c-3387-4674-9c03-6ea4ca648208/payload-processing/0.log"
Sep 11 04:16:52.864654 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:16:52.864617    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-7ff5fb48d-7h989_2695af89-0e52-4bd4-9bc5-da707f50efc9/router/0.log"
Sep 11 04:16:56.840482 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:16:56.840432    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-8b4d697c5-k7vtl_7d444dfd-481f-4684-a469-94371b9331fc/migrator/0.log"
Sep 11 04:16:56.868283 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:16:56.868248    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-8b4d697c5-k7vtl_7d444dfd-481f-4684-a469-94371b9331fc/graceful-termination/0.log"
Sep 11 04:16:57.245314 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:16:57.245284    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-6c8f84f897-6hjz5_9dbdbcd2-cfbf-447b-8064-35a1852b1a9d/kube-storage-version-migrator-operator/1.log"
Sep 11 04:16:57.246480 ip-10-0-141-105 kubenswrapper[2579]: I0911 04:16:57.246455    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-6c8f84f897-6hjz5_9dbdbcd2-cfbf-447b-8064-35a1852b1a9d/kube-storage-version-migrator-operator/0.log"