Sep 16 02:18:41.420835 ip-10-0-128-111 systemd[1]: Starting Kubernetes Kubelet...
Sep 16 02:18:41.823998 ip-10-0-128-111 kubenswrapper[2583]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 16 02:18:41.823998 ip-10-0-128-111 kubenswrapper[2583]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Sep 16 02:18:41.823998 ip-10-0-128-111 kubenswrapper[2583]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 16 02:18:41.823998 ip-10-0-128-111 kubenswrapper[2583]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI.
Sep 16 02:18:41.823998 ip-10-0-128-111 kubenswrapper[2583]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 16 02:18:41.825705 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:18:41.825615    2583 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime"
Sep 16 02:18:41.831990 ip-10-0-128-111 kubenswrapper[2583]: W0916 02:18:41.831965    2583 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 16 02:18:41.831990 ip-10-0-128-111 kubenswrapper[2583]: W0916 02:18:41.831984    2583 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 16 02:18:41.831990 ip-10-0-128-111 kubenswrapper[2583]: W0916 02:18:41.831988    2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 16 02:18:41.831990 ip-10-0-128-111 kubenswrapper[2583]: W0916 02:18:41.831992    2583 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 16 02:18:41.831990 ip-10-0-128-111 kubenswrapper[2583]: W0916 02:18:41.831995    2583 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 16 02:18:41.831990 ip-10-0-128-111 kubenswrapper[2583]: W0916 02:18:41.831998    2583 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 16 02:18:41.832220 ip-10-0-128-111 kubenswrapper[2583]: W0916 02:18:41.832002    2583 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 16 02:18:41.832220 ip-10-0-128-111 kubenswrapper[2583]: W0916 02:18:41.832005    2583 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 16 02:18:41.832220 ip-10-0-128-111 kubenswrapper[2583]: W0916 02:18:41.832008    2583 feature_gate.go:328] unrecognized feature gate: Example2
Sep 16 02:18:41.832220 ip-10-0-128-111 kubenswrapper[2583]: W0916 02:18:41.832011    2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 16 02:18:41.832220 ip-10-0-128-111 kubenswrapper[2583]: W0916 02:18:41.832013    2583 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 16 02:18:41.832220 ip-10-0-128-111 kubenswrapper[2583]: W0916 02:18:41.832018    2583 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 16 02:18:41.832220 ip-10-0-128-111 kubenswrapper[2583]: W0916 02:18:41.832022    2583 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 16 02:18:41.832220 ip-10-0-128-111 kubenswrapper[2583]: W0916 02:18:41.832025    2583 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 16 02:18:41.832220 ip-10-0-128-111 kubenswrapper[2583]: W0916 02:18:41.832029    2583 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 16 02:18:41.832220 ip-10-0-128-111 kubenswrapper[2583]: W0916 02:18:41.832032    2583 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 16 02:18:41.832220 ip-10-0-128-111 kubenswrapper[2583]: W0916 02:18:41.832035    2583 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 16 02:18:41.832220 ip-10-0-128-111 kubenswrapper[2583]: W0916 02:18:41.832038    2583 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 16 02:18:41.832220 ip-10-0-128-111 kubenswrapper[2583]: W0916 02:18:41.832041    2583 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 16 02:18:41.832220 ip-10-0-128-111 kubenswrapper[2583]: W0916 02:18:41.832043    2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 16 02:18:41.832220 ip-10-0-128-111 kubenswrapper[2583]: W0916 02:18:41.832047    2583 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 16 02:18:41.832220 ip-10-0-128-111 kubenswrapper[2583]: W0916 02:18:41.832051    2583 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 16 02:18:41.832220 ip-10-0-128-111 kubenswrapper[2583]: W0916 02:18:41.832054    2583 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 16 02:18:41.832220 ip-10-0-128-111 kubenswrapper[2583]: W0916 02:18:41.832058    2583 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 16 02:18:41.832220 ip-10-0-128-111 kubenswrapper[2583]: W0916 02:18:41.832060    2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 16 02:18:41.832220 ip-10-0-128-111 kubenswrapper[2583]: W0916 02:18:41.832063    2583 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 16 02:18:41.832763 ip-10-0-128-111 kubenswrapper[2583]: W0916 02:18:41.832066    2583 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 16 02:18:41.832763 ip-10-0-128-111 kubenswrapper[2583]: W0916 02:18:41.832068    2583 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 16 02:18:41.832763 ip-10-0-128-111 kubenswrapper[2583]: W0916 02:18:41.832071    2583 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 16 02:18:41.832763 ip-10-0-128-111 kubenswrapper[2583]: W0916 02:18:41.832074    2583 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 16 02:18:41.832763 ip-10-0-128-111 kubenswrapper[2583]: W0916 02:18:41.832076    2583 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 16 02:18:41.832763 ip-10-0-128-111 kubenswrapper[2583]: W0916 02:18:41.832079    2583 feature_gate.go:328] unrecognized feature gate: Example
Sep 16 02:18:41.832763 ip-10-0-128-111 kubenswrapper[2583]: W0916 02:18:41.832082    2583 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 16 02:18:41.832763 ip-10-0-128-111 kubenswrapper[2583]: W0916 02:18:41.832084    2583 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 16 02:18:41.832763 ip-10-0-128-111 kubenswrapper[2583]: W0916 02:18:41.832087    2583 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 16 02:18:41.832763 ip-10-0-128-111 kubenswrapper[2583]: W0916 02:18:41.832089    2583 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 16 02:18:41.832763 ip-10-0-128-111 kubenswrapper[2583]: W0916 02:18:41.832092    2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 16 02:18:41.832763 ip-10-0-128-111 kubenswrapper[2583]: W0916 02:18:41.832095    2583 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 16 02:18:41.832763 ip-10-0-128-111 kubenswrapper[2583]: W0916 02:18:41.832098    2583 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 16 02:18:41.832763 ip-10-0-128-111 kubenswrapper[2583]: W0916 02:18:41.832101    2583 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 16 02:18:41.832763 ip-10-0-128-111 kubenswrapper[2583]: W0916 02:18:41.832104    2583 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 16 02:18:41.832763 ip-10-0-128-111 kubenswrapper[2583]: W0916 02:18:41.832106    2583 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 16 02:18:41.832763 ip-10-0-128-111 kubenswrapper[2583]: W0916 02:18:41.832109    2583 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 16 02:18:41.832763 ip-10-0-128-111 kubenswrapper[2583]: W0916 02:18:41.832111    2583 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 16 02:18:41.832763 ip-10-0-128-111 kubenswrapper[2583]: W0916 02:18:41.832114    2583 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 16 02:18:41.832763 ip-10-0-128-111 kubenswrapper[2583]: W0916 02:18:41.832116    2583 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 16 02:18:41.833261 ip-10-0-128-111 kubenswrapper[2583]: W0916 02:18:41.832119    2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 16 02:18:41.833261 ip-10-0-128-111 kubenswrapper[2583]: W0916 02:18:41.832122    2583 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 16 02:18:41.833261 ip-10-0-128-111 kubenswrapper[2583]: W0916 02:18:41.832124    2583 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 16 02:18:41.833261 ip-10-0-128-111 kubenswrapper[2583]: W0916 02:18:41.832127    2583 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 16 02:18:41.833261 ip-10-0-128-111 kubenswrapper[2583]: W0916 02:18:41.832130    2583 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 16 02:18:41.833261 ip-10-0-128-111 kubenswrapper[2583]: W0916 02:18:41.832132    2583 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 16 02:18:41.833261 ip-10-0-128-111 kubenswrapper[2583]: W0916 02:18:41.832135    2583 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 16 02:18:41.833261 ip-10-0-128-111 kubenswrapper[2583]: W0916 02:18:41.832138    2583 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 16 02:18:41.833261 ip-10-0-128-111 kubenswrapper[2583]: W0916 02:18:41.832140    2583 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 16 02:18:41.833261 ip-10-0-128-111 kubenswrapper[2583]: W0916 02:18:41.832143    2583 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 16 02:18:41.833261 ip-10-0-128-111 kubenswrapper[2583]: W0916 02:18:41.832145    2583 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 16 02:18:41.833261 ip-10-0-128-111 kubenswrapper[2583]: W0916 02:18:41.832148    2583 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 16 02:18:41.833261 ip-10-0-128-111 kubenswrapper[2583]: W0916 02:18:41.832150    2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 16 02:18:41.833261 ip-10-0-128-111 kubenswrapper[2583]: W0916 02:18:41.832153    2583 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 16 02:18:41.833261 ip-10-0-128-111 kubenswrapper[2583]: W0916 02:18:41.832155    2583 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 16 02:18:41.833261 ip-10-0-128-111 kubenswrapper[2583]: W0916 02:18:41.832158    2583 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 16 02:18:41.833261 ip-10-0-128-111 kubenswrapper[2583]: W0916 02:18:41.832161    2583 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 16 02:18:41.833261 ip-10-0-128-111 kubenswrapper[2583]: W0916 02:18:41.832164    2583 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 16 02:18:41.833261 ip-10-0-128-111 kubenswrapper[2583]: W0916 02:18:41.832167    2583 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 16 02:18:41.833747 ip-10-0-128-111 kubenswrapper[2583]: W0916 02:18:41.832169    2583 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 16 02:18:41.833747 ip-10-0-128-111 kubenswrapper[2583]: W0916 02:18:41.832172    2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 16 02:18:41.833747 ip-10-0-128-111 kubenswrapper[2583]: W0916 02:18:41.832174    2583 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 16 02:18:41.833747 ip-10-0-128-111 kubenswrapper[2583]: W0916 02:18:41.832177    2583 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 16 02:18:41.833747 ip-10-0-128-111 kubenswrapper[2583]: W0916 02:18:41.832179    2583 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 16 02:18:41.833747 ip-10-0-128-111 kubenswrapper[2583]: W0916 02:18:41.832182    2583 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 16 02:18:41.833747 ip-10-0-128-111 kubenswrapper[2583]: W0916 02:18:41.832186    2583 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 16 02:18:41.833747 ip-10-0-128-111 kubenswrapper[2583]: W0916 02:18:41.832188    2583 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 16 02:18:41.833747 ip-10-0-128-111 kubenswrapper[2583]: W0916 02:18:41.832191    2583 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 16 02:18:41.833747 ip-10-0-128-111 kubenswrapper[2583]: W0916 02:18:41.832193    2583 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 16 02:18:41.833747 ip-10-0-128-111 kubenswrapper[2583]: W0916 02:18:41.832196    2583 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 16 02:18:41.833747 ip-10-0-128-111 kubenswrapper[2583]: W0916 02:18:41.832200    2583 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 16 02:18:41.833747 ip-10-0-128-111 kubenswrapper[2583]: W0916 02:18:41.832204    2583 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 16 02:18:41.833747 ip-10-0-128-111 kubenswrapper[2583]: W0916 02:18:41.832207    2583 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 16 02:18:41.833747 ip-10-0-128-111 kubenswrapper[2583]: W0916 02:18:41.832211    2583 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 16 02:18:41.833747 ip-10-0-128-111 kubenswrapper[2583]: W0916 02:18:41.832214    2583 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 16 02:18:41.833747 ip-10-0-128-111 kubenswrapper[2583]: W0916 02:18:41.832216    2583 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 16 02:18:41.833747 ip-10-0-128-111 kubenswrapper[2583]: W0916 02:18:41.832219    2583 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 16 02:18:41.833747 ip-10-0-128-111 kubenswrapper[2583]: W0916 02:18:41.832222    2583 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 16 02:18:41.834208 ip-10-0-128-111 kubenswrapper[2583]: W0916 02:18:41.832224    2583 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 16 02:18:41.834208 ip-10-0-128-111 kubenswrapper[2583]: W0916 02:18:41.832227    2583 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 16 02:18:41.834208 ip-10-0-128-111 kubenswrapper[2583]: W0916 02:18:41.832230    2583 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 16 02:18:41.834208 ip-10-0-128-111 kubenswrapper[2583]: W0916 02:18:41.832233    2583 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 16 02:18:41.834208 ip-10-0-128-111 kubenswrapper[2583]: W0916 02:18:41.832235    2583 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 16 02:18:41.834208 ip-10-0-128-111 kubenswrapper[2583]: W0916 02:18:41.832238    2583 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 16 02:18:41.834208 ip-10-0-128-111 kubenswrapper[2583]: W0916 02:18:41.832240    2583 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 16 02:18:41.834208 ip-10-0-128-111 kubenswrapper[2583]: W0916 02:18:41.832243    2583 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 16 02:18:41.834208 ip-10-0-128-111 kubenswrapper[2583]: W0916 02:18:41.832245    2583 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 16 02:18:41.834208 ip-10-0-128-111 kubenswrapper[2583]: W0916 02:18:41.832248    2583 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 16 02:18:41.834208 ip-10-0-128-111 kubenswrapper[2583]: W0916 02:18:41.832251    2583 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 16 02:18:41.834208 ip-10-0-128-111 kubenswrapper[2583]: W0916 02:18:41.832253    2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 16 02:18:41.834208 ip-10-0-128-111 kubenswrapper[2583]: W0916 02:18:41.832255    2583 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 16 02:18:41.834208 ip-10-0-128-111 kubenswrapper[2583]: W0916 02:18:41.832258    2583 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 16 02:18:41.834208 ip-10-0-128-111 kubenswrapper[2583]: W0916 02:18:41.832261    2583 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 16 02:18:41.834208 ip-10-0-128-111 kubenswrapper[2583]: W0916 02:18:41.832980    2583 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 16 02:18:41.834208 ip-10-0-128-111 kubenswrapper[2583]: W0916 02:18:41.832987    2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 16 02:18:41.834208 ip-10-0-128-111 kubenswrapper[2583]: W0916 02:18:41.832990    2583 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 16 02:18:41.834208 ip-10-0-128-111 kubenswrapper[2583]: W0916 02:18:41.832994    2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 16 02:18:41.834208 ip-10-0-128-111 kubenswrapper[2583]: W0916 02:18:41.832997    2583 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 16 02:18:41.834705 ip-10-0-128-111 kubenswrapper[2583]: W0916 02:18:41.833000    2583 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 16 02:18:41.834705 ip-10-0-128-111 kubenswrapper[2583]: W0916 02:18:41.833002    2583 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 16 02:18:41.834705 ip-10-0-128-111 kubenswrapper[2583]: W0916 02:18:41.833005    2583 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 16 02:18:41.834705 ip-10-0-128-111 kubenswrapper[2583]: W0916 02:18:41.833008    2583 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 16 02:18:41.834705 ip-10-0-128-111 kubenswrapper[2583]: W0916 02:18:41.833011    2583 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 16 02:18:41.834705 ip-10-0-128-111 kubenswrapper[2583]: W0916 02:18:41.833013    2583 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 16 02:18:41.834705 ip-10-0-128-111 kubenswrapper[2583]: W0916 02:18:41.833017    2583 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 16 02:18:41.834705 ip-10-0-128-111 kubenswrapper[2583]: W0916 02:18:41.833020    2583 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 16 02:18:41.834705 ip-10-0-128-111 kubenswrapper[2583]: W0916 02:18:41.833022    2583 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 16 02:18:41.834705 ip-10-0-128-111 kubenswrapper[2583]: W0916 02:18:41.833025    2583 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 16 02:18:41.834705 ip-10-0-128-111 kubenswrapper[2583]: W0916 02:18:41.833027    2583 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 16 02:18:41.834705 ip-10-0-128-111 kubenswrapper[2583]: W0916 02:18:41.833030    2583 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 16 02:18:41.834705 ip-10-0-128-111 kubenswrapper[2583]: W0916 02:18:41.833034    2583 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 16 02:18:41.834705 ip-10-0-128-111 kubenswrapper[2583]: W0916 02:18:41.833037    2583 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 16 02:18:41.834705 ip-10-0-128-111 kubenswrapper[2583]: W0916 02:18:41.833039    2583 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 16 02:18:41.834705 ip-10-0-128-111 kubenswrapper[2583]: W0916 02:18:41.833042    2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 16 02:18:41.834705 ip-10-0-128-111 kubenswrapper[2583]: W0916 02:18:41.833045    2583 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 16 02:18:41.834705 ip-10-0-128-111 kubenswrapper[2583]: W0916 02:18:41.833047    2583 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 16 02:18:41.834705 ip-10-0-128-111 kubenswrapper[2583]: W0916 02:18:41.833050    2583 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 16 02:18:41.834705 ip-10-0-128-111 kubenswrapper[2583]: W0916 02:18:41.833053    2583 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 16 02:18:41.835234 ip-10-0-128-111 kubenswrapper[2583]: W0916 02:18:41.833055    2583 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 16 02:18:41.835234 ip-10-0-128-111 kubenswrapper[2583]: W0916 02:18:41.833058    2583 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 16 02:18:41.835234 ip-10-0-128-111 kubenswrapper[2583]: W0916 02:18:41.833060    2583 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 16 02:18:41.835234 ip-10-0-128-111 kubenswrapper[2583]: W0916 02:18:41.833063    2583 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 16 02:18:41.835234 ip-10-0-128-111 kubenswrapper[2583]: W0916 02:18:41.833065    2583 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 16 02:18:41.835234 ip-10-0-128-111 kubenswrapper[2583]: W0916 02:18:41.833070    2583 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 16 02:18:41.835234 ip-10-0-128-111 kubenswrapper[2583]: W0916 02:18:41.833074    2583 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 16 02:18:41.835234 ip-10-0-128-111 kubenswrapper[2583]: W0916 02:18:41.833077    2583 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 16 02:18:41.835234 ip-10-0-128-111 kubenswrapper[2583]: W0916 02:18:41.833081    2583 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 16 02:18:41.835234 ip-10-0-128-111 kubenswrapper[2583]: W0916 02:18:41.833084    2583 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 16 02:18:41.835234 ip-10-0-128-111 kubenswrapper[2583]: W0916 02:18:41.833088    2583 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 16 02:18:41.835234 ip-10-0-128-111 kubenswrapper[2583]: W0916 02:18:41.833091    2583 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 16 02:18:41.835234 ip-10-0-128-111 kubenswrapper[2583]: W0916 02:18:41.833095    2583 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 16 02:18:41.835234 ip-10-0-128-111 kubenswrapper[2583]: W0916 02:18:41.833098    2583 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 16 02:18:41.835234 ip-10-0-128-111 kubenswrapper[2583]: W0916 02:18:41.833100    2583 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 16 02:18:41.835234 ip-10-0-128-111 kubenswrapper[2583]: W0916 02:18:41.833103    2583 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 16 02:18:41.835234 ip-10-0-128-111 kubenswrapper[2583]: W0916 02:18:41.833105    2583 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 16 02:18:41.835234 ip-10-0-128-111 kubenswrapper[2583]: W0916 02:18:41.833108    2583 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 16 02:18:41.835234 ip-10-0-128-111 kubenswrapper[2583]: W0916 02:18:41.833111    2583 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 16 02:18:41.835718 ip-10-0-128-111 kubenswrapper[2583]: W0916 02:18:41.833113    2583 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 16 02:18:41.835718 ip-10-0-128-111 kubenswrapper[2583]: W0916 02:18:41.833116    2583 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 16 02:18:41.835718 ip-10-0-128-111 kubenswrapper[2583]: W0916 02:18:41.833118    2583 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 16 02:18:41.835718 ip-10-0-128-111 kubenswrapper[2583]: W0916 02:18:41.833121    2583 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 16 02:18:41.835718 ip-10-0-128-111 kubenswrapper[2583]: W0916 02:18:41.833123    2583 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 16 02:18:41.835718 ip-10-0-128-111 kubenswrapper[2583]: W0916 02:18:41.833126    2583 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 16 02:18:41.835718 ip-10-0-128-111 kubenswrapper[2583]: W0916 02:18:41.833129    2583 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 16 02:18:41.835718 ip-10-0-128-111 kubenswrapper[2583]: W0916 02:18:41.833132    2583 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 16 02:18:41.835718 ip-10-0-128-111 kubenswrapper[2583]: W0916 02:18:41.833136    2583 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 16 02:18:41.835718 ip-10-0-128-111 kubenswrapper[2583]: W0916 02:18:41.833139    2583 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 16 02:18:41.835718 ip-10-0-128-111 kubenswrapper[2583]: W0916 02:18:41.833142    2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 16 02:18:41.835718 ip-10-0-128-111 kubenswrapper[2583]: W0916 02:18:41.833144    2583 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 16 02:18:41.835718 ip-10-0-128-111 kubenswrapper[2583]: W0916 02:18:41.833147    2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 16 02:18:41.835718 ip-10-0-128-111 kubenswrapper[2583]: W0916 02:18:41.833150    2583 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 16 02:18:41.835718 ip-10-0-128-111 kubenswrapper[2583]: W0916 02:18:41.833152    2583 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 16 02:18:41.835718 ip-10-0-128-111 kubenswrapper[2583]: W0916 02:18:41.833155    2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 16 02:18:41.835718 ip-10-0-128-111 kubenswrapper[2583]: W0916 02:18:41.833157    2583 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 16 02:18:41.835718 ip-10-0-128-111 kubenswrapper[2583]: W0916 02:18:41.833160    2583 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 16 02:18:41.835718 ip-10-0-128-111 kubenswrapper[2583]: W0916 02:18:41.833163    2583 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 16 02:18:41.835718 ip-10-0-128-111 kubenswrapper[2583]: W0916 02:18:41.833165    2583 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 16 02:18:41.836211 ip-10-0-128-111 kubenswrapper[2583]: W0916 02:18:41.833168    2583 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 16 02:18:41.836211 ip-10-0-128-111 kubenswrapper[2583]: W0916 02:18:41.833171    2583 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 16 02:18:41.836211 ip-10-0-128-111 kubenswrapper[2583]: W0916 02:18:41.833174    2583 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 16 02:18:41.836211 ip-10-0-128-111 kubenswrapper[2583]: W0916 02:18:41.833176    2583 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 16 02:18:41.836211 ip-10-0-128-111 kubenswrapper[2583]: W0916 02:18:41.833179    2583 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 16 02:18:41.836211 ip-10-0-128-111 kubenswrapper[2583]: W0916 02:18:41.833182    2583 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 16 02:18:41.836211 ip-10-0-128-111 kubenswrapper[2583]: W0916 02:18:41.833184    2583 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 16 02:18:41.836211 ip-10-0-128-111 kubenswrapper[2583]: W0916 02:18:41.833187    2583 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 16 02:18:41.836211 ip-10-0-128-111 kubenswrapper[2583]: W0916 02:18:41.833189    2583 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 16 02:18:41.836211 ip-10-0-128-111 kubenswrapper[2583]: W0916 02:18:41.833192    2583 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 16 02:18:41.836211 ip-10-0-128-111 kubenswrapper[2583]: W0916 02:18:41.833196    2583 feature_gate.go:328] unrecognized feature gate: Example2
Sep 16 02:18:41.836211 ip-10-0-128-111 kubenswrapper[2583]: W0916 02:18:41.833198    2583 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 16 02:18:41.836211 ip-10-0-128-111 kubenswrapper[2583]: W0916 02:18:41.833201    2583 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 16 02:18:41.836211 ip-10-0-128-111 kubenswrapper[2583]: W0916 02:18:41.833203    2583 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 16 02:18:41.836211 ip-10-0-128-111 kubenswrapper[2583]: W0916 02:18:41.833206    2583 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 16 02:18:41.836211 ip-10-0-128-111 kubenswrapper[2583]: W0916 02:18:41.833208    2583 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 16 02:18:41.836211 ip-10-0-128-111 kubenswrapper[2583]: W0916 02:18:41.833211    2583 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 16 02:18:41.836211 ip-10-0-128-111 kubenswrapper[2583]: W0916 02:18:41.833213    2583 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 16 02:18:41.836211 ip-10-0-128-111 kubenswrapper[2583]: W0916 02:18:41.833216    2583 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 16 02:18:41.836211 ip-10-0-128-111 kubenswrapper[2583]: W0916 02:18:41.833219    2583 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 16 02:18:41.836725 ip-10-0-128-111 kubenswrapper[2583]: W0916 02:18:41.833222    2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 16 02:18:41.836725 ip-10-0-128-111 kubenswrapper[2583]: W0916 02:18:41.833224    2583 feature_gate.go:328] unrecognized feature gate: Example
Sep 16 02:18:41.836725 ip-10-0-128-111 kubenswrapper[2583]: W0916 02:18:41.833227    2583 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 16 02:18:41.836725 ip-10-0-128-111 kubenswrapper[2583]: W0916 02:18:41.833229    2583 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 16 02:18:41.836725 ip-10-0-128-111 kubenswrapper[2583]: W0916 02:18:41.833232    2583 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 16 02:18:41.836725 ip-10-0-128-111 kubenswrapper[2583]: W0916 02:18:41.833235    2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 16 02:18:41.836725 ip-10-0-128-111 kubenswrapper[2583]: W0916 02:18:41.833237    2583 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 16 02:18:41.836725 ip-10-0-128-111 kubenswrapper[2583]: W0916 02:18:41.833240    2583 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 16 02:18:41.836725 ip-10-0-128-111 kubenswrapper[2583]: W0916 02:18:41.833242    2583 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 16 02:18:41.836725 ip-10-0-128-111 kubenswrapper[2583]: W0916 02:18:41.833245    2583 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 16 02:18:41.836725 ip-10-0-128-111 kubenswrapper[2583]: W0916 02:18:41.833248    2583 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 16 02:18:41.836725 ip-10-0-128-111 kubenswrapper[2583]: W0916 02:18:41.833250    2583 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 16 02:18:41.836725 ip-10-0-128-111 kubenswrapper[2583]: W0916 02:18:41.833253    2583 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 16 02:18:41.836725 ip-10-0-128-111 kubenswrapper[2583]: W0916 02:18:41.833256    2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 16 02:18:41.836725 ip-10-0-128-111 kubenswrapper[2583]: W0916 02:18:41.833259    2583 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 16 02:18:41.836725 ip-10-0-128-111 kubenswrapper[2583]: W0916 02:18:41.833704    2583 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 16 02:18:41.836725 ip-10-0-128-111 kubenswrapper[2583]: W0916 02:18:41.833712    2583 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 16 02:18:41.836725 ip-10-0-128-111 kubenswrapper[2583]: W0916 02:18:41.833716    2583 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 16 02:18:41.836725 ip-10-0-128-111 kubenswrapper[2583]: W0916 02:18:41.833719    2583 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 16 02:18:41.837191 ip-10-0-128-111 kubenswrapper[2583]: W0916 02:18:41.833722    2583 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 16 02:18:41.837191 ip-10-0-128-111 kubenswrapper[2583]: W0916 02:18:41.833725    2583 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 16 02:18:41.837191 ip-10-0-128-111 kubenswrapper[2583]: W0916 02:18:41.833728    2583 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 16 02:18:41.837191 ip-10-0-128-111 kubenswrapper[2583]: W0916 02:18:41.833731    2583 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 16 02:18:41.837191 ip-10-0-128-111 kubenswrapper[2583]: W0916 02:18:41.833734    2583 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 16 02:18:41.837191 ip-10-0-128-111 kubenswrapper[2583]: W0916 02:18:41.833737    2583 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 16 02:18:41.837191 ip-10-0-128-111 kubenswrapper[2583]: W0916 02:18:41.833740    2583 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 16 02:18:41.837191 ip-10-0-128-111 kubenswrapper[2583]: W0916 02:18:41.833743    2583 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 16 02:18:41.837191 ip-10-0-128-111 kubenswrapper[2583]: W0916 02:18:41.833747    2583 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 16 02:18:41.837191 ip-10-0-128-111 kubenswrapper[2583]: W0916 02:18:41.833749    2583 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 16 02:18:41.837191 ip-10-0-128-111 kubenswrapper[2583]: W0916 02:18:41.833752    2583 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 16 02:18:41.837191 ip-10-0-128-111 kubenswrapper[2583]: W0916 02:18:41.833755    2583 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 16 02:18:41.837191 ip-10-0-128-111 kubenswrapper[2583]: W0916 02:18:41.833758    2583 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 16 02:18:41.837191 ip-10-0-128-111 kubenswrapper[2583]: W0916 02:18:41.833761    2583 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 16 02:18:41.837191 ip-10-0-128-111 kubenswrapper[2583]: W0916 02:18:41.833764    2583 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 16 02:18:41.837191 ip-10-0-128-111 kubenswrapper[2583]: W0916 02:18:41.833767    2583 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 16 02:18:41.837191 ip-10-0-128-111 kubenswrapper[2583]: W0916 02:18:41.833770    2583 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 16 02:18:41.837191 ip-10-0-128-111 kubenswrapper[2583]: W0916 02:18:41.833773    2583 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 16 02:18:41.837191 ip-10-0-128-111 kubenswrapper[2583]: W0916 02:18:41.833775    2583 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 16 02:18:41.837191 ip-10-0-128-111 kubenswrapper[2583]: W0916 02:18:41.833779    2583 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 16 02:18:41.837704 ip-10-0-128-111 kubenswrapper[2583]: W0916 02:18:41.833781    2583 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 16 02:18:41.837704 ip-10-0-128-111 kubenswrapper[2583]: W0916 02:18:41.833784    2583 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 16 02:18:41.837704 ip-10-0-128-111 kubenswrapper[2583]: W0916 02:18:41.833786    2583 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 16 02:18:41.837704 ip-10-0-128-111 kubenswrapper[2583]: W0916 02:18:41.833789    2583 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 16 02:18:41.837704 ip-10-0-128-111 kubenswrapper[2583]: W0916 02:18:41.833791    2583 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 16 02:18:41.837704 ip-10-0-128-111 kubenswrapper[2583]: W0916 02:18:41.833795    2583 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 16 02:18:41.837704 ip-10-0-128-111 kubenswrapper[2583]: W0916 02:18:41.833798    2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 16 02:18:41.837704 ip-10-0-128-111 kubenswrapper[2583]: W0916 02:18:41.833802    2583 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 16 02:18:41.837704 ip-10-0-128-111 kubenswrapper[2583]: W0916 02:18:41.833807    2583 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 16 02:18:41.837704 ip-10-0-128-111 kubenswrapper[2583]: W0916 02:18:41.833810    2583 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 16 02:18:41.837704 ip-10-0-128-111 kubenswrapper[2583]: W0916 02:18:41.833812    2583 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 16 02:18:41.837704 ip-10-0-128-111 kubenswrapper[2583]: W0916 02:18:41.833815    2583 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 16 02:18:41.837704 ip-10-0-128-111 kubenswrapper[2583]: W0916 02:18:41.833818    2583 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 16 02:18:41.837704 ip-10-0-128-111 kubenswrapper[2583]: W0916 02:18:41.833820    2583 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 16 02:18:41.837704 ip-10-0-128-111 kubenswrapper[2583]: W0916 02:18:41.833823    2583 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 16 02:18:41.837704 ip-10-0-128-111 kubenswrapper[2583]: W0916 02:18:41.833827    2583 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 16 02:18:41.837704 ip-10-0-128-111 kubenswrapper[2583]: W0916 02:18:41.833831    2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 16 02:18:41.837704 ip-10-0-128-111 kubenswrapper[2583]: W0916 02:18:41.833835    2583 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 16 02:18:41.837704 ip-10-0-128-111 kubenswrapper[2583]: W0916 02:18:41.833839    2583 feature_gate.go:328] unrecognized feature gate: Example
Sep 16 02:18:41.838167 ip-10-0-128-111 kubenswrapper[2583]: W0916 02:18:41.833842    2583 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 16 02:18:41.838167 ip-10-0-128-111 kubenswrapper[2583]: W0916 02:18:41.833844    2583 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 16 02:18:41.838167 ip-10-0-128-111 kubenswrapper[2583]: W0916 02:18:41.833847    2583 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 16 02:18:41.838167 ip-10-0-128-111 kubenswrapper[2583]: W0916 02:18:41.833850    2583 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 16 02:18:41.838167 ip-10-0-128-111 kubenswrapper[2583]: W0916 02:18:41.833852    2583 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 16 02:18:41.838167 ip-10-0-128-111 kubenswrapper[2583]: W0916 02:18:41.833855    2583 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 16 02:18:41.838167 ip-10-0-128-111 kubenswrapper[2583]: W0916 02:18:41.833858    2583 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 16 02:18:41.838167 ip-10-0-128-111 kubenswrapper[2583]: W0916 02:18:41.833864    2583 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 16 02:18:41.838167 ip-10-0-128-111 kubenswrapper[2583]: W0916 02:18:41.833867    2583 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 16 02:18:41.838167 ip-10-0-128-111 kubenswrapper[2583]: W0916 02:18:41.833869    2583 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 16 02:18:41.838167 ip-10-0-128-111 kubenswrapper[2583]: W0916 02:18:41.833872    2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 16 02:18:41.838167 ip-10-0-128-111 kubenswrapper[2583]: W0916 02:18:41.833874    2583 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 16 02:18:41.838167 ip-10-0-128-111 kubenswrapper[2583]: W0916 02:18:41.833877    2583 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 16 02:18:41.838167 ip-10-0-128-111 kubenswrapper[2583]: W0916 02:18:41.833880    2583 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 16 02:18:41.838167 ip-10-0-128-111 kubenswrapper[2583]: W0916 02:18:41.833882    2583 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 16 02:18:41.838167 ip-10-0-128-111 kubenswrapper[2583]: W0916 02:18:41.833885    2583 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 16 02:18:41.838167 ip-10-0-128-111 kubenswrapper[2583]: W0916 02:18:41.833888    2583 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 16 02:18:41.838167 ip-10-0-128-111 kubenswrapper[2583]: W0916 02:18:41.833890    2583 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 16 02:18:41.838167 ip-10-0-128-111 kubenswrapper[2583]: W0916 02:18:41.833893    2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 16 02:18:41.838167 ip-10-0-128-111 kubenswrapper[2583]: W0916 02:18:41.833897    2583 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 16 02:18:41.838683 ip-10-0-128-111 kubenswrapper[2583]: W0916 02:18:41.833899    2583 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 16 02:18:41.838683 ip-10-0-128-111 kubenswrapper[2583]: W0916 02:18:41.833901    2583 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 16 02:18:41.838683 ip-10-0-128-111 kubenswrapper[2583]: W0916 02:18:41.833904    2583 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 16 02:18:41.838683 ip-10-0-128-111 kubenswrapper[2583]: W0916 02:18:41.833907    2583 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 16 02:18:41.838683 ip-10-0-128-111 kubenswrapper[2583]: W0916 02:18:41.833910    2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 16 02:18:41.838683 ip-10-0-128-111 kubenswrapper[2583]: W0916 02:18:41.833913    2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 16 02:18:41.838683 ip-10-0-128-111 kubenswrapper[2583]: W0916 02:18:41.833916    2583 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 16 02:18:41.838683 ip-10-0-128-111 kubenswrapper[2583]: W0916 02:18:41.833918    2583 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 16 02:18:41.838683 ip-10-0-128-111 kubenswrapper[2583]: W0916 02:18:41.833921    2583 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 16 02:18:41.838683 ip-10-0-128-111 kubenswrapper[2583]: W0916 02:18:41.833923    2583 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 16 02:18:41.838683 ip-10-0-128-111 kubenswrapper[2583]: W0916 02:18:41.833926    2583 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 16 02:18:41.838683 ip-10-0-128-111 kubenswrapper[2583]: W0916 02:18:41.833929    2583 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 16 02:18:41.838683 ip-10-0-128-111 kubenswrapper[2583]: W0916 02:18:41.833931    2583 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 16 02:18:41.838683 ip-10-0-128-111 kubenswrapper[2583]: W0916 02:18:41.833933    2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 16 02:18:41.838683 ip-10-0-128-111 kubenswrapper[2583]: W0916 02:18:41.833936    2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 16 02:18:41.838683 ip-10-0-128-111 kubenswrapper[2583]: W0916 02:18:41.833938    2583 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 16 02:18:41.838683 ip-10-0-128-111 kubenswrapper[2583]: W0916 02:18:41.833941    2583 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 16 02:18:41.838683 ip-10-0-128-111 kubenswrapper[2583]: W0916 02:18:41.833944    2583 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 16 02:18:41.838683 ip-10-0-128-111 kubenswrapper[2583]: W0916 02:18:41.833946    2583 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 16 02:18:41.838683 ip-10-0-128-111 kubenswrapper[2583]: W0916 02:18:41.833952    2583 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 16 02:18:41.839177 ip-10-0-128-111 kubenswrapper[2583]: W0916 02:18:41.833955    2583 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 16 02:18:41.839177 ip-10-0-128-111 kubenswrapper[2583]: W0916 02:18:41.833957    2583 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 16 02:18:41.839177 ip-10-0-128-111 kubenswrapper[2583]: W0916 02:18:41.833960    2583 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 16 02:18:41.839177 ip-10-0-128-111 kubenswrapper[2583]: W0916 02:18:41.833962    2583 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 16 02:18:41.839177 ip-10-0-128-111 kubenswrapper[2583]: W0916 02:18:41.833965    2583 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 16 02:18:41.839177 ip-10-0-128-111 kubenswrapper[2583]: W0916 02:18:41.833967    2583 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 16 02:18:41.839177 ip-10-0-128-111 kubenswrapper[2583]: W0916 02:18:41.833971    2583 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 16 02:18:41.839177 ip-10-0-128-111 kubenswrapper[2583]: W0916 02:18:41.833974    2583 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 16 02:18:41.839177 ip-10-0-128-111 kubenswrapper[2583]: W0916 02:18:41.833977    2583 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 16 02:18:41.839177 ip-10-0-128-111 kubenswrapper[2583]: W0916 02:18:41.833980    2583 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 16 02:18:41.839177 ip-10-0-128-111 kubenswrapper[2583]: W0916 02:18:41.833983    2583 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 16 02:18:41.839177 ip-10-0-128-111 kubenswrapper[2583]: W0916 02:18:41.833987    2583 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 16 02:18:41.839177 ip-10-0-128-111 kubenswrapper[2583]: W0916 02:18:41.833989    2583 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 16 02:18:41.839177 ip-10-0-128-111 kubenswrapper[2583]: W0916 02:18:41.833992    2583 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 16 02:18:41.839177 ip-10-0-128-111 kubenswrapper[2583]: W0916 02:18:41.833994    2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 16 02:18:41.839177 ip-10-0-128-111 kubenswrapper[2583]: W0916 02:18:41.833997    2583 feature_gate.go:328] unrecognized feature gate: Example2
Sep 16 02:18:41.839177 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:18:41.834068    2583 flags.go:64] FLAG: --address="0.0.0.0"
Sep 16 02:18:41.839177 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:18:41.834106    2583 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Sep 16 02:18:41.839177 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:18:41.834115    2583 flags.go:64] FLAG: --anonymous-auth="true"
Sep 16 02:18:41.839177 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:18:41.834119    2583 flags.go:64] FLAG: --application-metrics-count-limit="100"
Sep 16 02:18:41.839177 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:18:41.834123    2583 flags.go:64] FLAG: --authentication-token-webhook="false"
Sep 16 02:18:41.839694 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:18:41.834127    2583 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Sep 16 02:18:41.839694 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:18:41.834131    2583 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Sep 16 02:18:41.839694 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:18:41.834136    2583 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Sep 16 02:18:41.839694 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:18:41.834139    2583 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Sep 16 02:18:41.839694 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:18:41.834142    2583 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Sep 16 02:18:41.839694 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:18:41.834145    2583 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Sep 16 02:18:41.839694 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:18:41.834149    2583 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Sep 16 02:18:41.839694 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:18:41.834152    2583 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Sep 16 02:18:41.839694 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:18:41.834155    2583 flags.go:64] FLAG: --cgroup-root=""
Sep 16 02:18:41.839694 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:18:41.834158    2583 flags.go:64] FLAG: --cgroups-per-qos="true"
Sep 16 02:18:41.839694 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:18:41.834163    2583 flags.go:64] FLAG: --client-ca-file=""
Sep 16 02:18:41.839694 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:18:41.834166    2583 flags.go:64] FLAG: --cloud-provider="external"
Sep 16 02:18:41.839694 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:18:41.834169    2583 flags.go:64] FLAG: --cluster-dns="[]"
Sep 16 02:18:41.839694 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:18:41.834173    2583 flags.go:64] FLAG: --cluster-domain=""
Sep 16 02:18:41.839694 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:18:41.834176    2583 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Sep 16 02:18:41.839694 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:18:41.834179    2583 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Sep 16 02:18:41.839694 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:18:41.834182    2583 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Sep 16 02:18:41.839694 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:18:41.834186    2583 flags.go:64] FLAG: --container-log-max-files="5"
Sep 16 02:18:41.839694 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:18:41.834190    2583 flags.go:64] FLAG: --container-log-max-size="10Mi"
Sep 16 02:18:41.839694 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:18:41.834193    2583 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Sep 16 02:18:41.839694 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:18:41.834196    2583 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Sep 16 02:18:41.839694 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:18:41.834199    2583 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Sep 16 02:18:41.839694 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:18:41.834203    2583 flags.go:64] FLAG: --contention-profiling="false"
Sep 16 02:18:41.839694 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:18:41.834206    2583 flags.go:64] FLAG: --cpu-cfs-quota="true"
Sep 16 02:18:41.840299 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:18:41.834209    2583 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Sep 16 02:18:41.840299 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:18:41.834213    2583 flags.go:64] FLAG: --cpu-manager-policy="none"
Sep 16 02:18:41.840299 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:18:41.834216    2583 flags.go:64] FLAG: --cpu-manager-policy-options=""
Sep 16 02:18:41.840299 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:18:41.834220    2583 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Sep 16 02:18:41.840299 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:18:41.834223    2583 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Sep 16 02:18:41.840299 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:18:41.834226    2583 flags.go:64] FLAG: --enable-debugging-handlers="true"
Sep 16 02:18:41.840299 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:18:41.834229    2583 flags.go:64] FLAG: --enable-load-reader="false"
Sep 16 02:18:41.840299 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:18:41.834232    2583 flags.go:64] FLAG: --enable-server="true"
Sep 16 02:18:41.840299 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:18:41.834235    2583 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Sep 16 02:18:41.840299 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:18:41.834239    2583 flags.go:64] FLAG: --event-burst="100"
Sep 16 02:18:41.840299 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:18:41.834243    2583 flags.go:64] FLAG: --event-qps="50"
Sep 16 02:18:41.840299 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:18:41.834246    2583 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Sep 16 02:18:41.840299 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:18:41.834248    2583 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Sep 16 02:18:41.840299 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:18:41.834252    2583 flags.go:64] FLAG: --eviction-hard=""
Sep 16 02:18:41.840299 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:18:41.834256    2583 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Sep 16 02:18:41.840299 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:18:41.834259    2583 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Sep 16 02:18:41.840299 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:18:41.834261    2583 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Sep 16 02:18:41.840299 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:18:41.834264    2583 flags.go:64] FLAG: --eviction-soft=""
Sep 16 02:18:41.840299 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:18:41.834269    2583 flags.go:64] FLAG: --eviction-soft-grace-period=""
Sep 16 02:18:41.840299 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:18:41.834272    2583 flags.go:64] FLAG: --exit-on-lock-contention="false"
Sep 16 02:18:41.840299 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:18:41.834276    2583 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Sep 16 02:18:41.840299 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:18:41.834278    2583 flags.go:64] FLAG: --experimental-mounter-path=""
Sep 16 02:18:41.840299 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:18:41.834281    2583 flags.go:64] FLAG: --fail-cgroupv1="false"
Sep 16 02:18:41.840299 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:18:41.834284    2583 flags.go:64] FLAG: --fail-swap-on="true"
Sep 16 02:18:41.840299 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:18:41.834287    2583 flags.go:64] FLAG: --feature-gates=""
Sep 16 02:18:41.840934 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:18:41.834293    2583 flags.go:64] FLAG: --file-check-frequency="20s"
Sep 16 02:18:41.840934 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:18:41.834296    2583 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Sep 16 02:18:41.840934 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:18:41.834299    2583 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Sep 16 02:18:41.840934 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:18:41.834302    2583 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Sep 16 02:18:41.840934 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:18:41.834305    2583 flags.go:64] FLAG: --healthz-port="10248"
Sep 16 02:18:41.840934 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:18:41.834309    2583 flags.go:64] FLAG: --help="false"
Sep 16 02:18:41.840934 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:18:41.834312    2583 flags.go:64] FLAG: --hostname-override="ip-10-0-128-111.ec2.internal"
Sep 16 02:18:41.840934 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:18:41.834316    2583 flags.go:64] FLAG: --housekeeping-interval="10s"
Sep 16 02:18:41.840934 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:18:41.834319    2583 flags.go:64] FLAG: --http-check-frequency="20s"
Sep 16 02:18:41.840934 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:18:41.834321    2583 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Sep 16 02:18:41.840934 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:18:41.834325    2583 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Sep 16 02:18:41.840934 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:18:41.834328    2583 flags.go:64] FLAG: --image-gc-high-threshold="85"
Sep 16 02:18:41.840934 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:18:41.834331    2583 flags.go:64] FLAG: --image-gc-low-threshold="80"
Sep 16 02:18:41.840934 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:18:41.834334    2583 flags.go:64] FLAG: --image-service-endpoint=""
Sep 16 02:18:41.840934 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:18:41.834336    2583 flags.go:64] FLAG: --kernel-memcg-notification="false"
Sep 16 02:18:41.840934 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:18:41.834339    2583 flags.go:64] FLAG: --kube-api-burst="100"
Sep 16 02:18:41.840934 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:18:41.834342    2583 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Sep 16 02:18:41.840934 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:18:41.834346    2583 flags.go:64] FLAG: --kube-api-qps="50"
Sep 16 02:18:41.840934 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:18:41.834348    2583 flags.go:64] FLAG: --kube-reserved=""
Sep 16 02:18:41.840934 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:18:41.834351    2583 flags.go:64] FLAG: --kube-reserved-cgroup=""
Sep 16 02:18:41.840934 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:18:41.834354    2583 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Sep 16 02:18:41.840934 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:18:41.834357    2583 flags.go:64] FLAG: --kubelet-cgroups=""
Sep 16 02:18:41.840934 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:18:41.834360    2583 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Sep 16 02:18:41.840934 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:18:41.834363    2583 flags.go:64] FLAG: --lock-file=""
Sep 16 02:18:41.841525 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:18:41.834366    2583 flags.go:64] FLAG: --log-cadvisor-usage="false"
Sep 16 02:18:41.841525 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:18:41.834370    2583 flags.go:64] FLAG: --log-flush-frequency="5s"
Sep 16 02:18:41.841525 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:18:41.834373    2583 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Sep 16 02:18:41.841525 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:18:41.834379    2583 flags.go:64] FLAG: --log-json-split-stream="false"
Sep 16 02:18:41.841525 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:18:41.834382    2583 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Sep 16 02:18:41.841525 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:18:41.834385    2583 flags.go:64] FLAG: --log-text-split-stream="false"
Sep 16 02:18:41.841525 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:18:41.834387    2583 flags.go:64] FLAG: --logging-format="text"
Sep 16 02:18:41.841525 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:18:41.834390    2583 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Sep 16 02:18:41.841525 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:18:41.834394    2583 flags.go:64] FLAG: --make-iptables-util-chains="true"
Sep 16 02:18:41.841525 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:18:41.834396    2583 flags.go:64] FLAG: --manifest-url=""
Sep 16 02:18:41.841525 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:18:41.834399    2583 flags.go:64] FLAG: --manifest-url-header=""
Sep 16 02:18:41.841525 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:18:41.834403    2583 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Sep 16 02:18:41.841525 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:18:41.834407    2583 flags.go:64] FLAG: --max-open-files="1000000"
Sep 16 02:18:41.841525 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:18:41.834412    2583 flags.go:64] FLAG: --max-pods="110"
Sep 16 02:18:41.841525 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:18:41.834415    2583 flags.go:64] FLAG: --maximum-dead-containers="-1"
Sep 16 02:18:41.841525 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:18:41.834417    2583 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Sep 16 02:18:41.841525 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:18:41.834420    2583 flags.go:64] FLAG: --memory-manager-policy="None"
Sep 16 02:18:41.841525 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:18:41.834423    2583 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Sep 16 02:18:41.841525 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:18:41.834426    2583 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Sep 16 02:18:41.841525 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:18:41.834429    2583 flags.go:64] FLAG: --node-ip="0.0.0.0"
Sep 16 02:18:41.841525 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:18:41.834432    2583 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Sep 16 02:18:41.841525 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:18:41.834439    2583 flags.go:64] FLAG: --node-status-max-images="50"
Sep 16 02:18:41.841525 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:18:41.834442    2583 flags.go:64] FLAG: --node-status-update-frequency="10s"
Sep 16 02:18:41.841525 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:18:41.834445    2583 flags.go:64] FLAG: --oom-score-adj="-999"
Sep 16 02:18:41.841525 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:18:41.834448    2583 flags.go:64] FLAG: --pod-cidr=""
Sep 16 02:18:41.842198 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:18:41.834451    2583 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:eee392dbe3ac2ed4a38bca29ae3bd486f5b7780406c62c600154cd1906871b10"
Sep 16 02:18:41.842198 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:18:41.834457    2583 flags.go:64] FLAG: --pod-manifest-path=""
Sep 16 02:18:41.842198 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:18:41.834460    2583 flags.go:64] FLAG: --pod-max-pids="-1"
Sep 16 02:18:41.842198 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:18:41.834463    2583 flags.go:64] FLAG: --pods-per-core="0"
Sep 16 02:18:41.842198 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:18:41.834466    2583 flags.go:64] FLAG: --port="10250"
Sep 16 02:18:41.842198 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:18:41.834469    2583 flags.go:64] FLAG: --protect-kernel-defaults="false"
Sep 16 02:18:41.842198 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:18:41.834472    2583 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-01b996b7e581d50ac"
Sep 16 02:18:41.842198 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:18:41.834475    2583 flags.go:64] FLAG: --qos-reserved=""
Sep 16 02:18:41.842198 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:18:41.834478    2583 flags.go:64] FLAG: --read-only-port="10255"
Sep 16 02:18:41.842198 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:18:41.834482    2583 flags.go:64] FLAG: --register-node="true"
Sep 16 02:18:41.842198 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:18:41.834486    2583 flags.go:64] FLAG: --register-with-taints=""
Sep 16 02:18:41.842198 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:18:41.834489    2583 flags.go:64] FLAG: --registry-burst="10"
Sep 16 02:18:41.842198 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:18:41.834493    2583 flags.go:64] FLAG: --registry-qps="5"
Sep 16 02:18:41.842198 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:18:41.834496    2583 flags.go:64] FLAG: --reserved-cpus=""
Sep 16 02:18:41.842198 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:18:41.834499    2583 flags.go:64] FLAG: --reserved-memory=""
Sep 16 02:18:41.842198 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:18:41.834503    2583 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Sep 16 02:18:41.842198 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:18:41.834506    2583 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Sep 16 02:18:41.842198 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:18:41.834509    2583 flags.go:64] FLAG: --rotate-certificates="false"
Sep 16 02:18:41.842198 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:18:41.834512    2583 flags.go:64] FLAG: --rotate-server-certificates="false"
Sep 16 02:18:41.842198 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:18:41.834515    2583 flags.go:64] FLAG: --runonce="false"
Sep 16 02:18:41.842198 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:18:41.834520    2583 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Sep 16 02:18:41.842198 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:18:41.834523    2583 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Sep 16 02:18:41.842198 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:18:41.834527    2583 flags.go:64] FLAG: --seccomp-default="false"
Sep 16 02:18:41.842198 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:18:41.834530    2583 flags.go:64] FLAG: --serialize-image-pulls="true"
Sep 16 02:18:41.842198 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:18:41.834533    2583 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Sep 16 02:18:41.843370 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:18:41.834536    2583 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Sep 16 02:18:41.843370 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:18:41.834539    2583 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Sep 16 02:18:41.843370 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:18:41.834542    2583 flags.go:64] FLAG: --storage-driver-password="root"
Sep 16 02:18:41.843370 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:18:41.834545    2583 flags.go:64] FLAG: --storage-driver-secure="false"
Sep 16 02:18:41.843370 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:18:41.834564    2583 flags.go:64] FLAG: --storage-driver-table="stats"
Sep 16 02:18:41.843370 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:18:41.834568    2583 flags.go:64] FLAG: --storage-driver-user="root"
Sep 16 02:18:41.843370 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:18:41.834571    2583 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Sep 16 02:18:41.843370 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:18:41.834574    2583 flags.go:64] FLAG: --sync-frequency="1m0s"
Sep 16 02:18:41.843370 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:18:41.834577    2583 flags.go:64] FLAG: --system-cgroups=""
Sep 16 02:18:41.843370 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:18:41.834580    2583 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Sep 16 02:18:41.843370 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:18:41.834588    2583 flags.go:64] FLAG: --system-reserved-cgroup=""
Sep 16 02:18:41.843370 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:18:41.834591    2583 flags.go:64] FLAG: --tls-cert-file=""
Sep 16 02:18:41.843370 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:18:41.834593    2583 flags.go:64] FLAG: --tls-cipher-suites="[]"
Sep 16 02:18:41.843370 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:18:41.834597    2583 flags.go:64] FLAG: --tls-min-version=""
Sep 16 02:18:41.843370 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:18:41.834600    2583 flags.go:64] FLAG: --tls-private-key-file=""
Sep 16 02:18:41.843370 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:18:41.834603    2583 flags.go:64] FLAG: --topology-manager-policy="none"
Sep 16 02:18:41.843370 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:18:41.834608    2583 flags.go:64] FLAG: --topology-manager-policy-options=""
Sep 16 02:18:41.843370 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:18:41.834610    2583 flags.go:64] FLAG: --topology-manager-scope="container"
Sep 16 02:18:41.843370 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:18:41.834613    2583 flags.go:64] FLAG: --v="2"
Sep 16 02:18:41.843370 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:18:41.834618    2583 flags.go:64] FLAG: --version="false"
Sep 16 02:18:41.843370 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:18:41.834622    2583 flags.go:64] FLAG: --vmodule=""
Sep 16 02:18:41.843370 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:18:41.834625    2583 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Sep 16 02:18:41.843370 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:18:41.834629    2583 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Sep 16 02:18:41.843370 ip-10-0-128-111 kubenswrapper[2583]: W0916 02:18:41.834739    2583 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 16 02:18:41.844432 ip-10-0-128-111 kubenswrapper[2583]: W0916 02:18:41.834743    2583 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 16 02:18:41.844432 ip-10-0-128-111 kubenswrapper[2583]: W0916 02:18:41.834746    2583 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 16 02:18:41.844432 ip-10-0-128-111 kubenswrapper[2583]: W0916 02:18:41.834748    2583 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 16 02:18:41.844432 ip-10-0-128-111 kubenswrapper[2583]: W0916 02:18:41.834752    2583 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 16 02:18:41.844432 ip-10-0-128-111 kubenswrapper[2583]: W0916 02:18:41.834755    2583 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 16 02:18:41.844432 ip-10-0-128-111 kubenswrapper[2583]: W0916 02:18:41.834757    2583 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 16 02:18:41.844432 ip-10-0-128-111 kubenswrapper[2583]: W0916 02:18:41.834760    2583 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 16 02:18:41.844432 ip-10-0-128-111 kubenswrapper[2583]: W0916 02:18:41.834763    2583 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 16 02:18:41.844432 ip-10-0-128-111 kubenswrapper[2583]: W0916 02:18:41.834765    2583 feature_gate.go:328] unrecognized feature gate: Example2
Sep 16 02:18:41.844432 ip-10-0-128-111 kubenswrapper[2583]: W0916 02:18:41.834768    2583 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 16 02:18:41.844432 ip-10-0-128-111 kubenswrapper[2583]: W0916 02:18:41.834770    2583 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 16 02:18:41.844432 ip-10-0-128-111 kubenswrapper[2583]: W0916 02:18:41.834773    2583 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 16 02:18:41.844432 ip-10-0-128-111 kubenswrapper[2583]: W0916 02:18:41.834775    2583 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 16 02:18:41.844432 ip-10-0-128-111 kubenswrapper[2583]: W0916 02:18:41.834778    2583 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 16 02:18:41.844432 ip-10-0-128-111 kubenswrapper[2583]: W0916 02:18:41.834781    2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 16 02:18:41.844432 ip-10-0-128-111 kubenswrapper[2583]: W0916 02:18:41.834784    2583 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 16 02:18:41.844432 ip-10-0-128-111 kubenswrapper[2583]: W0916 02:18:41.834787    2583 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 16 02:18:41.844432 ip-10-0-128-111 kubenswrapper[2583]: W0916 02:18:41.834790    2583 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 16 02:18:41.844432 ip-10-0-128-111 kubenswrapper[2583]: W0916 02:18:41.834794    2583 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 16 02:18:41.844432 ip-10-0-128-111 kubenswrapper[2583]: W0916 02:18:41.834797    2583 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 16 02:18:41.845160 ip-10-0-128-111 kubenswrapper[2583]: W0916 02:18:41.834800    2583 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 16 02:18:41.845160 ip-10-0-128-111 kubenswrapper[2583]: W0916 02:18:41.834802    2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 16 02:18:41.845160 ip-10-0-128-111 kubenswrapper[2583]: W0916 02:18:41.834805    2583 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 16 02:18:41.845160 ip-10-0-128-111 kubenswrapper[2583]: W0916 02:18:41.834807    2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 16 02:18:41.845160 ip-10-0-128-111 kubenswrapper[2583]: W0916 02:18:41.834811    2583 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 16 02:18:41.845160 ip-10-0-128-111 kubenswrapper[2583]: W0916 02:18:41.834814    2583 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 16 02:18:41.845160 ip-10-0-128-111 kubenswrapper[2583]: W0916 02:18:41.834816    2583 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 16 02:18:41.845160 ip-10-0-128-111 kubenswrapper[2583]: W0916 02:18:41.834819    2583 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 16 02:18:41.845160 ip-10-0-128-111 kubenswrapper[2583]: W0916 02:18:41.834822    2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 16 02:18:41.845160 ip-10-0-128-111 kubenswrapper[2583]: W0916 02:18:41.834825    2583 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 16 02:18:41.845160 ip-10-0-128-111 kubenswrapper[2583]: W0916 02:18:41.834827    2583 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 16 02:18:41.845160 ip-10-0-128-111 kubenswrapper[2583]: W0916 02:18:41.834830    2583 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 16 02:18:41.845160 ip-10-0-128-111 kubenswrapper[2583]: W0916 02:18:41.834834    2583 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 16 02:18:41.845160 ip-10-0-128-111 kubenswrapper[2583]: W0916 02:18:41.834838    2583 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 16 02:18:41.845160 ip-10-0-128-111 kubenswrapper[2583]: W0916 02:18:41.834842    2583 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 16 02:18:41.845160 ip-10-0-128-111 kubenswrapper[2583]: W0916 02:18:41.834846    2583 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 16 02:18:41.845160 ip-10-0-128-111 kubenswrapper[2583]: W0916 02:18:41.834848    2583 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 16 02:18:41.845160 ip-10-0-128-111 kubenswrapper[2583]: W0916 02:18:41.834851    2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 16 02:18:41.845160 ip-10-0-128-111 kubenswrapper[2583]: W0916 02:18:41.834854    2583 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 16 02:18:41.845160 ip-10-0-128-111 kubenswrapper[2583]: W0916 02:18:41.834856    2583 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 16 02:18:41.845807 ip-10-0-128-111 kubenswrapper[2583]: W0916 02:18:41.834859    2583 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 16 02:18:41.845807 ip-10-0-128-111 kubenswrapper[2583]: W0916 02:18:41.834862    2583 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 16 02:18:41.845807 ip-10-0-128-111 kubenswrapper[2583]: W0916 02:18:41.834864    2583 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 16 02:18:41.845807 ip-10-0-128-111 kubenswrapper[2583]: W0916 02:18:41.834867    2583 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 16 02:18:41.845807 ip-10-0-128-111 kubenswrapper[2583]: W0916 02:18:41.834870    2583 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 16 02:18:41.845807 ip-10-0-128-111 kubenswrapper[2583]: W0916 02:18:41.834872    2583 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 16 02:18:41.845807 ip-10-0-128-111 kubenswrapper[2583]: W0916 02:18:41.834875    2583 feature_gate.go:328] unrecognized feature gate: Example
Sep 16 02:18:41.845807 ip-10-0-128-111 kubenswrapper[2583]: W0916 02:18:41.834878    2583 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 16 02:18:41.845807 ip-10-0-128-111 kubenswrapper[2583]: W0916 02:18:41.834880    2583 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 16 02:18:41.845807 ip-10-0-128-111 kubenswrapper[2583]: W0916 02:18:41.834883    2583 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 16 02:18:41.845807 ip-10-0-128-111 kubenswrapper[2583]: W0916 02:18:41.834886    2583 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 16 02:18:41.845807 ip-10-0-128-111 kubenswrapper[2583]: W0916 02:18:41.834888    2583 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 16 02:18:41.845807 ip-10-0-128-111 kubenswrapper[2583]: W0916 02:18:41.834891    2583 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 16 02:18:41.845807 ip-10-0-128-111 kubenswrapper[2583]: W0916 02:18:41.834894    2583 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 16 02:18:41.845807 ip-10-0-128-111 kubenswrapper[2583]: W0916 02:18:41.834897    2583 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 16 02:18:41.845807 ip-10-0-128-111 kubenswrapper[2583]: W0916 02:18:41.834899    2583 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 16 02:18:41.845807 ip-10-0-128-111 kubenswrapper[2583]: W0916 02:18:41.834904    2583 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 16 02:18:41.845807 ip-10-0-128-111 kubenswrapper[2583]: W0916 02:18:41.834908    2583 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 16 02:18:41.845807 ip-10-0-128-111 kubenswrapper[2583]: W0916 02:18:41.834911    2583 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 16 02:18:41.846679 ip-10-0-128-111 kubenswrapper[2583]: W0916 02:18:41.834914    2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 16 02:18:41.846679 ip-10-0-128-111 kubenswrapper[2583]: W0916 02:18:41.834917    2583 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 16 02:18:41.846679 ip-10-0-128-111 kubenswrapper[2583]: W0916 02:18:41.834920    2583 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 16 02:18:41.846679 ip-10-0-128-111 kubenswrapper[2583]: W0916 02:18:41.834923    2583 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 16 02:18:41.846679 ip-10-0-128-111 kubenswrapper[2583]: W0916 02:18:41.834926    2583 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 16 02:18:41.846679 ip-10-0-128-111 kubenswrapper[2583]: W0916 02:18:41.834929    2583 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 16 02:18:41.846679 ip-10-0-128-111 kubenswrapper[2583]: W0916 02:18:41.834931    2583 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 16 02:18:41.846679 ip-10-0-128-111 kubenswrapper[2583]: W0916 02:18:41.834934    2583 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 16 02:18:41.846679 ip-10-0-128-111 kubenswrapper[2583]: W0916 02:18:41.834938    2583 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 16 02:18:41.846679 ip-10-0-128-111 kubenswrapper[2583]: W0916 02:18:41.834941    2583 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 16 02:18:41.846679 ip-10-0-128-111 kubenswrapper[2583]: W0916 02:18:41.834943    2583 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 16 02:18:41.846679 ip-10-0-128-111 kubenswrapper[2583]: W0916 02:18:41.834946    2583 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 16 02:18:41.846679 ip-10-0-128-111 kubenswrapper[2583]: W0916 02:18:41.834949    2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 16 02:18:41.846679 ip-10-0-128-111 kubenswrapper[2583]: W0916 02:18:41.834952    2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 16 02:18:41.846679 ip-10-0-128-111 kubenswrapper[2583]: W0916 02:18:41.834954    2583 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 16 02:18:41.846679 ip-10-0-128-111 kubenswrapper[2583]: W0916 02:18:41.834957    2583 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 16 02:18:41.846679 ip-10-0-128-111 kubenswrapper[2583]: W0916 02:18:41.834959    2583 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 16 02:18:41.846679 ip-10-0-128-111 kubenswrapper[2583]: W0916 02:18:41.834962    2583 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 16 02:18:41.846679 ip-10-0-128-111 kubenswrapper[2583]: W0916 02:18:41.834964    2583 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 16 02:18:41.846679 ip-10-0-128-111 kubenswrapper[2583]: W0916 02:18:41.834967    2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 16 02:18:41.847353 ip-10-0-128-111 kubenswrapper[2583]: W0916 02:18:41.834970    2583 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 16 02:18:41.847353 ip-10-0-128-111 kubenswrapper[2583]: W0916 02:18:41.834972    2583 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 16 02:18:41.847353 ip-10-0-128-111 kubenswrapper[2583]: W0916 02:18:41.834975    2583 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 16 02:18:41.847353 ip-10-0-128-111 kubenswrapper[2583]: W0916 02:18:41.834977    2583 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 16 02:18:41.847353 ip-10-0-128-111 kubenswrapper[2583]: W0916 02:18:41.834980    2583 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 16 02:18:41.847353 ip-10-0-128-111 kubenswrapper[2583]: W0916 02:18:41.834982    2583 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 16 02:18:41.847353 ip-10-0-128-111 kubenswrapper[2583]: W0916 02:18:41.834985    2583 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 16 02:18:41.847353 ip-10-0-128-111 kubenswrapper[2583]: W0916 02:18:41.834987    2583 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 16 02:18:41.847353 ip-10-0-128-111 kubenswrapper[2583]: W0916 02:18:41.834990    2583 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 16 02:18:41.847353 ip-10-0-128-111 kubenswrapper[2583]: W0916 02:18:41.834994    2583 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 16 02:18:41.847353 ip-10-0-128-111 kubenswrapper[2583]: W0916 02:18:41.834997    2583 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 16 02:18:41.847353 ip-10-0-128-111 kubenswrapper[2583]: W0916 02:18:41.834999    2583 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 16 02:18:41.847353 ip-10-0-128-111 kubenswrapper[2583]: W0916 02:18:41.835002    2583 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 16 02:18:41.847353 ip-10-0-128-111 kubenswrapper[2583]: W0916 02:18:41.835004    2583 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 16 02:18:41.847353 ip-10-0-128-111 kubenswrapper[2583]: W0916 02:18:41.835007    2583 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 16 02:18:41.847353 ip-10-0-128-111 kubenswrapper[2583]: W0916 02:18:41.835009    2583 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 16 02:18:41.847353 ip-10-0-128-111 kubenswrapper[2583]: W0916 02:18:41.835012    2583 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 16 02:18:41.847353 ip-10-0-128-111 kubenswrapper[2583]: W0916 02:18:41.835015    2583 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 16 02:18:41.847353 ip-10-0-128-111 kubenswrapper[2583]: W0916 02:18:41.835019    2583 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 16 02:18:41.848035 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:18:41.835713    2583 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 16 02:18:41.848035 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:18:41.844792    2583 server.go:529] "Kubelet version" kubeletVersion="v1.34.9"
Sep 16 02:18:41.848035 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:18:41.844810    2583 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Sep 16 02:18:41.848035 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:18:41.845636    2583 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Sep 16 02:18:41.848035 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:18:41.845652    2583 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Sep 16 02:18:41.848035 ip-10-0-128-111 kubenswrapper[2583]: W0916 02:18:41.845740    2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 16 02:18:41.848035 ip-10-0-128-111 kubenswrapper[2583]: W0916 02:18:41.845749    2583 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 16 02:18:41.848035 ip-10-0-128-111 kubenswrapper[2583]: W0916 02:18:41.845754    2583 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 16 02:18:41.848035 ip-10-0-128-111 kubenswrapper[2583]: W0916 02:18:41.845759    2583 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 16 02:18:41.848035 ip-10-0-128-111 kubenswrapper[2583]: W0916 02:18:41.845764    2583 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 16 02:18:41.848035 ip-10-0-128-111 kubenswrapper[2583]: W0916 02:18:41.845768    2583 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 16 02:18:41.848035 ip-10-0-128-111 kubenswrapper[2583]: W0916 02:18:41.845772    2583 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 16 02:18:41.848035 ip-10-0-128-111 kubenswrapper[2583]: W0916 02:18:41.845777    2583 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 16 02:18:41.848035 ip-10-0-128-111 kubenswrapper[2583]: W0916 02:18:41.845781    2583 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 16 02:18:41.848035 ip-10-0-128-111 kubenswrapper[2583]: W0916 02:18:41.845786    2583 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 16 02:18:41.848705 ip-10-0-128-111 kubenswrapper[2583]: W0916 02:18:41.845790    2583 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 16 02:18:41.848705 ip-10-0-128-111 kubenswrapper[2583]: W0916 02:18:41.845795    2583 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 16 02:18:41.848705 ip-10-0-128-111 kubenswrapper[2583]: W0916 02:18:41.845799    2583 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 16 02:18:41.848705 ip-10-0-128-111 kubenswrapper[2583]: W0916 02:18:41.845803    2583 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 16 02:18:41.848705 ip-10-0-128-111 kubenswrapper[2583]: W0916 02:18:41.845807    2583 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 16 02:18:41.848705 ip-10-0-128-111 kubenswrapper[2583]: W0916 02:18:41.845811    2583 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 16 02:18:41.848705 ip-10-0-128-111 kubenswrapper[2583]: W0916 02:18:41.845815    2583 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 16 02:18:41.848705 ip-10-0-128-111 kubenswrapper[2583]: W0916 02:18:41.845819    2583 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 16 02:18:41.848705 ip-10-0-128-111 kubenswrapper[2583]: W0916 02:18:41.845823    2583 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 16 02:18:41.848705 ip-10-0-128-111 kubenswrapper[2583]: W0916 02:18:41.845827    2583 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 16 02:18:41.848705 ip-10-0-128-111 kubenswrapper[2583]: W0916 02:18:41.845831    2583 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 16 02:18:41.848705 ip-10-0-128-111 kubenswrapper[2583]: W0916 02:18:41.845835    2583 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 16 02:18:41.848705 ip-10-0-128-111 kubenswrapper[2583]: W0916 02:18:41.845839    2583 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 16 02:18:41.848705 ip-10-0-128-111 kubenswrapper[2583]: W0916 02:18:41.845843    2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 16 02:18:41.848705 ip-10-0-128-111 kubenswrapper[2583]: W0916 02:18:41.845849    2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 16 02:18:41.848705 ip-10-0-128-111 kubenswrapper[2583]: W0916 02:18:41.845854    2583 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 16 02:18:41.848705 ip-10-0-128-111 kubenswrapper[2583]: W0916 02:18:41.845858    2583 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 16 02:18:41.848705 ip-10-0-128-111 kubenswrapper[2583]: W0916 02:18:41.845863    2583 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 16 02:18:41.848705 ip-10-0-128-111 kubenswrapper[2583]: W0916 02:18:41.845868    2583 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 16 02:18:41.848705 ip-10-0-128-111 kubenswrapper[2583]: W0916 02:18:41.845872    2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 16 02:18:41.849343 ip-10-0-128-111 kubenswrapper[2583]: W0916 02:18:41.845880    2583 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 16 02:18:41.849343 ip-10-0-128-111 kubenswrapper[2583]: W0916 02:18:41.845886    2583 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 16 02:18:41.849343 ip-10-0-128-111 kubenswrapper[2583]: W0916 02:18:41.845892    2583 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 16 02:18:41.849343 ip-10-0-128-111 kubenswrapper[2583]: W0916 02:18:41.845896    2583 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 16 02:18:41.849343 ip-10-0-128-111 kubenswrapper[2583]: W0916 02:18:41.845901    2583 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 16 02:18:41.849343 ip-10-0-128-111 kubenswrapper[2583]: W0916 02:18:41.845905    2583 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 16 02:18:41.849343 ip-10-0-128-111 kubenswrapper[2583]: W0916 02:18:41.845910    2583 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 16 02:18:41.849343 ip-10-0-128-111 kubenswrapper[2583]: W0916 02:18:41.845915    2583 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 16 02:18:41.849343 ip-10-0-128-111 kubenswrapper[2583]: W0916 02:18:41.845919    2583 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 16 02:18:41.849343 ip-10-0-128-111 kubenswrapper[2583]: W0916 02:18:41.845923    2583 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 16 02:18:41.849343 ip-10-0-128-111 kubenswrapper[2583]: W0916 02:18:41.845927    2583 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 16 02:18:41.849343 ip-10-0-128-111 kubenswrapper[2583]: W0916 02:18:41.845932    2583 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 16 02:18:41.849343 ip-10-0-128-111 kubenswrapper[2583]: W0916 02:18:41.845936    2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 16 02:18:41.849343 ip-10-0-128-111 kubenswrapper[2583]: W0916 02:18:41.845940    2583 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 16 02:18:41.849343 ip-10-0-128-111 kubenswrapper[2583]: W0916 02:18:41.845944    2583 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 16 02:18:41.849343 ip-10-0-128-111 kubenswrapper[2583]: W0916 02:18:41.845948    2583 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 16 02:18:41.849343 ip-10-0-128-111 kubenswrapper[2583]: W0916 02:18:41.845952    2583 feature_gate.go:328] unrecognized feature gate: Example
Sep 16 02:18:41.849343 ip-10-0-128-111 kubenswrapper[2583]: W0916 02:18:41.845956    2583 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 16 02:18:41.849343 ip-10-0-128-111 kubenswrapper[2583]: W0916 02:18:41.845960    2583 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 16 02:18:41.850020 ip-10-0-128-111 kubenswrapper[2583]: W0916 02:18:41.845964    2583 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 16 02:18:41.850020 ip-10-0-128-111 kubenswrapper[2583]: W0916 02:18:41.845968    2583 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 16 02:18:41.850020 ip-10-0-128-111 kubenswrapper[2583]: W0916 02:18:41.845973    2583 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 16 02:18:41.850020 ip-10-0-128-111 kubenswrapper[2583]: W0916 02:18:41.845977    2583 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 16 02:18:41.850020 ip-10-0-128-111 kubenswrapper[2583]: W0916 02:18:41.845981    2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 16 02:18:41.850020 ip-10-0-128-111 kubenswrapper[2583]: W0916 02:18:41.845985    2583 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 16 02:18:41.850020 ip-10-0-128-111 kubenswrapper[2583]: W0916 02:18:41.845989    2583 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 16 02:18:41.850020 ip-10-0-128-111 kubenswrapper[2583]: W0916 02:18:41.845994    2583 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 16 02:18:41.850020 ip-10-0-128-111 kubenswrapper[2583]: W0916 02:18:41.845999    2583 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 16 02:18:41.850020 ip-10-0-128-111 kubenswrapper[2583]: W0916 02:18:41.846003    2583 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 16 02:18:41.850020 ip-10-0-128-111 kubenswrapper[2583]: W0916 02:18:41.846007    2583 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 16 02:18:41.850020 ip-10-0-128-111 kubenswrapper[2583]: W0916 02:18:41.846011    2583 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 16 02:18:41.850020 ip-10-0-128-111 kubenswrapper[2583]: W0916 02:18:41.846018    2583 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 16 02:18:41.850020 ip-10-0-128-111 kubenswrapper[2583]: W0916 02:18:41.846025    2583 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 16 02:18:41.850020 ip-10-0-128-111 kubenswrapper[2583]: W0916 02:18:41.846029    2583 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 16 02:18:41.850020 ip-10-0-128-111 kubenswrapper[2583]: W0916 02:18:41.846033    2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 16 02:18:41.850020 ip-10-0-128-111 kubenswrapper[2583]: W0916 02:18:41.846037    2583 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 16 02:18:41.850020 ip-10-0-128-111 kubenswrapper[2583]: W0916 02:18:41.846041    2583 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 16 02:18:41.850020 ip-10-0-128-111 kubenswrapper[2583]: W0916 02:18:41.846046    2583 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 16 02:18:41.850020 ip-10-0-128-111 kubenswrapper[2583]: W0916 02:18:41.846050    2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 16 02:18:41.850506 ip-10-0-128-111 kubenswrapper[2583]: W0916 02:18:41.846053    2583 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 16 02:18:41.850506 ip-10-0-128-111 kubenswrapper[2583]: W0916 02:18:41.846057    2583 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 16 02:18:41.850506 ip-10-0-128-111 kubenswrapper[2583]: W0916 02:18:41.846062    2583 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 16 02:18:41.850506 ip-10-0-128-111 kubenswrapper[2583]: W0916 02:18:41.846066    2583 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 16 02:18:41.850506 ip-10-0-128-111 kubenswrapper[2583]: W0916 02:18:41.846070    2583 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 16 02:18:41.850506 ip-10-0-128-111 kubenswrapper[2583]: W0916 02:18:41.846074    2583 feature_gate.go:328] unrecognized feature gate: Example2
Sep 16 02:18:41.850506 ip-10-0-128-111 kubenswrapper[2583]: W0916 02:18:41.846079    2583 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 16 02:18:41.850506 ip-10-0-128-111 kubenswrapper[2583]: W0916 02:18:41.846083    2583 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 16 02:18:41.850506 ip-10-0-128-111 kubenswrapper[2583]: W0916 02:18:41.846087    2583 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 16 02:18:41.850506 ip-10-0-128-111 kubenswrapper[2583]: W0916 02:18:41.846092    2583 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 16 02:18:41.850506 ip-10-0-128-111 kubenswrapper[2583]: W0916 02:18:41.846096    2583 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 16 02:18:41.850506 ip-10-0-128-111 kubenswrapper[2583]: W0916 02:18:41.846100    2583 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 16 02:18:41.850506 ip-10-0-128-111 kubenswrapper[2583]: W0916 02:18:41.846104    2583 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 16 02:18:41.850506 ip-10-0-128-111 kubenswrapper[2583]: W0916 02:18:41.846109    2583 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 16 02:18:41.850506 ip-10-0-128-111 kubenswrapper[2583]: W0916 02:18:41.846113    2583 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 16 02:18:41.850506 ip-10-0-128-111 kubenswrapper[2583]: W0916 02:18:41.846117    2583 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 16 02:18:41.850506 ip-10-0-128-111 kubenswrapper[2583]: W0916 02:18:41.846121    2583 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 16 02:18:41.850506 ip-10-0-128-111 kubenswrapper[2583]: W0916 02:18:41.846126    2583 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 16 02:18:41.850506 ip-10-0-128-111 kubenswrapper[2583]: W0916 02:18:41.846130    2583 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 16 02:18:41.851023 ip-10-0-128-111 kubenswrapper[2583]: W0916 02:18:41.846134    2583 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 16 02:18:41.851023 ip-10-0-128-111 kubenswrapper[2583]: W0916 02:18:41.846138    2583 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 16 02:18:41.851023 ip-10-0-128-111 kubenswrapper[2583]: W0916 02:18:41.846143    2583 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 16 02:18:41.851023 ip-10-0-128-111 kubenswrapper[2583]: W0916 02:18:41.846148    2583 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 16 02:18:41.851023 ip-10-0-128-111 kubenswrapper[2583]: W0916 02:18:41.846152    2583 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 16 02:18:41.851023 ip-10-0-128-111 kubenswrapper[2583]: W0916 02:18:41.846157    2583 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 16 02:18:41.851023 ip-10-0-128-111 kubenswrapper[2583]: W0916 02:18:41.846164    2583 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 16 02:18:41.851023 ip-10-0-128-111 kubenswrapper[2583]: W0916 02:18:41.846169    2583 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 16 02:18:41.851023 ip-10-0-128-111 kubenswrapper[2583]: W0916 02:18:41.846173    2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 16 02:18:41.851023 ip-10-0-128-111 kubenswrapper[2583]: W0916 02:18:41.846179    2583 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 16 02:18:41.851023 ip-10-0-128-111 kubenswrapper[2583]: W0916 02:18:41.846183    2583 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 16 02:18:41.851023 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:18:41.846191    2583 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 16 02:18:41.851023 ip-10-0-128-111 kubenswrapper[2583]: W0916 02:18:41.846359    2583 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 16 02:18:41.851023 ip-10-0-128-111 kubenswrapper[2583]: W0916 02:18:41.846367    2583 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 16 02:18:41.851434 ip-10-0-128-111 kubenswrapper[2583]: W0916 02:18:41.846372    2583 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 16 02:18:41.851434 ip-10-0-128-111 kubenswrapper[2583]: W0916 02:18:41.846377    2583 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 16 02:18:41.851434 ip-10-0-128-111 kubenswrapper[2583]: W0916 02:18:41.846381    2583 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 16 02:18:41.851434 ip-10-0-128-111 kubenswrapper[2583]: W0916 02:18:41.846386    2583 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 16 02:18:41.851434 ip-10-0-128-111 kubenswrapper[2583]: W0916 02:18:41.846390    2583 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 16 02:18:41.851434 ip-10-0-128-111 kubenswrapper[2583]: W0916 02:18:41.846394    2583 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 16 02:18:41.851434 ip-10-0-128-111 kubenswrapper[2583]: W0916 02:18:41.846398    2583 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 16 02:18:41.851434 ip-10-0-128-111 kubenswrapper[2583]: W0916 02:18:41.846406    2583 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 16 02:18:41.851434 ip-10-0-128-111 kubenswrapper[2583]: W0916 02:18:41.846412    2583 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 16 02:18:41.851434 ip-10-0-128-111 kubenswrapper[2583]: W0916 02:18:41.846417    2583 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 16 02:18:41.851434 ip-10-0-128-111 kubenswrapper[2583]: W0916 02:18:41.846422    2583 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 16 02:18:41.851434 ip-10-0-128-111 kubenswrapper[2583]: W0916 02:18:41.846426    2583 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 16 02:18:41.851434 ip-10-0-128-111 kubenswrapper[2583]: W0916 02:18:41.846431    2583 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 16 02:18:41.851434 ip-10-0-128-111 kubenswrapper[2583]: W0916 02:18:41.846435    2583 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 16 02:18:41.851434 ip-10-0-128-111 kubenswrapper[2583]: W0916 02:18:41.846439    2583 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 16 02:18:41.851434 ip-10-0-128-111 kubenswrapper[2583]: W0916 02:18:41.846444    2583 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 16 02:18:41.851434 ip-10-0-128-111 kubenswrapper[2583]: W0916 02:18:41.846448    2583 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 16 02:18:41.851434 ip-10-0-128-111 kubenswrapper[2583]: W0916 02:18:41.846452    2583 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 16 02:18:41.851434 ip-10-0-128-111 kubenswrapper[2583]: W0916 02:18:41.846457    2583 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 16 02:18:41.851919 ip-10-0-128-111 kubenswrapper[2583]: W0916 02:18:41.846461    2583 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 16 02:18:41.851919 ip-10-0-128-111 kubenswrapper[2583]: W0916 02:18:41.846465    2583 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 16 02:18:41.851919 ip-10-0-128-111 kubenswrapper[2583]: W0916 02:18:41.846470    2583 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 16 02:18:41.851919 ip-10-0-128-111 kubenswrapper[2583]: W0916 02:18:41.846474    2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 16 02:18:41.851919 ip-10-0-128-111 kubenswrapper[2583]: W0916 02:18:41.846478    2583 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 16 02:18:41.851919 ip-10-0-128-111 kubenswrapper[2583]: W0916 02:18:41.846482    2583 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 16 02:18:41.851919 ip-10-0-128-111 kubenswrapper[2583]: W0916 02:18:41.846486    2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 16 02:18:41.851919 ip-10-0-128-111 kubenswrapper[2583]: W0916 02:18:41.846490    2583 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 16 02:18:41.851919 ip-10-0-128-111 kubenswrapper[2583]: W0916 02:18:41.846495    2583 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 16 02:18:41.851919 ip-10-0-128-111 kubenswrapper[2583]: W0916 02:18:41.846499    2583 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 16 02:18:41.851919 ip-10-0-128-111 kubenswrapper[2583]: W0916 02:18:41.846503    2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 16 02:18:41.851919 ip-10-0-128-111 kubenswrapper[2583]: W0916 02:18:41.846507    2583 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 16 02:18:41.851919 ip-10-0-128-111 kubenswrapper[2583]: W0916 02:18:41.846511    2583 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 16 02:18:41.851919 ip-10-0-128-111 kubenswrapper[2583]: W0916 02:18:41.846515    2583 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 16 02:18:41.851919 ip-10-0-128-111 kubenswrapper[2583]: W0916 02:18:41.846519    2583 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 16 02:18:41.851919 ip-10-0-128-111 kubenswrapper[2583]: W0916 02:18:41.846523    2583 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 16 02:18:41.851919 ip-10-0-128-111 kubenswrapper[2583]: W0916 02:18:41.846528    2583 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 16 02:18:41.851919 ip-10-0-128-111 kubenswrapper[2583]: W0916 02:18:41.846532    2583 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 16 02:18:41.851919 ip-10-0-128-111 kubenswrapper[2583]: W0916 02:18:41.846536    2583 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 16 02:18:41.851919 ip-10-0-128-111 kubenswrapper[2583]: W0916 02:18:41.846540    2583 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 16 02:18:41.852426 ip-10-0-128-111 kubenswrapper[2583]: W0916 02:18:41.846545    2583 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 16 02:18:41.852426 ip-10-0-128-111 kubenswrapper[2583]: W0916 02:18:41.846569    2583 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 16 02:18:41.852426 ip-10-0-128-111 kubenswrapper[2583]: W0916 02:18:41.846573    2583 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 16 02:18:41.852426 ip-10-0-128-111 kubenswrapper[2583]: W0916 02:18:41.846577    2583 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 16 02:18:41.852426 ip-10-0-128-111 kubenswrapper[2583]: W0916 02:18:41.846582    2583 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 16 02:18:41.852426 ip-10-0-128-111 kubenswrapper[2583]: W0916 02:18:41.846586    2583 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 16 02:18:41.852426 ip-10-0-128-111 kubenswrapper[2583]: W0916 02:18:41.846590    2583 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 16 02:18:41.852426 ip-10-0-128-111 kubenswrapper[2583]: W0916 02:18:41.846594    2583 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 16 02:18:41.852426 ip-10-0-128-111 kubenswrapper[2583]: W0916 02:18:41.846598    2583 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 16 02:18:41.852426 ip-10-0-128-111 kubenswrapper[2583]: W0916 02:18:41.846602    2583 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 16 02:18:41.852426 ip-10-0-128-111 kubenswrapper[2583]: W0916 02:18:41.846606    2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 16 02:18:41.852426 ip-10-0-128-111 kubenswrapper[2583]: W0916 02:18:41.846610    2583 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 16 02:18:41.852426 ip-10-0-128-111 kubenswrapper[2583]: W0916 02:18:41.846614    2583 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 16 02:18:41.852426 ip-10-0-128-111 kubenswrapper[2583]: W0916 02:18:41.846618    2583 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 16 02:18:41.852426 ip-10-0-128-111 kubenswrapper[2583]: W0916 02:18:41.846622    2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 16 02:18:41.852426 ip-10-0-128-111 kubenswrapper[2583]: W0916 02:18:41.846627    2583 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 16 02:18:41.852426 ip-10-0-128-111 kubenswrapper[2583]: W0916 02:18:41.846632    2583 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 16 02:18:41.852426 ip-10-0-128-111 kubenswrapper[2583]: W0916 02:18:41.846635    2583 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 16 02:18:41.852426 ip-10-0-128-111 kubenswrapper[2583]: W0916 02:18:41.846639    2583 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 16 02:18:41.852426 ip-10-0-128-111 kubenswrapper[2583]: W0916 02:18:41.846643    2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 16 02:18:41.853053 ip-10-0-128-111 kubenswrapper[2583]: W0916 02:18:41.846650    2583 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 16 02:18:41.853053 ip-10-0-128-111 kubenswrapper[2583]: W0916 02:18:41.846656    2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 16 02:18:41.853053 ip-10-0-128-111 kubenswrapper[2583]: W0916 02:18:41.846660    2583 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 16 02:18:41.853053 ip-10-0-128-111 kubenswrapper[2583]: W0916 02:18:41.846665    2583 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 16 02:18:41.853053 ip-10-0-128-111 kubenswrapper[2583]: W0916 02:18:41.846669    2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 16 02:18:41.853053 ip-10-0-128-111 kubenswrapper[2583]: W0916 02:18:41.846673    2583 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 16 02:18:41.853053 ip-10-0-128-111 kubenswrapper[2583]: W0916 02:18:41.846676    2583 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 16 02:18:41.853053 ip-10-0-128-111 kubenswrapper[2583]: W0916 02:18:41.846680    2583 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 16 02:18:41.853053 ip-10-0-128-111 kubenswrapper[2583]: W0916 02:18:41.846684    2583 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 16 02:18:41.853053 ip-10-0-128-111 kubenswrapper[2583]: W0916 02:18:41.846688    2583 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 16 02:18:41.853053 ip-10-0-128-111 kubenswrapper[2583]: W0916 02:18:41.846692    2583 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 16 02:18:41.853053 ip-10-0-128-111 kubenswrapper[2583]: W0916 02:18:41.846696    2583 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 16 02:18:41.853053 ip-10-0-128-111 kubenswrapper[2583]: W0916 02:18:41.846700    2583 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 16 02:18:41.853053 ip-10-0-128-111 kubenswrapper[2583]: W0916 02:18:41.846704    2583 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 16 02:18:41.853053 ip-10-0-128-111 kubenswrapper[2583]: W0916 02:18:41.846708    2583 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 16 02:18:41.853053 ip-10-0-128-111 kubenswrapper[2583]: W0916 02:18:41.846712    2583 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 16 02:18:41.853053 ip-10-0-128-111 kubenswrapper[2583]: W0916 02:18:41.846717    2583 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 16 02:18:41.853053 ip-10-0-128-111 kubenswrapper[2583]: W0916 02:18:41.846721    2583 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 16 02:18:41.853053 ip-10-0-128-111 kubenswrapper[2583]: W0916 02:18:41.846725    2583 feature_gate.go:328] unrecognized feature gate: Example
Sep 16 02:18:41.853539 ip-10-0-128-111 kubenswrapper[2583]: W0916 02:18:41.846729    2583 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 16 02:18:41.853539 ip-10-0-128-111 kubenswrapper[2583]: W0916 02:18:41.846733    2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 16 02:18:41.853539 ip-10-0-128-111 kubenswrapper[2583]: W0916 02:18:41.846737    2583 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 16 02:18:41.853539 ip-10-0-128-111 kubenswrapper[2583]: W0916 02:18:41.846741    2583 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 16 02:18:41.853539 ip-10-0-128-111 kubenswrapper[2583]: W0916 02:18:41.846746    2583 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 16 02:18:41.853539 ip-10-0-128-111 kubenswrapper[2583]: W0916 02:18:41.846749    2583 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 16 02:18:41.853539 ip-10-0-128-111 kubenswrapper[2583]: W0916 02:18:41.846753    2583 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 16 02:18:41.853539 ip-10-0-128-111 kubenswrapper[2583]: W0916 02:18:41.846758    2583 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 16 02:18:41.853539 ip-10-0-128-111 kubenswrapper[2583]: W0916 02:18:41.846764    2583 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 16 02:18:41.853539 ip-10-0-128-111 kubenswrapper[2583]: W0916 02:18:41.846770    2583 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 16 02:18:41.853539 ip-10-0-128-111 kubenswrapper[2583]: W0916 02:18:41.846775    2583 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 16 02:18:41.853539 ip-10-0-128-111 kubenswrapper[2583]: W0916 02:18:41.846780    2583 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 16 02:18:41.853539 ip-10-0-128-111 kubenswrapper[2583]: W0916 02:18:41.846784    2583 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 16 02:18:41.853539 ip-10-0-128-111 kubenswrapper[2583]: W0916 02:18:41.846788    2583 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 16 02:18:41.853539 ip-10-0-128-111 kubenswrapper[2583]: W0916 02:18:41.846792    2583 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 16 02:18:41.853539 ip-10-0-128-111 kubenswrapper[2583]: W0916 02:18:41.846797    2583 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 16 02:18:41.853539 ip-10-0-128-111 kubenswrapper[2583]: W0916 02:18:41.846801    2583 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 16 02:18:41.853539 ip-10-0-128-111 kubenswrapper[2583]: W0916 02:18:41.846805    2583 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 16 02:18:41.853539 ip-10-0-128-111 kubenswrapper[2583]: W0916 02:18:41.846809    2583 feature_gate.go:328] unrecognized feature gate: Example2
Sep 16 02:18:41.854098 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:18:41.846817    2583 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 16 02:18:41.854098 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:18:41.846959    2583 server.go:962] "Client rotation is on, will bootstrap in background"
Sep 16 02:18:41.854098 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:18:41.849520    2583 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Sep 16 02:18:41.854098 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:18:41.850541    2583 server.go:1019] "Starting client certificate rotation"
Sep 16 02:18:41.854098 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:18:41.850662    2583 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 16 02:18:41.854098 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:18:41.850702    2583 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 16 02:18:41.854098 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:18:41.853164    2583 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Sep 16 02:18:41.859511 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:18:41.859474    2583 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Sep 16 02:18:41.870229 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:18:41.870208    2583 log.go:25] "Validated CRI v1 runtime API"
Sep 16 02:18:41.875324 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:18:41.875299    2583 log.go:25] "Validated CRI v1 image API"
Sep 16 02:18:41.876134 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:18:41.876118    2583 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Sep 16 02:18:41.878366 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:18:41.878342    2583 fs.go:135] Filesystem UUIDs: map[51c24c6b-b464-4299-8d04-f296a54a4357:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2 7cabc624-11ee-496c-9d5e-32c3d795298e:/dev/nvme0n1p4]
Sep 16 02:18:41.878423 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:18:41.878360    2583 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}]
Sep 16 02:18:41.883286 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:18:41.883170    2583 manager.go:216] Machine: {Timestamp:2026-09-16 02:18:41.881937891 +0000 UTC m=+0.349726664 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3099868 MemoryCapacity:33164480512 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2b00377ce6da9f4e4ac6d7e1d0c467 SystemUUID:ec2b0037-7ce6-da9f-4e4a-c6d7e1d0c467 BootID:19361b8f-e96d-4a99-bc5a-ef4a2f9b2961 Filesystems:[{Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582242304 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582238208 Type:vfs Inodes:4048398 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632898560 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:ff:e2:0e:21:f1 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:ff:e2:0e:21:f1 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:ba:09:0c:26:87:a1 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164480512 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Sep 16 02:18:41.883286 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:18:41.883273    2583 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Sep 16 02:18:41.883404 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:18:41.883348    2583 manager.go:232] Version: {KernelVersion:5.14.0-570.140.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260908-1 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Sep 16 02:18:41.884939 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:18:41.884920    2583 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Sep 16 02:18:41.886176 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:18:41.886148    2583 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Sep 16 02:18:41.886302 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:18:41.886178    2583 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-128-111.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Sep 16 02:18:41.887307 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:18:41.887295    2583 topology_manager.go:138] "Creating topology manager with none policy"
Sep 16 02:18:41.887341 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:18:41.887310    2583 container_manager_linux.go:309] "Creating device plugin manager"
Sep 16 02:18:41.887341 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:18:41.887317    2583 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 16 02:18:41.888464 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:18:41.888454    2583 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 16 02:18:41.889236 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:18:41.889226    2583 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager"
Sep 16 02:18:41.891989 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:18:41.891977    2583 state_mem.go:36] "Initialized new in-memory state store"
Sep 16 02:18:41.892097 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:18:41.892087    2583 server.go:1240] "Using root directory" path="/var/lib/kubelet"
Sep 16 02:18:41.892126 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:18:41.892116    2583 kubelet.go:486] "Attempting to sync node with API server"
Sep 16 02:18:41.892156 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:18:41.892126    2583 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests"
Sep 16 02:18:41.892156 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:18:41.892140    2583 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Sep 16 02:18:41.892156 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:18:41.892151    2583 kubelet.go:398] "Adding apiserver pod source"
Sep 16 02:18:41.892248 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:18:41.892161    2583 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Sep 16 02:18:41.892573 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:18:41.892543    2583 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Sep 16 02:18:41.892618 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:18:41.892587    2583 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Sep 16 02:18:41.894891 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:18:41.894874    2583 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.12-5.rhaos4.21.gitd1ec63c.el9" apiVersion="v1"
Sep 16 02:18:41.896869 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:18:41.896855    2583 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Sep 16 02:18:41.896943 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:18:41.896885    2583 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Sep 16 02:18:41.898953 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:18:41.898940    2583 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Sep 16 02:18:41.898953 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:18:41.898956    2583 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Sep 16 02:18:41.899058 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:18:41.898962    2583 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Sep 16 02:18:41.899058 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:18:41.898967    2583 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Sep 16 02:18:41.899058 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:18:41.898975    2583 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Sep 16 02:18:41.899058 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:18:41.898981    2583 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Sep 16 02:18:41.899058 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:18:41.898994    2583 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Sep 16 02:18:41.899058 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:18:41.899000    2583 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Sep 16 02:18:41.899058 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:18:41.899006    2583 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Sep 16 02:18:41.899058 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:18:41.899012    2583 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Sep 16 02:18:41.899058 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:18:41.899025    2583 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Sep 16 02:18:41.899058 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:18:41.899031    2583 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Sep 16 02:18:41.901999 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:18:41.901985    2583 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Sep 16 02:18:41.901999 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:18:41.901996    2583 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Sep 16 02:18:41.903882 ip-10-0-128-111 kubenswrapper[2583]: E0916 02:18:41.903614    2583 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-128-111.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
Sep 16 02:18:41.904051 ip-10-0-128-111 kubenswrapper[2583]: E0916 02:18:41.904031    2583 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
Sep 16 02:18:41.906654 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:18:41.906636    2583 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-128-111.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 16 02:18:41.907184 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:18:41.907167    2583 server.go:1268] "Started kubelet"
Sep 16 02:18:41.907280 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:18:41.907235    2583 server.go:180] "Starting to listen" address="0.0.0.0" port=10250
Sep 16 02:18:41.907375 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:18:41.907324    2583 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Sep 16 02:18:41.907422 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:18:41.907398    2583 server_v1.go:49] "podresources" method="list" useActivePods=true
Sep 16 02:18:41.908057 ip-10-0-128-111 systemd[1]: Started Kubernetes Kubelet.
Sep 16 02:18:41.909265 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:18:41.908434    2583 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Sep 16 02:18:41.909265 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:18:41.909188    2583 server.go:310] "Adding debug handlers to kubelet server"
Sep 16 02:18:41.912375 ip-10-0-128-111 kubenswrapper[2583]: E0916 02:18:41.912350    2583 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Sep 16 02:18:41.913364 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:18:41.913351    2583 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
Sep 16 02:18:41.913804 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:18:41.913418    2583 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Sep 16 02:18:41.913804 ip-10-0-128-111 kubenswrapper[2583]: E0916 02:18:41.913476    2583 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-111.ec2.internal\" not found"
Sep 16 02:18:41.913964 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:18:41.913948    2583 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Sep 16 02:18:41.914028 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:18:41.913952    2583 volume_manager.go:311] "The desired_state_of_world populator starts"
Sep 16 02:18:41.914028 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:18:41.913998    2583 volume_manager.go:313] "Starting Kubelet Volume Manager"
Sep 16 02:18:41.914117 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:18:41.914081    2583 reconstruct.go:97] "Volume reconstruction finished"
Sep 16 02:18:41.914117 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:18:41.914091    2583 reconciler.go:29] "Reconciler: start to sync state"
Sep 16 02:18:41.914221 ip-10-0-128-111 kubenswrapper[2583]: E0916 02:18:41.912936    2583 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-128-111.ec2.internal.18d5ab9df7e97ff9  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-128-111.ec2.internal,UID:ip-10-0-128-111.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-128-111.ec2.internal,},FirstTimestamp:2026-09-16 02:18:41.907130361 +0000 UTC m=+0.374919137,LastTimestamp:2026-09-16 02:18:41.907130361 +0000 UTC m=+0.374919137,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-128-111.ec2.internal,}"
Sep 16 02:18:41.914368 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:18:41.914348    2583 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Sep 16 02:18:41.914368 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:18:41.914365    2583 factory.go:55] Registering systemd factory
Sep 16 02:18:41.914525 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:18:41.914373    2583 factory.go:223] Registration of the systemd container factory successfully
Sep 16 02:18:41.914652 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:18:41.914629    2583 factory.go:153] Registering CRI-O factory
Sep 16 02:18:41.914652 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:18:41.914645    2583 factory.go:223] Registration of the crio container factory successfully
Sep 16 02:18:41.914781 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:18:41.914659    2583 factory.go:103] Registering Raw factory
Sep 16 02:18:41.914781 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:18:41.914673    2583 manager.go:1196] Started watching for new ooms in manager
Sep 16 02:18:41.915067 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:18:41.915056    2583 manager.go:349] Starting recovery of all containers
Sep 16 02:18:41.918260 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:18:41.918238    2583 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-nkcx9"
Sep 16 02:18:41.918675 ip-10-0-128-111 kubenswrapper[2583]: E0916 02:18:41.918651    2583 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver"
Sep 16 02:18:41.918935 ip-10-0-128-111 kubenswrapper[2583]: E0916 02:18:41.918895    2583 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-128-111.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
Sep 16 02:18:41.925490 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:18:41.925471    2583 manager.go:354] Recovery completed
Sep 16 02:18:41.930690 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:18:41.930673    2583 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-nkcx9"
Sep 16 02:18:41.931775 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:18:41.931760    2583 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 16 02:18:41.934310 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:18:41.934287    2583 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-111.ec2.internal" event="NodeHasSufficientMemory"
Sep 16 02:18:41.934387 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:18:41.934326    2583 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-111.ec2.internal" event="NodeHasNoDiskPressure"
Sep 16 02:18:41.934387 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:18:41.934344    2583 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-111.ec2.internal" event="NodeHasSufficientPID"
Sep 16 02:18:41.934963 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:18:41.934948    2583 cpu_manager.go:222] "Starting CPU manager" policy="none"
Sep 16 02:18:41.934963 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:18:41.934962    2583 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 16 02:18:41.934963 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:18:41.934963    2583 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s"
Sep 16 02:18:41.935118 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:18:41.935007    2583 state_mem.go:36] "Initialized new in-memory state store"
Sep 16 02:18:41.935631 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:18:41.935613    2583 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-111.ec2.internal" event="NodeHasSufficientMemory"
Sep 16 02:18:41.935713 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:18:41.935644    2583 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-111.ec2.internal" event="NodeHasNoDiskPressure"
Sep 16 02:18:41.935713 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:18:41.935660    2583 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-111.ec2.internal" event="NodeHasSufficientPID"
Sep 16 02:18:41.935986 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:18:41.935974    2583 policy_none.go:49] "None policy: Start"
Sep 16 02:18:41.936015 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:18:41.935990    2583 memory_manager.go:187] "Starting memorymanager" policy="None"
Sep 16 02:18:41.936015 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:18:41.936000    2583 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Sep 16 02:18:41.936487 ip-10-0-128-111 kubenswrapper[2583]: E0916 02:18:41.936428    2583 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-128-111.ec2.internal.18d5ab9df9883425  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-128-111.ec2.internal,UID:ip-10-0-128-111.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-128-111.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-128-111.ec2.internal,},FirstTimestamp:2026-09-16 02:18:41.934308389 +0000 UTC m=+0.402097167,LastTimestamp:2026-09-16 02:18:41.934308389 +0000 UTC m=+0.402097167,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-128-111.ec2.internal,}"
Sep 16 02:18:41.936997 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:18:41.936982    2583 policy_none.go:47] "Start"
Sep 16 02:18:41.960133 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:18:41.960113    2583 manager.go:337] "Starting Device Plugin manager"
Sep 16 02:18:41.960209 ip-10-0-128-111 kubenswrapper[2583]: E0916 02:18:41.960151    2583 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Sep 16 02:18:41.960209 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:18:41.960164    2583 server.go:87] "Starting device plugin registration server"
Sep 16 02:18:41.960441 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:18:41.960428    2583 eviction_manager.go:189] "Eviction manager: starting control loop"
Sep 16 02:18:41.960508 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:18:41.960442    2583 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Sep 16 02:18:41.961130 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:18:41.960621    2583 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Sep 16 02:18:41.961130 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:18:41.960729    2583 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts"
Sep 16 02:18:41.961130 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:18:41.960742    2583 plugin_manager.go:118] "Starting Kubelet Plugin Manager"
Sep 16 02:18:41.962041 ip-10-0-128-111 kubenswrapper[2583]: E0916 02:18:41.962023    2583 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Sep 16 02:18:41.962093 ip-10-0-128-111 kubenswrapper[2583]: E0916 02:18:41.962076    2583 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-128-111.ec2.internal\" not found"
Sep 16 02:18:42.037483 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:18:42.037443    2583 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Sep 16 02:18:42.038602 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:18:42.038583    2583 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Sep 16 02:18:42.038677 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:18:42.038642    2583 status_manager.go:244] "Starting to sync pod status with apiserver"
Sep 16 02:18:42.038677 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:18:42.038670    2583 kubelet.go:2448] "Starting kubelet main sync loop"
Sep 16 02:18:42.038756 ip-10-0-128-111 kubenswrapper[2583]: E0916 02:18:42.038715    2583 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Sep 16 02:18:42.040792 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:18:42.040776    2583 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 16 02:18:42.061531 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:18:42.061505    2583 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 16 02:18:42.062316 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:18:42.062299    2583 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-111.ec2.internal" event="NodeHasSufficientMemory"
Sep 16 02:18:42.062387 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:18:42.062335    2583 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-111.ec2.internal" event="NodeHasNoDiskPressure"
Sep 16 02:18:42.062387 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:18:42.062354    2583 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-111.ec2.internal" event="NodeHasSufficientPID"
Sep 16 02:18:42.062387 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:18:42.062376    2583 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-128-111.ec2.internal"
Sep 16 02:18:42.071632 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:18:42.071615    2583 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-128-111.ec2.internal"
Sep 16 02:18:42.071679 ip-10-0-128-111 kubenswrapper[2583]: E0916 02:18:42.071639    2583 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-128-111.ec2.internal\": node \"ip-10-0-128-111.ec2.internal\" not found"
Sep 16 02:18:42.112004 ip-10-0-128-111 kubenswrapper[2583]: E0916 02:18:42.111930    2583 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-111.ec2.internal\" not found"
Sep 16 02:18:42.139306 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:18:42.139285    2583 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-111.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-128-111.ec2.internal"]
Sep 16 02:18:42.139405 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:18:42.139346    2583 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 16 02:18:42.140777 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:18:42.140760    2583 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-111.ec2.internal" event="NodeHasSufficientMemory"
Sep 16 02:18:42.140887 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:18:42.140788    2583 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-111.ec2.internal" event="NodeHasNoDiskPressure"
Sep 16 02:18:42.140887 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:18:42.140798    2583 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-111.ec2.internal" event="NodeHasSufficientPID"
Sep 16 02:18:42.143060 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:18:42.143048    2583 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 16 02:18:42.143179 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:18:42.143164    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-111.ec2.internal"
Sep 16 02:18:42.143234 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:18:42.143201    2583 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 16 02:18:42.143839 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:18:42.143821    2583 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-111.ec2.internal" event="NodeHasSufficientMemory"
Sep 16 02:18:42.143935 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:18:42.143842    2583 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-111.ec2.internal" event="NodeHasSufficientMemory"
Sep 16 02:18:42.143935 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:18:42.143866    2583 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-111.ec2.internal" event="NodeHasNoDiskPressure"
Sep 16 02:18:42.143935 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:18:42.143881    2583 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-111.ec2.internal" event="NodeHasSufficientPID"
Sep 16 02:18:42.143935 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:18:42.143846    2583 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-111.ec2.internal" event="NodeHasNoDiskPressure"
Sep 16 02:18:42.143935 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:18:42.143927    2583 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-111.ec2.internal" event="NodeHasSufficientPID"
Sep 16 02:18:42.146135 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:18:42.146117    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-111.ec2.internal"
Sep 16 02:18:42.146226 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:18:42.146141    2583 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 16 02:18:42.146840 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:18:42.146823    2583 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-111.ec2.internal" event="NodeHasSufficientMemory"
Sep 16 02:18:42.146948 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:18:42.146849    2583 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-111.ec2.internal" event="NodeHasNoDiskPressure"
Sep 16 02:18:42.146948 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:18:42.146859    2583 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-111.ec2.internal" event="NodeHasSufficientPID"
Sep 16 02:18:42.176167 ip-10-0-128-111 kubenswrapper[2583]: E0916 02:18:42.176148    2583 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-128-111.ec2.internal\" not found" node="ip-10-0-128-111.ec2.internal"
Sep 16 02:18:42.179524 ip-10-0-128-111 kubenswrapper[2583]: E0916 02:18:42.179504    2583 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-128-111.ec2.internal\" not found" node="ip-10-0-128-111.ec2.internal"
Sep 16 02:18:42.212181 ip-10-0-128-111 kubenswrapper[2583]: E0916 02:18:42.212162    2583 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-111.ec2.internal\" not found"
Sep 16 02:18:42.215425 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:18:42.215408    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/ac09b6665582e7a386f06843c363096b-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-128-111.ec2.internal\" (UID: \"ac09b6665582e7a386f06843c363096b\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-111.ec2.internal"
Sep 16 02:18:42.215507 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:18:42.215432    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ac09b6665582e7a386f06843c363096b-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-128-111.ec2.internal\" (UID: \"ac09b6665582e7a386f06843c363096b\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-111.ec2.internal"
Sep 16 02:18:42.215507 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:18:42.215450    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/b7568fa8cea4626b154e4750bd44461f-config\") pod \"kube-apiserver-proxy-ip-10-0-128-111.ec2.internal\" (UID: \"b7568fa8cea4626b154e4750bd44461f\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-128-111.ec2.internal"
Sep 16 02:18:42.313080 ip-10-0-128-111 kubenswrapper[2583]: E0916 02:18:42.313054    2583 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-111.ec2.internal\" not found"
Sep 16 02:18:42.316341 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:18:42.316324    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/ac09b6665582e7a386f06843c363096b-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-128-111.ec2.internal\" (UID: \"ac09b6665582e7a386f06843c363096b\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-111.ec2.internal"
Sep 16 02:18:42.316389 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:18:42.316341    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/ac09b6665582e7a386f06843c363096b-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-128-111.ec2.internal\" (UID: \"ac09b6665582e7a386f06843c363096b\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-111.ec2.internal"
Sep 16 02:18:42.316389 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:18:42.316361    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ac09b6665582e7a386f06843c363096b-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-128-111.ec2.internal\" (UID: \"ac09b6665582e7a386f06843c363096b\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-111.ec2.internal"
Sep 16 02:18:42.316450 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:18:42.316395    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/b7568fa8cea4626b154e4750bd44461f-config\") pod \"kube-apiserver-proxy-ip-10-0-128-111.ec2.internal\" (UID: \"b7568fa8cea4626b154e4750bd44461f\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-128-111.ec2.internal"
Sep 16 02:18:42.316450 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:18:42.316418    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/b7568fa8cea4626b154e4750bd44461f-config\") pod \"kube-apiserver-proxy-ip-10-0-128-111.ec2.internal\" (UID: \"b7568fa8cea4626b154e4750bd44461f\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-128-111.ec2.internal"
Sep 16 02:18:42.316513 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:18:42.316450    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ac09b6665582e7a386f06843c363096b-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-128-111.ec2.internal\" (UID: \"ac09b6665582e7a386f06843c363096b\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-111.ec2.internal"
Sep 16 02:18:42.413607 ip-10-0-128-111 kubenswrapper[2583]: E0916 02:18:42.413517    2583 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-111.ec2.internal\" not found"
Sep 16 02:18:42.479185 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:18:42.479137    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-111.ec2.internal"
Sep 16 02:18:42.482582 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:18:42.482544    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-111.ec2.internal"
Sep 16 02:18:42.514683 ip-10-0-128-111 kubenswrapper[2583]: E0916 02:18:42.514654    2583 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-111.ec2.internal\" not found"
Sep 16 02:18:42.620270 ip-10-0-128-111 kubenswrapper[2583]: E0916 02:18:42.620208    2583 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-111.ec2.internal\" not found"
Sep 16 02:18:42.721197 ip-10-0-128-111 kubenswrapper[2583]: E0916 02:18:42.721110    2583 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-111.ec2.internal\" not found"
Sep 16 02:18:42.821239 ip-10-0-128-111 kubenswrapper[2583]: E0916 02:18:42.821203    2583 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-111.ec2.internal\" not found"
Sep 16 02:18:42.849960 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:18:42.849931    2583 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Sep 16 02:18:42.850375 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:18:42.850092    2583 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Sep 16 02:18:42.919342 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:18:42.919316    2583 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 16 02:18:42.926640 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:18:42.926609    2583 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 16 02:18:42.933832 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:18:42.933806    2583 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-09-15 02:13:42 +0000 UTC" deadline="2028-04-05 08:17:34.661455103 +0000 UTC"
Sep 16 02:18:42.933832 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:18:42.933830    2583 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="13613h58m51.727628036s"
Sep 16 02:18:43.014765 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:18:43.014682    2583 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-111.ec2.internal"
Sep 16 02:18:43.026340 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:18:43.026323    2583 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Sep 16 02:18:43.027299 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:18:43.027285    2583 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-111.ec2.internal"
Sep 16 02:18:43.035150 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:18:43.035133    2583 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Sep 16 02:18:43.248124 ip-10-0-128-111 kubenswrapper[2583]: W0916 02:18:43.248094    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podac09b6665582e7a386f06843c363096b.slice/crio-566d16305d110fce7369ca972fcf287e420e20284da1922790c3306ac7c0d7c9 WatchSource:0}: Error finding container 566d16305d110fce7369ca972fcf287e420e20284da1922790c3306ac7c0d7c9: Status 404 returned error can't find the container with id 566d16305d110fce7369ca972fcf287e420e20284da1922790c3306ac7c0d7c9
Sep 16 02:18:43.248590 ip-10-0-128-111 kubenswrapper[2583]: W0916 02:18:43.248569    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb7568fa8cea4626b154e4750bd44461f.slice/crio-097e7632dfd1eb8f93b795d5ae0f6a398a2cf5caac1e3995aef9d75acaca710b WatchSource:0}: Error finding container 097e7632dfd1eb8f93b795d5ae0f6a398a2cf5caac1e3995aef9d75acaca710b: Status 404 returned error can't find the container with id 097e7632dfd1eb8f93b795d5ae0f6a398a2cf5caac1e3995aef9d75acaca710b
Sep 16 02:18:43.254676 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:18:43.254621    2583 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 16 02:18:43.510060 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:18:43.509969    2583 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 16 02:18:43.893396 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:18:43.893312    2583 apiserver.go:52] "Watching apiserver"
Sep 16 02:18:43.900742 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:18:43.900706    2583 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Sep 16 02:18:43.901275 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:18:43.901240    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-multus/multus-additional-cni-plugins-ll7tp","openshift-network-diagnostics/network-check-target-55zcf","openshift-ovn-kubernetes/ovnkube-node-hgb4m","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dqkz9","openshift-dns/node-resolver-kf6w7","openshift-image-registry/node-ca-5nwmf","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-111.ec2.internal","openshift-multus/multus-7nk9m","openshift-multus/network-metrics-daemon-bdwgr","openshift-network-operator/iptables-alerter-lmvcp","kube-system/global-pull-secret-syncer-mbxmd","kube-system/konnectivity-agent-psqcm","kube-system/kube-apiserver-proxy-ip-10-0-128-111.ec2.internal","openshift-cluster-node-tuning-operator/tuned-rx99l"]
Sep 16 02:18:43.904099 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:18:43.904076    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bdwgr"
Sep 16 02:18:43.904208 ip-10-0-128-111 kubenswrapper[2583]: E0916 02:18:43.904156    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bdwgr" podUID="67a44fbe-95d7-4808-bcff-f7d214c6d82c"
Sep 16 02:18:43.906562 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:18:43.906524    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-55zcf"
Sep 16 02:18:43.906662 ip-10-0-128-111 kubenswrapper[2583]: E0916 02:18:43.906617    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-55zcf" podUID="5f66bb1b-6d4f-42c5-9b63-f7f5f301b6f9"
Sep 16 02:18:43.909082 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:18:43.909058    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-hgb4m"
Sep 16 02:18:43.911404 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:18:43.911377    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Sep 16 02:18:43.912079 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:18:43.911572    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-2cb98\""
Sep 16 02:18:43.912079 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:18:43.911678    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Sep 16 02:18:43.912079 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:18:43.911750    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Sep 16 02:18:43.912079 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:18:43.911801    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-rx99l"
Sep 16 02:18:43.912079 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:18:43.912060    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Sep 16 02:18:43.912437 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:18:43.912085    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Sep 16 02:18:43.912437 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:18:43.912216    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Sep 16 02:18:43.914522 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:18:43.914075    2583 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Sep 16 02:18:43.914641 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:18:43.914528    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Sep 16 02:18:43.914641 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:18:43.914543    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-p94bc\""
Sep 16 02:18:43.914768 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:18:43.914754    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Sep 16 02:18:43.914856 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:18:43.914770    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-kf6w7"
Sep 16 02:18:43.917681 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:18:43.917051    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Sep 16 02:18:43.917681 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:18:43.917115    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-5nwmf"
Sep 16 02:18:43.917818 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:18:43.917169    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Sep 16 02:18:43.917985 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:18:43.917190    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-xb5rd\""
Sep 16 02:18:43.919647 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:18:43.919563    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-7nk9m"
Sep 16 02:18:43.920075 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:18:43.920054    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Sep 16 02:18:43.920174 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:18:43.920055    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Sep 16 02:18:43.920337 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:18:43.920320    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-4wvg9\""
Sep 16 02:18:43.920420 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:18:43.920356    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Sep 16 02:18:43.922596 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:18:43.922371    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Sep 16 02:18:43.922596 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:18:43.922536    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-ll7tp"
Sep 16 02:18:43.922932 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:18:43.922911    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Sep 16 02:18:43.923993 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:18:43.923373    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Sep 16 02:18:43.923993 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:18:43.923636    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Sep 16 02:18:43.923993 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:18:43.923872    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-nkq6s\""
Sep 16 02:18:43.925316 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:18:43.924937    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-5z8vk\""
Sep 16 02:18:43.925316 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:18:43.925135    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Sep 16 02:18:43.926760 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:18:43.925755    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-lmvcp"
Sep 16 02:18:43.926760 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:18:43.925885    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Sep 16 02:18:43.928019 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:18:43.927993    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/66c3846d-d0d2-4b46-8c59-f6d97a92f7ad-lib-modules\") pod \"tuned-rx99l\" (UID: \"66c3846d-d0d2-4b46-8c59-f6d97a92f7ad\") " pod="openshift-cluster-node-tuning-operator/tuned-rx99l"
Sep 16 02:18:43.928133 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:18:43.928037    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qwztf\" (UniqueName: \"kubernetes.io/projected/5f66bb1b-6d4f-42c5-9b63-f7f5f301b6f9-kube-api-access-qwztf\") pod \"network-check-target-55zcf\" (UID: \"5f66bb1b-6d4f-42c5-9b63-f7f5f301b6f9\") " pod="openshift-network-diagnostics/network-check-target-55zcf"
Sep 16 02:18:43.928133 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:18:43.928064    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/66c3846d-d0d2-4b46-8c59-f6d97a92f7ad-tmp\") pod \"tuned-rx99l\" (UID: \"66c3846d-d0d2-4b46-8c59-f6d97a92f7ad\") " pod="openshift-cluster-node-tuning-operator/tuned-rx99l"
Sep 16 02:18:43.928133 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:18:43.928098    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/d45dcef5-f6f3-45d7-b29e-1125a402eeb3-systemd-units\") pod \"ovnkube-node-hgb4m\" (UID: \"d45dcef5-f6f3-45d7-b29e-1125a402eeb3\") " pod="openshift-ovn-kubernetes/ovnkube-node-hgb4m"
Sep 16 02:18:43.928133 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:18:43.928123    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/d45dcef5-f6f3-45d7-b29e-1125a402eeb3-run-systemd\") pod \"ovnkube-node-hgb4m\" (UID: \"d45dcef5-f6f3-45d7-b29e-1125a402eeb3\") " pod="openshift-ovn-kubernetes/ovnkube-node-hgb4m"
Sep 16 02:18:43.928325 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:18:43.928148    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/d45dcef5-f6f3-45d7-b29e-1125a402eeb3-var-lib-openvswitch\") pod \"ovnkube-node-hgb4m\" (UID: \"d45dcef5-f6f3-45d7-b29e-1125a402eeb3\") " pod="openshift-ovn-kubernetes/ovnkube-node-hgb4m"
Sep 16 02:18:43.928325 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:18:43.928172    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/d45dcef5-f6f3-45d7-b29e-1125a402eeb3-run-openvswitch\") pod \"ovnkube-node-hgb4m\" (UID: \"d45dcef5-f6f3-45d7-b29e-1125a402eeb3\") " pod="openshift-ovn-kubernetes/ovnkube-node-hgb4m"
Sep 16 02:18:43.928325 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:18:43.928223    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d45dcef5-f6f3-45d7-b29e-1125a402eeb3-host-run-ovn-kubernetes\") pod \"ovnkube-node-hgb4m\" (UID: \"d45dcef5-f6f3-45d7-b29e-1125a402eeb3\") " pod="openshift-ovn-kubernetes/ovnkube-node-hgb4m"
Sep 16 02:18:43.928325 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:18:43.928305    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/d45dcef5-f6f3-45d7-b29e-1125a402eeb3-host-cni-bin\") pod \"ovnkube-node-hgb4m\" (UID: \"d45dcef5-f6f3-45d7-b29e-1125a402eeb3\") " pod="openshift-ovn-kubernetes/ovnkube-node-hgb4m"
Sep 16 02:18:43.928325 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:18:43.928314    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Sep 16 02:18:43.928553 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:18:43.928355    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-nb2cv\""
Sep 16 02:18:43.928620 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:18:43.928553    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/66c3846d-d0d2-4b46-8c59-f6d97a92f7ad-etc-tuned\") pod \"tuned-rx99l\" (UID: \"66c3846d-d0d2-4b46-8c59-f6d97a92f7ad\") " pod="openshift-cluster-node-tuning-operator/tuned-rx99l"
Sep 16 02:18:43.928620 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:18:43.928598    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Sep 16 02:18:43.928752 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:18:43.928717    2583 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Sep 16 02:18:43.928752 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:18:43.928731    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Sep 16 02:18:43.928867 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:18:43.928736    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4cl2x\" (UniqueName: \"kubernetes.io/projected/66c3846d-d0d2-4b46-8c59-f6d97a92f7ad-kube-api-access-4cl2x\") pod \"tuned-rx99l\" (UID: \"66c3846d-d0d2-4b46-8c59-f6d97a92f7ad\") " pod="openshift-cluster-node-tuning-operator/tuned-rx99l"
Sep 16 02:18:43.928867 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:18:43.928792    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/d45dcef5-f6f3-45d7-b29e-1125a402eeb3-ovn-node-metrics-cert\") pod \"ovnkube-node-hgb4m\" (UID: \"d45dcef5-f6f3-45d7-b29e-1125a402eeb3\") " pod="openshift-ovn-kubernetes/ovnkube-node-hgb4m"
Sep 16 02:18:43.928867 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:18:43.928824    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/67a44fbe-95d7-4808-bcff-f7d214c6d82c-metrics-certs\") pod \"network-metrics-daemon-bdwgr\" (UID: \"67a44fbe-95d7-4808-bcff-f7d214c6d82c\") " pod="openshift-multus/network-metrics-daemon-bdwgr"
Sep 16 02:18:43.928867 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:18:43.928849    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/d45dcef5-f6f3-45d7-b29e-1125a402eeb3-host-kubelet\") pod \"ovnkube-node-hgb4m\" (UID: \"d45dcef5-f6f3-45d7-b29e-1125a402eeb3\") " pod="openshift-ovn-kubernetes/ovnkube-node-hgb4m"
Sep 16 02:18:43.929055 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:18:43.928873    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/66c3846d-d0d2-4b46-8c59-f6d97a92f7ad-etc-sysctl-conf\") pod \"tuned-rx99l\" (UID: \"66c3846d-d0d2-4b46-8c59-f6d97a92f7ad\") " pod="openshift-cluster-node-tuning-operator/tuned-rx99l"
Sep 16 02:18:43.929055 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:18:43.928909    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/66c3846d-d0d2-4b46-8c59-f6d97a92f7ad-etc-kubernetes\") pod \"tuned-rx99l\" (UID: \"66c3846d-d0d2-4b46-8c59-f6d97a92f7ad\") " pod="openshift-cluster-node-tuning-operator/tuned-rx99l"
Sep 16 02:18:43.929055 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:18:43.928975    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/66c3846d-d0d2-4b46-8c59-f6d97a92f7ad-sys\") pod \"tuned-rx99l\" (UID: \"66c3846d-d0d2-4b46-8c59-f6d97a92f7ad\") " pod="openshift-cluster-node-tuning-operator/tuned-rx99l"
Sep 16 02:18:43.929055 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:18:43.929007    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/66c3846d-d0d2-4b46-8c59-f6d97a92f7ad-var-lib-kubelet\") pod \"tuned-rx99l\" (UID: \"66c3846d-d0d2-4b46-8c59-f6d97a92f7ad\") " pod="openshift-cluster-node-tuning-operator/tuned-rx99l"
Sep 16 02:18:43.929235 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:18:43.929058    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mbxmd"
Sep 16 02:18:43.929235 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:18:43.929077    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/66c3846d-d0d2-4b46-8c59-f6d97a92f7ad-host\") pod \"tuned-rx99l\" (UID: \"66c3846d-d0d2-4b46-8c59-f6d97a92f7ad\") " pod="openshift-cluster-node-tuning-operator/tuned-rx99l"
Sep 16 02:18:43.929235 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:18:43.929103    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2c5d542f-5718-4aeb-8bf5-680a69ddecbc-tmp-dir\") pod \"node-resolver-kf6w7\" (UID: \"2c5d542f-5718-4aeb-8bf5-680a69ddecbc\") " pod="openshift-dns/node-resolver-kf6w7"
Sep 16 02:18:43.929235 ip-10-0-128-111 kubenswrapper[2583]: E0916 02:18:43.929134    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mbxmd" podUID="2fbf440b-4c09-428e-b6bf-1236d95f97cc"
Sep 16 02:18:43.929235 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:18:43.929138    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4k94z\" (UniqueName: \"kubernetes.io/projected/67a44fbe-95d7-4808-bcff-f7d214c6d82c-kube-api-access-4k94z\") pod \"network-metrics-daemon-bdwgr\" (UID: \"67a44fbe-95d7-4808-bcff-f7d214c6d82c\") " pod="openshift-multus/network-metrics-daemon-bdwgr"
Sep 16 02:18:43.929235 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:18:43.929201    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/d45dcef5-f6f3-45d7-b29e-1125a402eeb3-run-ovn\") pod \"ovnkube-node-hgb4m\" (UID: \"d45dcef5-f6f3-45d7-b29e-1125a402eeb3\") " pod="openshift-ovn-kubernetes/ovnkube-node-hgb4m"
Sep 16 02:18:43.929517 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:18:43.929230    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/66c3846d-d0d2-4b46-8c59-f6d97a92f7ad-etc-modprobe-d\") pod \"tuned-rx99l\" (UID: \"66c3846d-d0d2-4b46-8c59-f6d97a92f7ad\") " pod="openshift-cluster-node-tuning-operator/tuned-rx99l"
Sep 16 02:18:43.929517 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:18:43.929285    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/d45dcef5-f6f3-45d7-b29e-1125a402eeb3-ovnkube-config\") pod \"ovnkube-node-hgb4m\" (UID: \"d45dcef5-f6f3-45d7-b29e-1125a402eeb3\") " pod="openshift-ovn-kubernetes/ovnkube-node-hgb4m"
Sep 16 02:18:43.929517 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:18:43.929310    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/d45dcef5-f6f3-45d7-b29e-1125a402eeb3-env-overrides\") pod \"ovnkube-node-hgb4m\" (UID: \"d45dcef5-f6f3-45d7-b29e-1125a402eeb3\") " pod="openshift-ovn-kubernetes/ovnkube-node-hgb4m"
Sep 16 02:18:43.929517 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:18:43.929374    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/d45dcef5-f6f3-45d7-b29e-1125a402eeb3-host-slash\") pod \"ovnkube-node-hgb4m\" (UID: \"d45dcef5-f6f3-45d7-b29e-1125a402eeb3\") " pod="openshift-ovn-kubernetes/ovnkube-node-hgb4m"
Sep 16 02:18:43.929517 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:18:43.929408    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/d45dcef5-f6f3-45d7-b29e-1125a402eeb3-etc-openvswitch\") pod \"ovnkube-node-hgb4m\" (UID: \"d45dcef5-f6f3-45d7-b29e-1125a402eeb3\") " pod="openshift-ovn-kubernetes/ovnkube-node-hgb4m"
Sep 16 02:18:43.929517 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:18:43.929426    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/d45dcef5-f6f3-45d7-b29e-1125a402eeb3-node-log\") pod \"ovnkube-node-hgb4m\" (UID: \"d45dcef5-f6f3-45d7-b29e-1125a402eeb3\") " pod="openshift-ovn-kubernetes/ovnkube-node-hgb4m"
Sep 16 02:18:43.929517 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:18:43.929448    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d45dcef5-f6f3-45d7-b29e-1125a402eeb3-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-hgb4m\" (UID: \"d45dcef5-f6f3-45d7-b29e-1125a402eeb3\") " pod="openshift-ovn-kubernetes/ovnkube-node-hgb4m"
Sep 16 02:18:43.929517 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:18:43.929471    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/66c3846d-d0d2-4b46-8c59-f6d97a92f7ad-run\") pod \"tuned-rx99l\" (UID: \"66c3846d-d0d2-4b46-8c59-f6d97a92f7ad\") " pod="openshift-cluster-node-tuning-operator/tuned-rx99l"
Sep 16 02:18:43.929517 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:18:43.929494    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/2c5d542f-5718-4aeb-8bf5-680a69ddecbc-hosts-file\") pod \"node-resolver-kf6w7\" (UID: \"2c5d542f-5718-4aeb-8bf5-680a69ddecbc\") " pod="openshift-dns/node-resolver-kf6w7"
Sep 16 02:18:43.929946 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:18:43.929536    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/d45dcef5-f6f3-45d7-b29e-1125a402eeb3-ovnkube-script-lib\") pod \"ovnkube-node-hgb4m\" (UID: \"d45dcef5-f6f3-45d7-b29e-1125a402eeb3\") " pod="openshift-ovn-kubernetes/ovnkube-node-hgb4m"
Sep 16 02:18:43.929946 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:18:43.929556    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cd96q\" (UniqueName: \"kubernetes.io/projected/d45dcef5-f6f3-45d7-b29e-1125a402eeb3-kube-api-access-cd96q\") pod \"ovnkube-node-hgb4m\" (UID: \"d45dcef5-f6f3-45d7-b29e-1125a402eeb3\") " pod="openshift-ovn-kubernetes/ovnkube-node-hgb4m"
Sep 16 02:18:43.929946 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:18:43.929614    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/66c3846d-d0d2-4b46-8c59-f6d97a92f7ad-etc-sysconfig\") pod \"tuned-rx99l\" (UID: \"66c3846d-d0d2-4b46-8c59-f6d97a92f7ad\") " pod="openshift-cluster-node-tuning-operator/tuned-rx99l"
Sep 16 02:18:43.929946 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:18:43.929667    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/d45dcef5-f6f3-45d7-b29e-1125a402eeb3-host-run-netns\") pod \"ovnkube-node-hgb4m\" (UID: \"d45dcef5-f6f3-45d7-b29e-1125a402eeb3\") " pod="openshift-ovn-kubernetes/ovnkube-node-hgb4m"
Sep 16 02:18:43.929946 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:18:43.929694    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/d45dcef5-f6f3-45d7-b29e-1125a402eeb3-log-socket\") pod \"ovnkube-node-hgb4m\" (UID: \"d45dcef5-f6f3-45d7-b29e-1125a402eeb3\") " pod="openshift-ovn-kubernetes/ovnkube-node-hgb4m"
Sep 16 02:18:43.929946 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:18:43.929720    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/66c3846d-d0d2-4b46-8c59-f6d97a92f7ad-etc-systemd\") pod \"tuned-rx99l\" (UID: \"66c3846d-d0d2-4b46-8c59-f6d97a92f7ad\") " pod="openshift-cluster-node-tuning-operator/tuned-rx99l"
Sep 16 02:18:43.929946 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:18:43.929748    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x94vc\" (UniqueName: \"kubernetes.io/projected/2c5d542f-5718-4aeb-8bf5-680a69ddecbc-kube-api-access-x94vc\") pod \"node-resolver-kf6w7\" (UID: \"2c5d542f-5718-4aeb-8bf5-680a69ddecbc\") " pod="openshift-dns/node-resolver-kf6w7"
Sep 16 02:18:43.929946 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:18:43.929787    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/d45dcef5-f6f3-45d7-b29e-1125a402eeb3-host-cni-netd\") pod \"ovnkube-node-hgb4m\" (UID: \"d45dcef5-f6f3-45d7-b29e-1125a402eeb3\") " pod="openshift-ovn-kubernetes/ovnkube-node-hgb4m"
Sep 16 02:18:43.929946 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:18:43.929819    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/66c3846d-d0d2-4b46-8c59-f6d97a92f7ad-etc-sysctl-d\") pod \"tuned-rx99l\" (UID: \"66c3846d-d0d2-4b46-8c59-f6d97a92f7ad\") " pod="openshift-cluster-node-tuning-operator/tuned-rx99l"
Sep 16 02:18:43.931602 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:18:43.931580    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-psqcm"
Sep 16 02:18:43.933842 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:18:43.933820    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-k5js4\""
Sep 16 02:18:43.933947 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:18:43.933898    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Sep 16 02:18:43.934213 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:18:43.934195    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dqkz9"
Sep 16 02:18:43.934642 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:18:43.934442    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Sep 16 02:18:43.936345 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:18:43.936325    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Sep 16 02:18:43.936548 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:18:43.936524    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Sep 16 02:18:43.936805 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:18:43.936786    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Sep 16 02:18:43.936932 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:18:43.936791    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-sv6k9\""
Sep 16 02:18:43.951689 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:18:43.951664    2583 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-htmnx"
Sep 16 02:18:43.960780 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:18:43.960761    2583 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-htmnx"
Sep 16 02:18:44.015547 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:18:44.015521    2583 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Sep 16 02:18:44.030527 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:18:44.030502    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/66c3846d-d0d2-4b46-8c59-f6d97a92f7ad-sys\") pod \"tuned-rx99l\" (UID: \"66c3846d-d0d2-4b46-8c59-f6d97a92f7ad\") " pod="openshift-cluster-node-tuning-operator/tuned-rx99l"
Sep 16 02:18:44.030527 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:18:44.030541    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/66c3846d-d0d2-4b46-8c59-f6d97a92f7ad-host\") pod \"tuned-rx99l\" (UID: \"66c3846d-d0d2-4b46-8c59-f6d97a92f7ad\") " pod="openshift-cluster-node-tuning-operator/tuned-rx99l"
Sep 16 02:18:44.030729 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:18:44.030588    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/e34369f4-88ba-4859-ad03-80219c62e272-host-run-k8s-cni-cncf-io\") pod \"multus-7nk9m\" (UID: \"e34369f4-88ba-4859-ad03-80219c62e272\") " pod="openshift-multus/multus-7nk9m"
Sep 16 02:18:44.030729 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:18:44.030640    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/3a1ad832-d1ab-4676-8eee-31d2eccce648-tuning-conf-dir\") pod \"multus-additional-cni-plugins-ll7tp\" (UID: \"3a1ad832-d1ab-4676-8eee-31d2eccce648\") " pod="openshift-multus/multus-additional-cni-plugins-ll7tp"
Sep 16 02:18:44.030729 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:18:44.030668    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/d45dcef5-f6f3-45d7-b29e-1125a402eeb3-run-ovn\") pod \"ovnkube-node-hgb4m\" (UID: \"d45dcef5-f6f3-45d7-b29e-1125a402eeb3\") " pod="openshift-ovn-kubernetes/ovnkube-node-hgb4m"
Sep 16 02:18:44.030729 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:18:44.030671    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/66c3846d-d0d2-4b46-8c59-f6d97a92f7ad-sys\") pod \"tuned-rx99l\" (UID: \"66c3846d-d0d2-4b46-8c59-f6d97a92f7ad\") " pod="openshift-cluster-node-tuning-operator/tuned-rx99l"
Sep 16 02:18:44.030729 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:18:44.030693    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/66c3846d-d0d2-4b46-8c59-f6d97a92f7ad-etc-modprobe-d\") pod \"tuned-rx99l\" (UID: \"66c3846d-d0d2-4b46-8c59-f6d97a92f7ad\") " pod="openshift-cluster-node-tuning-operator/tuned-rx99l"
Sep 16 02:18:44.030729 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:18:44.030726    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/b9f0b2c7-b0c8-49bf-9761-c387496f9207-kubelet-dir\") pod \"aws-ebs-csi-driver-node-dqkz9\" (UID: \"b9f0b2c7-b0c8-49bf-9761-c387496f9207\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dqkz9"
Sep 16 02:18:44.030976 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:18:44.030745    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/66c3846d-d0d2-4b46-8c59-f6d97a92f7ad-host\") pod \"tuned-rx99l\" (UID: \"66c3846d-d0d2-4b46-8c59-f6d97a92f7ad\") " pod="openshift-cluster-node-tuning-operator/tuned-rx99l"
Sep 16 02:18:44.030976 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:18:44.030766    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vkdx2\" (UniqueName: \"kubernetes.io/projected/b9f0b2c7-b0c8-49bf-9761-c387496f9207-kube-api-access-vkdx2\") pod \"aws-ebs-csi-driver-node-dqkz9\" (UID: \"b9f0b2c7-b0c8-49bf-9761-c387496f9207\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dqkz9"
Sep 16 02:18:44.030976 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:18:44.030793    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/2fbf440b-4c09-428e-b6bf-1236d95f97cc-dbus\") pod \"global-pull-secret-syncer-mbxmd\" (UID: \"2fbf440b-4c09-428e-b6bf-1236d95f97cc\") " pod="kube-system/global-pull-secret-syncer-mbxmd"
Sep 16 02:18:44.030976 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:18:44.030816    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/d45dcef5-f6f3-45d7-b29e-1125a402eeb3-ovnkube-config\") pod \"ovnkube-node-hgb4m\" (UID: \"d45dcef5-f6f3-45d7-b29e-1125a402eeb3\") " pod="openshift-ovn-kubernetes/ovnkube-node-hgb4m"
Sep 16 02:18:44.030976 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:18:44.030842    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/d45dcef5-f6f3-45d7-b29e-1125a402eeb3-env-overrides\") pod \"ovnkube-node-hgb4m\" (UID: \"d45dcef5-f6f3-45d7-b29e-1125a402eeb3\") " pod="openshift-ovn-kubernetes/ovnkube-node-hgb4m"
Sep 16 02:18:44.030976 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:18:44.030867    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/66c3846d-d0d2-4b46-8c59-f6d97a92f7ad-etc-modprobe-d\") pod \"tuned-rx99l\" (UID: \"66c3846d-d0d2-4b46-8c59-f6d97a92f7ad\") " pod="openshift-cluster-node-tuning-operator/tuned-rx99l"
Sep 16 02:18:44.030976 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:18:44.030880    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/d45dcef5-f6f3-45d7-b29e-1125a402eeb3-node-log\") pod \"ovnkube-node-hgb4m\" (UID: \"d45dcef5-f6f3-45d7-b29e-1125a402eeb3\") " pod="openshift-ovn-kubernetes/ovnkube-node-hgb4m"
Sep 16 02:18:44.030976 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:18:44.030921    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d45dcef5-f6f3-45d7-b29e-1125a402eeb3-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-hgb4m\" (UID: \"d45dcef5-f6f3-45d7-b29e-1125a402eeb3\") " pod="openshift-ovn-kubernetes/ovnkube-node-hgb4m"
Sep 16 02:18:44.030976 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:18:44.030949    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/66c3846d-d0d2-4b46-8c59-f6d97a92f7ad-run\") pod \"tuned-rx99l\" (UID: \"66c3846d-d0d2-4b46-8c59-f6d97a92f7ad\") " pod="openshift-cluster-node-tuning-operator/tuned-rx99l"
Sep 16 02:18:44.031443 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:18:44.030990    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/d45dcef5-f6f3-45d7-b29e-1125a402eeb3-node-log\") pod \"ovnkube-node-hgb4m\" (UID: \"d45dcef5-f6f3-45d7-b29e-1125a402eeb3\") " pod="openshift-ovn-kubernetes/ovnkube-node-hgb4m"
Sep 16 02:18:44.031554 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:18:44.030987    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/b9f0b2c7-b0c8-49bf-9761-c387496f9207-registration-dir\") pod \"aws-ebs-csi-driver-node-dqkz9\" (UID: \"b9f0b2c7-b0c8-49bf-9761-c387496f9207\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dqkz9"
Sep 16 02:18:44.031554 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:18:44.031459    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/d45dcef5-f6f3-45d7-b29e-1125a402eeb3-ovnkube-config\") pod \"ovnkube-node-hgb4m\" (UID: \"d45dcef5-f6f3-45d7-b29e-1125a402eeb3\") " pod="openshift-ovn-kubernetes/ovnkube-node-hgb4m"
Sep 16 02:18:44.031554 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:18:44.031473    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hznzt\" (UniqueName: \"kubernetes.io/projected/82d1b4b2-7910-42f8-b2f3-0d675dc2c530-kube-api-access-hznzt\") pod \"node-ca-5nwmf\" (UID: \"82d1b4b2-7910-42f8-b2f3-0d675dc2c530\") " pod="openshift-image-registry/node-ca-5nwmf"
Sep 16 02:18:44.031554 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:18:44.031509    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/3a1ad832-d1ab-4676-8eee-31d2eccce648-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-ll7tp\" (UID: \"3a1ad832-d1ab-4676-8eee-31d2eccce648\") " pod="openshift-multus/multus-additional-cni-plugins-ll7tp"
Sep 16 02:18:44.031554 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:18:44.031552    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d45dcef5-f6f3-45d7-b29e-1125a402eeb3-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-hgb4m\" (UID: \"d45dcef5-f6f3-45d7-b29e-1125a402eeb3\") " pod="openshift-ovn-kubernetes/ovnkube-node-hgb4m"
Sep 16 02:18:44.031860 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:18:44.030924    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/d45dcef5-f6f3-45d7-b29e-1125a402eeb3-run-ovn\") pod \"ovnkube-node-hgb4m\" (UID: \"d45dcef5-f6f3-45d7-b29e-1125a402eeb3\") " pod="openshift-ovn-kubernetes/ovnkube-node-hgb4m"
Sep 16 02:18:44.031860 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:18:44.031619    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/66c3846d-d0d2-4b46-8c59-f6d97a92f7ad-etc-sysconfig\") pod \"tuned-rx99l\" (UID: \"66c3846d-d0d2-4b46-8c59-f6d97a92f7ad\") " pod="openshift-cluster-node-tuning-operator/tuned-rx99l"
Sep 16 02:18:44.031860 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:18:44.031800    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/66c3846d-d0d2-4b46-8c59-f6d97a92f7ad-run\") pod \"tuned-rx99l\" (UID: \"66c3846d-d0d2-4b46-8c59-f6d97a92f7ad\") " pod="openshift-cluster-node-tuning-operator/tuned-rx99l"
Sep 16 02:18:44.031860 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:18:44.031555    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/66c3846d-d0d2-4b46-8c59-f6d97a92f7ad-etc-sysconfig\") pod \"tuned-rx99l\" (UID: \"66c3846d-d0d2-4b46-8c59-f6d97a92f7ad\") " pod="openshift-cluster-node-tuning-operator/tuned-rx99l"
Sep 16 02:18:44.031860 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:18:44.031839    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/d45dcef5-f6f3-45d7-b29e-1125a402eeb3-host-run-netns\") pod \"ovnkube-node-hgb4m\" (UID: \"d45dcef5-f6f3-45d7-b29e-1125a402eeb3\") " pod="openshift-ovn-kubernetes/ovnkube-node-hgb4m"
Sep 16 02:18:44.031860 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:18:44.031857    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/66c3846d-d0d2-4b46-8c59-f6d97a92f7ad-etc-systemd\") pod \"tuned-rx99l\" (UID: \"66c3846d-d0d2-4b46-8c59-f6d97a92f7ad\") " pod="openshift-cluster-node-tuning-operator/tuned-rx99l"
Sep 16 02:18:44.032048 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:18:44.031878    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/3a1ad832-d1ab-4676-8eee-31d2eccce648-cni-binary-copy\") pod \"multus-additional-cni-plugins-ll7tp\" (UID: \"3a1ad832-d1ab-4676-8eee-31d2eccce648\") " pod="openshift-multus/multus-additional-cni-plugins-ll7tp"
Sep 16 02:18:44.032048 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:18:44.031905    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/2fbf440b-4c09-428e-b6bf-1236d95f97cc-original-pull-secret\") pod \"global-pull-secret-syncer-mbxmd\" (UID: \"2fbf440b-4c09-428e-b6bf-1236d95f97cc\") " pod="kube-system/global-pull-secret-syncer-mbxmd"
Sep 16 02:18:44.032048 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:18:44.031922    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/66c3846d-d0d2-4b46-8c59-f6d97a92f7ad-etc-sysctl-d\") pod \"tuned-rx99l\" (UID: \"66c3846d-d0d2-4b46-8c59-f6d97a92f7ad\") " pod="openshift-cluster-node-tuning-operator/tuned-rx99l"
Sep 16 02:18:44.032048 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:18:44.031936    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/66c3846d-d0d2-4b46-8c59-f6d97a92f7ad-lib-modules\") pod \"tuned-rx99l\" (UID: \"66c3846d-d0d2-4b46-8c59-f6d97a92f7ad\") " pod="openshift-cluster-node-tuning-operator/tuned-rx99l"
Sep 16 02:18:44.032048 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:18:44.031952    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/e34369f4-88ba-4859-ad03-80219c62e272-host-var-lib-cni-bin\") pod \"multus-7nk9m\" (UID: \"e34369f4-88ba-4859-ad03-80219c62e272\") " pod="openshift-multus/multus-7nk9m"
Sep 16 02:18:44.032048 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:18:44.031967    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/e34369f4-88ba-4859-ad03-80219c62e272-host-var-lib-cni-multus\") pod \"multus-7nk9m\" (UID: \"e34369f4-88ba-4859-ad03-80219c62e272\") " pod="openshift-multus/multus-7nk9m"
Sep 16 02:18:44.032048 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:18:44.031968    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/d45dcef5-f6f3-45d7-b29e-1125a402eeb3-env-overrides\") pod \"ovnkube-node-hgb4m\" (UID: \"d45dcef5-f6f3-45d7-b29e-1125a402eeb3\") " pod="openshift-ovn-kubernetes/ovnkube-node-hgb4m"
Sep 16 02:18:44.032048 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:18:44.031981    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e34369f4-88ba-4859-ad03-80219c62e272-etc-kubernetes\") pod \"multus-7nk9m\" (UID: \"e34369f4-88ba-4859-ad03-80219c62e272\") " pod="openshift-multus/multus-7nk9m"
Sep 16 02:18:44.032048 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:18:44.031997    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/b9f0b2c7-b0c8-49bf-9761-c387496f9207-sys-fs\") pod \"aws-ebs-csi-driver-node-dqkz9\" (UID: \"b9f0b2c7-b0c8-49bf-9761-c387496f9207\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dqkz9"
Sep 16 02:18:44.032048 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:18:44.032013    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/66c3846d-d0d2-4b46-8c59-f6d97a92f7ad-tmp\") pod \"tuned-rx99l\" (UID: \"66c3846d-d0d2-4b46-8c59-f6d97a92f7ad\") " pod="openshift-cluster-node-tuning-operator/tuned-rx99l"
Sep 16 02:18:44.032048 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:18:44.032037    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/e34369f4-88ba-4859-ad03-80219c62e272-system-cni-dir\") pod \"multus-7nk9m\" (UID: \"e34369f4-88ba-4859-ad03-80219c62e272\") " pod="openshift-multus/multus-7nk9m"
Sep 16 02:18:44.032557 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:18:44.032055    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/e34369f4-88ba-4859-ad03-80219c62e272-multus-socket-dir-parent\") pod \"multus-7nk9m\" (UID: \"e34369f4-88ba-4859-ad03-80219c62e272\") " pod="openshift-multus/multus-7nk9m"
Sep 16 02:18:44.032557 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:18:44.032089    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kzksh\" (UniqueName: \"kubernetes.io/projected/0e5e9542-45a1-4c13-8868-6a73d01f381d-kube-api-access-kzksh\") pod \"iptables-alerter-lmvcp\" (UID: \"0e5e9542-45a1-4c13-8868-6a73d01f381d\") " pod="openshift-network-operator/iptables-alerter-lmvcp"
Sep 16 02:18:44.032557 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:18:44.032092    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/66c3846d-d0d2-4b46-8c59-f6d97a92f7ad-etc-sysctl-d\") pod \"tuned-rx99l\" (UID: \"66c3846d-d0d2-4b46-8c59-f6d97a92f7ad\") " pod="openshift-cluster-node-tuning-operator/tuned-rx99l"
Sep 16 02:18:44.032557 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:18:44.032150    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/d45dcef5-f6f3-45d7-b29e-1125a402eeb3-host-run-netns\") pod \"ovnkube-node-hgb4m\" (UID: \"d45dcef5-f6f3-45d7-b29e-1125a402eeb3\") " pod="openshift-ovn-kubernetes/ovnkube-node-hgb4m"
Sep 16 02:18:44.032557 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:18:44.032220    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/66c3846d-d0d2-4b46-8c59-f6d97a92f7ad-etc-systemd\") pod \"tuned-rx99l\" (UID: \"66c3846d-d0d2-4b46-8c59-f6d97a92f7ad\") " pod="openshift-cluster-node-tuning-operator/tuned-rx99l"
Sep 16 02:18:44.032557 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:18:44.032250    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/66c3846d-d0d2-4b46-8c59-f6d97a92f7ad-lib-modules\") pod \"tuned-rx99l\" (UID: \"66c3846d-d0d2-4b46-8c59-f6d97a92f7ad\") " pod="openshift-cluster-node-tuning-operator/tuned-rx99l"
Sep 16 02:18:44.032557 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:18:44.032263    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/82d1b4b2-7910-42f8-b2f3-0d675dc2c530-serviceca\") pod \"node-ca-5nwmf\" (UID: \"82d1b4b2-7910-42f8-b2f3-0d675dc2c530\") " pod="openshift-image-registry/node-ca-5nwmf"
Sep 16 02:18:44.032557 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:18:44.032292    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/d45dcef5-f6f3-45d7-b29e-1125a402eeb3-run-systemd\") pod \"ovnkube-node-hgb4m\" (UID: \"d45dcef5-f6f3-45d7-b29e-1125a402eeb3\") " pod="openshift-ovn-kubernetes/ovnkube-node-hgb4m"
Sep 16 02:18:44.032557 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:18:44.032318    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/d45dcef5-f6f3-45d7-b29e-1125a402eeb3-var-lib-openvswitch\") pod \"ovnkube-node-hgb4m\" (UID: \"d45dcef5-f6f3-45d7-b29e-1125a402eeb3\") " pod="openshift-ovn-kubernetes/ovnkube-node-hgb4m"
Sep 16 02:18:44.032557 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:18:44.032346    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d45dcef5-f6f3-45d7-b29e-1125a402eeb3-host-run-ovn-kubernetes\") pod \"ovnkube-node-hgb4m\" (UID: \"d45dcef5-f6f3-45d7-b29e-1125a402eeb3\") " pod="openshift-ovn-kubernetes/ovnkube-node-hgb4m"
Sep 16 02:18:44.032557 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:18:44.032382    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4cl2x\" (UniqueName: \"kubernetes.io/projected/66c3846d-d0d2-4b46-8c59-f6d97a92f7ad-kube-api-access-4cl2x\") pod \"tuned-rx99l\" (UID: \"66c3846d-d0d2-4b46-8c59-f6d97a92f7ad\") " pod="openshift-cluster-node-tuning-operator/tuned-rx99l"
Sep 16 02:18:44.032557 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:18:44.032407    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/e34369f4-88ba-4859-ad03-80219c62e272-hostroot\") pod \"multus-7nk9m\" (UID: \"e34369f4-88ba-4859-ad03-80219c62e272\") " pod="openshift-multus/multus-7nk9m"
Sep 16 02:18:44.032557 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:18:44.032432    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/b9f0b2c7-b0c8-49bf-9761-c387496f9207-etc-selinux\") pod \"aws-ebs-csi-driver-node-dqkz9\" (UID: \"b9f0b2c7-b0c8-49bf-9761-c387496f9207\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dqkz9"
Sep 16 02:18:44.032557 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:18:44.032461    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/d45dcef5-f6f3-45d7-b29e-1125a402eeb3-ovn-node-metrics-cert\") pod \"ovnkube-node-hgb4m\" (UID: \"d45dcef5-f6f3-45d7-b29e-1125a402eeb3\") " pod="openshift-ovn-kubernetes/ovnkube-node-hgb4m"
Sep 16 02:18:44.032557 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:18:44.032512    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/66c3846d-d0d2-4b46-8c59-f6d97a92f7ad-etc-sysctl-conf\") pod \"tuned-rx99l\" (UID: \"66c3846d-d0d2-4b46-8c59-f6d97a92f7ad\") " pod="openshift-cluster-node-tuning-operator/tuned-rx99l"
Sep 16 02:18:44.032557 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:18:44.032537    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/e34369f4-88ba-4859-ad03-80219c62e272-multus-daemon-config\") pod \"multus-7nk9m\" (UID: \"e34369f4-88ba-4859-ad03-80219c62e272\") " pod="openshift-multus/multus-7nk9m"
Sep 16 02:18:44.032557 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:18:44.032562    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/bdf274ad-3983-42ba-b712-8ce11da63210-agent-certs\") pod \"konnectivity-agent-psqcm\" (UID: \"bdf274ad-3983-42ba-b712-8ce11da63210\") " pod="kube-system/konnectivity-agent-psqcm"
Sep 16 02:18:44.033449 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:18:44.032587    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/bdf274ad-3983-42ba-b712-8ce11da63210-konnectivity-ca\") pod \"konnectivity-agent-psqcm\" (UID: \"bdf274ad-3983-42ba-b712-8ce11da63210\") " pod="kube-system/konnectivity-agent-psqcm"
Sep 16 02:18:44.033449 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:18:44.032617    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/b9f0b2c7-b0c8-49bf-9761-c387496f9207-socket-dir\") pod \"aws-ebs-csi-driver-node-dqkz9\" (UID: \"b9f0b2c7-b0c8-49bf-9761-c387496f9207\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dqkz9"
Sep 16 02:18:44.033449 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:18:44.032665    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/66c3846d-d0d2-4b46-8c59-f6d97a92f7ad-etc-kubernetes\") pod \"tuned-rx99l\" (UID: \"66c3846d-d0d2-4b46-8c59-f6d97a92f7ad\") " pod="openshift-cluster-node-tuning-operator/tuned-rx99l"
Sep 16 02:18:44.033449 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:18:44.032692    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/66c3846d-d0d2-4b46-8c59-f6d97a92f7ad-var-lib-kubelet\") pod \"tuned-rx99l\" (UID: \"66c3846d-d0d2-4b46-8c59-f6d97a92f7ad\") " pod="openshift-cluster-node-tuning-operator/tuned-rx99l"
Sep 16 02:18:44.033449 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:18:44.032732    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2c5d542f-5718-4aeb-8bf5-680a69ddecbc-tmp-dir\") pod \"node-resolver-kf6w7\" (UID: \"2c5d542f-5718-4aeb-8bf5-680a69ddecbc\") " pod="openshift-dns/node-resolver-kf6w7"
Sep 16 02:18:44.033449 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:18:44.032762    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/0e5e9542-45a1-4c13-8868-6a73d01f381d-iptables-alerter-script\") pod \"iptables-alerter-lmvcp\" (UID: \"0e5e9542-45a1-4c13-8868-6a73d01f381d\") " pod="openshift-network-operator/iptables-alerter-lmvcp"
Sep 16 02:18:44.033449 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:18:44.032795    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4k94z\" (UniqueName: \"kubernetes.io/projected/67a44fbe-95d7-4808-bcff-f7d214c6d82c-kube-api-access-4k94z\") pod \"network-metrics-daemon-bdwgr\" (UID: \"67a44fbe-95d7-4808-bcff-f7d214c6d82c\") " pod="openshift-multus/network-metrics-daemon-bdwgr"
Sep 16 02:18:44.033449 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:18:44.032820    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/e34369f4-88ba-4859-ad03-80219c62e272-host-run-netns\") pod \"multus-7nk9m\" (UID: \"e34369f4-88ba-4859-ad03-80219c62e272\") " pod="openshift-multus/multus-7nk9m"
Sep 16 02:18:44.033449 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:18:44.032846    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/e34369f4-88ba-4859-ad03-80219c62e272-multus-conf-dir\") pod \"multus-7nk9m\" (UID: \"e34369f4-88ba-4859-ad03-80219c62e272\") " pod="openshift-multus/multus-7nk9m"
Sep 16 02:18:44.033449 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:18:44.032845    2583 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Sep 16 02:18:44.033449 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:18:44.032872    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/d45dcef5-f6f3-45d7-b29e-1125a402eeb3-host-slash\") pod \"ovnkube-node-hgb4m\" (UID: \"d45dcef5-f6f3-45d7-b29e-1125a402eeb3\") " pod="openshift-ovn-kubernetes/ovnkube-node-hgb4m"
Sep 16 02:18:44.033449 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:18:44.032896    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/d45dcef5-f6f3-45d7-b29e-1125a402eeb3-etc-openvswitch\") pod \"ovnkube-node-hgb4m\" (UID: \"d45dcef5-f6f3-45d7-b29e-1125a402eeb3\") " pod="openshift-ovn-kubernetes/ovnkube-node-hgb4m"
Sep 16 02:18:44.033449 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:18:44.032919    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/2c5d542f-5718-4aeb-8bf5-680a69ddecbc-hosts-file\") pod \"node-resolver-kf6w7\" (UID: \"2c5d542f-5718-4aeb-8bf5-680a69ddecbc\") " pod="openshift-dns/node-resolver-kf6w7"
Sep 16 02:18:44.033449 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:18:44.032945    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x94vc\" (UniqueName: \"kubernetes.io/projected/2c5d542f-5718-4aeb-8bf5-680a69ddecbc-kube-api-access-x94vc\") pod \"node-resolver-kf6w7\" (UID: \"2c5d542f-5718-4aeb-8bf5-680a69ddecbc\") " pod="openshift-dns/node-resolver-kf6w7"
Sep 16 02:18:44.033449 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:18:44.032969    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/b9f0b2c7-b0c8-49bf-9761-c387496f9207-device-dir\") pod \"aws-ebs-csi-driver-node-dqkz9\" (UID: \"b9f0b2c7-b0c8-49bf-9761-c387496f9207\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dqkz9"
Sep 16 02:18:44.033449 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:18:44.032992    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/3a1ad832-d1ab-4676-8eee-31d2eccce648-cnibin\") pod \"multus-additional-cni-plugins-ll7tp\" (UID: \"3a1ad832-d1ab-4676-8eee-31d2eccce648\") " pod="openshift-multus/multus-additional-cni-plugins-ll7tp"
Sep 16 02:18:44.033449 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:18:44.033017    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lhw4b\" (UniqueName: \"kubernetes.io/projected/3a1ad832-d1ab-4676-8eee-31d2eccce648-kube-api-access-lhw4b\") pod \"multus-additional-cni-plugins-ll7tp\" (UID: \"3a1ad832-d1ab-4676-8eee-31d2eccce648\") " pod="openshift-multus/multus-additional-cni-plugins-ll7tp"
Sep 16 02:18:44.034227 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:18:44.033043    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/e34369f4-88ba-4859-ad03-80219c62e272-multus-cni-dir\") pod \"multus-7nk9m\" (UID: \"e34369f4-88ba-4859-ad03-80219c62e272\") " pod="openshift-multus/multus-7nk9m"
Sep 16 02:18:44.034227 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:18:44.033067    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/e34369f4-88ba-4859-ad03-80219c62e272-host-var-lib-kubelet\") pod \"multus-7nk9m\" (UID: \"e34369f4-88ba-4859-ad03-80219c62e272\") " pod="openshift-multus/multus-7nk9m"
Sep 16 02:18:44.034227 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:18:44.033105    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/d45dcef5-f6f3-45d7-b29e-1125a402eeb3-ovnkube-script-lib\") pod \"ovnkube-node-hgb4m\" (UID: \"d45dcef5-f6f3-45d7-b29e-1125a402eeb3\") " pod="openshift-ovn-kubernetes/ovnkube-node-hgb4m"
Sep 16 02:18:44.034227 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:18:44.033131    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cd96q\" (UniqueName: \"kubernetes.io/projected/d45dcef5-f6f3-45d7-b29e-1125a402eeb3-kube-api-access-cd96q\") pod \"ovnkube-node-hgb4m\" (UID: \"d45dcef5-f6f3-45d7-b29e-1125a402eeb3\") " pod="openshift-ovn-kubernetes/ovnkube-node-hgb4m"
Sep 16 02:18:44.034227 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:18:44.033162    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/d45dcef5-f6f3-45d7-b29e-1125a402eeb3-log-socket\") pod \"ovnkube-node-hgb4m\" (UID: \"d45dcef5-f6f3-45d7-b29e-1125a402eeb3\") " pod="openshift-ovn-kubernetes/ovnkube-node-hgb4m"
Sep 16 02:18:44.034227 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:18:44.033186    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/d45dcef5-f6f3-45d7-b29e-1125a402eeb3-host-cni-netd\") pod \"ovnkube-node-hgb4m\" (UID: \"d45dcef5-f6f3-45d7-b29e-1125a402eeb3\") " pod="openshift-ovn-kubernetes/ovnkube-node-hgb4m"
Sep 16 02:18:44.034227 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:18:44.033210    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/e34369f4-88ba-4859-ad03-80219c62e272-cni-binary-copy\") pod \"multus-7nk9m\" (UID: \"e34369f4-88ba-4859-ad03-80219c62e272\") " pod="openshift-multus/multus-7nk9m"
Sep 16 02:18:44.034227 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:18:44.033235    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/2fbf440b-4c09-428e-b6bf-1236d95f97cc-kubelet-config\") pod \"global-pull-secret-syncer-mbxmd\" (UID: \"2fbf440b-4c09-428e-b6bf-1236d95f97cc\") " pod="kube-system/global-pull-secret-syncer-mbxmd"
Sep 16 02:18:44.034227 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:18:44.033283    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qwztf\" (UniqueName: \"kubernetes.io/projected/5f66bb1b-6d4f-42c5-9b63-f7f5f301b6f9-kube-api-access-qwztf\") pod \"network-check-target-55zcf\" (UID: \"5f66bb1b-6d4f-42c5-9b63-f7f5f301b6f9\") " pod="openshift-network-diagnostics/network-check-target-55zcf"
Sep 16 02:18:44.034227 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:18:44.033309    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5xc9z\" (UniqueName: \"kubernetes.io/projected/e34369f4-88ba-4859-ad03-80219c62e272-kube-api-access-5xc9z\") pod \"multus-7nk9m\" (UID: \"e34369f4-88ba-4859-ad03-80219c62e272\") " pod="openshift-multus/multus-7nk9m"
Sep 16 02:18:44.034227 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:18:44.033331    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/82d1b4b2-7910-42f8-b2f3-0d675dc2c530-host\") pod \"node-ca-5nwmf\" (UID: \"82d1b4b2-7910-42f8-b2f3-0d675dc2c530\") " pod="openshift-image-registry/node-ca-5nwmf"
Sep 16 02:18:44.034227 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:18:44.033390    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/3a1ad832-d1ab-4676-8eee-31d2eccce648-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-ll7tp\" (UID: \"3a1ad832-d1ab-4676-8eee-31d2eccce648\") " pod="openshift-multus/multus-additional-cni-plugins-ll7tp"
Sep 16 02:18:44.034227 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:18:44.033424    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/d45dcef5-f6f3-45d7-b29e-1125a402eeb3-systemd-units\") pod \"ovnkube-node-hgb4m\" (UID: \"d45dcef5-f6f3-45d7-b29e-1125a402eeb3\") " pod="openshift-ovn-kubernetes/ovnkube-node-hgb4m"
Sep 16 02:18:44.034227 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:18:44.033450    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/d45dcef5-f6f3-45d7-b29e-1125a402eeb3-run-openvswitch\") pod \"ovnkube-node-hgb4m\" (UID: \"d45dcef5-f6f3-45d7-b29e-1125a402eeb3\") " pod="openshift-ovn-kubernetes/ovnkube-node-hgb4m"
Sep 16 02:18:44.034227 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:18:44.033476    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/d45dcef5-f6f3-45d7-b29e-1125a402eeb3-host-cni-bin\") pod \"ovnkube-node-hgb4m\" (UID: \"d45dcef5-f6f3-45d7-b29e-1125a402eeb3\") " pod="openshift-ovn-kubernetes/ovnkube-node-hgb4m"
Sep 16 02:18:44.034227 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:18:44.033498    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/66c3846d-d0d2-4b46-8c59-f6d97a92f7ad-etc-tuned\") pod \"tuned-rx99l\" (UID: \"66c3846d-d0d2-4b46-8c59-f6d97a92f7ad\") " pod="openshift-cluster-node-tuning-operator/tuned-rx99l"
Sep 16 02:18:44.034227 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:18:44.033525    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/e34369f4-88ba-4859-ad03-80219c62e272-host-run-multus-certs\") pod \"multus-7nk9m\" (UID: \"e34369f4-88ba-4859-ad03-80219c62e272\") " pod="openshift-multus/multus-7nk9m"
Sep 16 02:18:44.034870 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:18:44.033550    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/0e5e9542-45a1-4c13-8868-6a73d01f381d-host-slash\") pod \"iptables-alerter-lmvcp\" (UID: \"0e5e9542-45a1-4c13-8868-6a73d01f381d\") " pod="openshift-network-operator/iptables-alerter-lmvcp"
Sep 16 02:18:44.034870 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:18:44.033574    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/3a1ad832-d1ab-4676-8eee-31d2eccce648-system-cni-dir\") pod \"multus-additional-cni-plugins-ll7tp\" (UID: \"3a1ad832-d1ab-4676-8eee-31d2eccce648\") " pod="openshift-multus/multus-additional-cni-plugins-ll7tp"
Sep 16 02:18:44.034870 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:18:44.033604    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/67a44fbe-95d7-4808-bcff-f7d214c6d82c-metrics-certs\") pod \"network-metrics-daemon-bdwgr\" (UID: \"67a44fbe-95d7-4808-bcff-f7d214c6d82c\") " pod="openshift-multus/network-metrics-daemon-bdwgr"
Sep 16 02:18:44.034870 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:18:44.033645    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/d45dcef5-f6f3-45d7-b29e-1125a402eeb3-host-kubelet\") pod \"ovnkube-node-hgb4m\" (UID: \"d45dcef5-f6f3-45d7-b29e-1125a402eeb3\") " pod="openshift-ovn-kubernetes/ovnkube-node-hgb4m"
Sep 16 02:18:44.034870 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:18:44.033672    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/e34369f4-88ba-4859-ad03-80219c62e272-cnibin\") pod \"multus-7nk9m\" (UID: \"e34369f4-88ba-4859-ad03-80219c62e272\") " pod="openshift-multus/multus-7nk9m"
Sep 16 02:18:44.034870 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:18:44.033769    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/d45dcef5-f6f3-45d7-b29e-1125a402eeb3-run-systemd\") pod \"ovnkube-node-hgb4m\" (UID: \"d45dcef5-f6f3-45d7-b29e-1125a402eeb3\") " pod="openshift-ovn-kubernetes/ovnkube-node-hgb4m"
Sep 16 02:18:44.034870 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:18:44.033813    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/d45dcef5-f6f3-45d7-b29e-1125a402eeb3-var-lib-openvswitch\") pod \"ovnkube-node-hgb4m\" (UID: \"d45dcef5-f6f3-45d7-b29e-1125a402eeb3\") " pod="openshift-ovn-kubernetes/ovnkube-node-hgb4m"
Sep 16 02:18:44.034870 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:18:44.033853    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d45dcef5-f6f3-45d7-b29e-1125a402eeb3-host-run-ovn-kubernetes\") pod \"ovnkube-node-hgb4m\" (UID: \"d45dcef5-f6f3-45d7-b29e-1125a402eeb3\") " pod="openshift-ovn-kubernetes/ovnkube-node-hgb4m"
Sep 16 02:18:44.037829 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:18:44.036425    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/2c5d542f-5718-4aeb-8bf5-680a69ddecbc-hosts-file\") pod \"node-resolver-kf6w7\" (UID: \"2c5d542f-5718-4aeb-8bf5-680a69ddecbc\") " pod="openshift-dns/node-resolver-kf6w7"
Sep 16 02:18:44.037829 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:18:44.036531    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/d45dcef5-f6f3-45d7-b29e-1125a402eeb3-host-slash\") pod \"ovnkube-node-hgb4m\" (UID: \"d45dcef5-f6f3-45d7-b29e-1125a402eeb3\") " pod="openshift-ovn-kubernetes/ovnkube-node-hgb4m"
Sep 16 02:18:44.037829 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:18:44.036548    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/66c3846d-d0d2-4b46-8c59-f6d97a92f7ad-etc-sysctl-conf\") pod \"tuned-rx99l\" (UID: \"66c3846d-d0d2-4b46-8c59-f6d97a92f7ad\") " pod="openshift-cluster-node-tuning-operator/tuned-rx99l"
Sep 16 02:18:44.037829 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:18:44.036610    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/d45dcef5-f6f3-45d7-b29e-1125a402eeb3-etc-openvswitch\") pod \"ovnkube-node-hgb4m\" (UID: \"d45dcef5-f6f3-45d7-b29e-1125a402eeb3\") " pod="openshift-ovn-kubernetes/ovnkube-node-hgb4m"
Sep 16 02:18:44.038849 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:18:44.037996    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/d45dcef5-f6f3-45d7-b29e-1125a402eeb3-systemd-units\") pod \"ovnkube-node-hgb4m\" (UID: \"d45dcef5-f6f3-45d7-b29e-1125a402eeb3\") " pod="openshift-ovn-kubernetes/ovnkube-node-hgb4m"
Sep 16 02:18:44.038849 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:18:44.038057    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/d45dcef5-f6f3-45d7-b29e-1125a402eeb3-run-openvswitch\") pod \"ovnkube-node-hgb4m\" (UID: \"d45dcef5-f6f3-45d7-b29e-1125a402eeb3\") " pod="openshift-ovn-kubernetes/ovnkube-node-hgb4m"
Sep 16 02:18:44.038849 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:18:44.038205    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/66c3846d-d0d2-4b46-8c59-f6d97a92f7ad-etc-kubernetes\") pod \"tuned-rx99l\" (UID: \"66c3846d-d0d2-4b46-8c59-f6d97a92f7ad\") " pod="openshift-cluster-node-tuning-operator/tuned-rx99l"
Sep 16 02:18:44.038849 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:18:44.038215    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/66c3846d-d0d2-4b46-8c59-f6d97a92f7ad-var-lib-kubelet\") pod \"tuned-rx99l\" (UID: \"66c3846d-d0d2-4b46-8c59-f6d97a92f7ad\") " pod="openshift-cluster-node-tuning-operator/tuned-rx99l"
Sep 16 02:18:44.038849 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:18:44.038492    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2c5d542f-5718-4aeb-8bf5-680a69ddecbc-tmp-dir\") pod \"node-resolver-kf6w7\" (UID: \"2c5d542f-5718-4aeb-8bf5-680a69ddecbc\") " pod="openshift-dns/node-resolver-kf6w7"
Sep 16 02:18:44.039109 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:18:44.038920    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/d45dcef5-f6f3-45d7-b29e-1125a402eeb3-host-cni-netd\") pod \"ovnkube-node-hgb4m\" (UID: \"d45dcef5-f6f3-45d7-b29e-1125a402eeb3\") " pod="openshift-ovn-kubernetes/ovnkube-node-hgb4m"
Sep 16 02:18:44.039109 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:18:44.038969    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/d45dcef5-f6f3-45d7-b29e-1125a402eeb3-host-cni-bin\") pod \"ovnkube-node-hgb4m\" (UID: \"d45dcef5-f6f3-45d7-b29e-1125a402eeb3\") " pod="openshift-ovn-kubernetes/ovnkube-node-hgb4m"
Sep 16 02:18:44.039109 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:18:44.039009    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/d45dcef5-f6f3-45d7-b29e-1125a402eeb3-log-socket\") pod \"ovnkube-node-hgb4m\" (UID: \"d45dcef5-f6f3-45d7-b29e-1125a402eeb3\") " pod="openshift-ovn-kubernetes/ovnkube-node-hgb4m"
Sep 16 02:18:44.039109 ip-10-0-128-111 kubenswrapper[2583]: E0916 02:18:44.039085    2583 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 02:18:44.039336 ip-10-0-128-111 kubenswrapper[2583]: E0916 02:18:44.039165    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/67a44fbe-95d7-4808-bcff-f7d214c6d82c-metrics-certs podName:67a44fbe-95d7-4808-bcff-f7d214c6d82c nodeName:}" failed. No retries permitted until 2026-09-16 02:18:44.539144118 +0000 UTC m=+3.006932900 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/67a44fbe-95d7-4808-bcff-f7d214c6d82c-metrics-certs") pod "network-metrics-daemon-bdwgr" (UID: "67a44fbe-95d7-4808-bcff-f7d214c6d82c") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 02:18:44.042128 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:18:44.042063    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/66c3846d-d0d2-4b46-8c59-f6d97a92f7ad-etc-tuned\") pod \"tuned-rx99l\" (UID: \"66c3846d-d0d2-4b46-8c59-f6d97a92f7ad\") " pod="openshift-cluster-node-tuning-operator/tuned-rx99l"
Sep 16 02:18:44.043562 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:18:44.042513    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4cl2x\" (UniqueName: \"kubernetes.io/projected/66c3846d-d0d2-4b46-8c59-f6d97a92f7ad-kube-api-access-4cl2x\") pod \"tuned-rx99l\" (UID: \"66c3846d-d0d2-4b46-8c59-f6d97a92f7ad\") " pod="openshift-cluster-node-tuning-operator/tuned-rx99l"
Sep 16 02:18:44.043562 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:18:44.042914    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/d45dcef5-f6f3-45d7-b29e-1125a402eeb3-host-kubelet\") pod \"ovnkube-node-hgb4m\" (UID: \"d45dcef5-f6f3-45d7-b29e-1125a402eeb3\") " pod="openshift-ovn-kubernetes/ovnkube-node-hgb4m"
Sep 16 02:18:44.043562 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:18:44.043173    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/d45dcef5-f6f3-45d7-b29e-1125a402eeb3-ovnkube-script-lib\") pod \"ovnkube-node-hgb4m\" (UID: \"d45dcef5-f6f3-45d7-b29e-1125a402eeb3\") " pod="openshift-ovn-kubernetes/ovnkube-node-hgb4m"
Sep 16 02:18:44.044401 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:18:44.044167    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/66c3846d-d0d2-4b46-8c59-f6d97a92f7ad-tmp\") pod \"tuned-rx99l\" (UID: \"66c3846d-d0d2-4b46-8c59-f6d97a92f7ad\") " pod="openshift-cluster-node-tuning-operator/tuned-rx99l"
Sep 16 02:18:44.044401 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:18:44.044355    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4k94z\" (UniqueName: \"kubernetes.io/projected/67a44fbe-95d7-4808-bcff-f7d214c6d82c-kube-api-access-4k94z\") pod \"network-metrics-daemon-bdwgr\" (UID: \"67a44fbe-95d7-4808-bcff-f7d214c6d82c\") " pod="openshift-multus/network-metrics-daemon-bdwgr"
Sep 16 02:18:44.046364 ip-10-0-128-111 kubenswrapper[2583]: E0916 02:18:44.045505    2583 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 16 02:18:44.046364 ip-10-0-128-111 kubenswrapper[2583]: E0916 02:18:44.045545    2583 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 16 02:18:44.046364 ip-10-0-128-111 kubenswrapper[2583]: E0916 02:18:44.045563    2583 projected.go:196] Error preparing data for projected volume kube-api-access-qwztf for pod openshift-network-diagnostics/network-check-target-55zcf: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 02:18:44.046364 ip-10-0-128-111 kubenswrapper[2583]: E0916 02:18:44.045650    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/5f66bb1b-6d4f-42c5-9b63-f7f5f301b6f9-kube-api-access-qwztf podName:5f66bb1b-6d4f-42c5-9b63-f7f5f301b6f9 nodeName:}" failed. No retries permitted until 2026-09-16 02:18:44.545630674 +0000 UTC m=+3.013419460 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-qwztf" (UniqueName: "kubernetes.io/projected/5f66bb1b-6d4f-42c5-9b63-f7f5f301b6f9-kube-api-access-qwztf") pod "network-check-target-55zcf" (UID: "5f66bb1b-6d4f-42c5-9b63-f7f5f301b6f9") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 02:18:44.046364 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:18:44.045652    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/d45dcef5-f6f3-45d7-b29e-1125a402eeb3-ovn-node-metrics-cert\") pod \"ovnkube-node-hgb4m\" (UID: \"d45dcef5-f6f3-45d7-b29e-1125a402eeb3\") " pod="openshift-ovn-kubernetes/ovnkube-node-hgb4m"
Sep 16 02:18:44.047537 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:18:44.047480    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-111.ec2.internal" event={"ID":"b7568fa8cea4626b154e4750bd44461f","Type":"ContainerStarted","Data":"097e7632dfd1eb8f93b795d5ae0f6a398a2cf5caac1e3995aef9d75acaca710b"}
Sep 16 02:18:44.047656 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:18:44.047601    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-x94vc\" (UniqueName: \"kubernetes.io/projected/2c5d542f-5718-4aeb-8bf5-680a69ddecbc-kube-api-access-x94vc\") pod \"node-resolver-kf6w7\" (UID: \"2c5d542f-5718-4aeb-8bf5-680a69ddecbc\") " pod="openshift-dns/node-resolver-kf6w7"
Sep 16 02:18:44.048934 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:18:44.048910    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cd96q\" (UniqueName: \"kubernetes.io/projected/d45dcef5-f6f3-45d7-b29e-1125a402eeb3-kube-api-access-cd96q\") pod \"ovnkube-node-hgb4m\" (UID: \"d45dcef5-f6f3-45d7-b29e-1125a402eeb3\") " pod="openshift-ovn-kubernetes/ovnkube-node-hgb4m"
Sep 16 02:18:44.049127 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:18:44.049101    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-111.ec2.internal" event={"ID":"ac09b6665582e7a386f06843c363096b","Type":"ContainerStarted","Data":"566d16305d110fce7369ca972fcf287e420e20284da1922790c3306ac7c0d7c9"}
Sep 16 02:18:44.134770 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:18:44.134737    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/b9f0b2c7-b0c8-49bf-9761-c387496f9207-socket-dir\") pod \"aws-ebs-csi-driver-node-dqkz9\" (UID: \"b9f0b2c7-b0c8-49bf-9761-c387496f9207\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dqkz9"
Sep 16 02:18:44.135001 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:18:44.134777    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/0e5e9542-45a1-4c13-8868-6a73d01f381d-iptables-alerter-script\") pod \"iptables-alerter-lmvcp\" (UID: \"0e5e9542-45a1-4c13-8868-6a73d01f381d\") " pod="openshift-network-operator/iptables-alerter-lmvcp"
Sep 16 02:18:44.135001 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:18:44.134797    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/e34369f4-88ba-4859-ad03-80219c62e272-host-run-netns\") pod \"multus-7nk9m\" (UID: \"e34369f4-88ba-4859-ad03-80219c62e272\") " pod="openshift-multus/multus-7nk9m"
Sep 16 02:18:44.135001 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:18:44.134820    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/e34369f4-88ba-4859-ad03-80219c62e272-multus-conf-dir\") pod \"multus-7nk9m\" (UID: \"e34369f4-88ba-4859-ad03-80219c62e272\") " pod="openshift-multus/multus-7nk9m"
Sep 16 02:18:44.135001 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:18:44.134847    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/b9f0b2c7-b0c8-49bf-9761-c387496f9207-device-dir\") pod \"aws-ebs-csi-driver-node-dqkz9\" (UID: \"b9f0b2c7-b0c8-49bf-9761-c387496f9207\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dqkz9"
Sep 16 02:18:44.135001 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:18:44.134878    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/3a1ad832-d1ab-4676-8eee-31d2eccce648-cnibin\") pod \"multus-additional-cni-plugins-ll7tp\" (UID: \"3a1ad832-d1ab-4676-8eee-31d2eccce648\") " pod="openshift-multus/multus-additional-cni-plugins-ll7tp"
Sep 16 02:18:44.135001 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:18:44.134903    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lhw4b\" (UniqueName: \"kubernetes.io/projected/3a1ad832-d1ab-4676-8eee-31d2eccce648-kube-api-access-lhw4b\") pod \"multus-additional-cni-plugins-ll7tp\" (UID: \"3a1ad832-d1ab-4676-8eee-31d2eccce648\") " pod="openshift-multus/multus-additional-cni-plugins-ll7tp"
Sep 16 02:18:44.135001 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:18:44.134934    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/e34369f4-88ba-4859-ad03-80219c62e272-multus-cni-dir\") pod \"multus-7nk9m\" (UID: \"e34369f4-88ba-4859-ad03-80219c62e272\") " pod="openshift-multus/multus-7nk9m"
Sep 16 02:18:44.135001 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:18:44.134951    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/b9f0b2c7-b0c8-49bf-9761-c387496f9207-socket-dir\") pod \"aws-ebs-csi-driver-node-dqkz9\" (UID: \"b9f0b2c7-b0c8-49bf-9761-c387496f9207\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dqkz9"
Sep 16 02:18:44.135001 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:18:44.134962    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/e34369f4-88ba-4859-ad03-80219c62e272-multus-conf-dir\") pod \"multus-7nk9m\" (UID: \"e34369f4-88ba-4859-ad03-80219c62e272\") " pod="openshift-multus/multus-7nk9m"
Sep 16 02:18:44.135001 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:18:44.134960    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/e34369f4-88ba-4859-ad03-80219c62e272-host-var-lib-kubelet\") pod \"multus-7nk9m\" (UID: \"e34369f4-88ba-4859-ad03-80219c62e272\") " pod="openshift-multus/multus-7nk9m"
Sep 16 02:18:44.135001 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:18:44.135003    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/e34369f4-88ba-4859-ad03-80219c62e272-host-var-lib-kubelet\") pod \"multus-7nk9m\" (UID: \"e34369f4-88ba-4859-ad03-80219c62e272\") " pod="openshift-multus/multus-7nk9m"
Sep 16 02:18:44.135597 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:18:44.135024    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/3a1ad832-d1ab-4676-8eee-31d2eccce648-cnibin\") pod \"multus-additional-cni-plugins-ll7tp\" (UID: \"3a1ad832-d1ab-4676-8eee-31d2eccce648\") " pod="openshift-multus/multus-additional-cni-plugins-ll7tp"
Sep 16 02:18:44.135597 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:18:44.135031    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/e34369f4-88ba-4859-ad03-80219c62e272-cni-binary-copy\") pod \"multus-7nk9m\" (UID: \"e34369f4-88ba-4859-ad03-80219c62e272\") " pod="openshift-multus/multus-7nk9m"
Sep 16 02:18:44.135597 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:18:44.135066    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/b9f0b2c7-b0c8-49bf-9761-c387496f9207-device-dir\") pod \"aws-ebs-csi-driver-node-dqkz9\" (UID: \"b9f0b2c7-b0c8-49bf-9761-c387496f9207\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dqkz9"
Sep 16 02:18:44.135597 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:18:44.135061    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/2fbf440b-4c09-428e-b6bf-1236d95f97cc-kubelet-config\") pod \"global-pull-secret-syncer-mbxmd\" (UID: \"2fbf440b-4c09-428e-b6bf-1236d95f97cc\") " pod="kube-system/global-pull-secret-syncer-mbxmd"
Sep 16 02:18:44.135597 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:18:44.135111    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5xc9z\" (UniqueName: \"kubernetes.io/projected/e34369f4-88ba-4859-ad03-80219c62e272-kube-api-access-5xc9z\") pod \"multus-7nk9m\" (UID: \"e34369f4-88ba-4859-ad03-80219c62e272\") " pod="openshift-multus/multus-7nk9m"
Sep 16 02:18:44.135597 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:18:44.135128    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/2fbf440b-4c09-428e-b6bf-1236d95f97cc-kubelet-config\") pod \"global-pull-secret-syncer-mbxmd\" (UID: \"2fbf440b-4c09-428e-b6bf-1236d95f97cc\") " pod="kube-system/global-pull-secret-syncer-mbxmd"
Sep 16 02:18:44.135597 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:18:44.135136    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/82d1b4b2-7910-42f8-b2f3-0d675dc2c530-host\") pod \"node-ca-5nwmf\" (UID: \"82d1b4b2-7910-42f8-b2f3-0d675dc2c530\") " pod="openshift-image-registry/node-ca-5nwmf"
Sep 16 02:18:44.135597 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:18:44.135164    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/3a1ad832-d1ab-4676-8eee-31d2eccce648-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-ll7tp\" (UID: \"3a1ad832-d1ab-4676-8eee-31d2eccce648\") " pod="openshift-multus/multus-additional-cni-plugins-ll7tp"
Sep 16 02:18:44.135597 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:18:44.135192    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/e34369f4-88ba-4859-ad03-80219c62e272-host-run-netns\") pod \"multus-7nk9m\" (UID: \"e34369f4-88ba-4859-ad03-80219c62e272\") " pod="openshift-multus/multus-7nk9m"
Sep 16 02:18:44.135597 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:18:44.135211    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/e34369f4-88ba-4859-ad03-80219c62e272-host-run-multus-certs\") pod \"multus-7nk9m\" (UID: \"e34369f4-88ba-4859-ad03-80219c62e272\") " pod="openshift-multus/multus-7nk9m"
Sep 16 02:18:44.135597 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:18:44.135238    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/0e5e9542-45a1-4c13-8868-6a73d01f381d-host-slash\") pod \"iptables-alerter-lmvcp\" (UID: \"0e5e9542-45a1-4c13-8868-6a73d01f381d\") " pod="openshift-network-operator/iptables-alerter-lmvcp"
Sep 16 02:18:44.135597 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:18:44.135311    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/e34369f4-88ba-4859-ad03-80219c62e272-multus-cni-dir\") pod \"multus-7nk9m\" (UID: \"e34369f4-88ba-4859-ad03-80219c62e272\") " pod="openshift-multus/multus-7nk9m"
Sep 16 02:18:44.135597 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:18:44.135360    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/0e5e9542-45a1-4c13-8868-6a73d01f381d-host-slash\") pod \"iptables-alerter-lmvcp\" (UID: \"0e5e9542-45a1-4c13-8868-6a73d01f381d\") " pod="openshift-network-operator/iptables-alerter-lmvcp"
Sep 16 02:18:44.135597 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:18:44.135391    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/3a1ad832-d1ab-4676-8eee-31d2eccce648-system-cni-dir\") pod \"multus-additional-cni-plugins-ll7tp\" (UID: \"3a1ad832-d1ab-4676-8eee-31d2eccce648\") " pod="openshift-multus/multus-additional-cni-plugins-ll7tp"
Sep 16 02:18:44.135597 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:18:44.135430    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/0e5e9542-45a1-4c13-8868-6a73d01f381d-iptables-alerter-script\") pod \"iptables-alerter-lmvcp\" (UID: \"0e5e9542-45a1-4c13-8868-6a73d01f381d\") " pod="openshift-network-operator/iptables-alerter-lmvcp"
Sep 16 02:18:44.135597 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:18:44.135433    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/e34369f4-88ba-4859-ad03-80219c62e272-cnibin\") pod \"multus-7nk9m\" (UID: \"e34369f4-88ba-4859-ad03-80219c62e272\") " pod="openshift-multus/multus-7nk9m"
Sep 16 02:18:44.135597 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:18:44.135472    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/e34369f4-88ba-4859-ad03-80219c62e272-cnibin\") pod \"multus-7nk9m\" (UID: \"e34369f4-88ba-4859-ad03-80219c62e272\") " pod="openshift-multus/multus-7nk9m"
Sep 16 02:18:44.135597 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:18:44.135486    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/e34369f4-88ba-4859-ad03-80219c62e272-host-run-k8s-cni-cncf-io\") pod \"multus-7nk9m\" (UID: \"e34369f4-88ba-4859-ad03-80219c62e272\") " pod="openshift-multus/multus-7nk9m"
Sep 16 02:18:44.136528 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:18:44.135494    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/82d1b4b2-7910-42f8-b2f3-0d675dc2c530-host\") pod \"node-ca-5nwmf\" (UID: \"82d1b4b2-7910-42f8-b2f3-0d675dc2c530\") " pod="openshift-image-registry/node-ca-5nwmf"
Sep 16 02:18:44.136528 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:18:44.135511    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/3a1ad832-d1ab-4676-8eee-31d2eccce648-system-cni-dir\") pod \"multus-additional-cni-plugins-ll7tp\" (UID: \"3a1ad832-d1ab-4676-8eee-31d2eccce648\") " pod="openshift-multus/multus-additional-cni-plugins-ll7tp"
Sep 16 02:18:44.136528 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:18:44.135514    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/3a1ad832-d1ab-4676-8eee-31d2eccce648-tuning-conf-dir\") pod \"multus-additional-cni-plugins-ll7tp\" (UID: \"3a1ad832-d1ab-4676-8eee-31d2eccce648\") " pod="openshift-multus/multus-additional-cni-plugins-ll7tp"
Sep 16 02:18:44.136528 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:18:44.135544    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/b9f0b2c7-b0c8-49bf-9761-c387496f9207-kubelet-dir\") pod \"aws-ebs-csi-driver-node-dqkz9\" (UID: \"b9f0b2c7-b0c8-49bf-9761-c387496f9207\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dqkz9"
Sep 16 02:18:44.136528 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:18:44.135551    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/e34369f4-88ba-4859-ad03-80219c62e272-host-run-multus-certs\") pod \"multus-7nk9m\" (UID: \"e34369f4-88ba-4859-ad03-80219c62e272\") " pod="openshift-multus/multus-7nk9m"
Sep 16 02:18:44.136528 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:18:44.135571    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vkdx2\" (UniqueName: \"kubernetes.io/projected/b9f0b2c7-b0c8-49bf-9761-c387496f9207-kube-api-access-vkdx2\") pod \"aws-ebs-csi-driver-node-dqkz9\" (UID: \"b9f0b2c7-b0c8-49bf-9761-c387496f9207\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dqkz9"
Sep 16 02:18:44.136528 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:18:44.135598    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/2fbf440b-4c09-428e-b6bf-1236d95f97cc-dbus\") pod \"global-pull-secret-syncer-mbxmd\" (UID: \"2fbf440b-4c09-428e-b6bf-1236d95f97cc\") " pod="kube-system/global-pull-secret-syncer-mbxmd"
Sep 16 02:18:44.136528 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:18:44.135653    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/b9f0b2c7-b0c8-49bf-9761-c387496f9207-registration-dir\") pod \"aws-ebs-csi-driver-node-dqkz9\" (UID: \"b9f0b2c7-b0c8-49bf-9761-c387496f9207\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dqkz9"
Sep 16 02:18:44.136528 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:18:44.135678    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hznzt\" (UniqueName: \"kubernetes.io/projected/82d1b4b2-7910-42f8-b2f3-0d675dc2c530-kube-api-access-hznzt\") pod \"node-ca-5nwmf\" (UID: \"82d1b4b2-7910-42f8-b2f3-0d675dc2c530\") " pod="openshift-image-registry/node-ca-5nwmf"
Sep 16 02:18:44.136528 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:18:44.135703    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/3a1ad832-d1ab-4676-8eee-31d2eccce648-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-ll7tp\" (UID: \"3a1ad832-d1ab-4676-8eee-31d2eccce648\") " pod="openshift-multus/multus-additional-cni-plugins-ll7tp"
Sep 16 02:18:44.136528 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:18:44.135736    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/3a1ad832-d1ab-4676-8eee-31d2eccce648-cni-binary-copy\") pod \"multus-additional-cni-plugins-ll7tp\" (UID: \"3a1ad832-d1ab-4676-8eee-31d2eccce648\") " pod="openshift-multus/multus-additional-cni-plugins-ll7tp"
Sep 16 02:18:44.136528 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:18:44.135759    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/3a1ad832-d1ab-4676-8eee-31d2eccce648-tuning-conf-dir\") pod \"multus-additional-cni-plugins-ll7tp\" (UID: \"3a1ad832-d1ab-4676-8eee-31d2eccce648\") " pod="openshift-multus/multus-additional-cni-plugins-ll7tp"
Sep 16 02:18:44.136528 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:18:44.135762    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/2fbf440b-4c09-428e-b6bf-1236d95f97cc-original-pull-secret\") pod \"global-pull-secret-syncer-mbxmd\" (UID: \"2fbf440b-4c09-428e-b6bf-1236d95f97cc\") " pod="kube-system/global-pull-secret-syncer-mbxmd"
Sep 16 02:18:44.136528 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:18:44.135809    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/e34369f4-88ba-4859-ad03-80219c62e272-host-var-lib-cni-bin\") pod \"multus-7nk9m\" (UID: \"e34369f4-88ba-4859-ad03-80219c62e272\") " pod="openshift-multus/multus-7nk9m"
Sep 16 02:18:44.136528 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:18:44.135836    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/e34369f4-88ba-4859-ad03-80219c62e272-host-var-lib-cni-multus\") pod \"multus-7nk9m\" (UID: \"e34369f4-88ba-4859-ad03-80219c62e272\") " pod="openshift-multus/multus-7nk9m"
Sep 16 02:18:44.136528 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:18:44.135851    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/e34369f4-88ba-4859-ad03-80219c62e272-cni-binary-copy\") pod \"multus-7nk9m\" (UID: \"e34369f4-88ba-4859-ad03-80219c62e272\") " pod="openshift-multus/multus-7nk9m"
Sep 16 02:18:44.136528 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:18:44.135861    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e34369f4-88ba-4859-ad03-80219c62e272-etc-kubernetes\") pod \"multus-7nk9m\" (UID: \"e34369f4-88ba-4859-ad03-80219c62e272\") " pod="openshift-multus/multus-7nk9m"
Sep 16 02:18:44.137409 ip-10-0-128-111 kubenswrapper[2583]: E0916 02:18:44.135858    2583 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 16 02:18:44.137409 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:18:44.135887    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/b9f0b2c7-b0c8-49bf-9761-c387496f9207-sys-fs\") pod \"aws-ebs-csi-driver-node-dqkz9\" (UID: \"b9f0b2c7-b0c8-49bf-9761-c387496f9207\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dqkz9"
Sep 16 02:18:44.137409 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:18:44.135908    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/b9f0b2c7-b0c8-49bf-9761-c387496f9207-kubelet-dir\") pod \"aws-ebs-csi-driver-node-dqkz9\" (UID: \"b9f0b2c7-b0c8-49bf-9761-c387496f9207\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dqkz9"
Sep 16 02:18:44.137409 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:18:44.135918    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/e34369f4-88ba-4859-ad03-80219c62e272-system-cni-dir\") pod \"multus-7nk9m\" (UID: \"e34369f4-88ba-4859-ad03-80219c62e272\") " pod="openshift-multus/multus-7nk9m"
Sep 16 02:18:44.137409 ip-10-0-128-111 kubenswrapper[2583]: E0916 02:18:44.135958    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2fbf440b-4c09-428e-b6bf-1236d95f97cc-original-pull-secret podName:2fbf440b-4c09-428e-b6bf-1236d95f97cc nodeName:}" failed. No retries permitted until 2026-09-16 02:18:44.635939788 +0000 UTC m=+3.103728556 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/2fbf440b-4c09-428e-b6bf-1236d95f97cc-original-pull-secret") pod "global-pull-secret-syncer-mbxmd" (UID: "2fbf440b-4c09-428e-b6bf-1236d95f97cc") : object "kube-system"/"original-pull-secret" not registered
Sep 16 02:18:44.137409 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:18:44.135979    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/e34369f4-88ba-4859-ad03-80219c62e272-multus-socket-dir-parent\") pod \"multus-7nk9m\" (UID: \"e34369f4-88ba-4859-ad03-80219c62e272\") " pod="openshift-multus/multus-7nk9m"
Sep 16 02:18:44.137409 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:18:44.136028    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kzksh\" (UniqueName: \"kubernetes.io/projected/0e5e9542-45a1-4c13-8868-6a73d01f381d-kube-api-access-kzksh\") pod \"iptables-alerter-lmvcp\" (UID: \"0e5e9542-45a1-4c13-8868-6a73d01f381d\") " pod="openshift-network-operator/iptables-alerter-lmvcp"
Sep 16 02:18:44.137409 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:18:44.136057    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/82d1b4b2-7910-42f8-b2f3-0d675dc2c530-serviceca\") pod \"node-ca-5nwmf\" (UID: \"82d1b4b2-7910-42f8-b2f3-0d675dc2c530\") " pod="openshift-image-registry/node-ca-5nwmf"
Sep 16 02:18:44.137409 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:18:44.136086    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/e34369f4-88ba-4859-ad03-80219c62e272-hostroot\") pod \"multus-7nk9m\" (UID: \"e34369f4-88ba-4859-ad03-80219c62e272\") " pod="openshift-multus/multus-7nk9m"
Sep 16 02:18:44.137409 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:18:44.136095    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/3a1ad832-d1ab-4676-8eee-31d2eccce648-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-ll7tp\" (UID: \"3a1ad832-d1ab-4676-8eee-31d2eccce648\") " pod="openshift-multus/multus-additional-cni-plugins-ll7tp"
Sep 16 02:18:44.137409 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:18:44.136111    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/b9f0b2c7-b0c8-49bf-9761-c387496f9207-etc-selinux\") pod \"aws-ebs-csi-driver-node-dqkz9\" (UID: \"b9f0b2c7-b0c8-49bf-9761-c387496f9207\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dqkz9"
Sep 16 02:18:44.137409 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:18:44.136123    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/2fbf440b-4c09-428e-b6bf-1236d95f97cc-dbus\") pod \"global-pull-secret-syncer-mbxmd\" (UID: \"2fbf440b-4c09-428e-b6bf-1236d95f97cc\") " pod="kube-system/global-pull-secret-syncer-mbxmd"
Sep 16 02:18:44.137409 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:18:44.136146    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/e34369f4-88ba-4859-ad03-80219c62e272-multus-daemon-config\") pod \"multus-7nk9m\" (UID: \"e34369f4-88ba-4859-ad03-80219c62e272\") " pod="openshift-multus/multus-7nk9m"
Sep 16 02:18:44.137409 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:18:44.136158    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/b9f0b2c7-b0c8-49bf-9761-c387496f9207-sys-fs\") pod \"aws-ebs-csi-driver-node-dqkz9\" (UID: \"b9f0b2c7-b0c8-49bf-9761-c387496f9207\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dqkz9"
Sep 16 02:18:44.137409 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:18:44.136171    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/bdf274ad-3983-42ba-b712-8ce11da63210-agent-certs\") pod \"konnectivity-agent-psqcm\" (UID: \"bdf274ad-3983-42ba-b712-8ce11da63210\") " pod="kube-system/konnectivity-agent-psqcm"
Sep 16 02:18:44.137409 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:18:44.136204    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/e34369f4-88ba-4859-ad03-80219c62e272-system-cni-dir\") pod \"multus-7nk9m\" (UID: \"e34369f4-88ba-4859-ad03-80219c62e272\") " pod="openshift-multus/multus-7nk9m"
Sep 16 02:18:44.137409 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:18:44.136230    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/e34369f4-88ba-4859-ad03-80219c62e272-host-run-k8s-cni-cncf-io\") pod \"multus-7nk9m\" (UID: \"e34369f4-88ba-4859-ad03-80219c62e272\") " pod="openshift-multus/multus-7nk9m"
Sep 16 02:18:44.138286 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:18:44.136268    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/b9f0b2c7-b0c8-49bf-9761-c387496f9207-registration-dir\") pod \"aws-ebs-csi-driver-node-dqkz9\" (UID: \"b9f0b2c7-b0c8-49bf-9761-c387496f9207\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dqkz9"
Sep 16 02:18:44.138286 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:18:44.136271    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/e34369f4-88ba-4859-ad03-80219c62e272-host-var-lib-cni-bin\") pod \"multus-7nk9m\" (UID: \"e34369f4-88ba-4859-ad03-80219c62e272\") " pod="openshift-multus/multus-7nk9m"
Sep 16 02:18:44.138286 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:18:44.136302    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/e34369f4-88ba-4859-ad03-80219c62e272-host-var-lib-cni-multus\") pod \"multus-7nk9m\" (UID: \"e34369f4-88ba-4859-ad03-80219c62e272\") " pod="openshift-multus/multus-7nk9m"
Sep 16 02:18:44.138286 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:18:44.136332    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/e34369f4-88ba-4859-ad03-80219c62e272-multus-socket-dir-parent\") pod \"multus-7nk9m\" (UID: \"e34369f4-88ba-4859-ad03-80219c62e272\") " pod="openshift-multus/multus-7nk9m"
Sep 16 02:18:44.138286 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:18:44.136350    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e34369f4-88ba-4859-ad03-80219c62e272-etc-kubernetes\") pod \"multus-7nk9m\" (UID: \"e34369f4-88ba-4859-ad03-80219c62e272\") " pod="openshift-multus/multus-7nk9m"
Sep 16 02:18:44.138286 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:18:44.136519    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/e34369f4-88ba-4859-ad03-80219c62e272-hostroot\") pod \"multus-7nk9m\" (UID: \"e34369f4-88ba-4859-ad03-80219c62e272\") " pod="openshift-multus/multus-7nk9m"
Sep 16 02:18:44.138286 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:18:44.136569    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/b9f0b2c7-b0c8-49bf-9761-c387496f9207-etc-selinux\") pod \"aws-ebs-csi-driver-node-dqkz9\" (UID: \"b9f0b2c7-b0c8-49bf-9761-c387496f9207\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dqkz9"
Sep 16 02:18:44.138286 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:18:44.136915    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/3a1ad832-d1ab-4676-8eee-31d2eccce648-cni-binary-copy\") pod \"multus-additional-cni-plugins-ll7tp\" (UID: \"3a1ad832-d1ab-4676-8eee-31d2eccce648\") " pod="openshift-multus/multus-additional-cni-plugins-ll7tp"
Sep 16 02:18:44.138286 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:18:44.137021    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/3a1ad832-d1ab-4676-8eee-31d2eccce648-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-ll7tp\" (UID: \"3a1ad832-d1ab-4676-8eee-31d2eccce648\") " pod="openshift-multus/multus-additional-cni-plugins-ll7tp"
Sep 16 02:18:44.138286 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:18:44.137029    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/e34369f4-88ba-4859-ad03-80219c62e272-multus-daemon-config\") pod \"multus-7nk9m\" (UID: \"e34369f4-88ba-4859-ad03-80219c62e272\") " pod="openshift-multus/multus-7nk9m"
Sep 16 02:18:44.138286 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:18:44.137051    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/82d1b4b2-7910-42f8-b2f3-0d675dc2c530-serviceca\") pod \"node-ca-5nwmf\" (UID: \"82d1b4b2-7910-42f8-b2f3-0d675dc2c530\") " pod="openshift-image-registry/node-ca-5nwmf"
Sep 16 02:18:44.138286 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:18:44.137083    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/bdf274ad-3983-42ba-b712-8ce11da63210-konnectivity-ca\") pod \"konnectivity-agent-psqcm\" (UID: \"bdf274ad-3983-42ba-b712-8ce11da63210\") " pod="kube-system/konnectivity-agent-psqcm"
Sep 16 02:18:44.138286 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:18:44.137557    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/bdf274ad-3983-42ba-b712-8ce11da63210-konnectivity-ca\") pod \"konnectivity-agent-psqcm\" (UID: \"bdf274ad-3983-42ba-b712-8ce11da63210\") " pod="kube-system/konnectivity-agent-psqcm"
Sep 16 02:18:44.139731 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:18:44.139655    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/bdf274ad-3983-42ba-b712-8ce11da63210-agent-certs\") pod \"konnectivity-agent-psqcm\" (UID: \"bdf274ad-3983-42ba-b712-8ce11da63210\") " pod="kube-system/konnectivity-agent-psqcm"
Sep 16 02:18:44.144983 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:18:44.144926    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vkdx2\" (UniqueName: \"kubernetes.io/projected/b9f0b2c7-b0c8-49bf-9761-c387496f9207-kube-api-access-vkdx2\") pod \"aws-ebs-csi-driver-node-dqkz9\" (UID: \"b9f0b2c7-b0c8-49bf-9761-c387496f9207\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dqkz9"
Sep 16 02:18:44.145595 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:18:44.145571    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hznzt\" (UniqueName: \"kubernetes.io/projected/82d1b4b2-7910-42f8-b2f3-0d675dc2c530-kube-api-access-hznzt\") pod \"node-ca-5nwmf\" (UID: \"82d1b4b2-7910-42f8-b2f3-0d675dc2c530\") " pod="openshift-image-registry/node-ca-5nwmf"
Sep 16 02:18:44.145712 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:18:44.145697    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kzksh\" (UniqueName: \"kubernetes.io/projected/0e5e9542-45a1-4c13-8868-6a73d01f381d-kube-api-access-kzksh\") pod \"iptables-alerter-lmvcp\" (UID: \"0e5e9542-45a1-4c13-8868-6a73d01f381d\") " pod="openshift-network-operator/iptables-alerter-lmvcp"
Sep 16 02:18:44.146962 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:18:44.146916    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lhw4b\" (UniqueName: \"kubernetes.io/projected/3a1ad832-d1ab-4676-8eee-31d2eccce648-kube-api-access-lhw4b\") pod \"multus-additional-cni-plugins-ll7tp\" (UID: \"3a1ad832-d1ab-4676-8eee-31d2eccce648\") " pod="openshift-multus/multus-additional-cni-plugins-ll7tp"
Sep 16 02:18:44.147374 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:18:44.147334    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5xc9z\" (UniqueName: \"kubernetes.io/projected/e34369f4-88ba-4859-ad03-80219c62e272-kube-api-access-5xc9z\") pod \"multus-7nk9m\" (UID: \"e34369f4-88ba-4859-ad03-80219c62e272\") " pod="openshift-multus/multus-7nk9m"
Sep 16 02:18:44.164683 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:18:44.164654    2583 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 16 02:18:44.222234 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:18:44.222198    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-hgb4m"
Sep 16 02:18:44.231279 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:18:44.231249    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-rx99l"
Sep 16 02:18:44.240578 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:18:44.240558    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-kf6w7"
Sep 16 02:18:44.247139 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:18:44.247119    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-5nwmf"
Sep 16 02:18:44.255160 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:18:44.255139    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-7nk9m"
Sep 16 02:18:44.262038 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:18:44.262018    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-ll7tp"
Sep 16 02:18:44.271016 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:18:44.270997    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-lmvcp"
Sep 16 02:18:44.279151 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:18:44.279085    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-psqcm"
Sep 16 02:18:44.283930 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:18:44.283910    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dqkz9"
Sep 16 02:18:44.541347 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:18:44.541263    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/67a44fbe-95d7-4808-bcff-f7d214c6d82c-metrics-certs\") pod \"network-metrics-daemon-bdwgr\" (UID: \"67a44fbe-95d7-4808-bcff-f7d214c6d82c\") " pod="openshift-multus/network-metrics-daemon-bdwgr"
Sep 16 02:18:44.541487 ip-10-0-128-111 kubenswrapper[2583]: E0916 02:18:44.541425    2583 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 02:18:44.541528 ip-10-0-128-111 kubenswrapper[2583]: E0916 02:18:44.541495    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/67a44fbe-95d7-4808-bcff-f7d214c6d82c-metrics-certs podName:67a44fbe-95d7-4808-bcff-f7d214c6d82c nodeName:}" failed. No retries permitted until 2026-09-16 02:18:45.541478028 +0000 UTC m=+4.009266790 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/67a44fbe-95d7-4808-bcff-f7d214c6d82c-metrics-certs") pod "network-metrics-daemon-bdwgr" (UID: "67a44fbe-95d7-4808-bcff-f7d214c6d82c") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 02:18:44.642094 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:18:44.642056    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qwztf\" (UniqueName: \"kubernetes.io/projected/5f66bb1b-6d4f-42c5-9b63-f7f5f301b6f9-kube-api-access-qwztf\") pod \"network-check-target-55zcf\" (UID: \"5f66bb1b-6d4f-42c5-9b63-f7f5f301b6f9\") " pod="openshift-network-diagnostics/network-check-target-55zcf"
Sep 16 02:18:44.642263 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:18:44.642122    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/2fbf440b-4c09-428e-b6bf-1236d95f97cc-original-pull-secret\") pod \"global-pull-secret-syncer-mbxmd\" (UID: \"2fbf440b-4c09-428e-b6bf-1236d95f97cc\") " pod="kube-system/global-pull-secret-syncer-mbxmd"
Sep 16 02:18:44.642263 ip-10-0-128-111 kubenswrapper[2583]: E0916 02:18:44.642233    2583 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 16 02:18:44.642263 ip-10-0-128-111 kubenswrapper[2583]: E0916 02:18:44.642244    2583 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 16 02:18:44.642388 ip-10-0-128-111 kubenswrapper[2583]: E0916 02:18:44.642272    2583 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 16 02:18:44.642388 ip-10-0-128-111 kubenswrapper[2583]: E0916 02:18:44.642282    2583 projected.go:196] Error preparing data for projected volume kube-api-access-qwztf for pod openshift-network-diagnostics/network-check-target-55zcf: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 02:18:44.642388 ip-10-0-128-111 kubenswrapper[2583]: E0916 02:18:44.642299    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2fbf440b-4c09-428e-b6bf-1236d95f97cc-original-pull-secret podName:2fbf440b-4c09-428e-b6bf-1236d95f97cc nodeName:}" failed. No retries permitted until 2026-09-16 02:18:45.642282466 +0000 UTC m=+4.110071230 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/2fbf440b-4c09-428e-b6bf-1236d95f97cc-original-pull-secret") pod "global-pull-secret-syncer-mbxmd" (UID: "2fbf440b-4c09-428e-b6bf-1236d95f97cc") : object "kube-system"/"original-pull-secret" not registered
Sep 16 02:18:44.642388 ip-10-0-128-111 kubenswrapper[2583]: E0916 02:18:44.642329    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/5f66bb1b-6d4f-42c5-9b63-f7f5f301b6f9-kube-api-access-qwztf podName:5f66bb1b-6d4f-42c5-9b63-f7f5f301b6f9 nodeName:}" failed. No retries permitted until 2026-09-16 02:18:45.642315412 +0000 UTC m=+4.110104174 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-qwztf" (UniqueName: "kubernetes.io/projected/5f66bb1b-6d4f-42c5-9b63-f7f5f301b6f9-kube-api-access-qwztf") pod "network-check-target-55zcf" (UID: "5f66bb1b-6d4f-42c5-9b63-f7f5f301b6f9") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 02:18:44.964765 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:18:44.964672    2583 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-15 02:13:44 +0000 UTC" deadline="2028-02-24 20:24:01.912832021 +0000 UTC"
Sep 16 02:18:44.964765 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:18:44.964709    2583 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12642h5m16.948126421s"
Sep 16 02:18:45.039239 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:18:45.039198    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mbxmd"
Sep 16 02:18:45.039427 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:18:45.039198    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-55zcf"
Sep 16 02:18:45.039427 ip-10-0-128-111 kubenswrapper[2583]: E0916 02:18:45.039347    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mbxmd" podUID="2fbf440b-4c09-428e-b6bf-1236d95f97cc"
Sep 16 02:18:45.039527 ip-10-0-128-111 kubenswrapper[2583]: E0916 02:18:45.039449    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-55zcf" podUID="5f66bb1b-6d4f-42c5-9b63-f7f5f301b6f9"
Sep 16 02:18:45.267981 ip-10-0-128-111 kubenswrapper[2583]: W0916 02:18:45.267951    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0e5e9542_45a1_4c13_8868_6a73d01f381d.slice/crio-2f2dc77f516338195fb3b8fe01c1d0d70906a2ab26b184917d37478d2fe3e518 WatchSource:0}: Error finding container 2f2dc77f516338195fb3b8fe01c1d0d70906a2ab26b184917d37478d2fe3e518: Status 404 returned error can't find the container with id 2f2dc77f516338195fb3b8fe01c1d0d70906a2ab26b184917d37478d2fe3e518
Sep 16 02:18:45.268904 ip-10-0-128-111 kubenswrapper[2583]: W0916 02:18:45.268880    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd45dcef5_f6f3_45d7_b29e_1125a402eeb3.slice/crio-0008d04286f116694f7efc493fe20e34f55d592fc9ae0b1ef460bd0d494da4e7 WatchSource:0}: Error finding container 0008d04286f116694f7efc493fe20e34f55d592fc9ae0b1ef460bd0d494da4e7: Status 404 returned error can't find the container with id 0008d04286f116694f7efc493fe20e34f55d592fc9ae0b1ef460bd0d494da4e7
Sep 16 02:18:45.270217 ip-10-0-128-111 kubenswrapper[2583]: W0916 02:18:45.270035    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2c5d542f_5718_4aeb_8bf5_680a69ddecbc.slice/crio-d9b9a35ddaaf73e420be44208f93c2737eec4735b6518add5ab95e0d55aa0fb9 WatchSource:0}: Error finding container d9b9a35ddaaf73e420be44208f93c2737eec4735b6518add5ab95e0d55aa0fb9: Status 404 returned error can't find the container with id d9b9a35ddaaf73e420be44208f93c2737eec4735b6518add5ab95e0d55aa0fb9
Sep 16 02:18:45.271183 ip-10-0-128-111 kubenswrapper[2583]: W0916 02:18:45.271060    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbdf274ad_3983_42ba_b712_8ce11da63210.slice/crio-3f0d4b60e136dd6e6e45a941873534753e0447df4b21e58bf6db2ca7ec1cd20d WatchSource:0}: Error finding container 3f0d4b60e136dd6e6e45a941873534753e0447df4b21e58bf6db2ca7ec1cd20d: Status 404 returned error can't find the container with id 3f0d4b60e136dd6e6e45a941873534753e0447df4b21e58bf6db2ca7ec1cd20d
Sep 16 02:18:45.274192 ip-10-0-128-111 kubenswrapper[2583]: W0916 02:18:45.273906    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb9f0b2c7_b0c8_49bf_9761_c387496f9207.slice/crio-ab3f3f0f954a3ac51414b6fb508394694a493138644e9700a4caf6d4465358ee WatchSource:0}: Error finding container ab3f3f0f954a3ac51414b6fb508394694a493138644e9700a4caf6d4465358ee: Status 404 returned error can't find the container with id ab3f3f0f954a3ac51414b6fb508394694a493138644e9700a4caf6d4465358ee
Sep 16 02:18:45.276676 ip-10-0-128-111 kubenswrapper[2583]: W0916 02:18:45.276570    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod82d1b4b2_7910_42f8_b2f3_0d675dc2c530.slice/crio-2ad4aa9413baf39a346662913af82a3868af74c9588315d77540ae192c7a2d50 WatchSource:0}: Error finding container 2ad4aa9413baf39a346662913af82a3868af74c9588315d77540ae192c7a2d50: Status 404 returned error can't find the container with id 2ad4aa9413baf39a346662913af82a3868af74c9588315d77540ae192c7a2d50
Sep 16 02:18:45.278124 ip-10-0-128-111 kubenswrapper[2583]: W0916 02:18:45.278051    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod66c3846d_d0d2_4b46_8c59_f6d97a92f7ad.slice/crio-c1d77c39db3ce61130e1f0574726bc893896071e889007df899de22ac78705c7 WatchSource:0}: Error finding container c1d77c39db3ce61130e1f0574726bc893896071e889007df899de22ac78705c7: Status 404 returned error can't find the container with id c1d77c39db3ce61130e1f0574726bc893896071e889007df899de22ac78705c7
Sep 16 02:18:45.550897 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:18:45.550868    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/67a44fbe-95d7-4808-bcff-f7d214c6d82c-metrics-certs\") pod \"network-metrics-daemon-bdwgr\" (UID: \"67a44fbe-95d7-4808-bcff-f7d214c6d82c\") " pod="openshift-multus/network-metrics-daemon-bdwgr"
Sep 16 02:18:45.551007 ip-10-0-128-111 kubenswrapper[2583]: E0916 02:18:45.550994    2583 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 02:18:45.551054 ip-10-0-128-111 kubenswrapper[2583]: E0916 02:18:45.551046    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/67a44fbe-95d7-4808-bcff-f7d214c6d82c-metrics-certs podName:67a44fbe-95d7-4808-bcff-f7d214c6d82c nodeName:}" failed. No retries permitted until 2026-09-16 02:18:47.55103195 +0000 UTC m=+6.018820716 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/67a44fbe-95d7-4808-bcff-f7d214c6d82c-metrics-certs") pod "network-metrics-daemon-bdwgr" (UID: "67a44fbe-95d7-4808-bcff-f7d214c6d82c") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 02:18:45.651336 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:18:45.651281    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qwztf\" (UniqueName: \"kubernetes.io/projected/5f66bb1b-6d4f-42c5-9b63-f7f5f301b6f9-kube-api-access-qwztf\") pod \"network-check-target-55zcf\" (UID: \"5f66bb1b-6d4f-42c5-9b63-f7f5f301b6f9\") " pod="openshift-network-diagnostics/network-check-target-55zcf"
Sep 16 02:18:45.651496 ip-10-0-128-111 kubenswrapper[2583]: E0916 02:18:45.651460    2583 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 16 02:18:45.651496 ip-10-0-128-111 kubenswrapper[2583]: E0916 02:18:45.651492    2583 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 16 02:18:45.651638 ip-10-0-128-111 kubenswrapper[2583]: E0916 02:18:45.651508    2583 projected.go:196] Error preparing data for projected volume kube-api-access-qwztf for pod openshift-network-diagnostics/network-check-target-55zcf: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 02:18:45.651638 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:18:45.651517    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/2fbf440b-4c09-428e-b6bf-1236d95f97cc-original-pull-secret\") pod \"global-pull-secret-syncer-mbxmd\" (UID: \"2fbf440b-4c09-428e-b6bf-1236d95f97cc\") " pod="kube-system/global-pull-secret-syncer-mbxmd"
Sep 16 02:18:45.651638 ip-10-0-128-111 kubenswrapper[2583]: E0916 02:18:45.651561    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/5f66bb1b-6d4f-42c5-9b63-f7f5f301b6f9-kube-api-access-qwztf podName:5f66bb1b-6d4f-42c5-9b63-f7f5f301b6f9 nodeName:}" failed. No retries permitted until 2026-09-16 02:18:47.651544738 +0000 UTC m=+6.119333505 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-qwztf" (UniqueName: "kubernetes.io/projected/5f66bb1b-6d4f-42c5-9b63-f7f5f301b6f9-kube-api-access-qwztf") pod "network-check-target-55zcf" (UID: "5f66bb1b-6d4f-42c5-9b63-f7f5f301b6f9") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 02:18:45.651817 ip-10-0-128-111 kubenswrapper[2583]: E0916 02:18:45.651677    2583 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 16 02:18:45.651817 ip-10-0-128-111 kubenswrapper[2583]: E0916 02:18:45.651745    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2fbf440b-4c09-428e-b6bf-1236d95f97cc-original-pull-secret podName:2fbf440b-4c09-428e-b6bf-1236d95f97cc nodeName:}" failed. No retries permitted until 2026-09-16 02:18:47.651726168 +0000 UTC m=+6.119514931 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/2fbf440b-4c09-428e-b6bf-1236d95f97cc-original-pull-secret") pod "global-pull-secret-syncer-mbxmd" (UID: "2fbf440b-4c09-428e-b6bf-1236d95f97cc") : object "kube-system"/"original-pull-secret" not registered
Sep 16 02:18:45.965890 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:18:45.965719    2583 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-15 02:13:44 +0000 UTC" deadline="2028-05-23 19:47:00.510296794 +0000 UTC"
Sep 16 02:18:45.965890 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:18:45.965783    2583 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14777h28m14.54451763s"
Sep 16 02:18:46.043909 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:18:46.043349    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bdwgr"
Sep 16 02:18:46.043909 ip-10-0-128-111 kubenswrapper[2583]: E0916 02:18:46.043501    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bdwgr" podUID="67a44fbe-95d7-4808-bcff-f7d214c6d82c"
Sep 16 02:18:46.071926 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:18:46.071394    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-111.ec2.internal" event={"ID":"b7568fa8cea4626b154e4750bd44461f","Type":"ContainerStarted","Data":"23d05a66b0db7d2cd43618b908d3656227e75d8b4fe5e67a9a8b9ecc0a30f442"}
Sep 16 02:18:46.085860 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:18:46.085798    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-111.ec2.internal" podStartSLOduration=3.085776539 podStartE2EDuration="3.085776539s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 02:18:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 02:18:46.084852628 +0000 UTC m=+4.552641416" watchObservedRunningTime="2026-09-16 02:18:46.085776539 +0000 UTC m=+4.553565326"
Sep 16 02:18:46.095678 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:18:46.095216    2583 generic.go:356] "Generic (PLEG): container finished" podID="ac09b6665582e7a386f06843c363096b" containerID="ccd6d4227a200a2747209c50ec130062b46f92b7c5f205c7628d2f5a52cb591f" exitCode=0
Sep 16 02:18:46.095678 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:18:46.095304    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-111.ec2.internal" event={"ID":"ac09b6665582e7a386f06843c363096b","Type":"ContainerDied","Data":"ccd6d4227a200a2747209c50ec130062b46f92b7c5f205c7628d2f5a52cb591f"}
Sep 16 02:18:46.102154 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:18:46.102109    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-5nwmf" event={"ID":"82d1b4b2-7910-42f8-b2f3-0d675dc2c530","Type":"ContainerStarted","Data":"2ad4aa9413baf39a346662913af82a3868af74c9588315d77540ae192c7a2d50"}
Sep 16 02:18:46.106546 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:18:46.106517    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dqkz9" event={"ID":"b9f0b2c7-b0c8-49bf-9761-c387496f9207","Type":"ContainerStarted","Data":"ab3f3f0f954a3ac51414b6fb508394694a493138644e9700a4caf6d4465358ee"}
Sep 16 02:18:46.129134 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:18:46.129097    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-psqcm" event={"ID":"bdf274ad-3983-42ba-b712-8ce11da63210","Type":"ContainerStarted","Data":"3f0d4b60e136dd6e6e45a941873534753e0447df4b21e58bf6db2ca7ec1cd20d"}
Sep 16 02:18:46.141930 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:18:46.141834    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-kf6w7" event={"ID":"2c5d542f-5718-4aeb-8bf5-680a69ddecbc","Type":"ContainerStarted","Data":"d9b9a35ddaaf73e420be44208f93c2737eec4735b6518add5ab95e0d55aa0fb9"}
Sep 16 02:18:46.151585 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:18:46.151511    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-lmvcp" event={"ID":"0e5e9542-45a1-4c13-8868-6a73d01f381d","Type":"ContainerStarted","Data":"2f2dc77f516338195fb3b8fe01c1d0d70906a2ab26b184917d37478d2fe3e518"}
Sep 16 02:18:46.165955 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:18:46.165905    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-7nk9m" event={"ID":"e34369f4-88ba-4859-ad03-80219c62e272","Type":"ContainerStarted","Data":"efab2a0f0200857b6dd8b1ae673e54c8333915718543158ada9e338b2d1d218e"}
Sep 16 02:18:46.181062 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:18:46.181022    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-ll7tp" event={"ID":"3a1ad832-d1ab-4676-8eee-31d2eccce648","Type":"ContainerStarted","Data":"d3e6b9fcf41f46cca0fe4a755041faa4601757ff8259fb0225fc53ef2d5f2691"}
Sep 16 02:18:46.193028 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:18:46.192991    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-rx99l" event={"ID":"66c3846d-d0d2-4b46-8c59-f6d97a92f7ad","Type":"ContainerStarted","Data":"c1d77c39db3ce61130e1f0574726bc893896071e889007df899de22ac78705c7"}
Sep 16 02:18:46.197201 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:18:46.197108    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-hgb4m" event={"ID":"d45dcef5-f6f3-45d7-b29e-1125a402eeb3","Type":"ContainerStarted","Data":"0008d04286f116694f7efc493fe20e34f55d592fc9ae0b1ef460bd0d494da4e7"}
Sep 16 02:18:47.040031 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:18:47.039314    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-55zcf"
Sep 16 02:18:47.040031 ip-10-0-128-111 kubenswrapper[2583]: E0916 02:18:47.039449    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-55zcf" podUID="5f66bb1b-6d4f-42c5-9b63-f7f5f301b6f9"
Sep 16 02:18:47.040031 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:18:47.039530    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mbxmd"
Sep 16 02:18:47.040031 ip-10-0-128-111 kubenswrapper[2583]: E0916 02:18:47.039626    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mbxmd" podUID="2fbf440b-4c09-428e-b6bf-1236d95f97cc"
Sep 16 02:18:47.213728 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:18:47.213690    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-111.ec2.internal" event={"ID":"ac09b6665582e7a386f06843c363096b","Type":"ContainerStarted","Data":"39b0766259971e37c4b6cf91318e3cd7108921bcf90c47f4ab09442b6dc57730"}
Sep 16 02:18:47.567087 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:18:47.566492    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/67a44fbe-95d7-4808-bcff-f7d214c6d82c-metrics-certs\") pod \"network-metrics-daemon-bdwgr\" (UID: \"67a44fbe-95d7-4808-bcff-f7d214c6d82c\") " pod="openshift-multus/network-metrics-daemon-bdwgr"
Sep 16 02:18:47.567087 ip-10-0-128-111 kubenswrapper[2583]: E0916 02:18:47.566674    2583 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 02:18:47.567087 ip-10-0-128-111 kubenswrapper[2583]: E0916 02:18:47.566739    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/67a44fbe-95d7-4808-bcff-f7d214c6d82c-metrics-certs podName:67a44fbe-95d7-4808-bcff-f7d214c6d82c nodeName:}" failed. No retries permitted until 2026-09-16 02:18:51.566719895 +0000 UTC m=+10.034508714 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/67a44fbe-95d7-4808-bcff-f7d214c6d82c-metrics-certs") pod "network-metrics-daemon-bdwgr" (UID: "67a44fbe-95d7-4808-bcff-f7d214c6d82c") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 02:18:47.668209 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:18:47.667386    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/2fbf440b-4c09-428e-b6bf-1236d95f97cc-original-pull-secret\") pod \"global-pull-secret-syncer-mbxmd\" (UID: \"2fbf440b-4c09-428e-b6bf-1236d95f97cc\") " pod="kube-system/global-pull-secret-syncer-mbxmd"
Sep 16 02:18:47.668209 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:18:47.667475    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qwztf\" (UniqueName: \"kubernetes.io/projected/5f66bb1b-6d4f-42c5-9b63-f7f5f301b6f9-kube-api-access-qwztf\") pod \"network-check-target-55zcf\" (UID: \"5f66bb1b-6d4f-42c5-9b63-f7f5f301b6f9\") " pod="openshift-network-diagnostics/network-check-target-55zcf"
Sep 16 02:18:47.668209 ip-10-0-128-111 kubenswrapper[2583]: E0916 02:18:47.667691    2583 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 16 02:18:47.668209 ip-10-0-128-111 kubenswrapper[2583]: E0916 02:18:47.667717    2583 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 16 02:18:47.668209 ip-10-0-128-111 kubenswrapper[2583]: E0916 02:18:47.667732    2583 projected.go:196] Error preparing data for projected volume kube-api-access-qwztf for pod openshift-network-diagnostics/network-check-target-55zcf: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 02:18:47.668209 ip-10-0-128-111 kubenswrapper[2583]: E0916 02:18:47.667809    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/5f66bb1b-6d4f-42c5-9b63-f7f5f301b6f9-kube-api-access-qwztf podName:5f66bb1b-6d4f-42c5-9b63-f7f5f301b6f9 nodeName:}" failed. No retries permitted until 2026-09-16 02:18:51.667779641 +0000 UTC m=+10.135568407 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-qwztf" (UniqueName: "kubernetes.io/projected/5f66bb1b-6d4f-42c5-9b63-f7f5f301b6f9-kube-api-access-qwztf") pod "network-check-target-55zcf" (UID: "5f66bb1b-6d4f-42c5-9b63-f7f5f301b6f9") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 02:18:47.668209 ip-10-0-128-111 kubenswrapper[2583]: E0916 02:18:47.668017    2583 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 16 02:18:47.668209 ip-10-0-128-111 kubenswrapper[2583]: E0916 02:18:47.668085    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2fbf440b-4c09-428e-b6bf-1236d95f97cc-original-pull-secret podName:2fbf440b-4c09-428e-b6bf-1236d95f97cc nodeName:}" failed. No retries permitted until 2026-09-16 02:18:51.668068507 +0000 UTC m=+10.135857284 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/2fbf440b-4c09-428e-b6bf-1236d95f97cc-original-pull-secret") pod "global-pull-secret-syncer-mbxmd" (UID: "2fbf440b-4c09-428e-b6bf-1236d95f97cc") : object "kube-system"/"original-pull-secret" not registered
Sep 16 02:18:48.039508 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:18:48.039198    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bdwgr"
Sep 16 02:18:48.039711 ip-10-0-128-111 kubenswrapper[2583]: E0916 02:18:48.039584    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bdwgr" podUID="67a44fbe-95d7-4808-bcff-f7d214c6d82c"
Sep 16 02:18:49.039681 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:18:49.038940    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-55zcf"
Sep 16 02:18:49.039681 ip-10-0-128-111 kubenswrapper[2583]: E0916 02:18:49.039078    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-55zcf" podUID="5f66bb1b-6d4f-42c5-9b63-f7f5f301b6f9"
Sep 16 02:18:49.039681 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:18:49.039496    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mbxmd"
Sep 16 02:18:49.039681 ip-10-0-128-111 kubenswrapper[2583]: E0916 02:18:49.039602    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mbxmd" podUID="2fbf440b-4c09-428e-b6bf-1236d95f97cc"
Sep 16 02:18:50.039408 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:18:50.039370    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bdwgr"
Sep 16 02:18:50.039650 ip-10-0-128-111 kubenswrapper[2583]: E0916 02:18:50.039606    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bdwgr" podUID="67a44fbe-95d7-4808-bcff-f7d214c6d82c"
Sep 16 02:18:51.039160 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:18:51.039031    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-55zcf"
Sep 16 02:18:51.039160 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:18:51.039033    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mbxmd"
Sep 16 02:18:51.039695 ip-10-0-128-111 kubenswrapper[2583]: E0916 02:18:51.039158    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-55zcf" podUID="5f66bb1b-6d4f-42c5-9b63-f7f5f301b6f9"
Sep 16 02:18:51.039695 ip-10-0-128-111 kubenswrapper[2583]: E0916 02:18:51.039254    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mbxmd" podUID="2fbf440b-4c09-428e-b6bf-1236d95f97cc"
Sep 16 02:18:51.606361 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:18:51.606322    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/67a44fbe-95d7-4808-bcff-f7d214c6d82c-metrics-certs\") pod \"network-metrics-daemon-bdwgr\" (UID: \"67a44fbe-95d7-4808-bcff-f7d214c6d82c\") " pod="openshift-multus/network-metrics-daemon-bdwgr"
Sep 16 02:18:51.606537 ip-10-0-128-111 kubenswrapper[2583]: E0916 02:18:51.606466    2583 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 02:18:51.606537 ip-10-0-128-111 kubenswrapper[2583]: E0916 02:18:51.606535    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/67a44fbe-95d7-4808-bcff-f7d214c6d82c-metrics-certs podName:67a44fbe-95d7-4808-bcff-f7d214c6d82c nodeName:}" failed. No retries permitted until 2026-09-16 02:18:59.606515143 +0000 UTC m=+18.074303911 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/67a44fbe-95d7-4808-bcff-f7d214c6d82c-metrics-certs") pod "network-metrics-daemon-bdwgr" (UID: "67a44fbe-95d7-4808-bcff-f7d214c6d82c") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 02:18:51.706882 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:18:51.706844    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/2fbf440b-4c09-428e-b6bf-1236d95f97cc-original-pull-secret\") pod \"global-pull-secret-syncer-mbxmd\" (UID: \"2fbf440b-4c09-428e-b6bf-1236d95f97cc\") " pod="kube-system/global-pull-secret-syncer-mbxmd"
Sep 16 02:18:51.707064 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:18:51.706928    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qwztf\" (UniqueName: \"kubernetes.io/projected/5f66bb1b-6d4f-42c5-9b63-f7f5f301b6f9-kube-api-access-qwztf\") pod \"network-check-target-55zcf\" (UID: \"5f66bb1b-6d4f-42c5-9b63-f7f5f301b6f9\") " pod="openshift-network-diagnostics/network-check-target-55zcf"
Sep 16 02:18:51.707064 ip-10-0-128-111 kubenswrapper[2583]: E0916 02:18:51.707029    2583 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 16 02:18:51.707156 ip-10-0-128-111 kubenswrapper[2583]: E0916 02:18:51.707100    2583 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 16 02:18:51.707156 ip-10-0-128-111 kubenswrapper[2583]: E0916 02:18:51.707126    2583 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 16 02:18:51.707156 ip-10-0-128-111 kubenswrapper[2583]: E0916 02:18:51.707139    2583 projected.go:196] Error preparing data for projected volume kube-api-access-qwztf for pod openshift-network-diagnostics/network-check-target-55zcf: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 02:18:51.707278 ip-10-0-128-111 kubenswrapper[2583]: E0916 02:18:51.707104    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2fbf440b-4c09-428e-b6bf-1236d95f97cc-original-pull-secret podName:2fbf440b-4c09-428e-b6bf-1236d95f97cc nodeName:}" failed. No retries permitted until 2026-09-16 02:18:59.707083307 +0000 UTC m=+18.174872074 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/2fbf440b-4c09-428e-b6bf-1236d95f97cc-original-pull-secret") pod "global-pull-secret-syncer-mbxmd" (UID: "2fbf440b-4c09-428e-b6bf-1236d95f97cc") : object "kube-system"/"original-pull-secret" not registered
Sep 16 02:18:51.707278 ip-10-0-128-111 kubenswrapper[2583]: E0916 02:18:51.707202    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/5f66bb1b-6d4f-42c5-9b63-f7f5f301b6f9-kube-api-access-qwztf podName:5f66bb1b-6d4f-42c5-9b63-f7f5f301b6f9 nodeName:}" failed. No retries permitted until 2026-09-16 02:18:59.707185051 +0000 UTC m=+18.174973813 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-qwztf" (UniqueName: "kubernetes.io/projected/5f66bb1b-6d4f-42c5-9b63-f7f5f301b6f9-kube-api-access-qwztf") pod "network-check-target-55zcf" (UID: "5f66bb1b-6d4f-42c5-9b63-f7f5f301b6f9") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 02:18:52.040987 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:18:52.040911    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bdwgr"
Sep 16 02:18:52.041424 ip-10-0-128-111 kubenswrapper[2583]: E0916 02:18:52.041040    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bdwgr" podUID="67a44fbe-95d7-4808-bcff-f7d214c6d82c"
Sep 16 02:18:53.039131 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:18:53.039092    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mbxmd"
Sep 16 02:18:53.039311 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:18:53.039091    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-55zcf"
Sep 16 02:18:53.039311 ip-10-0-128-111 kubenswrapper[2583]: E0916 02:18:53.039230    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mbxmd" podUID="2fbf440b-4c09-428e-b6bf-1236d95f97cc"
Sep 16 02:18:53.039434 ip-10-0-128-111 kubenswrapper[2583]: E0916 02:18:53.039326    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-55zcf" podUID="5f66bb1b-6d4f-42c5-9b63-f7f5f301b6f9"
Sep 16 02:18:54.039499 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:18:54.038984    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bdwgr"
Sep 16 02:18:54.039499 ip-10-0-128-111 kubenswrapper[2583]: E0916 02:18:54.039127    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bdwgr" podUID="67a44fbe-95d7-4808-bcff-f7d214c6d82c"
Sep 16 02:18:55.038851 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:18:55.038822    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-55zcf"
Sep 16 02:18:55.039030 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:18:55.038826    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mbxmd"
Sep 16 02:18:55.039030 ip-10-0-128-111 kubenswrapper[2583]: E0916 02:18:55.038923    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-55zcf" podUID="5f66bb1b-6d4f-42c5-9b63-f7f5f301b6f9"
Sep 16 02:18:55.039030 ip-10-0-128-111 kubenswrapper[2583]: E0916 02:18:55.039005    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mbxmd" podUID="2fbf440b-4c09-428e-b6bf-1236d95f97cc"
Sep 16 02:18:56.039251 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:18:56.039214    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bdwgr"
Sep 16 02:18:56.039715 ip-10-0-128-111 kubenswrapper[2583]: E0916 02:18:56.039355    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bdwgr" podUID="67a44fbe-95d7-4808-bcff-f7d214c6d82c"
Sep 16 02:18:57.039904 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:18:57.039851    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-55zcf"
Sep 16 02:18:57.040379 ip-10-0-128-111 kubenswrapper[2583]: E0916 02:18:57.039977    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-55zcf" podUID="5f66bb1b-6d4f-42c5-9b63-f7f5f301b6f9"
Sep 16 02:18:57.040379 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:18:57.040042    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mbxmd"
Sep 16 02:18:57.040379 ip-10-0-128-111 kubenswrapper[2583]: E0916 02:18:57.040154    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mbxmd" podUID="2fbf440b-4c09-428e-b6bf-1236d95f97cc"
Sep 16 02:18:58.039530 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:18:58.039487    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bdwgr"
Sep 16 02:18:58.039743 ip-10-0-128-111 kubenswrapper[2583]: E0916 02:18:58.039646    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bdwgr" podUID="67a44fbe-95d7-4808-bcff-f7d214c6d82c"
Sep 16 02:18:59.039398 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:18:59.039368    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mbxmd"
Sep 16 02:18:59.039907 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:18:59.039373    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-55zcf"
Sep 16 02:18:59.039907 ip-10-0-128-111 kubenswrapper[2583]: E0916 02:18:59.039476    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mbxmd" podUID="2fbf440b-4c09-428e-b6bf-1236d95f97cc"
Sep 16 02:18:59.039907 ip-10-0-128-111 kubenswrapper[2583]: E0916 02:18:59.039570    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-55zcf" podUID="5f66bb1b-6d4f-42c5-9b63-f7f5f301b6f9"
Sep 16 02:18:59.664423 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:18:59.664384    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/67a44fbe-95d7-4808-bcff-f7d214c6d82c-metrics-certs\") pod \"network-metrics-daemon-bdwgr\" (UID: \"67a44fbe-95d7-4808-bcff-f7d214c6d82c\") " pod="openshift-multus/network-metrics-daemon-bdwgr"
Sep 16 02:18:59.664725 ip-10-0-128-111 kubenswrapper[2583]: E0916 02:18:59.664576    2583 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 02:18:59.664725 ip-10-0-128-111 kubenswrapper[2583]: E0916 02:18:59.664652    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/67a44fbe-95d7-4808-bcff-f7d214c6d82c-metrics-certs podName:67a44fbe-95d7-4808-bcff-f7d214c6d82c nodeName:}" failed. No retries permitted until 2026-09-16 02:19:15.664629408 +0000 UTC m=+34.132418186 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/67a44fbe-95d7-4808-bcff-f7d214c6d82c-metrics-certs") pod "network-metrics-daemon-bdwgr" (UID: "67a44fbe-95d7-4808-bcff-f7d214c6d82c") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 02:18:59.765683 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:18:59.765649    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qwztf\" (UniqueName: \"kubernetes.io/projected/5f66bb1b-6d4f-42c5-9b63-f7f5f301b6f9-kube-api-access-qwztf\") pod \"network-check-target-55zcf\" (UID: \"5f66bb1b-6d4f-42c5-9b63-f7f5f301b6f9\") " pod="openshift-network-diagnostics/network-check-target-55zcf"
Sep 16 02:18:59.765868 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:18:59.765722    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/2fbf440b-4c09-428e-b6bf-1236d95f97cc-original-pull-secret\") pod \"global-pull-secret-syncer-mbxmd\" (UID: \"2fbf440b-4c09-428e-b6bf-1236d95f97cc\") " pod="kube-system/global-pull-secret-syncer-mbxmd"
Sep 16 02:18:59.765868 ip-10-0-128-111 kubenswrapper[2583]: E0916 02:18:59.765818    2583 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 16 02:18:59.765868 ip-10-0-128-111 kubenswrapper[2583]: E0916 02:18:59.765832    2583 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 16 02:18:59.765868 ip-10-0-128-111 kubenswrapper[2583]: E0916 02:18:59.765846    2583 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 16 02:18:59.765868 ip-10-0-128-111 kubenswrapper[2583]: E0916 02:18:59.765860    2583 projected.go:196] Error preparing data for projected volume kube-api-access-qwztf for pod openshift-network-diagnostics/network-check-target-55zcf: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 02:18:59.766066 ip-10-0-128-111 kubenswrapper[2583]: E0916 02:18:59.765889    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2fbf440b-4c09-428e-b6bf-1236d95f97cc-original-pull-secret podName:2fbf440b-4c09-428e-b6bf-1236d95f97cc nodeName:}" failed. No retries permitted until 2026-09-16 02:19:15.765870446 +0000 UTC m=+34.233659211 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/2fbf440b-4c09-428e-b6bf-1236d95f97cc-original-pull-secret") pod "global-pull-secret-syncer-mbxmd" (UID: "2fbf440b-4c09-428e-b6bf-1236d95f97cc") : object "kube-system"/"original-pull-secret" not registered
Sep 16 02:18:59.766066 ip-10-0-128-111 kubenswrapper[2583]: E0916 02:18:59.765912    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/5f66bb1b-6d4f-42c5-9b63-f7f5f301b6f9-kube-api-access-qwztf podName:5f66bb1b-6d4f-42c5-9b63-f7f5f301b6f9 nodeName:}" failed. No retries permitted until 2026-09-16 02:19:15.765896255 +0000 UTC m=+34.233685030 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-qwztf" (UniqueName: "kubernetes.io/projected/5f66bb1b-6d4f-42c5-9b63-f7f5f301b6f9-kube-api-access-qwztf") pod "network-check-target-55zcf" (UID: "5f66bb1b-6d4f-42c5-9b63-f7f5f301b6f9") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 02:19:00.039777 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:19:00.039707    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bdwgr"
Sep 16 02:19:00.040131 ip-10-0-128-111 kubenswrapper[2583]: E0916 02:19:00.039820    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bdwgr" podUID="67a44fbe-95d7-4808-bcff-f7d214c6d82c"
Sep 16 02:19:01.039691 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:19:01.039645    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-55zcf"
Sep 16 02:19:01.039868 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:19:01.039645    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mbxmd"
Sep 16 02:19:01.039868 ip-10-0-128-111 kubenswrapper[2583]: E0916 02:19:01.039796    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-55zcf" podUID="5f66bb1b-6d4f-42c5-9b63-f7f5f301b6f9"
Sep 16 02:19:01.039868 ip-10-0-128-111 kubenswrapper[2583]: E0916 02:19:01.039843    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mbxmd" podUID="2fbf440b-4c09-428e-b6bf-1236d95f97cc"
Sep 16 02:19:02.040166 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:19:02.040120    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bdwgr"
Sep 16 02:19:02.040641 ip-10-0-128-111 kubenswrapper[2583]: E0916 02:19:02.040244    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bdwgr" podUID="67a44fbe-95d7-4808-bcff-f7d214c6d82c"
Sep 16 02:19:03.039855 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:19:03.039824    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-55zcf"
Sep 16 02:19:03.040040 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:19:03.039826    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mbxmd"
Sep 16 02:19:03.040040 ip-10-0-128-111 kubenswrapper[2583]: E0916 02:19:03.039926    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-55zcf" podUID="5f66bb1b-6d4f-42c5-9b63-f7f5f301b6f9"
Sep 16 02:19:03.040040 ip-10-0-128-111 kubenswrapper[2583]: E0916 02:19:03.040031    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mbxmd" podUID="2fbf440b-4c09-428e-b6bf-1236d95f97cc"
Sep 16 02:19:04.039955 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:19:04.039719    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bdwgr"
Sep 16 02:19:04.040512 ip-10-0-128-111 kubenswrapper[2583]: E0916 02:19:04.040083    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bdwgr" podUID="67a44fbe-95d7-4808-bcff-f7d214c6d82c"
Sep 16 02:19:04.251178 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:19:04.251086    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-psqcm" event={"ID":"bdf274ad-3983-42ba-b712-8ce11da63210","Type":"ContainerStarted","Data":"84dd0561be9b75241e27014b55625f95786c44d196c0cbe54ca10340e4e73bdf"}
Sep 16 02:19:04.252800 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:19:04.252777    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-kf6w7" event={"ID":"2c5d542f-5718-4aeb-8bf5-680a69ddecbc","Type":"ContainerStarted","Data":"61d28951d27f5dc3a78f3b4b76df2011e706fe8919db3fc38b39f6ba3a09be4d"}
Sep 16 02:19:04.254601 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:19:04.254569    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-7nk9m" event={"ID":"e34369f4-88ba-4859-ad03-80219c62e272","Type":"ContainerStarted","Data":"d6a356df3417d201cc6649b7ed2e77d75d550e690910ee11da61ccd784bb0917"}
Sep 16 02:19:04.257237 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:19:04.257189    2583 generic.go:356] "Generic (PLEG): container finished" podID="3a1ad832-d1ab-4676-8eee-31d2eccce648" containerID="ebbade48bd7826d081d7ac470eb5a6ef5827a985ffd7691dccd449c58a6e5e7b" exitCode=0
Sep 16 02:19:04.257327 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:19:04.257300    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-ll7tp" event={"ID":"3a1ad832-d1ab-4676-8eee-31d2eccce648","Type":"ContainerDied","Data":"ebbade48bd7826d081d7ac470eb5a6ef5827a985ffd7691dccd449c58a6e5e7b"}
Sep 16 02:19:04.260608 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:19:04.260579    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-rx99l" event={"ID":"66c3846d-d0d2-4b46-8c59-f6d97a92f7ad","Type":"ContainerStarted","Data":"1e0f0c6e28d350baa73ef82fddd5ac01b18e28e9a7a60f542f12a5dbacb75b7d"}
Sep 16 02:19:04.264254 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:19:04.264233    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-hgb4m" event={"ID":"d45dcef5-f6f3-45d7-b29e-1125a402eeb3","Type":"ContainerStarted","Data":"0f8a45c923b10976c6260a326c7fad2087d6d9f64aed47fd0df72ed5cc1cc6a7"}
Sep 16 02:19:04.264354 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:19:04.264303    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-hgb4m" event={"ID":"d45dcef5-f6f3-45d7-b29e-1125a402eeb3","Type":"ContainerStarted","Data":"4d71a9e5f08df35135de413cbad73671ff81f94790a2794fb3ba4fea4b5a622c"}
Sep 16 02:19:04.264354 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:19:04.264348    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-hgb4m" event={"ID":"d45dcef5-f6f3-45d7-b29e-1125a402eeb3","Type":"ContainerStarted","Data":"570ed7092e042b3b5209f21d399948ad7a412ff45b07ef7da4dfab9e7c87f6ee"}
Sep 16 02:19:04.264470 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:19:04.264384    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-hgb4m" event={"ID":"d45dcef5-f6f3-45d7-b29e-1125a402eeb3","Type":"ContainerStarted","Data":"494693538c335797e7a2c92d5d316e1f222565482e8077f1c641ec2b310249dc"}
Sep 16 02:19:04.264470 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:19:04.264412    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-hgb4m" event={"ID":"d45dcef5-f6f3-45d7-b29e-1125a402eeb3","Type":"ContainerStarted","Data":"d9ac0f55c495928a4dde6578c9cb02ae100fad5cca08c2748a7b72f7ef4cf156"}
Sep 16 02:19:04.264470 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:19:04.264449    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-hgb4m" event={"ID":"d45dcef5-f6f3-45d7-b29e-1125a402eeb3","Type":"ContainerStarted","Data":"03f53b943b189c37c3a44942539bb79e1860ecfa9dfcdee82453659ee8a4163b"}
Sep 16 02:19:04.265064 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:19:04.265021    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-psqcm" podStartSLOduration=10.237102018 podStartE2EDuration="21.26500785s" totalImagesPullingTime="11.027905832s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 02:18:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 02:19:04.264799559 +0000 UTC m=+22.732588347" watchObservedRunningTime="2026-09-16 02:19:04.26500785 +0000 UTC m=+22.732796634"
Sep 16 02:19:04.265695 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:19:04.265652    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-111.ec2.internal" podStartSLOduration=21.265639304 podStartE2EDuration="21.265639304s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 02:18:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 02:18:47.228904885 +0000 UTC m=+5.696693672" watchObservedRunningTime="2026-09-16 02:19:04.265639304 +0000 UTC m=+22.733428085"
Sep 16 02:19:04.266037 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:19:04.266008    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-5nwmf" event={"ID":"82d1b4b2-7910-42f8-b2f3-0d675dc2c530","Type":"ContainerStarted","Data":"cb99b3a4aaa96f203f7c92b70b39e3196a6e1e983b348dd9a47a3e086aad923e"}
Sep 16 02:19:04.267668 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:19:04.267642    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dqkz9" event={"ID":"b9f0b2c7-b0c8-49bf-9761-c387496f9207","Type":"ContainerStarted","Data":"6823d3daacdf3ba9a45e73b8927e053024e1468ea406cd4dd8125e59746e5cda"}
Sep 16 02:19:04.279182 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:19:04.279140    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-rx99l" podStartSLOduration=4.10525611 podStartE2EDuration="22.279128359s" totalImagesPullingTime="18.173872249s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 02:18:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 02:19:04.278695233 +0000 UTC m=+22.746484014" watchObservedRunningTime="2026-09-16 02:19:04.279128359 +0000 UTC m=+22.746917190"
Sep 16 02:19:04.291003 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:19:04.290968    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-kf6w7" podStartSLOduration=4.110682444 podStartE2EDuration="22.290957216s" totalImagesPullingTime="18.180274772s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 02:18:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 02:19:04.29063935 +0000 UTC m=+22.758428131" watchObservedRunningTime="2026-09-16 02:19:04.290957216 +0000 UTC m=+22.758745997"
Sep 16 02:19:04.332695 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:19:04.332650    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-7nk9m" podStartSLOduration=3.107150637 podStartE2EDuration="21.332635378s" totalImagesPullingTime="18.225484741s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 02:18:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 02:19:04.33217145 +0000 UTC m=+22.799960229" watchObservedRunningTime="2026-09-16 02:19:04.332635378 +0000 UTC m=+22.800424159"
Sep 16 02:19:04.346958 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:19:04.346919    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-5nwmf" podStartSLOduration=4.172218021 podStartE2EDuration="22.346905508s" totalImagesPullingTime="18.174687487s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 02:18:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 02:19:04.346655349 +0000 UTC m=+22.814444131" watchObservedRunningTime="2026-09-16 02:19:04.346905508 +0000 UTC m=+22.814694289"
Sep 16 02:19:04.900048 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:19:04.899896    2583 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Sep 16 02:19:04.976569 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:19:04.976449    2583 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-09-16T02:19:04.900042527Z","UUID":"49c11d37-1adc-4b34-995b-a1442783dd41","Handler":null,"Name":"","Endpoint":""}
Sep 16 02:19:04.978012 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:19:04.977987    2583 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Sep 16 02:19:04.978127 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:19:04.978022    2583 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Sep 16 02:19:05.039595 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:19:05.039494    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-55zcf"
Sep 16 02:19:05.039595 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:19:05.039494    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mbxmd"
Sep 16 02:19:05.039790 ip-10-0-128-111 kubenswrapper[2583]: E0916 02:19:05.039621    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-55zcf" podUID="5f66bb1b-6d4f-42c5-9b63-f7f5f301b6f9"
Sep 16 02:19:05.039790 ip-10-0-128-111 kubenswrapper[2583]: E0916 02:19:05.039683    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mbxmd" podUID="2fbf440b-4c09-428e-b6bf-1236d95f97cc"
Sep 16 02:19:05.272083 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:19:05.272049    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dqkz9" event={"ID":"b9f0b2c7-b0c8-49bf-9761-c387496f9207","Type":"ContainerStarted","Data":"681ff22da9565b21ec63213fdf32369b383032f13613c52e2259c1c1a84e8ed9"}
Sep 16 02:19:05.273887 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:19:05.273848    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-lmvcp" event={"ID":"0e5e9542-45a1-4c13-8868-6a73d01f381d","Type":"ContainerStarted","Data":"42172ac60e088616b0ceb918638e8713e777f6f0afa38780b82a03ce1f75970c"}
Sep 16 02:19:05.287532 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:19:05.287490    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-lmvcp" podStartSLOduration=4.104623311 podStartE2EDuration="22.287464693s" totalImagesPullingTime="18.182841382s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 02:18:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 02:19:05.287438971 +0000 UTC m=+23.755227756" watchObservedRunningTime="2026-09-16 02:19:05.287464693 +0000 UTC m=+23.755253471"
Sep 16 02:19:06.039439 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:19:06.039401    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bdwgr"
Sep 16 02:19:06.039618 ip-10-0-128-111 kubenswrapper[2583]: E0916 02:19:06.039581    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bdwgr" podUID="67a44fbe-95d7-4808-bcff-f7d214c6d82c"
Sep 16 02:19:06.280287 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:19:06.280253    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-hgb4m" event={"ID":"d45dcef5-f6f3-45d7-b29e-1125a402eeb3","Type":"ContainerStarted","Data":"08dc8a08635df50d4a3aff9c2a4515a350d902de5579fbdeb8f930febc897aec"}
Sep 16 02:19:06.282618 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:19:06.282585    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dqkz9" event={"ID":"b9f0b2c7-b0c8-49bf-9761-c387496f9207","Type":"ContainerStarted","Data":"233bb5f36f4dd807b9df5c6ebdfce9739f3b540b3d0828a699fa90452265c9ec"}
Sep 16 02:19:06.299798 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:19:06.299745    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dqkz9" podStartSLOduration=3.099296759 podStartE2EDuration="23.299725494s" totalImagesPullingTime="20.200428735s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 02:18:43 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 02:19:06.299676745 +0000 UTC m=+24.767465526" watchObservedRunningTime="2026-09-16 02:19:06.299725494 +0000 UTC m=+24.767514269"
Sep 16 02:19:06.366988 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:19:06.366949    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-psqcm"
Sep 16 02:19:06.367541 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:19:06.367519    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-psqcm"
Sep 16 02:19:07.039824 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:19:07.039788    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mbxmd"
Sep 16 02:19:07.040037 ip-10-0-128-111 kubenswrapper[2583]: E0916 02:19:07.039897    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mbxmd" podUID="2fbf440b-4c09-428e-b6bf-1236d95f97cc"
Sep 16 02:19:07.040037 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:19:07.039960    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-55zcf"
Sep 16 02:19:07.040159 ip-10-0-128-111 kubenswrapper[2583]: E0916 02:19:07.040071    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-55zcf" podUID="5f66bb1b-6d4f-42c5-9b63-f7f5f301b6f9"
Sep 16 02:19:07.285092 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:19:07.285061    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-psqcm"
Sep 16 02:19:07.285621 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:19:07.285470    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-psqcm"
Sep 16 02:19:08.039822 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:19:08.039793    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bdwgr"
Sep 16 02:19:08.039997 ip-10-0-128-111 kubenswrapper[2583]: E0916 02:19:08.039940    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bdwgr" podUID="67a44fbe-95d7-4808-bcff-f7d214c6d82c"
Sep 16 02:19:09.039449 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:19:09.039245    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-55zcf"
Sep 16 02:19:09.040024 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:19:09.039245    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mbxmd"
Sep 16 02:19:09.040024 ip-10-0-128-111 kubenswrapper[2583]: E0916 02:19:09.039477    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-55zcf" podUID="5f66bb1b-6d4f-42c5-9b63-f7f5f301b6f9"
Sep 16 02:19:09.040024 ip-10-0-128-111 kubenswrapper[2583]: E0916 02:19:09.039573    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mbxmd" podUID="2fbf440b-4c09-428e-b6bf-1236d95f97cc"
Sep 16 02:19:09.290241 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:19:09.290158    2583 generic.go:356] "Generic (PLEG): container finished" podID="3a1ad832-d1ab-4676-8eee-31d2eccce648" containerID="baab96c747b826b1f97d3d87ef692bc9372a9a06a602bf7e4be2ebddeabf5a14" exitCode=0
Sep 16 02:19:09.290386 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:19:09.290238    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-ll7tp" event={"ID":"3a1ad832-d1ab-4676-8eee-31d2eccce648","Type":"ContainerDied","Data":"baab96c747b826b1f97d3d87ef692bc9372a9a06a602bf7e4be2ebddeabf5a14"}
Sep 16 02:19:09.293997 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:19:09.293962    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-hgb4m" event={"ID":"d45dcef5-f6f3-45d7-b29e-1125a402eeb3","Type":"ContainerStarted","Data":"05abcb10dbe815a8f425eebc3410b7c22c422002936ff4dc834ae8dcaf51ba59"}
Sep 16 02:19:09.294266 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:19:09.294244    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-hgb4m"
Sep 16 02:19:09.309631 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:19:09.309607    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-hgb4m"
Sep 16 02:19:09.332417 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:19:09.332371    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-hgb4m" podStartSLOduration=9.097841034 podStartE2EDuration="27.332356541s" totalImagesPullingTime="18.234515507s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 02:18:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 02:19:09.332131109 +0000 UTC m=+27.799919894" watchObservedRunningTime="2026-09-16 02:19:09.332356541 +0000 UTC m=+27.800145305"
Sep 16 02:19:09.997054 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:19:09.997023    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-kf6w7_2c5d542f-5718-4aeb-8bf5-680a69ddecbc/dns-node-resolver/0.log"
Sep 16 02:19:10.039300 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:19:10.039271    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bdwgr"
Sep 16 02:19:10.039431 ip-10-0-128-111 kubenswrapper[2583]: E0916 02:19:10.039385    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bdwgr" podUID="67a44fbe-95d7-4808-bcff-f7d214c6d82c"
Sep 16 02:19:10.298529 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:19:10.298431    2583 generic.go:356] "Generic (PLEG): container finished" podID="3a1ad832-d1ab-4676-8eee-31d2eccce648" containerID="f151b55f755953f2fb50c355cb25dba6cd9ad21685b4a99694efc45d37d36a41" exitCode=0
Sep 16 02:19:10.299022 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:19:10.298525    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-ll7tp" event={"ID":"3a1ad832-d1ab-4676-8eee-31d2eccce648","Type":"ContainerDied","Data":"f151b55f755953f2fb50c355cb25dba6cd9ad21685b4a99694efc45d37d36a41"}
Sep 16 02:19:10.299022 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:19:10.298966    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-hgb4m"
Sep 16 02:19:10.299124 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:19:10.299030    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-hgb4m"
Sep 16 02:19:10.315327 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:19:10.315307    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-hgb4m"
Sep 16 02:19:10.335949 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:19:10.335923    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-55zcf"]
Sep 16 02:19:10.336071 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:19:10.336020    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-55zcf"
Sep 16 02:19:10.336140 ip-10-0-128-111 kubenswrapper[2583]: E0916 02:19:10.336116    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-55zcf" podUID="5f66bb1b-6d4f-42c5-9b63-f7f5f301b6f9"
Sep 16 02:19:10.339115 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:19:10.339093    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-mbxmd"]
Sep 16 02:19:10.339223 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:19:10.339190    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mbxmd"
Sep 16 02:19:10.339310 ip-10-0-128-111 kubenswrapper[2583]: E0916 02:19:10.339280    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mbxmd" podUID="2fbf440b-4c09-428e-b6bf-1236d95f97cc"
Sep 16 02:19:10.339730 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:19:10.339709    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-bdwgr"]
Sep 16 02:19:10.339855 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:19:10.339816    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bdwgr"
Sep 16 02:19:10.339926 ip-10-0-128-111 kubenswrapper[2583]: E0916 02:19:10.339906    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bdwgr" podUID="67a44fbe-95d7-4808-bcff-f7d214c6d82c"
Sep 16 02:19:10.779743 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:19:10.779666    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-5nwmf_82d1b4b2-7910-42f8-b2f3-0d675dc2c530/node-ca/0.log"
Sep 16 02:19:11.303257 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:19:11.303172    2583 generic.go:356] "Generic (PLEG): container finished" podID="3a1ad832-d1ab-4676-8eee-31d2eccce648" containerID="762c66d72a44ff0aa228ad45e3645e73f57b8f7f27a3e2cc92de6658799f9f8d" exitCode=0
Sep 16 02:19:11.303641 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:19:11.303256    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-ll7tp" event={"ID":"3a1ad832-d1ab-4676-8eee-31d2eccce648","Type":"ContainerDied","Data":"762c66d72a44ff0aa228ad45e3645e73f57b8f7f27a3e2cc92de6658799f9f8d"}
Sep 16 02:19:12.040391 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:19:12.040351    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mbxmd"
Sep 16 02:19:12.040580 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:19:12.040444    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bdwgr"
Sep 16 02:19:12.040580 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:19:12.040485    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-55zcf"
Sep 16 02:19:12.040580 ip-10-0-128-111 kubenswrapper[2583]: E0916 02:19:12.040499    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mbxmd" podUID="2fbf440b-4c09-428e-b6bf-1236d95f97cc"
Sep 16 02:19:12.040724 ip-10-0-128-111 kubenswrapper[2583]: E0916 02:19:12.040575    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-55zcf" podUID="5f66bb1b-6d4f-42c5-9b63-f7f5f301b6f9"
Sep 16 02:19:12.040724 ip-10-0-128-111 kubenswrapper[2583]: E0916 02:19:12.040663    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bdwgr" podUID="67a44fbe-95d7-4808-bcff-f7d214c6d82c"
Sep 16 02:19:14.039067 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:19:14.039030    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bdwgr"
Sep 16 02:19:14.039450 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:19:14.039169    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mbxmd"
Sep 16 02:19:14.039450 ip-10-0-128-111 kubenswrapper[2583]: E0916 02:19:14.039183    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bdwgr" podUID="67a44fbe-95d7-4808-bcff-f7d214c6d82c"
Sep 16 02:19:14.039450 ip-10-0-128-111 kubenswrapper[2583]: E0916 02:19:14.039260    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mbxmd" podUID="2fbf440b-4c09-428e-b6bf-1236d95f97cc"
Sep 16 02:19:14.039450 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:19:14.039296    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-55zcf"
Sep 16 02:19:14.039450 ip-10-0-128-111 kubenswrapper[2583]: E0916 02:19:14.039350    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-55zcf" podUID="5f66bb1b-6d4f-42c5-9b63-f7f5f301b6f9"
Sep 16 02:19:15.684144 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:19:15.683947    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/67a44fbe-95d7-4808-bcff-f7d214c6d82c-metrics-certs\") pod \"network-metrics-daemon-bdwgr\" (UID: \"67a44fbe-95d7-4808-bcff-f7d214c6d82c\") " pod="openshift-multus/network-metrics-daemon-bdwgr"
Sep 16 02:19:15.684144 ip-10-0-128-111 kubenswrapper[2583]: E0916 02:19:15.684131    2583 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 02:19:15.684714 ip-10-0-128-111 kubenswrapper[2583]: E0916 02:19:15.684211    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/67a44fbe-95d7-4808-bcff-f7d214c6d82c-metrics-certs podName:67a44fbe-95d7-4808-bcff-f7d214c6d82c nodeName:}" failed. No retries permitted until 2026-09-16 02:19:47.684191755 +0000 UTC m=+66.151980517 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/67a44fbe-95d7-4808-bcff-f7d214c6d82c-metrics-certs") pod "network-metrics-daemon-bdwgr" (UID: "67a44fbe-95d7-4808-bcff-f7d214c6d82c") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 02:19:15.784832 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:19:15.784744    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/2fbf440b-4c09-428e-b6bf-1236d95f97cc-original-pull-secret\") pod \"global-pull-secret-syncer-mbxmd\" (UID: \"2fbf440b-4c09-428e-b6bf-1236d95f97cc\") " pod="kube-system/global-pull-secret-syncer-mbxmd"
Sep 16 02:19:15.785081 ip-10-0-128-111 kubenswrapper[2583]: E0916 02:19:15.784878    2583 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 16 02:19:15.785081 ip-10-0-128-111 kubenswrapper[2583]: E0916 02:19:15.784952    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2fbf440b-4c09-428e-b6bf-1236d95f97cc-original-pull-secret podName:2fbf440b-4c09-428e-b6bf-1236d95f97cc nodeName:}" failed. No retries permitted until 2026-09-16 02:19:47.784931254 +0000 UTC m=+66.252720018 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/2fbf440b-4c09-428e-b6bf-1236d95f97cc-original-pull-secret") pod "global-pull-secret-syncer-mbxmd" (UID: "2fbf440b-4c09-428e-b6bf-1236d95f97cc") : object "kube-system"/"original-pull-secret" not registered
Sep 16 02:19:15.785081 ip-10-0-128-111 kubenswrapper[2583]: E0916 02:19:15.784975    2583 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 16 02:19:15.785081 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:19:15.784883    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qwztf\" (UniqueName: \"kubernetes.io/projected/5f66bb1b-6d4f-42c5-9b63-f7f5f301b6f9-kube-api-access-qwztf\") pod \"network-check-target-55zcf\" (UID: \"5f66bb1b-6d4f-42c5-9b63-f7f5f301b6f9\") " pod="openshift-network-diagnostics/network-check-target-55zcf"
Sep 16 02:19:15.785081 ip-10-0-128-111 kubenswrapper[2583]: E0916 02:19:15.784994    2583 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 16 02:19:15.785081 ip-10-0-128-111 kubenswrapper[2583]: E0916 02:19:15.785005    2583 projected.go:196] Error preparing data for projected volume kube-api-access-qwztf for pod openshift-network-diagnostics/network-check-target-55zcf: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 02:19:15.785081 ip-10-0-128-111 kubenswrapper[2583]: E0916 02:19:15.785057    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/5f66bb1b-6d4f-42c5-9b63-f7f5f301b6f9-kube-api-access-qwztf podName:5f66bb1b-6d4f-42c5-9b63-f7f5f301b6f9 nodeName:}" failed. No retries permitted until 2026-09-16 02:19:47.78504258 +0000 UTC m=+66.252831344 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-qwztf" (UniqueName: "kubernetes.io/projected/5f66bb1b-6d4f-42c5-9b63-f7f5f301b6f9-kube-api-access-qwztf") pod "network-check-target-55zcf" (UID: "5f66bb1b-6d4f-42c5-9b63-f7f5f301b6f9") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 02:19:16.039670 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:19:16.039589    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mbxmd"
Sep 16 02:19:16.039670 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:19:16.039623    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-55zcf"
Sep 16 02:19:16.039989 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:19:16.039589    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bdwgr"
Sep 16 02:19:16.039989 ip-10-0-128-111 kubenswrapper[2583]: E0916 02:19:16.039702    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mbxmd" podUID="2fbf440b-4c09-428e-b6bf-1236d95f97cc"
Sep 16 02:19:16.039989 ip-10-0-128-111 kubenswrapper[2583]: E0916 02:19:16.039775    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bdwgr" podUID="67a44fbe-95d7-4808-bcff-f7d214c6d82c"
Sep 16 02:19:16.039989 ip-10-0-128-111 kubenswrapper[2583]: E0916 02:19:16.039871    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-55zcf" podUID="5f66bb1b-6d4f-42c5-9b63-f7f5f301b6f9"
Sep 16 02:19:17.320539 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:19:17.320452    2583 generic.go:356] "Generic (PLEG): container finished" podID="3a1ad832-d1ab-4676-8eee-31d2eccce648" containerID="50a2978a4655a739980b3a041b726dafc91018f15eef7c4542767e800a44fdfa" exitCode=0
Sep 16 02:19:17.320539 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:19:17.320496    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-ll7tp" event={"ID":"3a1ad832-d1ab-4676-8eee-31d2eccce648","Type":"ContainerDied","Data":"50a2978a4655a739980b3a041b726dafc91018f15eef7c4542767e800a44fdfa"}
Sep 16 02:19:18.039191 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:19:18.039160    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mbxmd"
Sep 16 02:19:18.039191 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:19:18.039186    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-55zcf"
Sep 16 02:19:18.039448 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:19:18.039160    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bdwgr"
Sep 16 02:19:18.039448 ip-10-0-128-111 kubenswrapper[2583]: E0916 02:19:18.039285    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mbxmd" podUID="2fbf440b-4c09-428e-b6bf-1236d95f97cc"
Sep 16 02:19:18.039448 ip-10-0-128-111 kubenswrapper[2583]: E0916 02:19:18.039387    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bdwgr" podUID="67a44fbe-95d7-4808-bcff-f7d214c6d82c"
Sep 16 02:19:18.039631 ip-10-0-128-111 kubenswrapper[2583]: E0916 02:19:18.039469    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-55zcf" podUID="5f66bb1b-6d4f-42c5-9b63-f7f5f301b6f9"
Sep 16 02:19:18.325614 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:19:18.325507    2583 generic.go:356] "Generic (PLEG): container finished" podID="3a1ad832-d1ab-4676-8eee-31d2eccce648" containerID="923cc6a3cd82f093c9515cdeceb6c4fcd5548b27495358825cbb4ff478621279" exitCode=0
Sep 16 02:19:18.325614 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:19:18.325570    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-ll7tp" event={"ID":"3a1ad832-d1ab-4676-8eee-31d2eccce648","Type":"ContainerDied","Data":"923cc6a3cd82f093c9515cdeceb6c4fcd5548b27495358825cbb4ff478621279"}
Sep 16 02:19:19.331495 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:19:19.331459    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-ll7tp" event={"ID":"3a1ad832-d1ab-4676-8eee-31d2eccce648","Type":"ContainerStarted","Data":"1bb431fd1aa1f3e08c16c509a177217dc99e010c414d3c9617922fd0250f1f0c"}
Sep 16 02:19:19.352892 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:19:19.352844    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-ll7tp" podStartSLOduration=6.958319672 podStartE2EDuration="36.352829584s" totalImagesPullingTime="29.394509912s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 02:18:43 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 02:19:19.352418069 +0000 UTC m=+37.820206852" watchObservedRunningTime="2026-09-16 02:19:19.352829584 +0000 UTC m=+37.820618365"
Sep 16 02:19:20.039037 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:19:20.038998    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bdwgr"
Sep 16 02:19:20.039192 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:19:20.038998    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mbxmd"
Sep 16 02:19:20.039192 ip-10-0-128-111 kubenswrapper[2583]: E0916 02:19:20.039121    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bdwgr" podUID="67a44fbe-95d7-4808-bcff-f7d214c6d82c"
Sep 16 02:19:20.039266 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:19:20.039003    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-55zcf"
Sep 16 02:19:20.039266 ip-10-0-128-111 kubenswrapper[2583]: E0916 02:19:20.039206    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mbxmd" podUID="2fbf440b-4c09-428e-b6bf-1236d95f97cc"
Sep 16 02:19:20.039342 ip-10-0-128-111 kubenswrapper[2583]: E0916 02:19:20.039323    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-55zcf" podUID="5f66bb1b-6d4f-42c5-9b63-f7f5f301b6f9"
Sep 16 02:19:22.040532 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:19:22.040490    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mbxmd"
Sep 16 02:19:22.041006 ip-10-0-128-111 kubenswrapper[2583]: E0916 02:19:22.040602    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mbxmd" podUID="2fbf440b-4c09-428e-b6bf-1236d95f97cc"
Sep 16 02:19:22.041006 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:19:22.040692    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-55zcf"
Sep 16 02:19:22.041006 ip-10-0-128-111 kubenswrapper[2583]: E0916 02:19:22.040779    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-55zcf" podUID="5f66bb1b-6d4f-42c5-9b63-f7f5f301b6f9"
Sep 16 02:19:22.041006 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:19:22.040868    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bdwgr"
Sep 16 02:19:22.041006 ip-10-0-128-111 kubenswrapper[2583]: E0916 02:19:22.040963    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bdwgr" podUID="67a44fbe-95d7-4808-bcff-f7d214c6d82c"
Sep 16 02:19:24.042757 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:19:24.042722    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bdwgr"
Sep 16 02:19:24.043149 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:19:24.042727    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-55zcf"
Sep 16 02:19:24.043149 ip-10-0-128-111 kubenswrapper[2583]: E0916 02:19:24.042831    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bdwgr" podUID="67a44fbe-95d7-4808-bcff-f7d214c6d82c"
Sep 16 02:19:24.043149 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:19:24.042843    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mbxmd"
Sep 16 02:19:24.043149 ip-10-0-128-111 kubenswrapper[2583]: E0916 02:19:24.042894    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-55zcf" podUID="5f66bb1b-6d4f-42c5-9b63-f7f5f301b6f9"
Sep 16 02:19:24.043149 ip-10-0-128-111 kubenswrapper[2583]: E0916 02:19:24.042944    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mbxmd" podUID="2fbf440b-4c09-428e-b6bf-1236d95f97cc"
Sep 16 02:19:26.039665 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:19:26.039623    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bdwgr"
Sep 16 02:19:26.040140 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:19:26.039623    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mbxmd"
Sep 16 02:19:26.040140 ip-10-0-128-111 kubenswrapper[2583]: E0916 02:19:26.039748    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bdwgr" podUID="67a44fbe-95d7-4808-bcff-f7d214c6d82c"
Sep 16 02:19:26.040140 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:19:26.039633    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-55zcf"
Sep 16 02:19:26.040140 ip-10-0-128-111 kubenswrapper[2583]: E0916 02:19:26.039809    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mbxmd" podUID="2fbf440b-4c09-428e-b6bf-1236d95f97cc"
Sep 16 02:19:26.040140 ip-10-0-128-111 kubenswrapper[2583]: E0916 02:19:26.039878    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-55zcf" podUID="5f66bb1b-6d4f-42c5-9b63-f7f5f301b6f9"
Sep 16 02:19:28.039511 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:19:28.039471    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mbxmd"
Sep 16 02:19:28.040056 ip-10-0-128-111 kubenswrapper[2583]: E0916 02:19:28.039600    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mbxmd" podUID="2fbf440b-4c09-428e-b6bf-1236d95f97cc"
Sep 16 02:19:28.040056 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:19:28.039482    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bdwgr"
Sep 16 02:19:28.040056 ip-10-0-128-111 kubenswrapper[2583]: E0916 02:19:28.039679    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bdwgr" podUID="67a44fbe-95d7-4808-bcff-f7d214c6d82c"
Sep 16 02:19:28.040056 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:19:28.039471    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-55zcf"
Sep 16 02:19:28.040056 ip-10-0-128-111 kubenswrapper[2583]: E0916 02:19:28.039774    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-55zcf" podUID="5f66bb1b-6d4f-42c5-9b63-f7f5f301b6f9"
Sep 16 02:19:30.039830 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:19:30.039792    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mbxmd"
Sep 16 02:19:30.039830 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:19:30.039807    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bdwgr"
Sep 16 02:19:30.040331 ip-10-0-128-111 kubenswrapper[2583]: E0916 02:19:30.039911    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mbxmd" podUID="2fbf440b-4c09-428e-b6bf-1236d95f97cc"
Sep 16 02:19:30.040331 ip-10-0-128-111 kubenswrapper[2583]: E0916 02:19:30.039954    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bdwgr" podUID="67a44fbe-95d7-4808-bcff-f7d214c6d82c"
Sep 16 02:19:30.040331 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:19:30.039994    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-55zcf"
Sep 16 02:19:30.040331 ip-10-0-128-111 kubenswrapper[2583]: E0916 02:19:30.040092    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-55zcf" podUID="5f66bb1b-6d4f-42c5-9b63-f7f5f301b6f9"
Sep 16 02:19:32.040158 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:19:32.040129    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mbxmd"
Sep 16 02:19:32.040666 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:19:32.040219    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bdwgr"
Sep 16 02:19:32.040666 ip-10-0-128-111 kubenswrapper[2583]: E0916 02:19:32.040243    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mbxmd" podUID="2fbf440b-4c09-428e-b6bf-1236d95f97cc"
Sep 16 02:19:32.040666 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:19:32.040348    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-55zcf"
Sep 16 02:19:32.040666 ip-10-0-128-111 kubenswrapper[2583]: E0916 02:19:32.040377    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bdwgr" podUID="67a44fbe-95d7-4808-bcff-f7d214c6d82c"
Sep 16 02:19:32.040666 ip-10-0-128-111 kubenswrapper[2583]: E0916 02:19:32.040434    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-55zcf" podUID="5f66bb1b-6d4f-42c5-9b63-f7f5f301b6f9"
Sep 16 02:19:34.039763 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:19:34.039727    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-55zcf"
Sep 16 02:19:34.039763 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:19:34.039745    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bdwgr"
Sep 16 02:19:34.040257 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:19:34.039729    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mbxmd"
Sep 16 02:19:34.040257 ip-10-0-128-111 kubenswrapper[2583]: E0916 02:19:34.039828    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-55zcf" podUID="5f66bb1b-6d4f-42c5-9b63-f7f5f301b6f9"
Sep 16 02:19:34.040257 ip-10-0-128-111 kubenswrapper[2583]: E0916 02:19:34.039900    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mbxmd" podUID="2fbf440b-4c09-428e-b6bf-1236d95f97cc"
Sep 16 02:19:34.040257 ip-10-0-128-111 kubenswrapper[2583]: E0916 02:19:34.039958    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bdwgr" podUID="67a44fbe-95d7-4808-bcff-f7d214c6d82c"
Sep 16 02:19:34.307179 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:19:34.307098    2583 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-111.ec2.internal" event="NodeReady"
Sep 16 02:19:34.307337 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:19:34.307237    2583 kubelet_node_status.go:489] "Fast updating node status as it just became ready"
Sep 16 02:19:34.355806 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:19:34.355770    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-ntjs4"]
Sep 16 02:19:34.357633 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:19:34.357618    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-l8hmx"]
Sep 16 02:19:34.357751 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:19:34.357734    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-ntjs4"
Sep 16 02:19:34.359096 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:19:34.359078    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-l8hmx"
Sep 16 02:19:34.360626 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:19:34.360605    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Sep 16 02:19:34.360730 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:19:34.360716    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Sep 16 02:19:34.360832 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:19:34.360817    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-74plt\""
Sep 16 02:19:34.361458 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:19:34.361424    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Sep 16 02:19:34.361566 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:19:34.361519    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Sep 16 02:19:34.361843 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:19:34.361818    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Sep 16 02:19:34.361843 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:19:34.361833    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Sep 16 02:19:34.361965 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:19:34.361834    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Sep 16 02:19:34.365403 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:19:34.363020    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-p58tv\""
Sep 16 02:19:34.370308 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:19:34.370285    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-l8hmx"]
Sep 16 02:19:34.373187 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:19:34.373129    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-ntjs4"]
Sep 16 02:19:34.377201 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:19:34.377172    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-v2h5n"]
Sep 16 02:19:34.378746 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:19:34.378732    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-v2h5n"
Sep 16 02:19:34.381124 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:19:34.381100    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Sep 16 02:19:34.381200 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:19:34.381146    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Sep 16 02:19:34.381270 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:19:34.381255    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-w4x8g\""
Sep 16 02:19:34.391651 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:19:34.391629    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-v2h5n"]
Sep 16 02:19:34.421111 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:19:34.421080    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vgstm\" (UniqueName: \"kubernetes.io/projected/93090997-c69e-4547-bf00-2e6650ac00eb-kube-api-access-vgstm\") pod \"insights-runtime-extractor-ntjs4\" (UID: \"93090997-c69e-4547-bf00-2e6650ac00eb\") " pod="openshift-insights/insights-runtime-extractor-ntjs4"
Sep 16 02:19:34.421111 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:19:34.421111    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/216004c9-119c-415e-b79e-634e7382c0ea-metrics-tls\") pod \"dns-default-v2h5n\" (UID: \"216004c9-119c-415e-b79e-634e7382c0ea\") " pod="openshift-dns/dns-default-v2h5n"
Sep 16 02:19:34.421283 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:19:34.421150    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vkmrh\" (UniqueName: \"kubernetes.io/projected/c66d34de-0500-4a6a-9fd7-056ecb784bf7-kube-api-access-vkmrh\") pod \"ingress-canary-l8hmx\" (UID: \"c66d34de-0500-4a6a-9fd7-056ecb784bf7\") " pod="openshift-ingress-canary/ingress-canary-l8hmx"
Sep 16 02:19:34.421283 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:19:34.421222    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/93090997-c69e-4547-bf00-2e6650ac00eb-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-ntjs4\" (UID: \"93090997-c69e-4547-bf00-2e6650ac00eb\") " pod="openshift-insights/insights-runtime-extractor-ntjs4"
Sep 16 02:19:34.421283 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:19:34.421256    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b2zw6\" (UniqueName: \"kubernetes.io/projected/216004c9-119c-415e-b79e-634e7382c0ea-kube-api-access-b2zw6\") pod \"dns-default-v2h5n\" (UID: \"216004c9-119c-415e-b79e-634e7382c0ea\") " pod="openshift-dns/dns-default-v2h5n"
Sep 16 02:19:34.421283 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:19:34.421274    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/93090997-c69e-4547-bf00-2e6650ac00eb-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-ntjs4\" (UID: \"93090997-c69e-4547-bf00-2e6650ac00eb\") " pod="openshift-insights/insights-runtime-extractor-ntjs4"
Sep 16 02:19:34.421416 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:19:34.421293    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/216004c9-119c-415e-b79e-634e7382c0ea-tmp-dir\") pod \"dns-default-v2h5n\" (UID: \"216004c9-119c-415e-b79e-634e7382c0ea\") " pod="openshift-dns/dns-default-v2h5n"
Sep 16 02:19:34.421416 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:19:34.421309    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/93090997-c69e-4547-bf00-2e6650ac00eb-crio-socket\") pod \"insights-runtime-extractor-ntjs4\" (UID: \"93090997-c69e-4547-bf00-2e6650ac00eb\") " pod="openshift-insights/insights-runtime-extractor-ntjs4"
Sep 16 02:19:34.421416 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:19:34.421326    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/93090997-c69e-4547-bf00-2e6650ac00eb-data-volume\") pod \"insights-runtime-extractor-ntjs4\" (UID: \"93090997-c69e-4547-bf00-2e6650ac00eb\") " pod="openshift-insights/insights-runtime-extractor-ntjs4"
Sep 16 02:19:34.421416 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:19:34.421354    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/216004c9-119c-415e-b79e-634e7382c0ea-config-volume\") pod \"dns-default-v2h5n\" (UID: \"216004c9-119c-415e-b79e-634e7382c0ea\") " pod="openshift-dns/dns-default-v2h5n"
Sep 16 02:19:34.421623 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:19:34.421415    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/c66d34de-0500-4a6a-9fd7-056ecb784bf7-cert\") pod \"ingress-canary-l8hmx\" (UID: \"c66d34de-0500-4a6a-9fd7-056ecb784bf7\") " pod="openshift-ingress-canary/ingress-canary-l8hmx"
Sep 16 02:19:34.522433 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:19:34.522400    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/c66d34de-0500-4a6a-9fd7-056ecb784bf7-cert\") pod \"ingress-canary-l8hmx\" (UID: \"c66d34de-0500-4a6a-9fd7-056ecb784bf7\") " pod="openshift-ingress-canary/ingress-canary-l8hmx"
Sep 16 02:19:34.522433 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:19:34.522436    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vgstm\" (UniqueName: \"kubernetes.io/projected/93090997-c69e-4547-bf00-2e6650ac00eb-kube-api-access-vgstm\") pod \"insights-runtime-extractor-ntjs4\" (UID: \"93090997-c69e-4547-bf00-2e6650ac00eb\") " pod="openshift-insights/insights-runtime-extractor-ntjs4"
Sep 16 02:19:34.522655 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:19:34.522455    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/216004c9-119c-415e-b79e-634e7382c0ea-metrics-tls\") pod \"dns-default-v2h5n\" (UID: \"216004c9-119c-415e-b79e-634e7382c0ea\") " pod="openshift-dns/dns-default-v2h5n"
Sep 16 02:19:34.522655 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:19:34.522493    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vkmrh\" (UniqueName: \"kubernetes.io/projected/c66d34de-0500-4a6a-9fd7-056ecb784bf7-kube-api-access-vkmrh\") pod \"ingress-canary-l8hmx\" (UID: \"c66d34de-0500-4a6a-9fd7-056ecb784bf7\") " pod="openshift-ingress-canary/ingress-canary-l8hmx"
Sep 16 02:19:34.522655 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:19:34.522522    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/93090997-c69e-4547-bf00-2e6650ac00eb-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-ntjs4\" (UID: \"93090997-c69e-4547-bf00-2e6650ac00eb\") " pod="openshift-insights/insights-runtime-extractor-ntjs4"
Sep 16 02:19:34.522655 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:19:34.522538    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b2zw6\" (UniqueName: \"kubernetes.io/projected/216004c9-119c-415e-b79e-634e7382c0ea-kube-api-access-b2zw6\") pod \"dns-default-v2h5n\" (UID: \"216004c9-119c-415e-b79e-634e7382c0ea\") " pod="openshift-dns/dns-default-v2h5n"
Sep 16 02:19:34.522655 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:19:34.522571    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/93090997-c69e-4547-bf00-2e6650ac00eb-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-ntjs4\" (UID: \"93090997-c69e-4547-bf00-2e6650ac00eb\") " pod="openshift-insights/insights-runtime-extractor-ntjs4"
Sep 16 02:19:34.522655 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:19:34.522592    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/216004c9-119c-415e-b79e-634e7382c0ea-tmp-dir\") pod \"dns-default-v2h5n\" (UID: \"216004c9-119c-415e-b79e-634e7382c0ea\") " pod="openshift-dns/dns-default-v2h5n"
Sep 16 02:19:34.522936 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:19:34.522659    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/93090997-c69e-4547-bf00-2e6650ac00eb-crio-socket\") pod \"insights-runtime-extractor-ntjs4\" (UID: \"93090997-c69e-4547-bf00-2e6650ac00eb\") " pod="openshift-insights/insights-runtime-extractor-ntjs4"
Sep 16 02:19:34.522936 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:19:34.522708    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/93090997-c69e-4547-bf00-2e6650ac00eb-data-volume\") pod \"insights-runtime-extractor-ntjs4\" (UID: \"93090997-c69e-4547-bf00-2e6650ac00eb\") " pod="openshift-insights/insights-runtime-extractor-ntjs4"
Sep 16 02:19:34.522936 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:19:34.522733    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/216004c9-119c-415e-b79e-634e7382c0ea-config-volume\") pod \"dns-default-v2h5n\" (UID: \"216004c9-119c-415e-b79e-634e7382c0ea\") " pod="openshift-dns/dns-default-v2h5n"
Sep 16 02:19:34.522936 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:19:34.522909    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/216004c9-119c-415e-b79e-634e7382c0ea-tmp-dir\") pod \"dns-default-v2h5n\" (UID: \"216004c9-119c-415e-b79e-634e7382c0ea\") " pod="openshift-dns/dns-default-v2h5n"
Sep 16 02:19:34.522936 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:19:34.522919    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/93090997-c69e-4547-bf00-2e6650ac00eb-crio-socket\") pod \"insights-runtime-extractor-ntjs4\" (UID: \"93090997-c69e-4547-bf00-2e6650ac00eb\") " pod="openshift-insights/insights-runtime-extractor-ntjs4"
Sep 16 02:19:34.523141 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:19:34.523075    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/93090997-c69e-4547-bf00-2e6650ac00eb-data-volume\") pod \"insights-runtime-extractor-ntjs4\" (UID: \"93090997-c69e-4547-bf00-2e6650ac00eb\") " pod="openshift-insights/insights-runtime-extractor-ntjs4"
Sep 16 02:19:34.523201 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:19:34.523189    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/93090997-c69e-4547-bf00-2e6650ac00eb-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-ntjs4\" (UID: \"93090997-c69e-4547-bf00-2e6650ac00eb\") " pod="openshift-insights/insights-runtime-extractor-ntjs4"
Sep 16 02:19:34.523369 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:19:34.523349    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/216004c9-119c-415e-b79e-634e7382c0ea-config-volume\") pod \"dns-default-v2h5n\" (UID: \"216004c9-119c-415e-b79e-634e7382c0ea\") " pod="openshift-dns/dns-default-v2h5n"
Sep 16 02:19:34.526470 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:19:34.526440    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/93090997-c69e-4547-bf00-2e6650ac00eb-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-ntjs4\" (UID: \"93090997-c69e-4547-bf00-2e6650ac00eb\") " pod="openshift-insights/insights-runtime-extractor-ntjs4"
Sep 16 02:19:34.526470 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:19:34.526460    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/216004c9-119c-415e-b79e-634e7382c0ea-metrics-tls\") pod \"dns-default-v2h5n\" (UID: \"216004c9-119c-415e-b79e-634e7382c0ea\") " pod="openshift-dns/dns-default-v2h5n"
Sep 16 02:19:34.526639 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:19:34.526483    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/c66d34de-0500-4a6a-9fd7-056ecb784bf7-cert\") pod \"ingress-canary-l8hmx\" (UID: \"c66d34de-0500-4a6a-9fd7-056ecb784bf7\") " pod="openshift-ingress-canary/ingress-canary-l8hmx"
Sep 16 02:19:34.532845 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:19:34.532812    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vgstm\" (UniqueName: \"kubernetes.io/projected/93090997-c69e-4547-bf00-2e6650ac00eb-kube-api-access-vgstm\") pod \"insights-runtime-extractor-ntjs4\" (UID: \"93090997-c69e-4547-bf00-2e6650ac00eb\") " pod="openshift-insights/insights-runtime-extractor-ntjs4"
Sep 16 02:19:34.533008 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:19:34.532986    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-b2zw6\" (UniqueName: \"kubernetes.io/projected/216004c9-119c-415e-b79e-634e7382c0ea-kube-api-access-b2zw6\") pod \"dns-default-v2h5n\" (UID: \"216004c9-119c-415e-b79e-634e7382c0ea\") " pod="openshift-dns/dns-default-v2h5n"
Sep 16 02:19:34.533102 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:19:34.533083    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vkmrh\" (UniqueName: \"kubernetes.io/projected/c66d34de-0500-4a6a-9fd7-056ecb784bf7-kube-api-access-vkmrh\") pod \"ingress-canary-l8hmx\" (UID: \"c66d34de-0500-4a6a-9fd7-056ecb784bf7\") " pod="openshift-ingress-canary/ingress-canary-l8hmx"
Sep 16 02:19:34.582479 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:19:34.582401    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-z28pr"]
Sep 16 02:19:34.584991 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:19:34.584971    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-z28pr"
Sep 16 02:19:34.587282 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:19:34.587262    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-tls\""
Sep 16 02:19:34.587282 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:19:34.587273    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-dockercfg-jb28x\""
Sep 16 02:19:34.594431 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:19:34.594413    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-z28pr"]
Sep 16 02:19:34.624023 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:19:34.623995    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/a97c117e-0ae7-48e7-ba8e-b100f553ab2b-tls-certificates\") pod \"prometheus-operator-admission-webhook-5864cd99d7-z28pr\" (UID: \"a97c117e-0ae7-48e7-ba8e-b100f553ab2b\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-z28pr"
Sep 16 02:19:34.670048 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:19:34.670017    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-ntjs4"
Sep 16 02:19:34.675817 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:19:34.675800    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-l8hmx"
Sep 16 02:19:34.686701 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:19:34.686673    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-v2h5n"
Sep 16 02:19:34.725823 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:19:34.725774    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/a97c117e-0ae7-48e7-ba8e-b100f553ab2b-tls-certificates\") pod \"prometheus-operator-admission-webhook-5864cd99d7-z28pr\" (UID: \"a97c117e-0ae7-48e7-ba8e-b100f553ab2b\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-z28pr"
Sep 16 02:19:34.726625 ip-10-0-128-111 kubenswrapper[2583]: E0916 02:19:34.725921    2583 secret.go:189] Couldn't get secret openshift-monitoring/prometheus-operator-admission-webhook-tls: secret "prometheus-operator-admission-webhook-tls" not found
Sep 16 02:19:34.726625 ip-10-0-128-111 kubenswrapper[2583]: E0916 02:19:34.726004    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a97c117e-0ae7-48e7-ba8e-b100f553ab2b-tls-certificates podName:a97c117e-0ae7-48e7-ba8e-b100f553ab2b nodeName:}" failed. No retries permitted until 2026-09-16 02:19:35.225974322 +0000 UTC m=+53.693763102 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tls-certificates" (UniqueName: "kubernetes.io/secret/a97c117e-0ae7-48e7-ba8e-b100f553ab2b-tls-certificates") pod "prometheus-operator-admission-webhook-5864cd99d7-z28pr" (UID: "a97c117e-0ae7-48e7-ba8e-b100f553ab2b") : secret "prometheus-operator-admission-webhook-tls" not found
Sep 16 02:19:34.949635 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:19:34.949601    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-ntjs4"]
Sep 16 02:19:34.952326 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:19:34.952302    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-l8hmx"]
Sep 16 02:19:34.953282 ip-10-0-128-111 kubenswrapper[2583]: W0916 02:19:34.953256    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod93090997_c69e_4547_bf00_2e6650ac00eb.slice/crio-e0e641b3780093e4b399a13f939b43a12245576111295b201495d291d7fe5bf5 WatchSource:0}: Error finding container e0e641b3780093e4b399a13f939b43a12245576111295b201495d291d7fe5bf5: Status 404 returned error can't find the container with id e0e641b3780093e4b399a13f939b43a12245576111295b201495d291d7fe5bf5
Sep 16 02:19:34.955507 ip-10-0-128-111 kubenswrapper[2583]: W0916 02:19:34.955485    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc66d34de_0500_4a6a_9fd7_056ecb784bf7.slice/crio-501f5af1bfa3318785d17b05660f1d074357c52fa814766e81cb683602a51f9e WatchSource:0}: Error finding container 501f5af1bfa3318785d17b05660f1d074357c52fa814766e81cb683602a51f9e: Status 404 returned error can't find the container with id 501f5af1bfa3318785d17b05660f1d074357c52fa814766e81cb683602a51f9e
Sep 16 02:19:34.982976 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:19:34.982953    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-v2h5n"]
Sep 16 02:19:34.985587 ip-10-0-128-111 kubenswrapper[2583]: W0916 02:19:34.985562    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod216004c9_119c_415e_b79e_634e7382c0ea.slice/crio-e6ebd3cd95358d78ee33886c056e34bce78d7e834323be538e74215009c6c54c WatchSource:0}: Error finding container e6ebd3cd95358d78ee33886c056e34bce78d7e834323be538e74215009c6c54c: Status 404 returned error can't find the container with id e6ebd3cd95358d78ee33886c056e34bce78d7e834323be538e74215009c6c54c
Sep 16 02:19:35.231446 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:19:35.231343    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/a97c117e-0ae7-48e7-ba8e-b100f553ab2b-tls-certificates\") pod \"prometheus-operator-admission-webhook-5864cd99d7-z28pr\" (UID: \"a97c117e-0ae7-48e7-ba8e-b100f553ab2b\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-z28pr"
Sep 16 02:19:35.233671 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:19:35.233654    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/a97c117e-0ae7-48e7-ba8e-b100f553ab2b-tls-certificates\") pod \"prometheus-operator-admission-webhook-5864cd99d7-z28pr\" (UID: \"a97c117e-0ae7-48e7-ba8e-b100f553ab2b\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-z28pr"
Sep 16 02:19:35.370860 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:19:35.370825    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-v2h5n" event={"ID":"216004c9-119c-415e-b79e-634e7382c0ea","Type":"ContainerStarted","Data":"e6ebd3cd95358d78ee33886c056e34bce78d7e834323be538e74215009c6c54c"}
Sep 16 02:19:35.371962 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:19:35.371941    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-l8hmx" event={"ID":"c66d34de-0500-4a6a-9fd7-056ecb784bf7","Type":"ContainerStarted","Data":"501f5af1bfa3318785d17b05660f1d074357c52fa814766e81cb683602a51f9e"}
Sep 16 02:19:35.373340 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:19:35.373314    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-ntjs4" event={"ID":"93090997-c69e-4547-bf00-2e6650ac00eb","Type":"ContainerStarted","Data":"2e245eca04a3119d382bc3c90ef2abf9170670333aad16748150d536563d4c43"}
Sep 16 02:19:35.373488 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:19:35.373360    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-ntjs4" event={"ID":"93090997-c69e-4547-bf00-2e6650ac00eb","Type":"ContainerStarted","Data":"e0e641b3780093e4b399a13f939b43a12245576111295b201495d291d7fe5bf5"}
Sep 16 02:19:35.493949 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:19:35.493869    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-z28pr"
Sep 16 02:19:35.887760 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:19:35.887723    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-z28pr"]
Sep 16 02:19:35.891246 ip-10-0-128-111 kubenswrapper[2583]: W0916 02:19:35.891214    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda97c117e_0ae7_48e7_ba8e_b100f553ab2b.slice/crio-8c25510522b139eab1a1b1db8db32ec674368001b82543c993a06733ee87cf87 WatchSource:0}: Error finding container 8c25510522b139eab1a1b1db8db32ec674368001b82543c993a06733ee87cf87: Status 404 returned error can't find the container with id 8c25510522b139eab1a1b1db8db32ec674368001b82543c993a06733ee87cf87
Sep 16 02:19:36.039108 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:19:36.039078    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mbxmd"
Sep 16 02:19:36.039530 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:19:36.039504    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-55zcf"
Sep 16 02:19:36.040505 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:19:36.040477    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bdwgr"
Sep 16 02:19:36.044512 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:19:36.041994    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Sep 16 02:19:36.044512 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:19:36.042100    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-7ktl5\""
Sep 16 02:19:36.044512 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:19:36.042545    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Sep 16 02:19:36.044512 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:19:36.043289    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Sep 16 02:19:36.044512 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:19:36.043498    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-ss74q\""
Sep 16 02:19:36.044512 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:19:36.043619    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Sep 16 02:19:36.379432 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:19:36.379391    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-ntjs4" event={"ID":"93090997-c69e-4547-bf00-2e6650ac00eb","Type":"ContainerStarted","Data":"a150d55ca5d932c9d7c6590101181a0a15e3e0dfa6584ac35ae89c1fb141eef1"}
Sep 16 02:19:36.381205 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:19:36.381174    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-z28pr" event={"ID":"a97c117e-0ae7-48e7-ba8e-b100f553ab2b","Type":"ContainerStarted","Data":"8c25510522b139eab1a1b1db8db32ec674368001b82543c993a06733ee87cf87"}
Sep 16 02:19:38.389112 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:19:38.389076    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-l8hmx" event={"ID":"c66d34de-0500-4a6a-9fd7-056ecb784bf7","Type":"ContainerStarted","Data":"f0e89c8cbc4ed97ce0ba82b1a5e1598a4175e5cd20ba7ba7cfdecaa6708defe7"}
Sep 16 02:19:38.391363 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:19:38.391335    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-ntjs4" event={"ID":"93090997-c69e-4547-bf00-2e6650ac00eb","Type":"ContainerStarted","Data":"19d077bfe4e86437fd34c138e8da13e38150bf6ba6a317bcc769d7c09239d180"}
Sep 16 02:19:38.393106 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:19:38.393081    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-z28pr" event={"ID":"a97c117e-0ae7-48e7-ba8e-b100f553ab2b","Type":"ContainerStarted","Data":"a59d705780ad6af8276a14d2d9b786edc4c205a8df7e2dde884410ea48dfa5ae"}
Sep 16 02:19:38.393225 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:19:38.393205    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-z28pr"
Sep 16 02:19:38.394926 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:19:38.394900    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-v2h5n" event={"ID":"216004c9-119c-415e-b79e-634e7382c0ea","Type":"ContainerStarted","Data":"b9d091a872ec9e7cd9506b950f5285f5f5137c288492d6c13486c27e178d35c0"}
Sep 16 02:19:38.395042 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:19:38.394947    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-v2h5n" event={"ID":"216004c9-119c-415e-b79e-634e7382c0ea","Type":"ContainerStarted","Data":"ccbdf44d42bbe9b05f19b0cc8b2f2ad23605e1bdb59c6f23bb7d087fb09093ba"}
Sep 16 02:19:38.395097 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:19:38.395043    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-v2h5n"
Sep 16 02:19:38.398493 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:19:38.398474    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-z28pr"
Sep 16 02:19:38.404220 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:19:38.404183    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-l8hmx" podStartSLOduration=1.296214854 podStartE2EDuration="4.404170834s" totalImagesPullingTime="3.10795598s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 02:19:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 02:19:38.403360722 +0000 UTC m=+56.871149501" watchObservedRunningTime="2026-09-16 02:19:38.404170834 +0000 UTC m=+56.871959616"
Sep 16 02:19:38.420061 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:19:38.420023    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-v2h5n" podStartSLOduration=1.345209923 podStartE2EDuration="4.420005767s" totalImagesPullingTime="3.074795844s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 02:19:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 02:19:38.418988726 +0000 UTC m=+56.886777508" watchObservedRunningTime="2026-09-16 02:19:38.420005767 +0000 UTC m=+56.887794550"
Sep 16 02:19:38.434625 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:19:38.434582    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-ntjs4" podStartSLOduration=1.470124053 podStartE2EDuration="4.434570209s" totalImagesPullingTime="2.964446156s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 02:19:34 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 02:19:38.433583793 +0000 UTC m=+56.901372576" watchObservedRunningTime="2026-09-16 02:19:38.434570209 +0000 UTC m=+56.902358987"
Sep 16 02:19:38.450275 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:19:38.450232    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-z28pr" podStartSLOduration=2.278591224 podStartE2EDuration="4.450219336s" totalImagesPullingTime="2.171628112s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 02:19:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 02:19:38.449793291 +0000 UTC m=+56.917582074" watchObservedRunningTime="2026-09-16 02:19:38.450219336 +0000 UTC m=+56.918008118"
Sep 16 02:19:38.631260 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:19:38.631219    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-68bcd7f987-fk5jv"]
Sep 16 02:19:38.661414 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:19:38.661345    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-68bcd7f987-fk5jv"]
Sep 16 02:19:38.661533 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:19:38.661439    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-68bcd7f987-fk5jv"
Sep 16 02:19:38.664306 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:19:38.664277    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-kube-rbac-proxy-config\""
Sep 16 02:19:38.664306 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:19:38.664305    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-tls\""
Sep 16 02:19:38.664513 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:19:38.664322    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-dockercfg-427pm\""
Sep 16 02:19:38.664513 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:19:38.664308    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Sep 16 02:19:38.664513 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:19:38.664281    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Sep 16 02:19:38.664513 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:19:38.664420    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Sep 16 02:19:38.760199 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:19:38.760163    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/529830e9-72ae-4032-899d-6af9f83105ea-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-68bcd7f987-fk5jv\" (UID: \"529830e9-72ae-4032-899d-6af9f83105ea\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-fk5jv"
Sep 16 02:19:38.760342 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:19:38.760212    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/529830e9-72ae-4032-899d-6af9f83105ea-prometheus-operator-tls\") pod \"prometheus-operator-68bcd7f987-fk5jv\" (UID: \"529830e9-72ae-4032-899d-6af9f83105ea\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-fk5jv"
Sep 16 02:19:38.760342 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:19:38.760302    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/529830e9-72ae-4032-899d-6af9f83105ea-metrics-client-ca\") pod \"prometheus-operator-68bcd7f987-fk5jv\" (UID: \"529830e9-72ae-4032-899d-6af9f83105ea\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-fk5jv"
Sep 16 02:19:38.760342 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:19:38.760333    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lz74q\" (UniqueName: \"kubernetes.io/projected/529830e9-72ae-4032-899d-6af9f83105ea-kube-api-access-lz74q\") pod \"prometheus-operator-68bcd7f987-fk5jv\" (UID: \"529830e9-72ae-4032-899d-6af9f83105ea\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-fk5jv"
Sep 16 02:19:38.860865 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:19:38.860828    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/529830e9-72ae-4032-899d-6af9f83105ea-metrics-client-ca\") pod \"prometheus-operator-68bcd7f987-fk5jv\" (UID: \"529830e9-72ae-4032-899d-6af9f83105ea\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-fk5jv"
Sep 16 02:19:38.861007 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:19:38.860882    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lz74q\" (UniqueName: \"kubernetes.io/projected/529830e9-72ae-4032-899d-6af9f83105ea-kube-api-access-lz74q\") pod \"prometheus-operator-68bcd7f987-fk5jv\" (UID: \"529830e9-72ae-4032-899d-6af9f83105ea\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-fk5jv"
Sep 16 02:19:38.861007 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:19:38.860934    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/529830e9-72ae-4032-899d-6af9f83105ea-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-68bcd7f987-fk5jv\" (UID: \"529830e9-72ae-4032-899d-6af9f83105ea\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-fk5jv"
Sep 16 02:19:38.861007 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:19:38.860968    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/529830e9-72ae-4032-899d-6af9f83105ea-prometheus-operator-tls\") pod \"prometheus-operator-68bcd7f987-fk5jv\" (UID: \"529830e9-72ae-4032-899d-6af9f83105ea\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-fk5jv"
Sep 16 02:19:38.861144 ip-10-0-128-111 kubenswrapper[2583]: E0916 02:19:38.861082    2583 secret.go:189] Couldn't get secret openshift-monitoring/prometheus-operator-tls: secret "prometheus-operator-tls" not found
Sep 16 02:19:38.861183 ip-10-0-128-111 kubenswrapper[2583]: E0916 02:19:38.861156    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/529830e9-72ae-4032-899d-6af9f83105ea-prometheus-operator-tls podName:529830e9-72ae-4032-899d-6af9f83105ea nodeName:}" failed. No retries permitted until 2026-09-16 02:19:39.361137286 +0000 UTC m=+57.828926062 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "prometheus-operator-tls" (UniqueName: "kubernetes.io/secret/529830e9-72ae-4032-899d-6af9f83105ea-prometheus-operator-tls") pod "prometheus-operator-68bcd7f987-fk5jv" (UID: "529830e9-72ae-4032-899d-6af9f83105ea") : secret "prometheus-operator-tls" not found
Sep 16 02:19:38.861542 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:19:38.861501    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/529830e9-72ae-4032-899d-6af9f83105ea-metrics-client-ca\") pod \"prometheus-operator-68bcd7f987-fk5jv\" (UID: \"529830e9-72ae-4032-899d-6af9f83105ea\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-fk5jv"
Sep 16 02:19:38.863343 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:19:38.863318    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/529830e9-72ae-4032-899d-6af9f83105ea-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-68bcd7f987-fk5jv\" (UID: \"529830e9-72ae-4032-899d-6af9f83105ea\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-fk5jv"
Sep 16 02:19:38.869198 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:19:38.869177    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lz74q\" (UniqueName: \"kubernetes.io/projected/529830e9-72ae-4032-899d-6af9f83105ea-kube-api-access-lz74q\") pod \"prometheus-operator-68bcd7f987-fk5jv\" (UID: \"529830e9-72ae-4032-899d-6af9f83105ea\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-fk5jv"
Sep 16 02:19:39.364416 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:19:39.364371    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/529830e9-72ae-4032-899d-6af9f83105ea-prometheus-operator-tls\") pod \"prometheus-operator-68bcd7f987-fk5jv\" (UID: \"529830e9-72ae-4032-899d-6af9f83105ea\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-fk5jv"
Sep 16 02:19:39.366647 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:19:39.366629    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/529830e9-72ae-4032-899d-6af9f83105ea-prometheus-operator-tls\") pod \"prometheus-operator-68bcd7f987-fk5jv\" (UID: \"529830e9-72ae-4032-899d-6af9f83105ea\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-fk5jv"
Sep 16 02:19:39.569991 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:19:39.569951    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-68bcd7f987-fk5jv"
Sep 16 02:19:39.851843 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:19:39.851810    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-68bcd7f987-fk5jv"]
Sep 16 02:19:39.854505 ip-10-0-128-111 kubenswrapper[2583]: W0916 02:19:39.854479    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod529830e9_72ae_4032_899d_6af9f83105ea.slice/crio-1d0e3f523db95091ae728b5a1a9d92e96d362b9128b4fe4f3f44c6c6e672b29a WatchSource:0}: Error finding container 1d0e3f523db95091ae728b5a1a9d92e96d362b9128b4fe4f3f44c6c6e672b29a: Status 404 returned error can't find the container with id 1d0e3f523db95091ae728b5a1a9d92e96d362b9128b4fe4f3f44c6c6e672b29a
Sep 16 02:19:40.402862 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:19:40.402826    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-68bcd7f987-fk5jv" event={"ID":"529830e9-72ae-4032-899d-6af9f83105ea","Type":"ContainerStarted","Data":"1d0e3f523db95091ae728b5a1a9d92e96d362b9128b4fe4f3f44c6c6e672b29a"}
Sep 16 02:19:42.316427 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:19:42.316397    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-hgb4m"
Sep 16 02:19:42.411756 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:19:42.411720    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-68bcd7f987-fk5jv" event={"ID":"529830e9-72ae-4032-899d-6af9f83105ea","Type":"ContainerStarted","Data":"d384d6d3c56a4b99092ec1f786037912cf13c6bf5ae4597c0a46fb148a01f98b"}
Sep 16 02:19:42.411994 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:19:42.411773    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-68bcd7f987-fk5jv" event={"ID":"529830e9-72ae-4032-899d-6af9f83105ea","Type":"ContainerStarted","Data":"fe7a8f42be65c6a64c2eaa7159298f748cb97dce32880ef8a53ba45c936027f4"}
Sep 16 02:19:42.427196 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:19:42.427140    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-68bcd7f987-fk5jv" podStartSLOduration=2.625182702 podStartE2EDuration="4.427125137s" totalImagesPullingTime="1.801942435s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 02:19:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 02:19:42.426045501 +0000 UTC m=+60.893834284" watchObservedRunningTime="2026-09-16 02:19:42.427125137 +0000 UTC m=+60.894913919"
Sep 16 02:19:43.979977 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:19:43.979947    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-hv9kk"]
Sep 16 02:19:43.982401 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:19:43.982383    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-hv9kk"
Sep 16 02:19:43.984588 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:19:43.984566    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Sep 16 02:19:43.984717 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:19:43.984702    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Sep 16 02:19:43.984717 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:19:43.984710    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-fkjtp\""
Sep 16 02:19:43.984841 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:19:43.984819    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Sep 16 02:19:43.992752 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:19:43.992723    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/kube-state-metrics-59fbdd77d-4c8ff"]
Sep 16 02:19:43.994702 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:19:43.994687    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-59fbdd77d-4c8ff"
Sep 16 02:19:43.996855 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:19:43.996834    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-kube-rbac-proxy-config\""
Sep 16 02:19:43.996855 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:19:43.996838    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-tls\""
Sep 16 02:19:43.997197 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:19:43.997176    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-custom-resource-state-configmap\""
Sep 16 02:19:43.997283 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:19:43.997233    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-dockercfg-6knnq\""
Sep 16 02:19:44.008521 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:19:44.008497    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-59fbdd77d-4c8ff"]
Sep 16 02:19:44.103670 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:19:44.103637    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/80cd7034-5b29-4116-aa53-74168a3a96ed-node-exporter-wtmp\") pod \"node-exporter-hv9kk\" (UID: \"80cd7034-5b29-4116-aa53-74168a3a96ed\") " pod="openshift-monitoring/node-exporter-hv9kk"
Sep 16 02:19:44.103830 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:19:44.103676    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/f7765761-5911-4ee7-9587-9454b2e15ddc-volume-directive-shadow\") pod \"kube-state-metrics-59fbdd77d-4c8ff\" (UID: \"f7765761-5911-4ee7-9587-9454b2e15ddc\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-4c8ff"
Sep 16 02:19:44.103830 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:19:44.103698    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/80cd7034-5b29-4116-aa53-74168a3a96ed-node-exporter-tls\") pod \"node-exporter-hv9kk\" (UID: \"80cd7034-5b29-4116-aa53-74168a3a96ed\") " pod="openshift-monitoring/node-exporter-hv9kk"
Sep 16 02:19:44.103830 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:19:44.103717    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/80cd7034-5b29-4116-aa53-74168a3a96ed-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-hv9kk\" (UID: \"80cd7034-5b29-4116-aa53-74168a3a96ed\") " pod="openshift-monitoring/node-exporter-hv9kk"
Sep 16 02:19:44.103830 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:19:44.103757    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/f7765761-5911-4ee7-9587-9454b2e15ddc-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-59fbdd77d-4c8ff\" (UID: \"f7765761-5911-4ee7-9587-9454b2e15ddc\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-4c8ff"
Sep 16 02:19:44.103830 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:19:44.103788    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/80cd7034-5b29-4116-aa53-74168a3a96ed-node-exporter-textfile\") pod \"node-exporter-hv9kk\" (UID: \"80cd7034-5b29-4116-aa53-74168a3a96ed\") " pod="openshift-monitoring/node-exporter-hv9kk"
Sep 16 02:19:44.103830 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:19:44.103813    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/80cd7034-5b29-4116-aa53-74168a3a96ed-node-exporter-accelerators-collector-config\") pod \"node-exporter-hv9kk\" (UID: \"80cd7034-5b29-4116-aa53-74168a3a96ed\") " pod="openshift-monitoring/node-exporter-hv9kk"
Sep 16 02:19:44.104021 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:19:44.103843    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/f7765761-5911-4ee7-9587-9454b2e15ddc-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-59fbdd77d-4c8ff\" (UID: \"f7765761-5911-4ee7-9587-9454b2e15ddc\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-4c8ff"
Sep 16 02:19:44.104021 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:19:44.103872    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/f7765761-5911-4ee7-9587-9454b2e15ddc-kube-state-metrics-tls\") pod \"kube-state-metrics-59fbdd77d-4c8ff\" (UID: \"f7765761-5911-4ee7-9587-9454b2e15ddc\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-4c8ff"
Sep 16 02:19:44.104021 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:19:44.103885    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f7765761-5911-4ee7-9587-9454b2e15ddc-metrics-client-ca\") pod \"kube-state-metrics-59fbdd77d-4c8ff\" (UID: \"f7765761-5911-4ee7-9587-9454b2e15ddc\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-4c8ff"
Sep 16 02:19:44.104021 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:19:44.103901    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dxzmd\" (UniqueName: \"kubernetes.io/projected/80cd7034-5b29-4116-aa53-74168a3a96ed-kube-api-access-dxzmd\") pod \"node-exporter-hv9kk\" (UID: \"80cd7034-5b29-4116-aa53-74168a3a96ed\") " pod="openshift-monitoring/node-exporter-hv9kk"
Sep 16 02:19:44.104021 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:19:44.103921    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/80cd7034-5b29-4116-aa53-74168a3a96ed-metrics-client-ca\") pod \"node-exporter-hv9kk\" (UID: \"80cd7034-5b29-4116-aa53-74168a3a96ed\") " pod="openshift-monitoring/node-exporter-hv9kk"
Sep 16 02:19:44.104021 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:19:44.103962    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pmd92\" (UniqueName: \"kubernetes.io/projected/f7765761-5911-4ee7-9587-9454b2e15ddc-kube-api-access-pmd92\") pod \"kube-state-metrics-59fbdd77d-4c8ff\" (UID: \"f7765761-5911-4ee7-9587-9454b2e15ddc\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-4c8ff"
Sep 16 02:19:44.104021 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:19:44.103981    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/80cd7034-5b29-4116-aa53-74168a3a96ed-sys\") pod \"node-exporter-hv9kk\" (UID: \"80cd7034-5b29-4116-aa53-74168a3a96ed\") " pod="openshift-monitoring/node-exporter-hv9kk"
Sep 16 02:19:44.104021 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:19:44.103994    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/80cd7034-5b29-4116-aa53-74168a3a96ed-root\") pod \"node-exporter-hv9kk\" (UID: \"80cd7034-5b29-4116-aa53-74168a3a96ed\") " pod="openshift-monitoring/node-exporter-hv9kk"
Sep 16 02:19:44.204784 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:19:44.204751    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/80cd7034-5b29-4116-aa53-74168a3a96ed-node-exporter-tls\") pod \"node-exporter-hv9kk\" (UID: \"80cd7034-5b29-4116-aa53-74168a3a96ed\") " pod="openshift-monitoring/node-exporter-hv9kk"
Sep 16 02:19:44.204784 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:19:44.204787    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/80cd7034-5b29-4116-aa53-74168a3a96ed-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-hv9kk\" (UID: \"80cd7034-5b29-4116-aa53-74168a3a96ed\") " pod="openshift-monitoring/node-exporter-hv9kk"
Sep 16 02:19:44.205020 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:19:44.204812    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/f7765761-5911-4ee7-9587-9454b2e15ddc-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-59fbdd77d-4c8ff\" (UID: \"f7765761-5911-4ee7-9587-9454b2e15ddc\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-4c8ff"
Sep 16 02:19:44.205020 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:19:44.204933    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/80cd7034-5b29-4116-aa53-74168a3a96ed-node-exporter-textfile\") pod \"node-exporter-hv9kk\" (UID: \"80cd7034-5b29-4116-aa53-74168a3a96ed\") " pod="openshift-monitoring/node-exporter-hv9kk"
Sep 16 02:19:44.205020 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:19:44.204986    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/80cd7034-5b29-4116-aa53-74168a3a96ed-node-exporter-accelerators-collector-config\") pod \"node-exporter-hv9kk\" (UID: \"80cd7034-5b29-4116-aa53-74168a3a96ed\") " pod="openshift-monitoring/node-exporter-hv9kk"
Sep 16 02:19:44.205180 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:19:44.205033    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/f7765761-5911-4ee7-9587-9454b2e15ddc-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-59fbdd77d-4c8ff\" (UID: \"f7765761-5911-4ee7-9587-9454b2e15ddc\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-4c8ff"
Sep 16 02:19:44.205180 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:19:44.205064    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/f7765761-5911-4ee7-9587-9454b2e15ddc-kube-state-metrics-tls\") pod \"kube-state-metrics-59fbdd77d-4c8ff\" (UID: \"f7765761-5911-4ee7-9587-9454b2e15ddc\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-4c8ff"
Sep 16 02:19:44.205180 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:19:44.205093    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f7765761-5911-4ee7-9587-9454b2e15ddc-metrics-client-ca\") pod \"kube-state-metrics-59fbdd77d-4c8ff\" (UID: \"f7765761-5911-4ee7-9587-9454b2e15ddc\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-4c8ff"
Sep 16 02:19:44.205180 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:19:44.205124    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dxzmd\" (UniqueName: \"kubernetes.io/projected/80cd7034-5b29-4116-aa53-74168a3a96ed-kube-api-access-dxzmd\") pod \"node-exporter-hv9kk\" (UID: \"80cd7034-5b29-4116-aa53-74168a3a96ed\") " pod="openshift-monitoring/node-exporter-hv9kk"
Sep 16 02:19:44.205180 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:19:44.205158    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/80cd7034-5b29-4116-aa53-74168a3a96ed-metrics-client-ca\") pod \"node-exporter-hv9kk\" (UID: \"80cd7034-5b29-4116-aa53-74168a3a96ed\") " pod="openshift-monitoring/node-exporter-hv9kk"
Sep 16 02:19:44.205420 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:19:44.205207    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pmd92\" (UniqueName: \"kubernetes.io/projected/f7765761-5911-4ee7-9587-9454b2e15ddc-kube-api-access-pmd92\") pod \"kube-state-metrics-59fbdd77d-4c8ff\" (UID: \"f7765761-5911-4ee7-9587-9454b2e15ddc\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-4c8ff"
Sep 16 02:19:44.205420 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:19:44.205246    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/80cd7034-5b29-4116-aa53-74168a3a96ed-sys\") pod \"node-exporter-hv9kk\" (UID: \"80cd7034-5b29-4116-aa53-74168a3a96ed\") " pod="openshift-monitoring/node-exporter-hv9kk"
Sep 16 02:19:44.205420 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:19:44.205262    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/80cd7034-5b29-4116-aa53-74168a3a96ed-node-exporter-textfile\") pod \"node-exporter-hv9kk\" (UID: \"80cd7034-5b29-4116-aa53-74168a3a96ed\") " pod="openshift-monitoring/node-exporter-hv9kk"
Sep 16 02:19:44.205420 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:19:44.205277    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/80cd7034-5b29-4116-aa53-74168a3a96ed-root\") pod \"node-exporter-hv9kk\" (UID: \"80cd7034-5b29-4116-aa53-74168a3a96ed\") " pod="openshift-monitoring/node-exporter-hv9kk"
Sep 16 02:19:44.205420 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:19:44.205336    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/80cd7034-5b29-4116-aa53-74168a3a96ed-node-exporter-wtmp\") pod \"node-exporter-hv9kk\" (UID: \"80cd7034-5b29-4116-aa53-74168a3a96ed\") " pod="openshift-monitoring/node-exporter-hv9kk"
Sep 16 02:19:44.205420 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:19:44.205366    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/f7765761-5911-4ee7-9587-9454b2e15ddc-volume-directive-shadow\") pod \"kube-state-metrics-59fbdd77d-4c8ff\" (UID: \"f7765761-5911-4ee7-9587-9454b2e15ddc\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-4c8ff"
Sep 16 02:19:44.205732 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:19:44.205622    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/80cd7034-5b29-4116-aa53-74168a3a96ed-node-exporter-accelerators-collector-config\") pod \"node-exporter-hv9kk\" (UID: \"80cd7034-5b29-4116-aa53-74168a3a96ed\") " pod="openshift-monitoring/node-exporter-hv9kk"
Sep 16 02:19:44.206275 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:19:44.206056    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/80cd7034-5b29-4116-aa53-74168a3a96ed-root\") pod \"node-exporter-hv9kk\" (UID: \"80cd7034-5b29-4116-aa53-74168a3a96ed\") " pod="openshift-monitoring/node-exporter-hv9kk"
Sep 16 02:19:44.206275 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:19:44.206131    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/80cd7034-5b29-4116-aa53-74168a3a96ed-sys\") pod \"node-exporter-hv9kk\" (UID: \"80cd7034-5b29-4116-aa53-74168a3a96ed\") " pod="openshift-monitoring/node-exporter-hv9kk"
Sep 16 02:19:44.206275 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:19:44.206176    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/80cd7034-5b29-4116-aa53-74168a3a96ed-metrics-client-ca\") pod \"node-exporter-hv9kk\" (UID: \"80cd7034-5b29-4116-aa53-74168a3a96ed\") " pod="openshift-monitoring/node-exporter-hv9kk"
Sep 16 02:19:44.206275 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:19:44.206222    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/f7765761-5911-4ee7-9587-9454b2e15ddc-volume-directive-shadow\") pod \"kube-state-metrics-59fbdd77d-4c8ff\" (UID: \"f7765761-5911-4ee7-9587-9454b2e15ddc\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-4c8ff"
Sep 16 02:19:44.206275 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:19:44.206247    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/80cd7034-5b29-4116-aa53-74168a3a96ed-node-exporter-wtmp\") pod \"node-exporter-hv9kk\" (UID: \"80cd7034-5b29-4116-aa53-74168a3a96ed\") " pod="openshift-monitoring/node-exporter-hv9kk"
Sep 16 02:19:44.206597 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:19:44.206298    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f7765761-5911-4ee7-9587-9454b2e15ddc-metrics-client-ca\") pod \"kube-state-metrics-59fbdd77d-4c8ff\" (UID: \"f7765761-5911-4ee7-9587-9454b2e15ddc\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-4c8ff"
Sep 16 02:19:44.206786 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:19:44.206761    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/f7765761-5911-4ee7-9587-9454b2e15ddc-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-59fbdd77d-4c8ff\" (UID: \"f7765761-5911-4ee7-9587-9454b2e15ddc\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-4c8ff"
Sep 16 02:19:44.207495 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:19:44.207462    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/80cd7034-5b29-4116-aa53-74168a3a96ed-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-hv9kk\" (UID: \"80cd7034-5b29-4116-aa53-74168a3a96ed\") " pod="openshift-monitoring/node-exporter-hv9kk"
Sep 16 02:19:44.207668 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:19:44.207612    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/80cd7034-5b29-4116-aa53-74168a3a96ed-node-exporter-tls\") pod \"node-exporter-hv9kk\" (UID: \"80cd7034-5b29-4116-aa53-74168a3a96ed\") " pod="openshift-monitoring/node-exporter-hv9kk"
Sep 16 02:19:44.207668 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:19:44.207620    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/f7765761-5911-4ee7-9587-9454b2e15ddc-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-59fbdd77d-4c8ff\" (UID: \"f7765761-5911-4ee7-9587-9454b2e15ddc\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-4c8ff"
Sep 16 02:19:44.207960 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:19:44.207939    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/f7765761-5911-4ee7-9587-9454b2e15ddc-kube-state-metrics-tls\") pod \"kube-state-metrics-59fbdd77d-4c8ff\" (UID: \"f7765761-5911-4ee7-9587-9454b2e15ddc\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-4c8ff"
Sep 16 02:19:44.212869 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:19:44.212844    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dxzmd\" (UniqueName: \"kubernetes.io/projected/80cd7034-5b29-4116-aa53-74168a3a96ed-kube-api-access-dxzmd\") pod \"node-exporter-hv9kk\" (UID: \"80cd7034-5b29-4116-aa53-74168a3a96ed\") " pod="openshift-monitoring/node-exporter-hv9kk"
Sep 16 02:19:44.213191 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:19:44.213156    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pmd92\" (UniqueName: \"kubernetes.io/projected/f7765761-5911-4ee7-9587-9454b2e15ddc-kube-api-access-pmd92\") pod \"kube-state-metrics-59fbdd77d-4c8ff\" (UID: \"f7765761-5911-4ee7-9587-9454b2e15ddc\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-4c8ff"
Sep 16 02:19:44.291108 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:19:44.291016    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-hv9kk"
Sep 16 02:19:44.302397 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:19:44.302373    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-59fbdd77d-4c8ff"
Sep 16 02:19:44.302678 ip-10-0-128-111 kubenswrapper[2583]: W0916 02:19:44.302658    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod80cd7034_5b29_4116_aa53_74168a3a96ed.slice/crio-5dbb71ac2472e0e6acc54fe4a74e44271a5a5e73cb64eb25b71c8d9da87e3996 WatchSource:0}: Error finding container 5dbb71ac2472e0e6acc54fe4a74e44271a5a5e73cb64eb25b71c8d9da87e3996: Status 404 returned error can't find the container with id 5dbb71ac2472e0e6acc54fe4a74e44271a5a5e73cb64eb25b71c8d9da87e3996
Sep 16 02:19:44.419250 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:19:44.419213    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-hv9kk" event={"ID":"80cd7034-5b29-4116-aa53-74168a3a96ed","Type":"ContainerStarted","Data":"5dbb71ac2472e0e6acc54fe4a74e44271a5a5e73cb64eb25b71c8d9da87e3996"}
Sep 16 02:19:44.571348 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:19:44.571264    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-59fbdd77d-4c8ff"]
Sep 16 02:19:44.574058 ip-10-0-128-111 kubenswrapper[2583]: W0916 02:19:44.574026    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf7765761_5911_4ee7_9587_9454b2e15ddc.slice/crio-11e11b8b726507b140509b24454c06f5410595b8628e1ce42303353f055cfa39 WatchSource:0}: Error finding container 11e11b8b726507b140509b24454c06f5410595b8628e1ce42303353f055cfa39: Status 404 returned error can't find the container with id 11e11b8b726507b140509b24454c06f5410595b8628e1ce42303353f055cfa39
Sep 16 02:19:45.029033 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:19:45.028993    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 16 02:19:45.031324 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:19:45.031298    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Sep 16 02:19:45.033822 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:19:45.033736    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\""
Sep 16 02:19:45.033822 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:19:45.033794    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\""
Sep 16 02:19:45.033822 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:19:45.033795    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-tv9h9\""
Sep 16 02:19:45.034063 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:19:45.033843    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\""
Sep 16 02:19:45.034210 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:19:45.034189    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\""
Sep 16 02:19:45.034317 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:19:45.034256    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\""
Sep 16 02:19:45.034317 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:19:45.034279    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\""
Sep 16 02:19:45.034317 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:19:45.034299    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\""
Sep 16 02:19:45.034501 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:19:45.034314    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\""
Sep 16 02:19:45.034501 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:19:45.034500    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\""
Sep 16 02:19:45.048173 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:19:45.048148    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 16 02:19:45.114125 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:19:45.114091    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/5bf4691b-1501-4d65-9538-52851f45d954-config-out\") pod \"alertmanager-main-0\" (UID: \"5bf4691b-1501-4d65-9538-52851f45d954\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 02:19:45.114312 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:19:45.114196    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/5bf4691b-1501-4d65-9538-52851f45d954-tls-assets\") pod \"alertmanager-main-0\" (UID: \"5bf4691b-1501-4d65-9538-52851f45d954\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 02:19:45.114312 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:19:45.114239    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/5bf4691b-1501-4d65-9538-52851f45d954-config-volume\") pod \"alertmanager-main-0\" (UID: \"5bf4691b-1501-4d65-9538-52851f45d954\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 02:19:45.114312 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:19:45.114265    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/5bf4691b-1501-4d65-9538-52851f45d954-web-config\") pod \"alertmanager-main-0\" (UID: \"5bf4691b-1501-4d65-9538-52851f45d954\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 02:19:45.114312 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:19:45.114297    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/5bf4691b-1501-4d65-9538-52851f45d954-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"5bf4691b-1501-4d65-9538-52851f45d954\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 02:19:45.114486 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:19:45.114324    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2fbnw\" (UniqueName: \"kubernetes.io/projected/5bf4691b-1501-4d65-9538-52851f45d954-kube-api-access-2fbnw\") pod \"alertmanager-main-0\" (UID: \"5bf4691b-1501-4d65-9538-52851f45d954\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 02:19:45.114486 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:19:45.114417    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/5bf4691b-1501-4d65-9538-52851f45d954-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"5bf4691b-1501-4d65-9538-52851f45d954\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 02:19:45.114486 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:19:45.114468    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/5bf4691b-1501-4d65-9538-52851f45d954-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"5bf4691b-1501-4d65-9538-52851f45d954\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 02:19:45.114670 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:19:45.114507    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/5bf4691b-1501-4d65-9538-52851f45d954-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"5bf4691b-1501-4d65-9538-52851f45d954\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 02:19:45.114670 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:19:45.114541    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/5bf4691b-1501-4d65-9538-52851f45d954-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"5bf4691b-1501-4d65-9538-52851f45d954\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 02:19:45.114670 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:19:45.114581    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/5bf4691b-1501-4d65-9538-52851f45d954-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"5bf4691b-1501-4d65-9538-52851f45d954\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 02:19:45.114670 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:19:45.114641    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/5bf4691b-1501-4d65-9538-52851f45d954-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"5bf4691b-1501-4d65-9538-52851f45d954\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 02:19:45.114824 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:19:45.114696    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5bf4691b-1501-4d65-9538-52851f45d954-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"5bf4691b-1501-4d65-9538-52851f45d954\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 02:19:45.215263 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:19:45.215222    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/5bf4691b-1501-4d65-9538-52851f45d954-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"5bf4691b-1501-4d65-9538-52851f45d954\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 02:19:45.215428 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:19:45.215274    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5bf4691b-1501-4d65-9538-52851f45d954-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"5bf4691b-1501-4d65-9538-52851f45d954\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 02:19:45.215428 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:19:45.215316    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/5bf4691b-1501-4d65-9538-52851f45d954-config-out\") pod \"alertmanager-main-0\" (UID: \"5bf4691b-1501-4d65-9538-52851f45d954\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 02:19:45.215428 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:19:45.215361    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/5bf4691b-1501-4d65-9538-52851f45d954-tls-assets\") pod \"alertmanager-main-0\" (UID: \"5bf4691b-1501-4d65-9538-52851f45d954\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 02:19:45.215641 ip-10-0-128-111 kubenswrapper[2583]: E0916 02:19:45.215480    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/5bf4691b-1501-4d65-9538-52851f45d954-alertmanager-trusted-ca-bundle podName:5bf4691b-1501-4d65-9538-52851f45d954 nodeName:}" failed. No retries permitted until 2026-09-16 02:19:45.715458384 +0000 UTC m=+64.183247147 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "alertmanager-trusted-ca-bundle" (UniqueName: "kubernetes.io/configmap/5bf4691b-1501-4d65-9538-52851f45d954-alertmanager-trusted-ca-bundle") pod "alertmanager-main-0" (UID: "5bf4691b-1501-4d65-9538-52851f45d954") : configmap references non-existent config key: ca-bundle.crt
Sep 16 02:19:45.215641 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:19:45.215509    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/5bf4691b-1501-4d65-9538-52851f45d954-config-volume\") pod \"alertmanager-main-0\" (UID: \"5bf4691b-1501-4d65-9538-52851f45d954\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 02:19:45.215641 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:19:45.215540    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/5bf4691b-1501-4d65-9538-52851f45d954-web-config\") pod \"alertmanager-main-0\" (UID: \"5bf4691b-1501-4d65-9538-52851f45d954\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 02:19:45.215641 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:19:45.215606    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/5bf4691b-1501-4d65-9538-52851f45d954-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"5bf4691b-1501-4d65-9538-52851f45d954\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 02:19:45.215641 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:19:45.215634    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2fbnw\" (UniqueName: \"kubernetes.io/projected/5bf4691b-1501-4d65-9538-52851f45d954-kube-api-access-2fbnw\") pod \"alertmanager-main-0\" (UID: \"5bf4691b-1501-4d65-9538-52851f45d954\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 02:19:45.215930 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:19:45.215682    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/5bf4691b-1501-4d65-9538-52851f45d954-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"5bf4691b-1501-4d65-9538-52851f45d954\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 02:19:45.215930 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:19:45.215714    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/5bf4691b-1501-4d65-9538-52851f45d954-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"5bf4691b-1501-4d65-9538-52851f45d954\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 02:19:45.215930 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:19:45.215751    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/5bf4691b-1501-4d65-9538-52851f45d954-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"5bf4691b-1501-4d65-9538-52851f45d954\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 02:19:45.215930 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:19:45.215777    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/5bf4691b-1501-4d65-9538-52851f45d954-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"5bf4691b-1501-4d65-9538-52851f45d954\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 02:19:45.215930 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:19:45.215802    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/5bf4691b-1501-4d65-9538-52851f45d954-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"5bf4691b-1501-4d65-9538-52851f45d954\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 02:19:45.216764 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:19:45.216723    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/5bf4691b-1501-4d65-9538-52851f45d954-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"5bf4691b-1501-4d65-9538-52851f45d954\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 02:19:45.217030 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:19:45.217007    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/5bf4691b-1501-4d65-9538-52851f45d954-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"5bf4691b-1501-4d65-9538-52851f45d954\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 02:19:45.219108 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:19:45.219083    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/5bf4691b-1501-4d65-9538-52851f45d954-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"5bf4691b-1501-4d65-9538-52851f45d954\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 02:19:45.219920 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:19:45.219875    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/5bf4691b-1501-4d65-9538-52851f45d954-config-out\") pod \"alertmanager-main-0\" (UID: \"5bf4691b-1501-4d65-9538-52851f45d954\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 02:19:45.220393 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:19:45.220353    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/5bf4691b-1501-4d65-9538-52851f45d954-web-config\") pod \"alertmanager-main-0\" (UID: \"5bf4691b-1501-4d65-9538-52851f45d954\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 02:19:45.220709 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:19:45.220685    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/5bf4691b-1501-4d65-9538-52851f45d954-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"5bf4691b-1501-4d65-9538-52851f45d954\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 02:19:45.221144 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:19:45.221120    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/5bf4691b-1501-4d65-9538-52851f45d954-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"5bf4691b-1501-4d65-9538-52851f45d954\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 02:19:45.221489 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:19:45.221468    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/5bf4691b-1501-4d65-9538-52851f45d954-config-volume\") pod \"alertmanager-main-0\" (UID: \"5bf4691b-1501-4d65-9538-52851f45d954\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 02:19:45.222155 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:19:45.222111    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/5bf4691b-1501-4d65-9538-52851f45d954-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"5bf4691b-1501-4d65-9538-52851f45d954\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 02:19:45.223544 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:19:45.223504    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/5bf4691b-1501-4d65-9538-52851f45d954-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"5bf4691b-1501-4d65-9538-52851f45d954\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 02:19:45.223968 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:19:45.223946    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/5bf4691b-1501-4d65-9538-52851f45d954-tls-assets\") pod \"alertmanager-main-0\" (UID: \"5bf4691b-1501-4d65-9538-52851f45d954\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 02:19:45.224796 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:19:45.224772    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2fbnw\" (UniqueName: \"kubernetes.io/projected/5bf4691b-1501-4d65-9538-52851f45d954-kube-api-access-2fbnw\") pod \"alertmanager-main-0\" (UID: \"5bf4691b-1501-4d65-9538-52851f45d954\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 02:19:45.426203 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:19:45.426164    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-hv9kk" event={"ID":"80cd7034-5b29-4116-aa53-74168a3a96ed","Type":"ContainerStarted","Data":"a2e6af503564e05fc6c4975920089996965678ec0b84ebc5b27b2431dc53fafd"}
Sep 16 02:19:45.427601 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:19:45.427575    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-59fbdd77d-4c8ff" event={"ID":"f7765761-5911-4ee7-9587-9454b2e15ddc","Type":"ContainerStarted","Data":"11e11b8b726507b140509b24454c06f5410595b8628e1ce42303353f055cfa39"}
Sep 16 02:19:45.719921 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:19:45.719877    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5bf4691b-1501-4d65-9538-52851f45d954-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"5bf4691b-1501-4d65-9538-52851f45d954\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 02:19:45.720890 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:19:45.720856    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5bf4691b-1501-4d65-9538-52851f45d954-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"5bf4691b-1501-4d65-9538-52851f45d954\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 02:19:45.941538 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:19:45.941504    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Sep 16 02:19:45.950767 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:19:45.950624    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/thanos-querier-79d4d88bf4-4nc9x"]
Sep 16 02:19:45.967246 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:19:45.967205    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-79d4d88bf4-4nc9x"]
Sep 16 02:19:45.967377 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:19:45.967307    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-79d4d88bf4-4nc9x"
Sep 16 02:19:45.969970 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:19:45.969770    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy\""
Sep 16 02:19:45.969970 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:19:45.969799    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-tls\""
Sep 16 02:19:45.969970 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:19:45.969775    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-rules\""
Sep 16 02:19:45.969970 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:19:45.969891    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-web\""
Sep 16 02:19:45.969970 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:19:45.969895    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-dockercfg-7rmrc\""
Sep 16 02:19:45.969970 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:19:45.969968    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-grpc-tls-f3fficpsvn1j0\""
Sep 16 02:19:45.970336 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:19:45.970050    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-metrics\""
Sep 16 02:19:46.123196 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:19:46.123171    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/bd366e5d-6d4a-45b9-b259-9d5488a3b7a6-secret-thanos-querier-tls\") pod \"thanos-querier-79d4d88bf4-4nc9x\" (UID: \"bd366e5d-6d4a-45b9-b259-9d5488a3b7a6\") " pod="openshift-monitoring/thanos-querier-79d4d88bf4-4nc9x"
Sep 16 02:19:46.123511 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:19:46.123212    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/bd366e5d-6d4a-45b9-b259-9d5488a3b7a6-secret-grpc-tls\") pod \"thanos-querier-79d4d88bf4-4nc9x\" (UID: \"bd366e5d-6d4a-45b9-b259-9d5488a3b7a6\") " pod="openshift-monitoring/thanos-querier-79d4d88bf4-4nc9x"
Sep 16 02:19:46.123511 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:19:46.123234    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l2ckl\" (UniqueName: \"kubernetes.io/projected/bd366e5d-6d4a-45b9-b259-9d5488a3b7a6-kube-api-access-l2ckl\") pod \"thanos-querier-79d4d88bf4-4nc9x\" (UID: \"bd366e5d-6d4a-45b9-b259-9d5488a3b7a6\") " pod="openshift-monitoring/thanos-querier-79d4d88bf4-4nc9x"
Sep 16 02:19:46.123511 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:19:46.123260    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/bd366e5d-6d4a-45b9-b259-9d5488a3b7a6-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-79d4d88bf4-4nc9x\" (UID: \"bd366e5d-6d4a-45b9-b259-9d5488a3b7a6\") " pod="openshift-monitoring/thanos-querier-79d4d88bf4-4nc9x"
Sep 16 02:19:46.123511 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:19:46.123291    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/bd366e5d-6d4a-45b9-b259-9d5488a3b7a6-metrics-client-ca\") pod \"thanos-querier-79d4d88bf4-4nc9x\" (UID: \"bd366e5d-6d4a-45b9-b259-9d5488a3b7a6\") " pod="openshift-monitoring/thanos-querier-79d4d88bf4-4nc9x"
Sep 16 02:19:46.123511 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:19:46.123332    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/bd366e5d-6d4a-45b9-b259-9d5488a3b7a6-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-79d4d88bf4-4nc9x\" (UID: \"bd366e5d-6d4a-45b9-b259-9d5488a3b7a6\") " pod="openshift-monitoring/thanos-querier-79d4d88bf4-4nc9x"
Sep 16 02:19:46.123511 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:19:46.123376    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/bd366e5d-6d4a-45b9-b259-9d5488a3b7a6-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-79d4d88bf4-4nc9x\" (UID: \"bd366e5d-6d4a-45b9-b259-9d5488a3b7a6\") " pod="openshift-monitoring/thanos-querier-79d4d88bf4-4nc9x"
Sep 16 02:19:46.123511 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:19:46.123455    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/bd366e5d-6d4a-45b9-b259-9d5488a3b7a6-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-79d4d88bf4-4nc9x\" (UID: \"bd366e5d-6d4a-45b9-b259-9d5488a3b7a6\") " pod="openshift-monitoring/thanos-querier-79d4d88bf4-4nc9x"
Sep 16 02:19:46.224735 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:19:46.224689    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/bd366e5d-6d4a-45b9-b259-9d5488a3b7a6-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-79d4d88bf4-4nc9x\" (UID: \"bd366e5d-6d4a-45b9-b259-9d5488a3b7a6\") " pod="openshift-monitoring/thanos-querier-79d4d88bf4-4nc9x"
Sep 16 02:19:46.224863 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:19:46.224761    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/bd366e5d-6d4a-45b9-b259-9d5488a3b7a6-secret-thanos-querier-tls\") pod \"thanos-querier-79d4d88bf4-4nc9x\" (UID: \"bd366e5d-6d4a-45b9-b259-9d5488a3b7a6\") " pod="openshift-monitoring/thanos-querier-79d4d88bf4-4nc9x"
Sep 16 02:19:46.224863 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:19:46.224804    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/bd366e5d-6d4a-45b9-b259-9d5488a3b7a6-secret-grpc-tls\") pod \"thanos-querier-79d4d88bf4-4nc9x\" (UID: \"bd366e5d-6d4a-45b9-b259-9d5488a3b7a6\") " pod="openshift-monitoring/thanos-querier-79d4d88bf4-4nc9x"
Sep 16 02:19:46.224863 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:19:46.224837    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l2ckl\" (UniqueName: \"kubernetes.io/projected/bd366e5d-6d4a-45b9-b259-9d5488a3b7a6-kube-api-access-l2ckl\") pod \"thanos-querier-79d4d88bf4-4nc9x\" (UID: \"bd366e5d-6d4a-45b9-b259-9d5488a3b7a6\") " pod="openshift-monitoring/thanos-querier-79d4d88bf4-4nc9x"
Sep 16 02:19:46.225016 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:19:46.224864    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/bd366e5d-6d4a-45b9-b259-9d5488a3b7a6-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-79d4d88bf4-4nc9x\" (UID: \"bd366e5d-6d4a-45b9-b259-9d5488a3b7a6\") " pod="openshift-monitoring/thanos-querier-79d4d88bf4-4nc9x"
Sep 16 02:19:46.225016 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:19:46.224892    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/bd366e5d-6d4a-45b9-b259-9d5488a3b7a6-metrics-client-ca\") pod \"thanos-querier-79d4d88bf4-4nc9x\" (UID: \"bd366e5d-6d4a-45b9-b259-9d5488a3b7a6\") " pod="openshift-monitoring/thanos-querier-79d4d88bf4-4nc9x"
Sep 16 02:19:46.225016 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:19:46.224925    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/bd366e5d-6d4a-45b9-b259-9d5488a3b7a6-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-79d4d88bf4-4nc9x\" (UID: \"bd366e5d-6d4a-45b9-b259-9d5488a3b7a6\") " pod="openshift-monitoring/thanos-querier-79d4d88bf4-4nc9x"
Sep 16 02:19:46.225016 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:19:46.224958    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/bd366e5d-6d4a-45b9-b259-9d5488a3b7a6-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-79d4d88bf4-4nc9x\" (UID: \"bd366e5d-6d4a-45b9-b259-9d5488a3b7a6\") " pod="openshift-monitoring/thanos-querier-79d4d88bf4-4nc9x"
Sep 16 02:19:46.226070 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:19:46.226008    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/bd366e5d-6d4a-45b9-b259-9d5488a3b7a6-metrics-client-ca\") pod \"thanos-querier-79d4d88bf4-4nc9x\" (UID: \"bd366e5d-6d4a-45b9-b259-9d5488a3b7a6\") " pod="openshift-monitoring/thanos-querier-79d4d88bf4-4nc9x"
Sep 16 02:19:46.228099 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:19:46.228050    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/bd366e5d-6d4a-45b9-b259-9d5488a3b7a6-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-79d4d88bf4-4nc9x\" (UID: \"bd366e5d-6d4a-45b9-b259-9d5488a3b7a6\") " pod="openshift-monitoring/thanos-querier-79d4d88bf4-4nc9x"
Sep 16 02:19:46.228099 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:19:46.228078    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/bd366e5d-6d4a-45b9-b259-9d5488a3b7a6-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-79d4d88bf4-4nc9x\" (UID: \"bd366e5d-6d4a-45b9-b259-9d5488a3b7a6\") " pod="openshift-monitoring/thanos-querier-79d4d88bf4-4nc9x"
Sep 16 02:19:46.228263 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:19:46.228195    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/bd366e5d-6d4a-45b9-b259-9d5488a3b7a6-secret-thanos-querier-tls\") pod \"thanos-querier-79d4d88bf4-4nc9x\" (UID: \"bd366e5d-6d4a-45b9-b259-9d5488a3b7a6\") " pod="openshift-monitoring/thanos-querier-79d4d88bf4-4nc9x"
Sep 16 02:19:46.228801 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:19:46.228773    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/bd366e5d-6d4a-45b9-b259-9d5488a3b7a6-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-79d4d88bf4-4nc9x\" (UID: \"bd366e5d-6d4a-45b9-b259-9d5488a3b7a6\") " pod="openshift-monitoring/thanos-querier-79d4d88bf4-4nc9x"
Sep 16 02:19:46.229942 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:19:46.229901    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/bd366e5d-6d4a-45b9-b259-9d5488a3b7a6-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-79d4d88bf4-4nc9x\" (UID: \"bd366e5d-6d4a-45b9-b259-9d5488a3b7a6\") " pod="openshift-monitoring/thanos-querier-79d4d88bf4-4nc9x"
Sep 16 02:19:46.230111 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:19:46.230088    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/bd366e5d-6d4a-45b9-b259-9d5488a3b7a6-secret-grpc-tls\") pod \"thanos-querier-79d4d88bf4-4nc9x\" (UID: \"bd366e5d-6d4a-45b9-b259-9d5488a3b7a6\") " pod="openshift-monitoring/thanos-querier-79d4d88bf4-4nc9x"
Sep 16 02:19:46.234787 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:19:46.234761    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-l2ckl\" (UniqueName: \"kubernetes.io/projected/bd366e5d-6d4a-45b9-b259-9d5488a3b7a6-kube-api-access-l2ckl\") pod \"thanos-querier-79d4d88bf4-4nc9x\" (UID: \"bd366e5d-6d4a-45b9-b259-9d5488a3b7a6\") " pod="openshift-monitoring/thanos-querier-79d4d88bf4-4nc9x"
Sep 16 02:19:46.276970 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:19:46.276942    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-79d4d88bf4-4nc9x"
Sep 16 02:19:46.337540 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:19:46.337444    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 16 02:19:46.342037 ip-10-0-128-111 kubenswrapper[2583]: W0916 02:19:46.342008    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5bf4691b_1501_4d65_9538_52851f45d954.slice/crio-ab42e31d318efbc55abf8700c15fbd4e31a7752a1056ce0ad46025b7124024b7 WatchSource:0}: Error finding container ab42e31d318efbc55abf8700c15fbd4e31a7752a1056ce0ad46025b7124024b7: Status 404 returned error can't find the container with id ab42e31d318efbc55abf8700c15fbd4e31a7752a1056ce0ad46025b7124024b7
Sep 16 02:19:46.433156 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:19:46.433125    2583 generic.go:356] "Generic (PLEG): container finished" podID="80cd7034-5b29-4116-aa53-74168a3a96ed" containerID="a2e6af503564e05fc6c4975920089996965678ec0b84ebc5b27b2431dc53fafd" exitCode=0
Sep 16 02:19:46.433269 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:19:46.433211    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-hv9kk" event={"ID":"80cd7034-5b29-4116-aa53-74168a3a96ed","Type":"ContainerDied","Data":"a2e6af503564e05fc6c4975920089996965678ec0b84ebc5b27b2431dc53fafd"}
Sep 16 02:19:46.434783 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:19:46.434696    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"5bf4691b-1501-4d65-9538-52851f45d954","Type":"ContainerStarted","Data":"ab42e31d318efbc55abf8700c15fbd4e31a7752a1056ce0ad46025b7124024b7"}
Sep 16 02:19:46.437428 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:19:46.437404    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-59fbdd77d-4c8ff" event={"ID":"f7765761-5911-4ee7-9587-9454b2e15ddc","Type":"ContainerStarted","Data":"35edb443bb52755fb1fa0dcca899c1a80b630c7f698121811dc63e3390e76a52"}
Sep 16 02:19:46.437519 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:19:46.437459    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-59fbdd77d-4c8ff" event={"ID":"f7765761-5911-4ee7-9587-9454b2e15ddc","Type":"ContainerStarted","Data":"1e597bec4d1ae583ec8d3232df3f274a05af2e5f1cf13f98162abd6dc23fcc2d"}
Sep 16 02:19:46.437519 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:19:46.437487    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-59fbdd77d-4c8ff" event={"ID":"f7765761-5911-4ee7-9587-9454b2e15ddc","Type":"ContainerStarted","Data":"6a714c2ae609e3ff20c6ad8bf5607a8d64f98f5e700bbede8707347cd769eaaf"}
Sep 16 02:19:46.464957 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:19:46.464028    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/kube-state-metrics-59fbdd77d-4c8ff" podStartSLOduration=1.97574533 podStartE2EDuration="3.464010142s" totalImagesPullingTime="1.488264812s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 02:19:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 02:19:46.463421566 +0000 UTC m=+64.931210361" watchObservedRunningTime="2026-09-16 02:19:46.464010142 +0000 UTC m=+64.931798925"
Sep 16 02:19:46.582410 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:19:46.582383    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-79d4d88bf4-4nc9x"]
Sep 16 02:19:46.584328 ip-10-0-128-111 kubenswrapper[2583]: W0916 02:19:46.584301    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbd366e5d_6d4a_45b9_b259_9d5488a3b7a6.slice/crio-0891ef3f5d0aecd20cfc5aef0eceac5302218d79c64ac3ffb01026da253192a6 WatchSource:0}: Error finding container 0891ef3f5d0aecd20cfc5aef0eceac5302218d79c64ac3ffb01026da253192a6: Status 404 returned error can't find the container with id 0891ef3f5d0aecd20cfc5aef0eceac5302218d79c64ac3ffb01026da253192a6
Sep 16 02:19:47.441790 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:19:47.441752    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-79d4d88bf4-4nc9x" event={"ID":"bd366e5d-6d4a-45b9-b259-9d5488a3b7a6","Type":"ContainerStarted","Data":"0891ef3f5d0aecd20cfc5aef0eceac5302218d79c64ac3ffb01026da253192a6"}
Sep 16 02:19:47.444060 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:19:47.444028    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-hv9kk" event={"ID":"80cd7034-5b29-4116-aa53-74168a3a96ed","Type":"ContainerStarted","Data":"823ad2f04ceda2a93daa19a153b3262587a9cea0ae26f9ca8fd010d8c3e212a4"}
Sep 16 02:19:47.444175 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:19:47.444086    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-hv9kk" event={"ID":"80cd7034-5b29-4116-aa53-74168a3a96ed","Type":"ContainerStarted","Data":"56f341810aae11b3d7d34c6cb1155a8376e03c39f4f41724f6e27ad4a7f4b8df"}
Sep 16 02:19:47.463793 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:19:47.463753    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-hv9kk" podStartSLOduration=3.410956102 podStartE2EDuration="4.463739772s" totalImagesPullingTime="984.068575ms" totalInitContainerRuntime="68.715095ms" isStatefulPod=true podCreationTimestamp="2026-09-16 02:19:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 02:19:47.462424447 +0000 UTC m=+65.930213231" watchObservedRunningTime="2026-09-16 02:19:47.463739772 +0000 UTC m=+65.931528554"
Sep 16 02:19:47.740516 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:19:47.740423    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/67a44fbe-95d7-4808-bcff-f7d214c6d82c-metrics-certs\") pod \"network-metrics-daemon-bdwgr\" (UID: \"67a44fbe-95d7-4808-bcff-f7d214c6d82c\") " pod="openshift-multus/network-metrics-daemon-bdwgr"
Sep 16 02:19:47.743275 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:19:47.743247    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Sep 16 02:19:47.753644 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:19:47.753620    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/67a44fbe-95d7-4808-bcff-f7d214c6d82c-metrics-certs\") pod \"network-metrics-daemon-bdwgr\" (UID: \"67a44fbe-95d7-4808-bcff-f7d214c6d82c\") " pod="openshift-multus/network-metrics-daemon-bdwgr"
Sep 16 02:19:47.771862 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:19:47.771839    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-ss74q\""
Sep 16 02:19:47.780723 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:19:47.780702    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bdwgr"
Sep 16 02:19:47.842160 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:19:47.841180    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/2fbf440b-4c09-428e-b6bf-1236d95f97cc-original-pull-secret\") pod \"global-pull-secret-syncer-mbxmd\" (UID: \"2fbf440b-4c09-428e-b6bf-1236d95f97cc\") " pod="kube-system/global-pull-secret-syncer-mbxmd"
Sep 16 02:19:47.842160 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:19:47.841247    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qwztf\" (UniqueName: \"kubernetes.io/projected/5f66bb1b-6d4f-42c5-9b63-f7f5f301b6f9-kube-api-access-qwztf\") pod \"network-check-target-55zcf\" (UID: \"5f66bb1b-6d4f-42c5-9b63-f7f5f301b6f9\") " pod="openshift-network-diagnostics/network-check-target-55zcf"
Sep 16 02:19:47.844186 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:19:47.844164    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Sep 16 02:19:47.844441 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:19:47.844427    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Sep 16 02:19:47.854678 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:19:47.854657    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Sep 16 02:19:47.855142 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:19:47.855116    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/2fbf440b-4c09-428e-b6bf-1236d95f97cc-original-pull-secret\") pod \"global-pull-secret-syncer-mbxmd\" (UID: \"2fbf440b-4c09-428e-b6bf-1236d95f97cc\") " pod="kube-system/global-pull-secret-syncer-mbxmd"
Sep 16 02:19:47.865415 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:19:47.865385    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qwztf\" (UniqueName: \"kubernetes.io/projected/5f66bb1b-6d4f-42c5-9b63-f7f5f301b6f9-kube-api-access-qwztf\") pod \"network-check-target-55zcf\" (UID: \"5f66bb1b-6d4f-42c5-9b63-f7f5f301b6f9\") " pod="openshift-network-diagnostics/network-check-target-55zcf"
Sep 16 02:19:48.055452 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:19:48.055344    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-bdwgr"]
Sep 16 02:19:48.055452 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:19:48.055382    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mbxmd"
Sep 16 02:19:48.059125 ip-10-0-128-111 kubenswrapper[2583]: W0916 02:19:48.059105    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod67a44fbe_95d7_4808_bcff_f7d214c6d82c.slice/crio-1dbdbd4b39d38e5474bdd8f4cce9242b4e40350a64ef967398cdd9fb6f07b8f9 WatchSource:0}: Error finding container 1dbdbd4b39d38e5474bdd8f4cce9242b4e40350a64ef967398cdd9fb6f07b8f9: Status 404 returned error can't find the container with id 1dbdbd4b39d38e5474bdd8f4cce9242b4e40350a64ef967398cdd9fb6f07b8f9
Sep 16 02:19:48.064853 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:19:48.064832    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-7ktl5\""
Sep 16 02:19:48.072818 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:19:48.072795    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-55zcf"
Sep 16 02:19:48.322979 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:19:48.322896    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-mbxmd"]
Sep 16 02:19:48.346482 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:19:48.346451    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-55zcf"]
Sep 16 02:19:48.375002 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:19:48.374971    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/metrics-server-f586c465f-6kvt2"]
Sep 16 02:19:48.377711 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:19:48.377687    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-f586c465f-6kvt2"
Sep 16 02:19:48.380270 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:19:48.380246    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-client-certs\""
Sep 16 02:19:48.380374 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:19:48.380270    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-server-audit-profiles\""
Sep 16 02:19:48.381604 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:19:48.381303    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-aion3muctknbr\""
Sep 16 02:19:48.381604 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:19:48.381414    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-dockercfg-gqtpq\""
Sep 16 02:19:48.381785 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:19:48.381609    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\""
Sep 16 02:19:48.381785 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:19:48.381637    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-tls\""
Sep 16 02:19:48.385546 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:19:48.385525    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-f586c465f-6kvt2"]
Sep 16 02:19:48.400069 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:19:48.400047    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-v2h5n"
Sep 16 02:19:48.444978 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:19:48.444948    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/41130ea6-39bb-4c30-956c-b94b1b1cc68a-secret-metrics-server-client-certs\") pod \"metrics-server-f586c465f-6kvt2\" (UID: \"41130ea6-39bb-4c30-956c-b94b1b1cc68a\") " pod="openshift-monitoring/metrics-server-f586c465f-6kvt2"
Sep 16 02:19:48.444978 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:19:48.444981    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/41130ea6-39bb-4c30-956c-b94b1b1cc68a-audit-log\") pod \"metrics-server-f586c465f-6kvt2\" (UID: \"41130ea6-39bb-4c30-956c-b94b1b1cc68a\") " pod="openshift-monitoring/metrics-server-f586c465f-6kvt2"
Sep 16 02:19:48.445482 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:19:48.445006    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/41130ea6-39bb-4c30-956c-b94b1b1cc68a-client-ca-bundle\") pod \"metrics-server-f586c465f-6kvt2\" (UID: \"41130ea6-39bb-4c30-956c-b94b1b1cc68a\") " pod="openshift-monitoring/metrics-server-f586c465f-6kvt2"
Sep 16 02:19:48.445482 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:19:48.445033    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/41130ea6-39bb-4c30-956c-b94b1b1cc68a-metrics-server-audit-profiles\") pod \"metrics-server-f586c465f-6kvt2\" (UID: \"41130ea6-39bb-4c30-956c-b94b1b1cc68a\") " pod="openshift-monitoring/metrics-server-f586c465f-6kvt2"
Sep 16 02:19:48.445482 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:19:48.445133    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tj79d\" (UniqueName: \"kubernetes.io/projected/41130ea6-39bb-4c30-956c-b94b1b1cc68a-kube-api-access-tj79d\") pod \"metrics-server-f586c465f-6kvt2\" (UID: \"41130ea6-39bb-4c30-956c-b94b1b1cc68a\") " pod="openshift-monitoring/metrics-server-f586c465f-6kvt2"
Sep 16 02:19:48.445482 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:19:48.445253    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/41130ea6-39bb-4c30-956c-b94b1b1cc68a-secret-metrics-server-tls\") pod \"metrics-server-f586c465f-6kvt2\" (UID: \"41130ea6-39bb-4c30-956c-b94b1b1cc68a\") " pod="openshift-monitoring/metrics-server-f586c465f-6kvt2"
Sep 16 02:19:48.445482 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:19:48.445291    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/41130ea6-39bb-4c30-956c-b94b1b1cc68a-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-f586c465f-6kvt2\" (UID: \"41130ea6-39bb-4c30-956c-b94b1b1cc68a\") " pod="openshift-monitoring/metrics-server-f586c465f-6kvt2"
Sep 16 02:19:48.450291 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:19:48.450259    2583 generic.go:356] "Generic (PLEG): container finished" podID="5bf4691b-1501-4d65-9538-52851f45d954" containerID="3d32533969f138ea53bd203a1293d9a78c198026d7dfc791758590a05e1aa896" exitCode=0
Sep 16 02:19:48.450438 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:19:48.450409    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"5bf4691b-1501-4d65-9538-52851f45d954","Type":"ContainerDied","Data":"3d32533969f138ea53bd203a1293d9a78c198026d7dfc791758590a05e1aa896"}
Sep 16 02:19:48.452173 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:19:48.452144    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-bdwgr" event={"ID":"67a44fbe-95d7-4808-bcff-f7d214c6d82c","Type":"ContainerStarted","Data":"1dbdbd4b39d38e5474bdd8f4cce9242b4e40350a64ef967398cdd9fb6f07b8f9"}
Sep 16 02:19:48.546200 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:19:48.546159    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tj79d\" (UniqueName: \"kubernetes.io/projected/41130ea6-39bb-4c30-956c-b94b1b1cc68a-kube-api-access-tj79d\") pod \"metrics-server-f586c465f-6kvt2\" (UID: \"41130ea6-39bb-4c30-956c-b94b1b1cc68a\") " pod="openshift-monitoring/metrics-server-f586c465f-6kvt2"
Sep 16 02:19:48.546386 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:19:48.546287    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/41130ea6-39bb-4c30-956c-b94b1b1cc68a-secret-metrics-server-tls\") pod \"metrics-server-f586c465f-6kvt2\" (UID: \"41130ea6-39bb-4c30-956c-b94b1b1cc68a\") " pod="openshift-monitoring/metrics-server-f586c465f-6kvt2"
Sep 16 02:19:48.546464 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:19:48.546435    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/41130ea6-39bb-4c30-956c-b94b1b1cc68a-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-f586c465f-6kvt2\" (UID: \"41130ea6-39bb-4c30-956c-b94b1b1cc68a\") " pod="openshift-monitoring/metrics-server-f586c465f-6kvt2"
Sep 16 02:19:48.546923 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:19:48.546754    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/41130ea6-39bb-4c30-956c-b94b1b1cc68a-secret-metrics-server-client-certs\") pod \"metrics-server-f586c465f-6kvt2\" (UID: \"41130ea6-39bb-4c30-956c-b94b1b1cc68a\") " pod="openshift-monitoring/metrics-server-f586c465f-6kvt2"
Sep 16 02:19:48.546923 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:19:48.546800    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/41130ea6-39bb-4c30-956c-b94b1b1cc68a-audit-log\") pod \"metrics-server-f586c465f-6kvt2\" (UID: \"41130ea6-39bb-4c30-956c-b94b1b1cc68a\") " pod="openshift-monitoring/metrics-server-f586c465f-6kvt2"
Sep 16 02:19:48.546923 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:19:48.546847    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/41130ea6-39bb-4c30-956c-b94b1b1cc68a-client-ca-bundle\") pod \"metrics-server-f586c465f-6kvt2\" (UID: \"41130ea6-39bb-4c30-956c-b94b1b1cc68a\") " pod="openshift-monitoring/metrics-server-f586c465f-6kvt2"
Sep 16 02:19:48.546923 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:19:48.546902    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/41130ea6-39bb-4c30-956c-b94b1b1cc68a-metrics-server-audit-profiles\") pod \"metrics-server-f586c465f-6kvt2\" (UID: \"41130ea6-39bb-4c30-956c-b94b1b1cc68a\") " pod="openshift-monitoring/metrics-server-f586c465f-6kvt2"
Sep 16 02:19:48.547937 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:19:48.547907    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/41130ea6-39bb-4c30-956c-b94b1b1cc68a-audit-log\") pod \"metrics-server-f586c465f-6kvt2\" (UID: \"41130ea6-39bb-4c30-956c-b94b1b1cc68a\") " pod="openshift-monitoring/metrics-server-f586c465f-6kvt2"
Sep 16 02:19:48.548031 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:19:48.548005    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/41130ea6-39bb-4c30-956c-b94b1b1cc68a-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-f586c465f-6kvt2\" (UID: \"41130ea6-39bb-4c30-956c-b94b1b1cc68a\") " pod="openshift-monitoring/metrics-server-f586c465f-6kvt2"
Sep 16 02:19:48.549536 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:19:48.549510    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/41130ea6-39bb-4c30-956c-b94b1b1cc68a-metrics-server-audit-profiles\") pod \"metrics-server-f586c465f-6kvt2\" (UID: \"41130ea6-39bb-4c30-956c-b94b1b1cc68a\") " pod="openshift-monitoring/metrics-server-f586c465f-6kvt2"
Sep 16 02:19:48.550030 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:19:48.550009    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/41130ea6-39bb-4c30-956c-b94b1b1cc68a-secret-metrics-server-tls\") pod \"metrics-server-f586c465f-6kvt2\" (UID: \"41130ea6-39bb-4c30-956c-b94b1b1cc68a\") " pod="openshift-monitoring/metrics-server-f586c465f-6kvt2"
Sep 16 02:19:48.553650 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:19:48.553265    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/41130ea6-39bb-4c30-956c-b94b1b1cc68a-secret-metrics-server-client-certs\") pod \"metrics-server-f586c465f-6kvt2\" (UID: \"41130ea6-39bb-4c30-956c-b94b1b1cc68a\") " pod="openshift-monitoring/metrics-server-f586c465f-6kvt2"
Sep 16 02:19:48.554654 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:19:48.554631    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tj79d\" (UniqueName: \"kubernetes.io/projected/41130ea6-39bb-4c30-956c-b94b1b1cc68a-kube-api-access-tj79d\") pod \"metrics-server-f586c465f-6kvt2\" (UID: \"41130ea6-39bb-4c30-956c-b94b1b1cc68a\") " pod="openshift-monitoring/metrics-server-f586c465f-6kvt2"
Sep 16 02:19:48.555576 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:19:48.555531    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/41130ea6-39bb-4c30-956c-b94b1b1cc68a-client-ca-bundle\") pod \"metrics-server-f586c465f-6kvt2\" (UID: \"41130ea6-39bb-4c30-956c-b94b1b1cc68a\") " pod="openshift-monitoring/metrics-server-f586c465f-6kvt2"
Sep 16 02:19:48.689528 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:19:48.689488    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-f586c465f-6kvt2"
Sep 16 02:19:48.749380 ip-10-0-128-111 kubenswrapper[2583]: W0916 02:19:48.749308    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2fbf440b_4c09_428e_b6bf_1236d95f97cc.slice/crio-25db0ce470ed015798fda8e0c24abd3601c1d543d31ae54864e3388c3149cc2d WatchSource:0}: Error finding container 25db0ce470ed015798fda8e0c24abd3601c1d543d31ae54864e3388c3149cc2d: Status 404 returned error can't find the container with id 25db0ce470ed015798fda8e0c24abd3601c1d543d31ae54864e3388c3149cc2d
Sep 16 02:19:48.750118 ip-10-0-128-111 kubenswrapper[2583]: W0916 02:19:48.750092    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5f66bb1b_6d4f_42c5_9b63_f7f5f301b6f9.slice/crio-61e2d8456c35930ae8e61d4e6442e9b10c867cb9d681afffa87ca44e0fdca13e WatchSource:0}: Error finding container 61e2d8456c35930ae8e61d4e6442e9b10c867cb9d681afffa87ca44e0fdca13e: Status 404 returned error can't find the container with id 61e2d8456c35930ae8e61d4e6442e9b10c867cb9d681afffa87ca44e0fdca13e
Sep 16 02:19:49.023860 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:19:49.023824    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-f586c465f-6kvt2"]
Sep 16 02:19:49.027164 ip-10-0-128-111 kubenswrapper[2583]: W0916 02:19:49.027133    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41130ea6_39bb_4c30_956c_b94b1b1cc68a.slice/crio-8feee80d85df2872c8b5702a5bb7ade381797960a486c4927b8ff99f6b3a35d8 WatchSource:0}: Error finding container 8feee80d85df2872c8b5702a5bb7ade381797960a486c4927b8ff99f6b3a35d8: Status 404 returned error can't find the container with id 8feee80d85df2872c8b5702a5bb7ade381797960a486c4927b8ff99f6b3a35d8
Sep 16 02:19:49.458429 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:19:49.458356    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-55zcf" event={"ID":"5f66bb1b-6d4f-42c5-9b63-f7f5f301b6f9","Type":"ContainerStarted","Data":"61e2d8456c35930ae8e61d4e6442e9b10c867cb9d681afffa87ca44e0fdca13e"}
Sep 16 02:19:49.460135 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:19:49.460062    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-mbxmd" event={"ID":"2fbf440b-4c09-428e-b6bf-1236d95f97cc","Type":"ContainerStarted","Data":"25db0ce470ed015798fda8e0c24abd3601c1d543d31ae54864e3388c3149cc2d"}
Sep 16 02:19:49.461659 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:19:49.461604    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-f586c465f-6kvt2" event={"ID":"41130ea6-39bb-4c30-956c-b94b1b1cc68a","Type":"ContainerStarted","Data":"8feee80d85df2872c8b5702a5bb7ade381797960a486c4927b8ff99f6b3a35d8"}
Sep 16 02:19:49.464574 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:19:49.464504    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-79d4d88bf4-4nc9x" event={"ID":"bd366e5d-6d4a-45b9-b259-9d5488a3b7a6","Type":"ContainerStarted","Data":"3473f36a73447403b5f6ce5a286219fad836b76e70311c56ac341e4870565ff1"}
Sep 16 02:19:49.464681 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:19:49.464591    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-79d4d88bf4-4nc9x" event={"ID":"bd366e5d-6d4a-45b9-b259-9d5488a3b7a6","Type":"ContainerStarted","Data":"f5766a5f9df7f1af45e440816438c2d7f01c5dfc66aa47535fc12a9150c89b5e"}
Sep 16 02:19:49.464681 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:19:49.464634    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-79d4d88bf4-4nc9x" event={"ID":"bd366e5d-6d4a-45b9-b259-9d5488a3b7a6","Type":"ContainerStarted","Data":"c03c37a58fc4c0313de116ae1392b3f5d8996054fad5a32cd977d44494a07f97"}
Sep 16 02:19:50.226298 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:19:50.226261    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 16 02:19:50.230148 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:19:50.230112    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 02:19:50.235140 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:19:50.235111    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\""
Sep 16 02:19:50.236668 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:19:50.235303    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\""
Sep 16 02:19:50.236668 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:19:50.235522    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\""
Sep 16 02:19:50.236668 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:19:50.235694    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\""
Sep 16 02:19:50.236668 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:19:50.235811    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\""
Sep 16 02:19:50.236668 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:19:50.236077    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\""
Sep 16 02:19:50.236668 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:19:50.236268    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-e3djpgrfpc0t\""
Sep 16 02:19:50.236668 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:19:50.236397    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\""
Sep 16 02:19:50.236668 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:19:50.236537    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-jjgfm\""
Sep 16 02:19:50.241141 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:19:50.238050    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\""
Sep 16 02:19:50.244630 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:19:50.241956    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\""
Sep 16 02:19:50.244630 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:19:50.242137    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\""
Sep 16 02:19:50.244630 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:19:50.243137    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\""
Sep 16 02:19:50.244630 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:19:50.243408    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\""
Sep 16 02:19:50.244630 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:19:50.243987    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\""
Sep 16 02:19:50.250419 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:19:50.246831    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\""
Sep 16 02:19:50.250419 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:19:50.247967    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 16 02:19:50.368147 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:19:50.367263    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/9cc00559-db17-4390-b926-cb912b947e7b-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"9cc00559-db17-4390-b926-cb912b947e7b\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 02:19:50.368147 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:19:50.367319    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9cc00559-db17-4390-b926-cb912b947e7b-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"9cc00559-db17-4390-b926-cb912b947e7b\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 02:19:50.368147 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:19:50.367351    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/9cc00559-db17-4390-b926-cb912b947e7b-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"9cc00559-db17-4390-b926-cb912b947e7b\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 02:19:50.368147 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:19:50.367386    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9cc00559-db17-4390-b926-cb912b947e7b-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"9cc00559-db17-4390-b926-cb912b947e7b\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 02:19:50.368147 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:19:50.367434    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/9cc00559-db17-4390-b926-cb912b947e7b-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"9cc00559-db17-4390-b926-cb912b947e7b\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 02:19:50.368147 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:19:50.367461    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/9cc00559-db17-4390-b926-cb912b947e7b-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"9cc00559-db17-4390-b926-cb912b947e7b\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 02:19:50.368147 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:19:50.367504    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/9cc00559-db17-4390-b926-cb912b947e7b-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"9cc00559-db17-4390-b926-cb912b947e7b\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 02:19:50.368147 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:19:50.367540    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/9cc00559-db17-4390-b926-cb912b947e7b-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"9cc00559-db17-4390-b926-cb912b947e7b\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 02:19:50.368147 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:19:50.367586    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/9cc00559-db17-4390-b926-cb912b947e7b-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"9cc00559-db17-4390-b926-cb912b947e7b\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 02:19:50.368147 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:19:50.367613    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/9cc00559-db17-4390-b926-cb912b947e7b-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"9cc00559-db17-4390-b926-cb912b947e7b\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 02:19:50.368147 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:19:50.367640    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/9cc00559-db17-4390-b926-cb912b947e7b-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"9cc00559-db17-4390-b926-cb912b947e7b\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 02:19:50.368147 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:19:50.367664    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/9cc00559-db17-4390-b926-cb912b947e7b-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"9cc00559-db17-4390-b926-cb912b947e7b\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 02:19:50.368147 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:19:50.367688    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/9cc00559-db17-4390-b926-cb912b947e7b-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"9cc00559-db17-4390-b926-cb912b947e7b\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 02:19:50.368147 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:19:50.367723    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/9cc00559-db17-4390-b926-cb912b947e7b-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"9cc00559-db17-4390-b926-cb912b947e7b\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 02:19:50.368147 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:19:50.367754    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9cc00559-db17-4390-b926-cb912b947e7b-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"9cc00559-db17-4390-b926-cb912b947e7b\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 02:19:50.368147 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:19:50.367778    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9jgx5\" (UniqueName: \"kubernetes.io/projected/9cc00559-db17-4390-b926-cb912b947e7b-kube-api-access-9jgx5\") pod \"prometheus-k8s-0\" (UID: \"9cc00559-db17-4390-b926-cb912b947e7b\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 02:19:50.369033 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:19:50.367811    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/9cc00559-db17-4390-b926-cb912b947e7b-config-out\") pod \"prometheus-k8s-0\" (UID: \"9cc00559-db17-4390-b926-cb912b947e7b\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 02:19:50.369033 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:19:50.367836    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/9cc00559-db17-4390-b926-cb912b947e7b-web-config\") pod \"prometheus-k8s-0\" (UID: \"9cc00559-db17-4390-b926-cb912b947e7b\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 02:19:50.369033 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:19:50.367863    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/9cc00559-db17-4390-b926-cb912b947e7b-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"9cc00559-db17-4390-b926-cb912b947e7b\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 02:19:50.369033 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:19:50.367906    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/9cc00559-db17-4390-b926-cb912b947e7b-config\") pod \"prometheus-k8s-0\" (UID: \"9cc00559-db17-4390-b926-cb912b947e7b\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 02:19:50.469901 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:19:50.468944    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/9cc00559-db17-4390-b926-cb912b947e7b-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"9cc00559-db17-4390-b926-cb912b947e7b\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 02:19:50.469901 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:19:50.468993    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9cc00559-db17-4390-b926-cb912b947e7b-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"9cc00559-db17-4390-b926-cb912b947e7b\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 02:19:50.469901 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:19:50.469026    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/9cc00559-db17-4390-b926-cb912b947e7b-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"9cc00559-db17-4390-b926-cb912b947e7b\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 02:19:50.469901 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:19:50.469065    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9cc00559-db17-4390-b926-cb912b947e7b-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"9cc00559-db17-4390-b926-cb912b947e7b\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 02:19:50.469901 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:19:50.469097    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/9cc00559-db17-4390-b926-cb912b947e7b-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"9cc00559-db17-4390-b926-cb912b947e7b\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 02:19:50.469901 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:19:50.469120    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/9cc00559-db17-4390-b926-cb912b947e7b-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"9cc00559-db17-4390-b926-cb912b947e7b\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 02:19:50.469901 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:19:50.469148    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/9cc00559-db17-4390-b926-cb912b947e7b-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"9cc00559-db17-4390-b926-cb912b947e7b\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 02:19:50.469901 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:19:50.469198    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/9cc00559-db17-4390-b926-cb912b947e7b-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"9cc00559-db17-4390-b926-cb912b947e7b\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 02:19:50.469901 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:19:50.469220    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/9cc00559-db17-4390-b926-cb912b947e7b-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"9cc00559-db17-4390-b926-cb912b947e7b\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 02:19:50.469901 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:19:50.469245    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/9cc00559-db17-4390-b926-cb912b947e7b-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"9cc00559-db17-4390-b926-cb912b947e7b\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 02:19:50.469901 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:19:50.469268    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/9cc00559-db17-4390-b926-cb912b947e7b-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"9cc00559-db17-4390-b926-cb912b947e7b\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 02:19:50.469901 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:19:50.469291    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/9cc00559-db17-4390-b926-cb912b947e7b-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"9cc00559-db17-4390-b926-cb912b947e7b\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 02:19:50.469901 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:19:50.469312    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/9cc00559-db17-4390-b926-cb912b947e7b-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"9cc00559-db17-4390-b926-cb912b947e7b\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 02:19:50.469901 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:19:50.469342    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/9cc00559-db17-4390-b926-cb912b947e7b-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"9cc00559-db17-4390-b926-cb912b947e7b\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 02:19:50.469901 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:19:50.469364    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9cc00559-db17-4390-b926-cb912b947e7b-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"9cc00559-db17-4390-b926-cb912b947e7b\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 02:19:50.469901 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:19:50.469386    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9jgx5\" (UniqueName: \"kubernetes.io/projected/9cc00559-db17-4390-b926-cb912b947e7b-kube-api-access-9jgx5\") pod \"prometheus-k8s-0\" (UID: \"9cc00559-db17-4390-b926-cb912b947e7b\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 02:19:50.469901 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:19:50.469421    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/9cc00559-db17-4390-b926-cb912b947e7b-config-out\") pod \"prometheus-k8s-0\" (UID: \"9cc00559-db17-4390-b926-cb912b947e7b\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 02:19:50.471200 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:19:50.469447    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/9cc00559-db17-4390-b926-cb912b947e7b-web-config\") pod \"prometheus-k8s-0\" (UID: \"9cc00559-db17-4390-b926-cb912b947e7b\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 02:19:50.471200 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:19:50.469482    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/9cc00559-db17-4390-b926-cb912b947e7b-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"9cc00559-db17-4390-b926-cb912b947e7b\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 02:19:50.471200 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:19:50.469524    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/9cc00559-db17-4390-b926-cb912b947e7b-config\") pod \"prometheus-k8s-0\" (UID: \"9cc00559-db17-4390-b926-cb912b947e7b\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 02:19:50.471200 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:19:50.470162    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/9cc00559-db17-4390-b926-cb912b947e7b-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"9cc00559-db17-4390-b926-cb912b947e7b\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 02:19:50.471410 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:19:50.471372    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9cc00559-db17-4390-b926-cb912b947e7b-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"9cc00559-db17-4390-b926-cb912b947e7b\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 02:19:50.472498 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:19:50.472178    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9cc00559-db17-4390-b926-cb912b947e7b-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"9cc00559-db17-4390-b926-cb912b947e7b\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 02:19:50.473155 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:19:50.473127    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/9cc00559-db17-4390-b926-cb912b947e7b-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"9cc00559-db17-4390-b926-cb912b947e7b\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 02:19:50.474773 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:19:50.474351    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/9cc00559-db17-4390-b926-cb912b947e7b-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"9cc00559-db17-4390-b926-cb912b947e7b\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 02:19:50.475049 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:19:50.475031    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/9cc00559-db17-4390-b926-cb912b947e7b-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"9cc00559-db17-4390-b926-cb912b947e7b\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 02:19:50.475785 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:19:50.475762    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9cc00559-db17-4390-b926-cb912b947e7b-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"9cc00559-db17-4390-b926-cb912b947e7b\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 02:19:50.479837 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:19:50.479183    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/9cc00559-db17-4390-b926-cb912b947e7b-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"9cc00559-db17-4390-b926-cb912b947e7b\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 02:19:50.479837 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:19:50.479410    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/9cc00559-db17-4390-b926-cb912b947e7b-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"9cc00559-db17-4390-b926-cb912b947e7b\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 02:19:50.479837 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:19:50.479735    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/9cc00559-db17-4390-b926-cb912b947e7b-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"9cc00559-db17-4390-b926-cb912b947e7b\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 02:19:50.480474 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:19:50.480427    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/9cc00559-db17-4390-b926-cb912b947e7b-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"9cc00559-db17-4390-b926-cb912b947e7b\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 02:19:50.481247 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:19:50.481202    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/9cc00559-db17-4390-b926-cb912b947e7b-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"9cc00559-db17-4390-b926-cb912b947e7b\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 02:19:50.484663 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:19:50.484641    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/9cc00559-db17-4390-b926-cb912b947e7b-config\") pod \"prometheus-k8s-0\" (UID: \"9cc00559-db17-4390-b926-cb912b947e7b\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 02:19:50.484768 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:19:50.484747    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/9cc00559-db17-4390-b926-cb912b947e7b-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"9cc00559-db17-4390-b926-cb912b947e7b\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 02:19:50.484768 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:19:50.484756    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9jgx5\" (UniqueName: \"kubernetes.io/projected/9cc00559-db17-4390-b926-cb912b947e7b-kube-api-access-9jgx5\") pod \"prometheus-k8s-0\" (UID: \"9cc00559-db17-4390-b926-cb912b947e7b\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 02:19:50.484948 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:19:50.484921    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/9cc00559-db17-4390-b926-cb912b947e7b-config-out\") pod \"prometheus-k8s-0\" (UID: \"9cc00559-db17-4390-b926-cb912b947e7b\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 02:19:50.485012 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:19:50.484950    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/9cc00559-db17-4390-b926-cb912b947e7b-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"9cc00559-db17-4390-b926-cb912b947e7b\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 02:19:50.485171 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:19:50.485151    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/9cc00559-db17-4390-b926-cb912b947e7b-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"9cc00559-db17-4390-b926-cb912b947e7b\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 02:19:50.485234 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:19:50.485182    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/9cc00559-db17-4390-b926-cb912b947e7b-web-config\") pod \"prometheus-k8s-0\" (UID: \"9cc00559-db17-4390-b926-cb912b947e7b\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 02:19:50.485234 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:19:50.485182    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/9cc00559-db17-4390-b926-cb912b947e7b-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"9cc00559-db17-4390-b926-cb912b947e7b\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 02:19:50.551641 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:19:50.551375    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 02:19:51.321418 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:19:51.321200    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 16 02:19:51.328266 ip-10-0-128-111 kubenswrapper[2583]: W0916 02:19:51.328232    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9cc00559_db17_4390_b926_cb912b947e7b.slice/crio-142fe8da9aa07e1b39f8e33d1a4de7860fa4dc52e215793fce1ffcd9c611f540 WatchSource:0}: Error finding container 142fe8da9aa07e1b39f8e33d1a4de7860fa4dc52e215793fce1ffcd9c611f540: Status 404 returned error can't find the container with id 142fe8da9aa07e1b39f8e33d1a4de7860fa4dc52e215793fce1ffcd9c611f540
Sep 16 02:19:51.478763 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:19:51.478339    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"9cc00559-db17-4390-b926-cb912b947e7b","Type":"ContainerStarted","Data":"335fb22f91f7fbe2f73e01a471295c3453e09ef84d66a4d843daa5af0f1cba2d"}
Sep 16 02:19:51.478763 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:19:51.478418    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"9cc00559-db17-4390-b926-cb912b947e7b","Type":"ContainerStarted","Data":"142fe8da9aa07e1b39f8e33d1a4de7860fa4dc52e215793fce1ffcd9c611f540"}
Sep 16 02:19:51.484519 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:19:51.484493    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"5bf4691b-1501-4d65-9538-52851f45d954","Type":"ContainerStarted","Data":"79420c355c9bdc0f2968e2c0d3d22e0c0651599f0f94a0732efe01bc15381592"}
Sep 16 02:19:51.484733 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:19:51.484714    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"5bf4691b-1501-4d65-9538-52851f45d954","Type":"ContainerStarted","Data":"63d9db423d59c98d2fce3f3c7335605fa71160e25015a75a29498b296f453caa"}
Sep 16 02:19:51.484892 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:19:51.484876    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"5bf4691b-1501-4d65-9538-52851f45d954","Type":"ContainerStarted","Data":"30eaae2482b7efd3dc0aadf8fddf4ed760b07e76b2aae9bc6752104c72cc2b52"}
Sep 16 02:19:51.485038 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:19:51.485023    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"5bf4691b-1501-4d65-9538-52851f45d954","Type":"ContainerStarted","Data":"8375df7a74ca45d311dc292b23d9289ff7bca198e49b7dd4251a1636003e07e7"}
Sep 16 02:19:51.487775 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:19:51.487700    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-bdwgr" event={"ID":"67a44fbe-95d7-4808-bcff-f7d214c6d82c","Type":"ContainerStarted","Data":"954edd6406ba28151e41ebfcc3b5247e19946b3ee56cd89d8d2cc1d287a492d7"}
Sep 16 02:19:51.487775 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:19:51.487746    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-bdwgr" event={"ID":"67a44fbe-95d7-4808-bcff-f7d214c6d82c","Type":"ContainerStarted","Data":"8a56d9a25d43bd7f6b08f17402745031be73b714baf2ab8c3524bfaa3bdebc1f"}
Sep 16 02:19:51.489877 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:19:51.489854    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-f586c465f-6kvt2" event={"ID":"41130ea6-39bb-4c30-956c-b94b1b1cc68a","Type":"ContainerStarted","Data":"4a1da014dc06ee32ad633a3ab3f7deff02ef4360f82d2ea92ae1c204dd07a4dd"}
Sep 16 02:19:51.496302 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:19:51.496278    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-79d4d88bf4-4nc9x" event={"ID":"bd366e5d-6d4a-45b9-b259-9d5488a3b7a6","Type":"ContainerStarted","Data":"e7081840adb67f67492c7e560e602e56b1dd71f30e757b95b6fec53227b70bfe"}
Sep 16 02:19:51.496389 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:19:51.496344    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-79d4d88bf4-4nc9x" event={"ID":"bd366e5d-6d4a-45b9-b259-9d5488a3b7a6","Type":"ContainerStarted","Data":"85f8060451a3e8a67f8160b40a72b2faf9797f2e6af6f9a226b26eb56898d6ec"}
Sep 16 02:19:51.496460 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:19:51.496387    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-79d4d88bf4-4nc9x" event={"ID":"bd366e5d-6d4a-45b9-b259-9d5488a3b7a6","Type":"ContainerStarted","Data":"eadde5104cb4f5da73fd8670f9c6a28bc2814b10d586015a9137b5fcab5b3afa"}
Sep 16 02:19:51.497151 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:19:51.497132    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/thanos-querier-79d4d88bf4-4nc9x"
Sep 16 02:19:51.521755 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:19:51.521637    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/thanos-querier-79d4d88bf4-4nc9x" podStartSLOduration=2.278062668 podStartE2EDuration="6.521616597s" totalImagesPullingTime="4.243553929s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 02:19:45 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 02:19:51.520960494 +0000 UTC m=+69.988749277" watchObservedRunningTime="2026-09-16 02:19:51.521616597 +0000 UTC m=+69.989405380"
Sep 16 02:19:51.539038 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:19:51.538952    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/metrics-server-f586c465f-6kvt2" podStartSLOduration=1.504864334 podStartE2EDuration="3.538934141s" totalImagesPullingTime="2.034069807s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 02:19:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 02:19:51.537514205 +0000 UTC m=+70.005303014" watchObservedRunningTime="2026-09-16 02:19:51.538934141 +0000 UTC m=+70.006722925"
Sep 16 02:19:51.552606 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:19:51.552571    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-bdwgr" podStartSLOduration=68.054071034 podStartE2EDuration="1m9.55254005s" totalImagesPullingTime="1.498469016s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 02:18:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 02:19:51.550345054 +0000 UTC m=+70.018133848" watchObservedRunningTime="2026-09-16 02:19:51.55254005 +0000 UTC m=+70.020328876"
Sep 16 02:19:52.504356 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:19:52.504314    2583 generic.go:356] "Generic (PLEG): container finished" podID="9cc00559-db17-4390-b926-cb912b947e7b" containerID="335fb22f91f7fbe2f73e01a471295c3453e09ef84d66a4d843daa5af0f1cba2d" exitCode=0
Sep 16 02:19:52.504861 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:19:52.504404    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"9cc00559-db17-4390-b926-cb912b947e7b","Type":"ContainerDied","Data":"335fb22f91f7fbe2f73e01a471295c3453e09ef84d66a4d843daa5af0f1cba2d"}
Sep 16 02:19:52.508839 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:19:52.508804    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"5bf4691b-1501-4d65-9538-52851f45d954","Type":"ContainerStarted","Data":"582f429a3fbde7376ca575553aafc85b416a57258ae0bc5f949f128e41dfcf64"}
Sep 16 02:19:52.508945 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:19:52.508889    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"5bf4691b-1501-4d65-9538-52851f45d954","Type":"ContainerStarted","Data":"f34d032e688ef59537b80645fb4184c9c133350be5f8c68a1515659d0bc1e472"}
Sep 16 02:19:52.554101 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:19:52.554048    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=3.842142022 podStartE2EDuration="7.554030947s" totalImagesPullingTime="3.688500897s" totalInitContainerRuntime="23.388028ms" isStatefulPod=false podCreationTimestamp="2026-09-16 02:19:45 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 02:19:52.552841944 +0000 UTC m=+71.020630726" watchObservedRunningTime="2026-09-16 02:19:52.554030947 +0000 UTC m=+71.021819728"
Sep 16 02:19:53.523129 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:19:53.520437    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/thanos-querier-79d4d88bf4-4nc9x"
Sep 16 02:19:55.525119 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:19:55.525080    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-55zcf" event={"ID":"5f66bb1b-6d4f-42c5-9b63-f7f5f301b6f9","Type":"ContainerStarted","Data":"d8d3701c00dd10a96205b279ef63bebaab5393e83dd4157651ae7b1c4c117a7a"}
Sep 16 02:19:55.525606 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:19:55.525212    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-55zcf"
Sep 16 02:19:55.526934 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:19:55.526912    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-mbxmd" event={"ID":"2fbf440b-4c09-428e-b6bf-1236d95f97cc","Type":"ContainerStarted","Data":"e7d35b6def58dd8d87fda31e3463bf161f9702227a36fcab72a2d103d84ba18c"}
Sep 16 02:19:55.539504 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:19:55.539055    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-55zcf" podStartSLOduration=67.673249593 podStartE2EDuration="1m13.539037692s" totalImagesPullingTime="5.865788099s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 02:18:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 02:19:55.538390251 +0000 UTC m=+74.006179035" watchObservedRunningTime="2026-09-16 02:19:55.539037692 +0000 UTC m=+74.006826474"
Sep 16 02:19:55.552017 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:19:55.551967    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-mbxmd" podStartSLOduration=66.686175456 podStartE2EDuration="1m12.551948493s" totalImagesPullingTime="5.865773037s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 02:18:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 02:19:55.551158437 +0000 UTC m=+74.018947220" watchObservedRunningTime="2026-09-16 02:19:55.551948493 +0000 UTC m=+74.019737276"
Sep 16 02:19:57.538873 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:19:57.538841    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"9cc00559-db17-4390-b926-cb912b947e7b","Type":"ContainerStarted","Data":"501ee12b90e989972c892e33d1b1754695561add0dd08622d66b99b6e2d38f54"}
Sep 16 02:19:57.539161 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:19:57.538962    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"9cc00559-db17-4390-b926-cb912b947e7b","Type":"ContainerStarted","Data":"1421871f4efae41cca1f0a74619eb934782244bf8a909df0351c27932b0b4150"}
Sep 16 02:19:57.539161 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:19:57.539005    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"9cc00559-db17-4390-b926-cb912b947e7b","Type":"ContainerStarted","Data":"36cb8aec3dcb4f882b28557545eb9fd3890739e6141f4e49482d18cadcac7677"}
Sep 16 02:19:57.539161 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:19:57.539046    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"9cc00559-db17-4390-b926-cb912b947e7b","Type":"ContainerStarted","Data":"84b39eb0e85e4cb68e8c22ec7d5ebd8349797f955edd9fa2ad7cc3835eaa4ef3"}
Sep 16 02:19:58.546086 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:19:58.546045    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"9cc00559-db17-4390-b926-cb912b947e7b","Type":"ContainerStarted","Data":"dc01d49bd843fdcdfc3fe05ac1ebd0d33a9e94874df1f3df550cc51825f83c01"}
Sep 16 02:19:58.546431 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:19:58.546126    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"9cc00559-db17-4390-b926-cb912b947e7b","Type":"ContainerStarted","Data":"b2da4cf4a935141217a1262560b078b359b0162860efdb9647353030e0090590"}
Sep 16 02:19:58.572347 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:19:58.572282    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=3.796118291 podStartE2EDuration="8.572262357s" totalImagesPullingTime="4.738411813s" totalInitContainerRuntime="37.732253ms" isStatefulPod=false podCreationTimestamp="2026-09-16 02:19:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 02:19:58.570785673 +0000 UTC m=+77.038574455" watchObservedRunningTime="2026-09-16 02:19:58.572262357 +0000 UTC m=+77.040051140"
Sep 16 02:20:00.551739 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:20:00.551704    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 02:20:08.690148 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:20:08.690023    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/metrics-server-f586c465f-6kvt2"
Sep 16 02:20:08.690148 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:20:08.690095    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/metrics-server-f586c465f-6kvt2"
Sep 16 02:20:26.534502 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:20:26.534464    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-55zcf"
Sep 16 02:20:28.695310 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:20:28.695278    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/metrics-server-f586c465f-6kvt2"
Sep 16 02:20:28.699266 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:20:28.699241    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/metrics-server-f586c465f-6kvt2"
Sep 16 02:20:50.552290 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:20:50.552241    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 02:20:50.571436 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:20:50.571411    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 02:20:50.765996 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:20:50.765971    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 02:21:04.284122 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:21:04.284089    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 16 02:21:04.284613 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:21:04.284357    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="5bf4691b-1501-4d65-9538-52851f45d954" containerName="alertmanager" containerID="cri-o://8375df7a74ca45d311dc292b23d9289ff7bca198e49b7dd4251a1636003e07e7" gracePeriod=120
Sep 16 02:21:04.284613 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:21:04.284428    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="5bf4691b-1501-4d65-9538-52851f45d954" containerName="kube-rbac-proxy-metric" containerID="cri-o://f34d032e688ef59537b80645fb4184c9c133350be5f8c68a1515659d0bc1e472" gracePeriod=120
Sep 16 02:21:04.284613 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:21:04.284443    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="5bf4691b-1501-4d65-9538-52851f45d954" containerName="kube-rbac-proxy-web" containerID="cri-o://63d9db423d59c98d2fce3f3c7335605fa71160e25015a75a29498b296f453caa" gracePeriod=120
Sep 16 02:21:04.284613 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:21:04.284454    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="5bf4691b-1501-4d65-9538-52851f45d954" containerName="config-reloader" containerID="cri-o://30eaae2482b7efd3dc0aadf8fddf4ed760b07e76b2aae9bc6752104c72cc2b52" gracePeriod=120
Sep 16 02:21:04.284613 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:21:04.284497    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="5bf4691b-1501-4d65-9538-52851f45d954" containerName="kube-rbac-proxy" containerID="cri-o://79420c355c9bdc0f2968e2c0d3d22e0c0651599f0f94a0732efe01bc15381592" gracePeriod=120
Sep 16 02:21:04.284613 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:21:04.284545    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="5bf4691b-1501-4d65-9538-52851f45d954" containerName="prom-label-proxy" containerID="cri-o://582f429a3fbde7376ca575553aafc85b416a57258ae0bc5f949f128e41dfcf64" gracePeriod=120
Sep 16 02:21:04.810666 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:21:04.810634    2583 generic.go:356] "Generic (PLEG): container finished" podID="5bf4691b-1501-4d65-9538-52851f45d954" containerID="582f429a3fbde7376ca575553aafc85b416a57258ae0bc5f949f128e41dfcf64" exitCode=0
Sep 16 02:21:04.810666 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:21:04.810658    2583 generic.go:356] "Generic (PLEG): container finished" podID="5bf4691b-1501-4d65-9538-52851f45d954" containerID="79420c355c9bdc0f2968e2c0d3d22e0c0651599f0f94a0732efe01bc15381592" exitCode=0
Sep 16 02:21:04.810666 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:21:04.810664    2583 generic.go:356] "Generic (PLEG): container finished" podID="5bf4691b-1501-4d65-9538-52851f45d954" containerID="30eaae2482b7efd3dc0aadf8fddf4ed760b07e76b2aae9bc6752104c72cc2b52" exitCode=0
Sep 16 02:21:04.810666 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:21:04.810670    2583 generic.go:356] "Generic (PLEG): container finished" podID="5bf4691b-1501-4d65-9538-52851f45d954" containerID="8375df7a74ca45d311dc292b23d9289ff7bca198e49b7dd4251a1636003e07e7" exitCode=0
Sep 16 02:21:04.810926 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:21:04.810703    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"5bf4691b-1501-4d65-9538-52851f45d954","Type":"ContainerDied","Data":"582f429a3fbde7376ca575553aafc85b416a57258ae0bc5f949f128e41dfcf64"}
Sep 16 02:21:04.810926 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:21:04.810767    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"5bf4691b-1501-4d65-9538-52851f45d954","Type":"ContainerDied","Data":"79420c355c9bdc0f2968e2c0d3d22e0c0651599f0f94a0732efe01bc15381592"}
Sep 16 02:21:04.810926 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:21:04.810795    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"5bf4691b-1501-4d65-9538-52851f45d954","Type":"ContainerDied","Data":"30eaae2482b7efd3dc0aadf8fddf4ed760b07e76b2aae9bc6752104c72cc2b52"}
Sep 16 02:21:04.810926 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:21:04.810820    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"5bf4691b-1501-4d65-9538-52851f45d954","Type":"ContainerDied","Data":"8375df7a74ca45d311dc292b23d9289ff7bca198e49b7dd4251a1636003e07e7"}
Sep 16 02:21:05.521045 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:21:05.521019    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Sep 16 02:21:05.600750 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:21:05.600721    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/5bf4691b-1501-4d65-9538-52851f45d954-config-volume\") pod \"5bf4691b-1501-4d65-9538-52851f45d954\" (UID: \"5bf4691b-1501-4d65-9538-52851f45d954\") "
Sep 16 02:21:05.600924 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:21:05.600761    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/5bf4691b-1501-4d65-9538-52851f45d954-cluster-tls-config\") pod \"5bf4691b-1501-4d65-9538-52851f45d954\" (UID: \"5bf4691b-1501-4d65-9538-52851f45d954\") "
Sep 16 02:21:05.600924 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:21:05.600799    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/5bf4691b-1501-4d65-9538-52851f45d954-alertmanager-main-db\") pod \"5bf4691b-1501-4d65-9538-52851f45d954\" (UID: \"5bf4691b-1501-4d65-9538-52851f45d954\") "
Sep 16 02:21:05.600924 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:21:05.600827    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/5bf4691b-1501-4d65-9538-52851f45d954-web-config\") pod \"5bf4691b-1501-4d65-9538-52851f45d954\" (UID: \"5bf4691b-1501-4d65-9538-52851f45d954\") "
Sep 16 02:21:05.600924 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:21:05.600862    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/5bf4691b-1501-4d65-9538-52851f45d954-tls-assets\") pod \"5bf4691b-1501-4d65-9538-52851f45d954\" (UID: \"5bf4691b-1501-4d65-9538-52851f45d954\") "
Sep 16 02:21:05.600924 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:21:05.600903    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/5bf4691b-1501-4d65-9538-52851f45d954-secret-alertmanager-kube-rbac-proxy-web\") pod \"5bf4691b-1501-4d65-9538-52851f45d954\" (UID: \"5bf4691b-1501-4d65-9538-52851f45d954\") "
Sep 16 02:21:05.601174 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:21:05.600936    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/5bf4691b-1501-4d65-9538-52851f45d954-secret-alertmanager-main-tls\") pod \"5bf4691b-1501-4d65-9538-52851f45d954\" (UID: \"5bf4691b-1501-4d65-9538-52851f45d954\") "
Sep 16 02:21:05.601174 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:21:05.600968    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/5bf4691b-1501-4d65-9538-52851f45d954-metrics-client-ca\") pod \"5bf4691b-1501-4d65-9538-52851f45d954\" (UID: \"5bf4691b-1501-4d65-9538-52851f45d954\") "
Sep 16 02:21:05.601174 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:21:05.601013    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2fbnw\" (UniqueName: \"kubernetes.io/projected/5bf4691b-1501-4d65-9538-52851f45d954-kube-api-access-2fbnw\") pod \"5bf4691b-1501-4d65-9538-52851f45d954\" (UID: \"5bf4691b-1501-4d65-9538-52851f45d954\") "
Sep 16 02:21:05.601174 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:21:05.601051    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/5bf4691b-1501-4d65-9538-52851f45d954-config-out\") pod \"5bf4691b-1501-4d65-9538-52851f45d954\" (UID: \"5bf4691b-1501-4d65-9538-52851f45d954\") "
Sep 16 02:21:05.601174 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:21:05.601118    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/5bf4691b-1501-4d65-9538-52851f45d954-secret-alertmanager-kube-rbac-proxy\") pod \"5bf4691b-1501-4d65-9538-52851f45d954\" (UID: \"5bf4691b-1501-4d65-9538-52851f45d954\") "
Sep 16 02:21:05.601174 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:21:05.601155    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/5bf4691b-1501-4d65-9538-52851f45d954-secret-alertmanager-kube-rbac-proxy-metric\") pod \"5bf4691b-1501-4d65-9538-52851f45d954\" (UID: \"5bf4691b-1501-4d65-9538-52851f45d954\") "
Sep 16 02:21:05.601456 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:21:05.601165    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5bf4691b-1501-4d65-9538-52851f45d954-alertmanager-main-db" (OuterVolumeSpecName: "alertmanager-main-db") pod "5bf4691b-1501-4d65-9538-52851f45d954" (UID: "5bf4691b-1501-4d65-9538-52851f45d954"). InnerVolumeSpecName "alertmanager-main-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 02:21:05.601456 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:21:05.601181    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5bf4691b-1501-4d65-9538-52851f45d954-alertmanager-trusted-ca-bundle\") pod \"5bf4691b-1501-4d65-9538-52851f45d954\" (UID: \"5bf4691b-1501-4d65-9538-52851f45d954\") "
Sep 16 02:21:05.601456 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:21:05.601399    2583 reconciler_common.go:299] "Volume detached for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/5bf4691b-1501-4d65-9538-52851f45d954-alertmanager-main-db\") on node \"ip-10-0-128-111.ec2.internal\" DevicePath \"\""
Sep 16 02:21:05.601456 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:21:05.601413    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5bf4691b-1501-4d65-9538-52851f45d954-metrics-client-ca" (OuterVolumeSpecName: "metrics-client-ca") pod "5bf4691b-1501-4d65-9538-52851f45d954" (UID: "5bf4691b-1501-4d65-9538-52851f45d954"). InnerVolumeSpecName "metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 02:21:05.602063 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:21:05.601777    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5bf4691b-1501-4d65-9538-52851f45d954-alertmanager-trusted-ca-bundle" (OuterVolumeSpecName: "alertmanager-trusted-ca-bundle") pod "5bf4691b-1501-4d65-9538-52851f45d954" (UID: "5bf4691b-1501-4d65-9538-52851f45d954"). InnerVolumeSpecName "alertmanager-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 02:21:05.607405 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:21:05.607382    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5bf4691b-1501-4d65-9538-52851f45d954-config-out" (OuterVolumeSpecName: "config-out") pod "5bf4691b-1501-4d65-9538-52851f45d954" (UID: "5bf4691b-1501-4d65-9538-52851f45d954"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 02:21:05.607531 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:21:05.607406    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5bf4691b-1501-4d65-9538-52851f45d954-secret-alertmanager-main-tls" (OuterVolumeSpecName: "secret-alertmanager-main-tls") pod "5bf4691b-1501-4d65-9538-52851f45d954" (UID: "5bf4691b-1501-4d65-9538-52851f45d954"). InnerVolumeSpecName "secret-alertmanager-main-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 02:21:05.607531 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:21:05.607492    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5bf4691b-1501-4d65-9538-52851f45d954-config-volume" (OuterVolumeSpecName: "config-volume") pod "5bf4691b-1501-4d65-9538-52851f45d954" (UID: "5bf4691b-1501-4d65-9538-52851f45d954"). InnerVolumeSpecName "config-volume". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 02:21:05.607672 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:21:05.607586    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5bf4691b-1501-4d65-9538-52851f45d954-secret-alertmanager-kube-rbac-proxy-metric" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy-metric") pod "5bf4691b-1501-4d65-9538-52851f45d954" (UID: "5bf4691b-1501-4d65-9538-52851f45d954"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-metric". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 02:21:05.607789 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:21:05.607772    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5bf4691b-1501-4d65-9538-52851f45d954-kube-api-access-2fbnw" (OuterVolumeSpecName: "kube-api-access-2fbnw") pod "5bf4691b-1501-4d65-9538-52851f45d954" (UID: "5bf4691b-1501-4d65-9538-52851f45d954"). InnerVolumeSpecName "kube-api-access-2fbnw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 02:21:05.608044 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:21:05.608014    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5bf4691b-1501-4d65-9538-52851f45d954-secret-alertmanager-kube-rbac-proxy-web" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy-web") pod "5bf4691b-1501-4d65-9538-52851f45d954" (UID: "5bf4691b-1501-4d65-9538-52851f45d954"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 02:21:05.608639 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:21:05.608598    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5bf4691b-1501-4d65-9538-52851f45d954-secret-alertmanager-kube-rbac-proxy" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy") pod "5bf4691b-1501-4d65-9538-52851f45d954" (UID: "5bf4691b-1501-4d65-9538-52851f45d954"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 02:21:05.608733 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:21:05.608692    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5bf4691b-1501-4d65-9538-52851f45d954-tls-assets" (OuterVolumeSpecName: "tls-assets") pod "5bf4691b-1501-4d65-9538-52851f45d954" (UID: "5bf4691b-1501-4d65-9538-52851f45d954"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 02:21:05.610962 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:21:05.610942    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5bf4691b-1501-4d65-9538-52851f45d954-cluster-tls-config" (OuterVolumeSpecName: "cluster-tls-config") pod "5bf4691b-1501-4d65-9538-52851f45d954" (UID: "5bf4691b-1501-4d65-9538-52851f45d954"). InnerVolumeSpecName "cluster-tls-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 02:21:05.616874 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:21:05.616848    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5bf4691b-1501-4d65-9538-52851f45d954-web-config" (OuterVolumeSpecName: "web-config") pod "5bf4691b-1501-4d65-9538-52851f45d954" (UID: "5bf4691b-1501-4d65-9538-52851f45d954"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 02:21:05.701783 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:21:05.701732    2583 reconciler_common.go:299] "Volume detached for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/5bf4691b-1501-4d65-9538-52851f45d954-config-volume\") on node \"ip-10-0-128-111.ec2.internal\" DevicePath \"\""
Sep 16 02:21:05.701783 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:21:05.701773    2583 reconciler_common.go:299] "Volume detached for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/5bf4691b-1501-4d65-9538-52851f45d954-cluster-tls-config\") on node \"ip-10-0-128-111.ec2.internal\" DevicePath \"\""
Sep 16 02:21:05.701783 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:21:05.701784    2583 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/5bf4691b-1501-4d65-9538-52851f45d954-web-config\") on node \"ip-10-0-128-111.ec2.internal\" DevicePath \"\""
Sep 16 02:21:05.701783 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:21:05.701793    2583 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/5bf4691b-1501-4d65-9538-52851f45d954-tls-assets\") on node \"ip-10-0-128-111.ec2.internal\" DevicePath \"\""
Sep 16 02:21:05.701783 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:21:05.701802    2583 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/5bf4691b-1501-4d65-9538-52851f45d954-secret-alertmanager-kube-rbac-proxy-web\") on node \"ip-10-0-128-111.ec2.internal\" DevicePath \"\""
Sep 16 02:21:05.702090 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:21:05.701812    2583 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/5bf4691b-1501-4d65-9538-52851f45d954-secret-alertmanager-main-tls\") on node \"ip-10-0-128-111.ec2.internal\" DevicePath \"\""
Sep 16 02:21:05.702090 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:21:05.701821    2583 reconciler_common.go:299] "Volume detached for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/5bf4691b-1501-4d65-9538-52851f45d954-metrics-client-ca\") on node \"ip-10-0-128-111.ec2.internal\" DevicePath \"\""
Sep 16 02:21:05.702090 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:21:05.701829    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2fbnw\" (UniqueName: \"kubernetes.io/projected/5bf4691b-1501-4d65-9538-52851f45d954-kube-api-access-2fbnw\") on node \"ip-10-0-128-111.ec2.internal\" DevicePath \"\""
Sep 16 02:21:05.702090 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:21:05.701837    2583 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/5bf4691b-1501-4d65-9538-52851f45d954-config-out\") on node \"ip-10-0-128-111.ec2.internal\" DevicePath \"\""
Sep 16 02:21:05.702090 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:21:05.701845    2583 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/5bf4691b-1501-4d65-9538-52851f45d954-secret-alertmanager-kube-rbac-proxy\") on node \"ip-10-0-128-111.ec2.internal\" DevicePath \"\""
Sep 16 02:21:05.702090 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:21:05.701855    2583 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/5bf4691b-1501-4d65-9538-52851f45d954-secret-alertmanager-kube-rbac-proxy-metric\") on node \"ip-10-0-128-111.ec2.internal\" DevicePath \"\""
Sep 16 02:21:05.702090 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:21:05.701863    2583 reconciler_common.go:299] "Volume detached for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5bf4691b-1501-4d65-9538-52851f45d954-alertmanager-trusted-ca-bundle\") on node \"ip-10-0-128-111.ec2.internal\" DevicePath \"\""
Sep 16 02:21:05.817060 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:21:05.817022    2583 generic.go:356] "Generic (PLEG): container finished" podID="5bf4691b-1501-4d65-9538-52851f45d954" containerID="f34d032e688ef59537b80645fb4184c9c133350be5f8c68a1515659d0bc1e472" exitCode=0
Sep 16 02:21:05.817060 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:21:05.817053    2583 generic.go:356] "Generic (PLEG): container finished" podID="5bf4691b-1501-4d65-9538-52851f45d954" containerID="63d9db423d59c98d2fce3f3c7335605fa71160e25015a75a29498b296f453caa" exitCode=0
Sep 16 02:21:05.817252 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:21:05.817103    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"5bf4691b-1501-4d65-9538-52851f45d954","Type":"ContainerDied","Data":"f34d032e688ef59537b80645fb4184c9c133350be5f8c68a1515659d0bc1e472"}
Sep 16 02:21:05.817252 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:21:05.817167    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"5bf4691b-1501-4d65-9538-52851f45d954","Type":"ContainerDied","Data":"63d9db423d59c98d2fce3f3c7335605fa71160e25015a75a29498b296f453caa"}
Sep 16 02:21:05.817252 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:21:05.817115    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Sep 16 02:21:05.817252 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:21:05.817195    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"5bf4691b-1501-4d65-9538-52851f45d954","Type":"ContainerDied","Data":"ab42e31d318efbc55abf8700c15fbd4e31a7752a1056ce0ad46025b7124024b7"}
Sep 16 02:21:05.817252 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:21:05.817240    2583 scope.go:117] "RemoveContainer" containerID="582f429a3fbde7376ca575553aafc85b416a57258ae0bc5f949f128e41dfcf64"
Sep 16 02:21:05.828606 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:21:05.828585    2583 scope.go:117] "RemoveContainer" containerID="f34d032e688ef59537b80645fb4184c9c133350be5f8c68a1515659d0bc1e472"
Sep 16 02:21:05.835756 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:21:05.835740    2583 scope.go:117] "RemoveContainer" containerID="79420c355c9bdc0f2968e2c0d3d22e0c0651599f0f94a0732efe01bc15381592"
Sep 16 02:21:05.842862 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:21:05.842836    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 16 02:21:05.843466 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:21:05.843441    2583 scope.go:117] "RemoveContainer" containerID="63d9db423d59c98d2fce3f3c7335605fa71160e25015a75a29498b296f453caa"
Sep 16 02:21:05.846511 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:21:05.846490    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 16 02:21:05.850723 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:21:05.850703    2583 scope.go:117] "RemoveContainer" containerID="30eaae2482b7efd3dc0aadf8fddf4ed760b07e76b2aae9bc6752104c72cc2b52"
Sep 16 02:21:05.857489 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:21:05.857471    2583 scope.go:117] "RemoveContainer" containerID="8375df7a74ca45d311dc292b23d9289ff7bca198e49b7dd4251a1636003e07e7"
Sep 16 02:21:05.864500 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:21:05.864484    2583 scope.go:117] "RemoveContainer" containerID="3d32533969f138ea53bd203a1293d9a78c198026d7dfc791758590a05e1aa896"
Sep 16 02:21:05.872911 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:21:05.872891    2583 scope.go:117] "RemoveContainer" containerID="582f429a3fbde7376ca575553aafc85b416a57258ae0bc5f949f128e41dfcf64"
Sep 16 02:21:05.873477 ip-10-0-128-111 kubenswrapper[2583]: E0916 02:21:05.873448    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"582f429a3fbde7376ca575553aafc85b416a57258ae0bc5f949f128e41dfcf64\": container with ID starting with 582f429a3fbde7376ca575553aafc85b416a57258ae0bc5f949f128e41dfcf64 not found: ID does not exist" containerID="582f429a3fbde7376ca575553aafc85b416a57258ae0bc5f949f128e41dfcf64"
Sep 16 02:21:05.873590 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:21:05.873488    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"582f429a3fbde7376ca575553aafc85b416a57258ae0bc5f949f128e41dfcf64"} err="failed to get container status \"582f429a3fbde7376ca575553aafc85b416a57258ae0bc5f949f128e41dfcf64\": rpc error: code = NotFound desc = could not find container \"582f429a3fbde7376ca575553aafc85b416a57258ae0bc5f949f128e41dfcf64\": container with ID starting with 582f429a3fbde7376ca575553aafc85b416a57258ae0bc5f949f128e41dfcf64 not found: ID does not exist"
Sep 16 02:21:05.873590 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:21:05.873534    2583 scope.go:117] "RemoveContainer" containerID="f34d032e688ef59537b80645fb4184c9c133350be5f8c68a1515659d0bc1e472"
Sep 16 02:21:05.873872 ip-10-0-128-111 kubenswrapper[2583]: E0916 02:21:05.873855    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f34d032e688ef59537b80645fb4184c9c133350be5f8c68a1515659d0bc1e472\": container with ID starting with f34d032e688ef59537b80645fb4184c9c133350be5f8c68a1515659d0bc1e472 not found: ID does not exist" containerID="f34d032e688ef59537b80645fb4184c9c133350be5f8c68a1515659d0bc1e472"
Sep 16 02:21:05.873918 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:21:05.873876    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f34d032e688ef59537b80645fb4184c9c133350be5f8c68a1515659d0bc1e472"} err="failed to get container status \"f34d032e688ef59537b80645fb4184c9c133350be5f8c68a1515659d0bc1e472\": rpc error: code = NotFound desc = could not find container \"f34d032e688ef59537b80645fb4184c9c133350be5f8c68a1515659d0bc1e472\": container with ID starting with f34d032e688ef59537b80645fb4184c9c133350be5f8c68a1515659d0bc1e472 not found: ID does not exist"
Sep 16 02:21:05.873918 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:21:05.873890    2583 scope.go:117] "RemoveContainer" containerID="79420c355c9bdc0f2968e2c0d3d22e0c0651599f0f94a0732efe01bc15381592"
Sep 16 02:21:05.874121 ip-10-0-128-111 kubenswrapper[2583]: E0916 02:21:05.874106    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"79420c355c9bdc0f2968e2c0d3d22e0c0651599f0f94a0732efe01bc15381592\": container with ID starting with 79420c355c9bdc0f2968e2c0d3d22e0c0651599f0f94a0732efe01bc15381592 not found: ID does not exist" containerID="79420c355c9bdc0f2968e2c0d3d22e0c0651599f0f94a0732efe01bc15381592"
Sep 16 02:21:05.874164 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:21:05.874124    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"79420c355c9bdc0f2968e2c0d3d22e0c0651599f0f94a0732efe01bc15381592"} err="failed to get container status \"79420c355c9bdc0f2968e2c0d3d22e0c0651599f0f94a0732efe01bc15381592\": rpc error: code = NotFound desc = could not find container \"79420c355c9bdc0f2968e2c0d3d22e0c0651599f0f94a0732efe01bc15381592\": container with ID starting with 79420c355c9bdc0f2968e2c0d3d22e0c0651599f0f94a0732efe01bc15381592 not found: ID does not exist"
Sep 16 02:21:05.874164 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:21:05.874136    2583 scope.go:117] "RemoveContainer" containerID="63d9db423d59c98d2fce3f3c7335605fa71160e25015a75a29498b296f453caa"
Sep 16 02:21:05.874338 ip-10-0-128-111 kubenswrapper[2583]: E0916 02:21:05.874322    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"63d9db423d59c98d2fce3f3c7335605fa71160e25015a75a29498b296f453caa\": container with ID starting with 63d9db423d59c98d2fce3f3c7335605fa71160e25015a75a29498b296f453caa not found: ID does not exist" containerID="63d9db423d59c98d2fce3f3c7335605fa71160e25015a75a29498b296f453caa"
Sep 16 02:21:05.874384 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:21:05.874340    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"63d9db423d59c98d2fce3f3c7335605fa71160e25015a75a29498b296f453caa"} err="failed to get container status \"63d9db423d59c98d2fce3f3c7335605fa71160e25015a75a29498b296f453caa\": rpc error: code = NotFound desc = could not find container \"63d9db423d59c98d2fce3f3c7335605fa71160e25015a75a29498b296f453caa\": container with ID starting with 63d9db423d59c98d2fce3f3c7335605fa71160e25015a75a29498b296f453caa not found: ID does not exist"
Sep 16 02:21:05.874384 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:21:05.874353    2583 scope.go:117] "RemoveContainer" containerID="30eaae2482b7efd3dc0aadf8fddf4ed760b07e76b2aae9bc6752104c72cc2b52"
Sep 16 02:21:05.874590 ip-10-0-128-111 kubenswrapper[2583]: E0916 02:21:05.874573    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"30eaae2482b7efd3dc0aadf8fddf4ed760b07e76b2aae9bc6752104c72cc2b52\": container with ID starting with 30eaae2482b7efd3dc0aadf8fddf4ed760b07e76b2aae9bc6752104c72cc2b52 not found: ID does not exist" containerID="30eaae2482b7efd3dc0aadf8fddf4ed760b07e76b2aae9bc6752104c72cc2b52"
Sep 16 02:21:05.874669 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:21:05.874591    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"30eaae2482b7efd3dc0aadf8fddf4ed760b07e76b2aae9bc6752104c72cc2b52"} err="failed to get container status \"30eaae2482b7efd3dc0aadf8fddf4ed760b07e76b2aae9bc6752104c72cc2b52\": rpc error: code = NotFound desc = could not find container \"30eaae2482b7efd3dc0aadf8fddf4ed760b07e76b2aae9bc6752104c72cc2b52\": container with ID starting with 30eaae2482b7efd3dc0aadf8fddf4ed760b07e76b2aae9bc6752104c72cc2b52 not found: ID does not exist"
Sep 16 02:21:05.874669 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:21:05.874602    2583 scope.go:117] "RemoveContainer" containerID="8375df7a74ca45d311dc292b23d9289ff7bca198e49b7dd4251a1636003e07e7"
Sep 16 02:21:05.874829 ip-10-0-128-111 kubenswrapper[2583]: E0916 02:21:05.874812    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8375df7a74ca45d311dc292b23d9289ff7bca198e49b7dd4251a1636003e07e7\": container with ID starting with 8375df7a74ca45d311dc292b23d9289ff7bca198e49b7dd4251a1636003e07e7 not found: ID does not exist" containerID="8375df7a74ca45d311dc292b23d9289ff7bca198e49b7dd4251a1636003e07e7"
Sep 16 02:21:05.874882 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:21:05.874832    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8375df7a74ca45d311dc292b23d9289ff7bca198e49b7dd4251a1636003e07e7"} err="failed to get container status \"8375df7a74ca45d311dc292b23d9289ff7bca198e49b7dd4251a1636003e07e7\": rpc error: code = NotFound desc = could not find container \"8375df7a74ca45d311dc292b23d9289ff7bca198e49b7dd4251a1636003e07e7\": container with ID starting with 8375df7a74ca45d311dc292b23d9289ff7bca198e49b7dd4251a1636003e07e7 not found: ID does not exist"
Sep 16 02:21:05.874882 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:21:05.874845    2583 scope.go:117] "RemoveContainer" containerID="3d32533969f138ea53bd203a1293d9a78c198026d7dfc791758590a05e1aa896"
Sep 16 02:21:05.875086 ip-10-0-128-111 kubenswrapper[2583]: E0916 02:21:05.875059    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3d32533969f138ea53bd203a1293d9a78c198026d7dfc791758590a05e1aa896\": container with ID starting with 3d32533969f138ea53bd203a1293d9a78c198026d7dfc791758590a05e1aa896 not found: ID does not exist" containerID="3d32533969f138ea53bd203a1293d9a78c198026d7dfc791758590a05e1aa896"
Sep 16 02:21:05.875130 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:21:05.875084    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3d32533969f138ea53bd203a1293d9a78c198026d7dfc791758590a05e1aa896"} err="failed to get container status \"3d32533969f138ea53bd203a1293d9a78c198026d7dfc791758590a05e1aa896\": rpc error: code = NotFound desc = could not find container \"3d32533969f138ea53bd203a1293d9a78c198026d7dfc791758590a05e1aa896\": container with ID starting with 3d32533969f138ea53bd203a1293d9a78c198026d7dfc791758590a05e1aa896 not found: ID does not exist"
Sep 16 02:21:05.875130 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:21:05.875101    2583 scope.go:117] "RemoveContainer" containerID="582f429a3fbde7376ca575553aafc85b416a57258ae0bc5f949f128e41dfcf64"
Sep 16 02:21:05.875341 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:21:05.875320    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"582f429a3fbde7376ca575553aafc85b416a57258ae0bc5f949f128e41dfcf64"} err="failed to get container status \"582f429a3fbde7376ca575553aafc85b416a57258ae0bc5f949f128e41dfcf64\": rpc error: code = NotFound desc = could not find container \"582f429a3fbde7376ca575553aafc85b416a57258ae0bc5f949f128e41dfcf64\": container with ID starting with 582f429a3fbde7376ca575553aafc85b416a57258ae0bc5f949f128e41dfcf64 not found: ID does not exist"
Sep 16 02:21:05.875419 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:21:05.875342    2583 scope.go:117] "RemoveContainer" containerID="f34d032e688ef59537b80645fb4184c9c133350be5f8c68a1515659d0bc1e472"
Sep 16 02:21:05.875608 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:21:05.875592    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f34d032e688ef59537b80645fb4184c9c133350be5f8c68a1515659d0bc1e472"} err="failed to get container status \"f34d032e688ef59537b80645fb4184c9c133350be5f8c68a1515659d0bc1e472\": rpc error: code = NotFound desc = could not find container \"f34d032e688ef59537b80645fb4184c9c133350be5f8c68a1515659d0bc1e472\": container with ID starting with f34d032e688ef59537b80645fb4184c9c133350be5f8c68a1515659d0bc1e472 not found: ID does not exist"
Sep 16 02:21:05.875658 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:21:05.875609    2583 scope.go:117] "RemoveContainer" containerID="79420c355c9bdc0f2968e2c0d3d22e0c0651599f0f94a0732efe01bc15381592"
Sep 16 02:21:05.875824 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:21:05.875807    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"79420c355c9bdc0f2968e2c0d3d22e0c0651599f0f94a0732efe01bc15381592"} err="failed to get container status \"79420c355c9bdc0f2968e2c0d3d22e0c0651599f0f94a0732efe01bc15381592\": rpc error: code = NotFound desc = could not find container \"79420c355c9bdc0f2968e2c0d3d22e0c0651599f0f94a0732efe01bc15381592\": container with ID starting with 79420c355c9bdc0f2968e2c0d3d22e0c0651599f0f94a0732efe01bc15381592 not found: ID does not exist"
Sep 16 02:21:05.875889 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:21:05.875826    2583 scope.go:117] "RemoveContainer" containerID="63d9db423d59c98d2fce3f3c7335605fa71160e25015a75a29498b296f453caa"
Sep 16 02:21:05.876040 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:21:05.876022    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"63d9db423d59c98d2fce3f3c7335605fa71160e25015a75a29498b296f453caa"} err="failed to get container status \"63d9db423d59c98d2fce3f3c7335605fa71160e25015a75a29498b296f453caa\": rpc error: code = NotFound desc = could not find container \"63d9db423d59c98d2fce3f3c7335605fa71160e25015a75a29498b296f453caa\": container with ID starting with 63d9db423d59c98d2fce3f3c7335605fa71160e25015a75a29498b296f453caa not found: ID does not exist"
Sep 16 02:21:05.876081 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:21:05.876041    2583 scope.go:117] "RemoveContainer" containerID="30eaae2482b7efd3dc0aadf8fddf4ed760b07e76b2aae9bc6752104c72cc2b52"
Sep 16 02:21:05.876275 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:21:05.876258    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"30eaae2482b7efd3dc0aadf8fddf4ed760b07e76b2aae9bc6752104c72cc2b52"} err="failed to get container status \"30eaae2482b7efd3dc0aadf8fddf4ed760b07e76b2aae9bc6752104c72cc2b52\": rpc error: code = NotFound desc = could not find container \"30eaae2482b7efd3dc0aadf8fddf4ed760b07e76b2aae9bc6752104c72cc2b52\": container with ID starting with 30eaae2482b7efd3dc0aadf8fddf4ed760b07e76b2aae9bc6752104c72cc2b52 not found: ID does not exist"
Sep 16 02:21:05.876376 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:21:05.876276    2583 scope.go:117] "RemoveContainer" containerID="8375df7a74ca45d311dc292b23d9289ff7bca198e49b7dd4251a1636003e07e7"
Sep 16 02:21:05.876608 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:21:05.876587    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8375df7a74ca45d311dc292b23d9289ff7bca198e49b7dd4251a1636003e07e7"} err="failed to get container status \"8375df7a74ca45d311dc292b23d9289ff7bca198e49b7dd4251a1636003e07e7\": rpc error: code = NotFound desc = could not find container \"8375df7a74ca45d311dc292b23d9289ff7bca198e49b7dd4251a1636003e07e7\": container with ID starting with 8375df7a74ca45d311dc292b23d9289ff7bca198e49b7dd4251a1636003e07e7 not found: ID does not exist"
Sep 16 02:21:05.876691 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:21:05.876609    2583 scope.go:117] "RemoveContainer" containerID="3d32533969f138ea53bd203a1293d9a78c198026d7dfc791758590a05e1aa896"
Sep 16 02:21:05.876888 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:21:05.876870    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3d32533969f138ea53bd203a1293d9a78c198026d7dfc791758590a05e1aa896"} err="failed to get container status \"3d32533969f138ea53bd203a1293d9a78c198026d7dfc791758590a05e1aa896\": rpc error: code = NotFound desc = could not find container \"3d32533969f138ea53bd203a1293d9a78c198026d7dfc791758590a05e1aa896\": container with ID starting with 3d32533969f138ea53bd203a1293d9a78c198026d7dfc791758590a05e1aa896 not found: ID does not exist"
Sep 16 02:21:06.044119 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:21:06.044086    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5bf4691b-1501-4d65-9538-52851f45d954" path="/var/lib/kubelet/pods/5bf4691b-1501-4d65-9538-52851f45d954/volumes"
Sep 16 02:21:08.477223 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:21:08.477189    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 16 02:21:08.477661 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:21:08.477484    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="9cc00559-db17-4390-b926-cb912b947e7b" containerName="prometheus" containerID="cri-o://84b39eb0e85e4cb68e8c22ec7d5ebd8349797f955edd9fa2ad7cc3835eaa4ef3" gracePeriod=600
Sep 16 02:21:08.477661 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:21:08.477514    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="9cc00559-db17-4390-b926-cb912b947e7b" containerName="kube-rbac-proxy" containerID="cri-o://b2da4cf4a935141217a1262560b078b359b0162860efdb9647353030e0090590" gracePeriod=600
Sep 16 02:21:08.477661 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:21:08.477572    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="9cc00559-db17-4390-b926-cb912b947e7b" containerName="kube-rbac-proxy-thanos" containerID="cri-o://dc01d49bd843fdcdfc3fe05ac1ebd0d33a9e94874df1f3df550cc51825f83c01" gracePeriod=600
Sep 16 02:21:08.477661 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:21:08.477600    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="9cc00559-db17-4390-b926-cb912b947e7b" containerName="config-reloader" containerID="cri-o://36cb8aec3dcb4f882b28557545eb9fd3890739e6141f4e49482d18cadcac7677" gracePeriod=600
Sep 16 02:21:08.477661 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:21:08.477585    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="9cc00559-db17-4390-b926-cb912b947e7b" containerName="thanos-sidecar" containerID="cri-o://1421871f4efae41cca1f0a74619eb934782244bf8a909df0351c27932b0b4150" gracePeriod=600
Sep 16 02:21:08.477921 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:21:08.477576    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="9cc00559-db17-4390-b926-cb912b947e7b" containerName="kube-rbac-proxy-web" containerID="cri-o://501ee12b90e989972c892e33d1b1754695561add0dd08622d66b99b6e2d38f54" gracePeriod=600
Sep 16 02:21:08.833342 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:21:08.833244    2583 generic.go:356] "Generic (PLEG): container finished" podID="9cc00559-db17-4390-b926-cb912b947e7b" containerID="dc01d49bd843fdcdfc3fe05ac1ebd0d33a9e94874df1f3df550cc51825f83c01" exitCode=0
Sep 16 02:21:08.833342 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:21:08.833267    2583 generic.go:356] "Generic (PLEG): container finished" podID="9cc00559-db17-4390-b926-cb912b947e7b" containerID="b2da4cf4a935141217a1262560b078b359b0162860efdb9647353030e0090590" exitCode=0
Sep 16 02:21:08.833342 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:21:08.833278    2583 generic.go:356] "Generic (PLEG): container finished" podID="9cc00559-db17-4390-b926-cb912b947e7b" containerID="1421871f4efae41cca1f0a74619eb934782244bf8a909df0351c27932b0b4150" exitCode=0
Sep 16 02:21:08.833342 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:21:08.833287    2583 generic.go:356] "Generic (PLEG): container finished" podID="9cc00559-db17-4390-b926-cb912b947e7b" containerID="36cb8aec3dcb4f882b28557545eb9fd3890739e6141f4e49482d18cadcac7677" exitCode=0
Sep 16 02:21:08.833342 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:21:08.833294    2583 generic.go:356] "Generic (PLEG): container finished" podID="9cc00559-db17-4390-b926-cb912b947e7b" containerID="84b39eb0e85e4cb68e8c22ec7d5ebd8349797f955edd9fa2ad7cc3835eaa4ef3" exitCode=0
Sep 16 02:21:08.833342 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:21:08.833322    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"9cc00559-db17-4390-b926-cb912b947e7b","Type":"ContainerDied","Data":"dc01d49bd843fdcdfc3fe05ac1ebd0d33a9e94874df1f3df550cc51825f83c01"}
Sep 16 02:21:08.833716 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:21:08.833405    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"9cc00559-db17-4390-b926-cb912b947e7b","Type":"ContainerDied","Data":"b2da4cf4a935141217a1262560b078b359b0162860efdb9647353030e0090590"}
Sep 16 02:21:08.833716 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:21:08.833443    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"9cc00559-db17-4390-b926-cb912b947e7b","Type":"ContainerDied","Data":"1421871f4efae41cca1f0a74619eb934782244bf8a909df0351c27932b0b4150"}
Sep 16 02:21:08.833716 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:21:08.833479    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"9cc00559-db17-4390-b926-cb912b947e7b","Type":"ContainerDied","Data":"36cb8aec3dcb4f882b28557545eb9fd3890739e6141f4e49482d18cadcac7677"}
Sep 16 02:21:08.833716 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:21:08.833515    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"9cc00559-db17-4390-b926-cb912b947e7b","Type":"ContainerDied","Data":"84b39eb0e85e4cb68e8c22ec7d5ebd8349797f955edd9fa2ad7cc3835eaa4ef3"}
Sep 16 02:21:09.721115 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:21:09.721091    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 02:21:09.732944 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:21:09.732923    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/9cc00559-db17-4390-b926-cb912b947e7b-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"9cc00559-db17-4390-b926-cb912b947e7b\" (UID: \"9cc00559-db17-4390-b926-cb912b947e7b\") "
Sep 16 02:21:09.733029 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:21:09.732952    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/9cc00559-db17-4390-b926-cb912b947e7b-prometheus-k8s-db\") pod \"9cc00559-db17-4390-b926-cb912b947e7b\" (UID: \"9cc00559-db17-4390-b926-cb912b947e7b\") "
Sep 16 02:21:09.733029 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:21:09.732974    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/9cc00559-db17-4390-b926-cb912b947e7b-configmap-metrics-client-ca\") pod \"9cc00559-db17-4390-b926-cb912b947e7b\" (UID: \"9cc00559-db17-4390-b926-cb912b947e7b\") "
Sep 16 02:21:09.733029 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:21:09.732995    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/9cc00559-db17-4390-b926-cb912b947e7b-secret-metrics-client-certs\") pod \"9cc00559-db17-4390-b926-cb912b947e7b\" (UID: \"9cc00559-db17-4390-b926-cb912b947e7b\") "
Sep 16 02:21:09.733029 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:21:09.733014    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/9cc00559-db17-4390-b926-cb912b947e7b-web-config\") pod \"9cc00559-db17-4390-b926-cb912b947e7b\" (UID: \"9cc00559-db17-4390-b926-cb912b947e7b\") "
Sep 16 02:21:09.733224 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:21:09.733039    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/9cc00559-db17-4390-b926-cb912b947e7b-config\") pod \"9cc00559-db17-4390-b926-cb912b947e7b\" (UID: \"9cc00559-db17-4390-b926-cb912b947e7b\") "
Sep 16 02:21:09.733224 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:21:09.733065    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/9cc00559-db17-4390-b926-cb912b947e7b-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"9cc00559-db17-4390-b926-cb912b947e7b\" (UID: \"9cc00559-db17-4390-b926-cb912b947e7b\") "
Sep 16 02:21:09.733224 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:21:09.733097    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/9cc00559-db17-4390-b926-cb912b947e7b-secret-prometheus-k8s-tls\") pod \"9cc00559-db17-4390-b926-cb912b947e7b\" (UID: \"9cc00559-db17-4390-b926-cb912b947e7b\") "
Sep 16 02:21:09.733224 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:21:09.733122    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/9cc00559-db17-4390-b926-cb912b947e7b-prometheus-k8s-rulefiles-1\") pod \"9cc00559-db17-4390-b926-cb912b947e7b\" (UID: \"9cc00559-db17-4390-b926-cb912b947e7b\") "
Sep 16 02:21:09.733224 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:21:09.733150    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/9cc00559-db17-4390-b926-cb912b947e7b-prometheus-k8s-rulefiles-0\") pod \"9cc00559-db17-4390-b926-cb912b947e7b\" (UID: \"9cc00559-db17-4390-b926-cb912b947e7b\") "
Sep 16 02:21:09.733224 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:21:09.733179    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/9cc00559-db17-4390-b926-cb912b947e7b-prometheus-k8s-rulefiles-2\") pod \"9cc00559-db17-4390-b926-cb912b947e7b\" (UID: \"9cc00559-db17-4390-b926-cb912b947e7b\") "
Sep 16 02:21:09.733224 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:21:09.733199    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9cc00559-db17-4390-b926-cb912b947e7b-configmap-serving-certs-ca-bundle\") pod \"9cc00559-db17-4390-b926-cb912b947e7b\" (UID: \"9cc00559-db17-4390-b926-cb912b947e7b\") "
Sep 16 02:21:09.733224 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:21:09.733222    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/9cc00559-db17-4390-b926-cb912b947e7b-secret-kube-rbac-proxy\") pod \"9cc00559-db17-4390-b926-cb912b947e7b\" (UID: \"9cc00559-db17-4390-b926-cb912b947e7b\") "
Sep 16 02:21:09.733866 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:21:09.733241    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/9cc00559-db17-4390-b926-cb912b947e7b-config-out\") pod \"9cc00559-db17-4390-b926-cb912b947e7b\" (UID: \"9cc00559-db17-4390-b926-cb912b947e7b\") "
Sep 16 02:21:09.733866 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:21:09.733269    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9jgx5\" (UniqueName: \"kubernetes.io/projected/9cc00559-db17-4390-b926-cb912b947e7b-kube-api-access-9jgx5\") pod \"9cc00559-db17-4390-b926-cb912b947e7b\" (UID: \"9cc00559-db17-4390-b926-cb912b947e7b\") "
Sep 16 02:21:09.733866 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:21:09.733297    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9cc00559-db17-4390-b926-cb912b947e7b-configmap-kubelet-serving-ca-bundle\") pod \"9cc00559-db17-4390-b926-cb912b947e7b\" (UID: \"9cc00559-db17-4390-b926-cb912b947e7b\") "
Sep 16 02:21:09.733866 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:21:09.733325    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/9cc00559-db17-4390-b926-cb912b947e7b-thanos-prometheus-http-client-file\") pod \"9cc00559-db17-4390-b926-cb912b947e7b\" (UID: \"9cc00559-db17-4390-b926-cb912b947e7b\") "
Sep 16 02:21:09.733866 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:21:09.733354    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9cc00559-db17-4390-b926-cb912b947e7b-prometheus-trusted-ca-bundle\") pod \"9cc00559-db17-4390-b926-cb912b947e7b\" (UID: \"9cc00559-db17-4390-b926-cb912b947e7b\") "
Sep 16 02:21:09.733866 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:21:09.733403    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/9cc00559-db17-4390-b926-cb912b947e7b-tls-assets\") pod \"9cc00559-db17-4390-b926-cb912b947e7b\" (UID: \"9cc00559-db17-4390-b926-cb912b947e7b\") "
Sep 16 02:21:09.733866 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:21:09.733436    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9cc00559-db17-4390-b926-cb912b947e7b-configmap-metrics-client-ca" (OuterVolumeSpecName: "configmap-metrics-client-ca") pod "9cc00559-db17-4390-b926-cb912b947e7b" (UID: "9cc00559-db17-4390-b926-cb912b947e7b"). InnerVolumeSpecName "configmap-metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 02:21:09.733866 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:21:09.733482    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/9cc00559-db17-4390-b926-cb912b947e7b-secret-grpc-tls\") pod \"9cc00559-db17-4390-b926-cb912b947e7b\" (UID: \"9cc00559-db17-4390-b926-cb912b947e7b\") "
Sep 16 02:21:09.733866 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:21:09.733706    2583 reconciler_common.go:299] "Volume detached for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/9cc00559-db17-4390-b926-cb912b947e7b-configmap-metrics-client-ca\") on node \"ip-10-0-128-111.ec2.internal\" DevicePath \"\""
Sep 16 02:21:09.734306 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:21:09.734145    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9cc00559-db17-4390-b926-cb912b947e7b-prometheus-k8s-db" (OuterVolumeSpecName: "prometheus-k8s-db") pod "9cc00559-db17-4390-b926-cb912b947e7b" (UID: "9cc00559-db17-4390-b926-cb912b947e7b"). InnerVolumeSpecName "prometheus-k8s-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 02:21:09.734609 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:21:09.734528    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9cc00559-db17-4390-b926-cb912b947e7b-configmap-kubelet-serving-ca-bundle" (OuterVolumeSpecName: "configmap-kubelet-serving-ca-bundle") pod "9cc00559-db17-4390-b926-cb912b947e7b" (UID: "9cc00559-db17-4390-b926-cb912b947e7b"). InnerVolumeSpecName "configmap-kubelet-serving-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 02:21:09.734962 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:21:09.734923    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9cc00559-db17-4390-b926-cb912b947e7b-prometheus-trusted-ca-bundle" (OuterVolumeSpecName: "prometheus-trusted-ca-bundle") pod "9cc00559-db17-4390-b926-cb912b947e7b" (UID: "9cc00559-db17-4390-b926-cb912b947e7b"). InnerVolumeSpecName "prometheus-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 02:21:09.736346 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:21:09.736263    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9cc00559-db17-4390-b926-cb912b947e7b-prometheus-k8s-rulefiles-1" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-1") pod "9cc00559-db17-4390-b926-cb912b947e7b" (UID: "9cc00559-db17-4390-b926-cb912b947e7b"). InnerVolumeSpecName "prometheus-k8s-rulefiles-1". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 02:21:09.736622 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:21:09.736594    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9cc00559-db17-4390-b926-cb912b947e7b-prometheus-k8s-rulefiles-2" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-2") pod "9cc00559-db17-4390-b926-cb912b947e7b" (UID: "9cc00559-db17-4390-b926-cb912b947e7b"). InnerVolumeSpecName "prometheus-k8s-rulefiles-2". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 02:21:09.736727 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:21:09.736618    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9cc00559-db17-4390-b926-cb912b947e7b-configmap-serving-certs-ca-bundle" (OuterVolumeSpecName: "configmap-serving-certs-ca-bundle") pod "9cc00559-db17-4390-b926-cb912b947e7b" (UID: "9cc00559-db17-4390-b926-cb912b947e7b"). InnerVolumeSpecName "configmap-serving-certs-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 02:21:09.736994 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:21:09.736969    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9cc00559-db17-4390-b926-cb912b947e7b-secret-prometheus-k8s-thanos-sidecar-tls" (OuterVolumeSpecName: "secret-prometheus-k8s-thanos-sidecar-tls") pod "9cc00559-db17-4390-b926-cb912b947e7b" (UID: "9cc00559-db17-4390-b926-cb912b947e7b"). InnerVolumeSpecName "secret-prometheus-k8s-thanos-sidecar-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 02:21:09.737175 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:21:09.737147    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9cc00559-db17-4390-b926-cb912b947e7b-secret-metrics-client-certs" (OuterVolumeSpecName: "secret-metrics-client-certs") pod "9cc00559-db17-4390-b926-cb912b947e7b" (UID: "9cc00559-db17-4390-b926-cb912b947e7b"). InnerVolumeSpecName "secret-metrics-client-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 02:21:09.737409 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:21:09.737383    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9cc00559-db17-4390-b926-cb912b947e7b-secret-kube-rbac-proxy" (OuterVolumeSpecName: "secret-kube-rbac-proxy") pod "9cc00559-db17-4390-b926-cb912b947e7b" (UID: "9cc00559-db17-4390-b926-cb912b947e7b"). InnerVolumeSpecName "secret-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 02:21:09.737570 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:21:09.737527    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9cc00559-db17-4390-b926-cb912b947e7b-secret-grpc-tls" (OuterVolumeSpecName: "secret-grpc-tls") pod "9cc00559-db17-4390-b926-cb912b947e7b" (UID: "9cc00559-db17-4390-b926-cb912b947e7b"). InnerVolumeSpecName "secret-grpc-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 02:21:09.737670 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:21:09.737647    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9cc00559-db17-4390-b926-cb912b947e7b-prometheus-k8s-rulefiles-0" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-0") pod "9cc00559-db17-4390-b926-cb912b947e7b" (UID: "9cc00559-db17-4390-b926-cb912b947e7b"). InnerVolumeSpecName "prometheus-k8s-rulefiles-0". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 02:21:09.738253 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:21:09.738224    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9cc00559-db17-4390-b926-cb912b947e7b-kube-api-access-9jgx5" (OuterVolumeSpecName: "kube-api-access-9jgx5") pod "9cc00559-db17-4390-b926-cb912b947e7b" (UID: "9cc00559-db17-4390-b926-cb912b947e7b"). InnerVolumeSpecName "kube-api-access-9jgx5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 02:21:09.738347 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:21:09.738331    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9cc00559-db17-4390-b926-cb912b947e7b-secret-prometheus-k8s-kube-rbac-proxy-web" (OuterVolumeSpecName: "secret-prometheus-k8s-kube-rbac-proxy-web") pod "9cc00559-db17-4390-b926-cb912b947e7b" (UID: "9cc00559-db17-4390-b926-cb912b947e7b"). InnerVolumeSpecName "secret-prometheus-k8s-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 02:21:09.738441 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:21:09.738412    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9cc00559-db17-4390-b926-cb912b947e7b-config" (OuterVolumeSpecName: "config") pod "9cc00559-db17-4390-b926-cb912b947e7b" (UID: "9cc00559-db17-4390-b926-cb912b947e7b"). InnerVolumeSpecName "config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 02:21:09.739004 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:21:09.738979    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9cc00559-db17-4390-b926-cb912b947e7b-secret-prometheus-k8s-tls" (OuterVolumeSpecName: "secret-prometheus-k8s-tls") pod "9cc00559-db17-4390-b926-cb912b947e7b" (UID: "9cc00559-db17-4390-b926-cb912b947e7b"). InnerVolumeSpecName "secret-prometheus-k8s-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 02:21:09.739107 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:21:09.739048    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9cc00559-db17-4390-b926-cb912b947e7b-config-out" (OuterVolumeSpecName: "config-out") pod "9cc00559-db17-4390-b926-cb912b947e7b" (UID: "9cc00559-db17-4390-b926-cb912b947e7b"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 02:21:09.739381 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:21:09.739359    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9cc00559-db17-4390-b926-cb912b947e7b-thanos-prometheus-http-client-file" (OuterVolumeSpecName: "thanos-prometheus-http-client-file") pod "9cc00559-db17-4390-b926-cb912b947e7b" (UID: "9cc00559-db17-4390-b926-cb912b947e7b"). InnerVolumeSpecName "thanos-prometheus-http-client-file". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 02:21:09.739674 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:21:09.739653    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9cc00559-db17-4390-b926-cb912b947e7b-tls-assets" (OuterVolumeSpecName: "tls-assets") pod "9cc00559-db17-4390-b926-cb912b947e7b" (UID: "9cc00559-db17-4390-b926-cb912b947e7b"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 02:21:09.748019 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:21:09.747984    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9cc00559-db17-4390-b926-cb912b947e7b-web-config" (OuterVolumeSpecName: "web-config") pod "9cc00559-db17-4390-b926-cb912b947e7b" (UID: "9cc00559-db17-4390-b926-cb912b947e7b"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 02:21:09.834519 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:21:09.834439    2583 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/9cc00559-db17-4390-b926-cb912b947e7b-prometheus-k8s-rulefiles-0\") on node \"ip-10-0-128-111.ec2.internal\" DevicePath \"\""
Sep 16 02:21:09.834519 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:21:09.834470    2583 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/9cc00559-db17-4390-b926-cb912b947e7b-prometheus-k8s-rulefiles-2\") on node \"ip-10-0-128-111.ec2.internal\" DevicePath \"\""
Sep 16 02:21:09.834519 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:21:09.834486    2583 reconciler_common.go:299] "Volume detached for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9cc00559-db17-4390-b926-cb912b947e7b-configmap-serving-certs-ca-bundle\") on node \"ip-10-0-128-111.ec2.internal\" DevicePath \"\""
Sep 16 02:21:09.834519 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:21:09.834496    2583 reconciler_common.go:299] "Volume detached for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/9cc00559-db17-4390-b926-cb912b947e7b-secret-kube-rbac-proxy\") on node \"ip-10-0-128-111.ec2.internal\" DevicePath \"\""
Sep 16 02:21:09.834519 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:21:09.834508    2583 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/9cc00559-db17-4390-b926-cb912b947e7b-config-out\") on node \"ip-10-0-128-111.ec2.internal\" DevicePath \"\""
Sep 16 02:21:09.834519 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:21:09.834522    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9jgx5\" (UniqueName: \"kubernetes.io/projected/9cc00559-db17-4390-b926-cb912b947e7b-kube-api-access-9jgx5\") on node \"ip-10-0-128-111.ec2.internal\" DevicePath \"\""
Sep 16 02:21:09.834868 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:21:09.834531    2583 reconciler_common.go:299] "Volume detached for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9cc00559-db17-4390-b926-cb912b947e7b-configmap-kubelet-serving-ca-bundle\") on node \"ip-10-0-128-111.ec2.internal\" DevicePath \"\""
Sep 16 02:21:09.834868 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:21:09.834540    2583 reconciler_common.go:299] "Volume detached for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/9cc00559-db17-4390-b926-cb912b947e7b-thanos-prometheus-http-client-file\") on node \"ip-10-0-128-111.ec2.internal\" DevicePath \"\""
Sep 16 02:21:09.834868 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:21:09.834568    2583 reconciler_common.go:299] "Volume detached for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9cc00559-db17-4390-b926-cb912b947e7b-prometheus-trusted-ca-bundle\") on node \"ip-10-0-128-111.ec2.internal\" DevicePath \"\""
Sep 16 02:21:09.834868 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:21:09.834579    2583 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/9cc00559-db17-4390-b926-cb912b947e7b-tls-assets\") on node \"ip-10-0-128-111.ec2.internal\" DevicePath \"\""
Sep 16 02:21:09.834868 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:21:09.834592    2583 reconciler_common.go:299] "Volume detached for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/9cc00559-db17-4390-b926-cb912b947e7b-secret-grpc-tls\") on node \"ip-10-0-128-111.ec2.internal\" DevicePath \"\""
Sep 16 02:21:09.834868 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:21:09.834602    2583 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/9cc00559-db17-4390-b926-cb912b947e7b-secret-prometheus-k8s-thanos-sidecar-tls\") on node \"ip-10-0-128-111.ec2.internal\" DevicePath \"\""
Sep 16 02:21:09.834868 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:21:09.834611    2583 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/9cc00559-db17-4390-b926-cb912b947e7b-prometheus-k8s-db\") on node \"ip-10-0-128-111.ec2.internal\" DevicePath \"\""
Sep 16 02:21:09.834868 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:21:09.834619    2583 reconciler_common.go:299] "Volume detached for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/9cc00559-db17-4390-b926-cb912b947e7b-secret-metrics-client-certs\") on node \"ip-10-0-128-111.ec2.internal\" DevicePath \"\""
Sep 16 02:21:09.834868 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:21:09.834648    2583 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/9cc00559-db17-4390-b926-cb912b947e7b-web-config\") on node \"ip-10-0-128-111.ec2.internal\" DevicePath \"\""
Sep 16 02:21:09.834868 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:21:09.834656    2583 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/secret/9cc00559-db17-4390-b926-cb912b947e7b-config\") on node \"ip-10-0-128-111.ec2.internal\" DevicePath \"\""
Sep 16 02:21:09.834868 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:21:09.834664    2583 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/9cc00559-db17-4390-b926-cb912b947e7b-secret-prometheus-k8s-kube-rbac-proxy-web\") on node \"ip-10-0-128-111.ec2.internal\" DevicePath \"\""
Sep 16 02:21:09.834868 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:21:09.834672    2583 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/9cc00559-db17-4390-b926-cb912b947e7b-secret-prometheus-k8s-tls\") on node \"ip-10-0-128-111.ec2.internal\" DevicePath \"\""
Sep 16 02:21:09.834868 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:21:09.834680    2583 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/9cc00559-db17-4390-b926-cb912b947e7b-prometheus-k8s-rulefiles-1\") on node \"ip-10-0-128-111.ec2.internal\" DevicePath \"\""
Sep 16 02:21:09.840858 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:21:09.840825    2583 generic.go:356] "Generic (PLEG): container finished" podID="9cc00559-db17-4390-b926-cb912b947e7b" containerID="501ee12b90e989972c892e33d1b1754695561add0dd08622d66b99b6e2d38f54" exitCode=0
Sep 16 02:21:09.840993 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:21:09.840898    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"9cc00559-db17-4390-b926-cb912b947e7b","Type":"ContainerDied","Data":"501ee12b90e989972c892e33d1b1754695561add0dd08622d66b99b6e2d38f54"}
Sep 16 02:21:09.840993 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:21:09.840911    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 02:21:09.841076 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:21:09.840997    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"9cc00559-db17-4390-b926-cb912b947e7b","Type":"ContainerDied","Data":"142fe8da9aa07e1b39f8e33d1a4de7860fa4dc52e215793fce1ffcd9c611f540"}
Sep 16 02:21:09.841076 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:21:09.841030    2583 scope.go:117] "RemoveContainer" containerID="dc01d49bd843fdcdfc3fe05ac1ebd0d33a9e94874df1f3df550cc51825f83c01"
Sep 16 02:21:09.848613 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:21:09.848596    2583 scope.go:117] "RemoveContainer" containerID="b2da4cf4a935141217a1262560b078b359b0162860efdb9647353030e0090590"
Sep 16 02:21:09.855705 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:21:09.855691    2583 scope.go:117] "RemoveContainer" containerID="501ee12b90e989972c892e33d1b1754695561add0dd08622d66b99b6e2d38f54"
Sep 16 02:21:09.862562 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:21:09.862527    2583 scope.go:117] "RemoveContainer" containerID="1421871f4efae41cca1f0a74619eb934782244bf8a909df0351c27932b0b4150"
Sep 16 02:21:09.863389 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:21:09.863371    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 16 02:21:09.867038 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:21:09.867017    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 16 02:21:09.869575 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:21:09.869544    2583 scope.go:117] "RemoveContainer" containerID="36cb8aec3dcb4f882b28557545eb9fd3890739e6141f4e49482d18cadcac7677"
Sep 16 02:21:09.876121 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:21:09.876097    2583 scope.go:117] "RemoveContainer" containerID="84b39eb0e85e4cb68e8c22ec7d5ebd8349797f955edd9fa2ad7cc3835eaa4ef3"
Sep 16 02:21:09.883266 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:21:09.883241    2583 scope.go:117] "RemoveContainer" containerID="335fb22f91f7fbe2f73e01a471295c3453e09ef84d66a4d843daa5af0f1cba2d"
Sep 16 02:21:09.889764 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:21:09.889656    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 16 02:21:09.889819 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:21:09.889805    2583 scope.go:117] "RemoveContainer" containerID="dc01d49bd843fdcdfc3fe05ac1ebd0d33a9e94874df1f3df550cc51825f83c01"
Sep 16 02:21:09.889906 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:21:09.889894    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9cc00559-db17-4390-b926-cb912b947e7b" containerName="init-config-reloader"
Sep 16 02:21:09.889969 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:21:09.889908    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="9cc00559-db17-4390-b926-cb912b947e7b" containerName="init-config-reloader"
Sep 16 02:21:09.889969 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:21:09.889921    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5bf4691b-1501-4d65-9538-52851f45d954" containerName="config-reloader"
Sep 16 02:21:09.889969 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:21:09.889929    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="5bf4691b-1501-4d65-9538-52851f45d954" containerName="config-reloader"
Sep 16 02:21:09.889969 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:21:09.889943    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9cc00559-db17-4390-b926-cb912b947e7b" containerName="prometheus"
Sep 16 02:21:09.889969 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:21:09.889951    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="9cc00559-db17-4390-b926-cb912b947e7b" containerName="prometheus"
Sep 16 02:21:09.889969 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:21:09.889962    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9cc00559-db17-4390-b926-cb912b947e7b" containerName="config-reloader"
Sep 16 02:21:09.889969 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:21:09.889969    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="9cc00559-db17-4390-b926-cb912b947e7b" containerName="config-reloader"
Sep 16 02:21:09.890206 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:21:09.889979    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5bf4691b-1501-4d65-9538-52851f45d954" containerName="alertmanager"
Sep 16 02:21:09.890206 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:21:09.889984    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="5bf4691b-1501-4d65-9538-52851f45d954" containerName="alertmanager"
Sep 16 02:21:09.890206 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:21:09.889991    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5bf4691b-1501-4d65-9538-52851f45d954" containerName="prom-label-proxy"
Sep 16 02:21:09.890206 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:21:09.889996    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="5bf4691b-1501-4d65-9538-52851f45d954" containerName="prom-label-proxy"
Sep 16 02:21:09.890206 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:21:09.890006    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9cc00559-db17-4390-b926-cb912b947e7b" containerName="kube-rbac-proxy-web"
Sep 16 02:21:09.890206 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:21:09.890013    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="9cc00559-db17-4390-b926-cb912b947e7b" containerName="kube-rbac-proxy-web"
Sep 16 02:21:09.890206 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:21:09.890026    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9cc00559-db17-4390-b926-cb912b947e7b" containerName="kube-rbac-proxy-thanos"
Sep 16 02:21:09.890206 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:21:09.890033    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="9cc00559-db17-4390-b926-cb912b947e7b" containerName="kube-rbac-proxy-thanos"
Sep 16 02:21:09.890206 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:21:09.890047    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5bf4691b-1501-4d65-9538-52851f45d954" containerName="kube-rbac-proxy-web"
Sep 16 02:21:09.890206 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:21:09.890053    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="5bf4691b-1501-4d65-9538-52851f45d954" containerName="kube-rbac-proxy-web"
Sep 16 02:21:09.890206 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:21:09.890060    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9cc00559-db17-4390-b926-cb912b947e7b" containerName="kube-rbac-proxy"
Sep 16 02:21:09.890206 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:21:09.890065    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="9cc00559-db17-4390-b926-cb912b947e7b" containerName="kube-rbac-proxy"
Sep 16 02:21:09.890206 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:21:09.890071    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5bf4691b-1501-4d65-9538-52851f45d954" containerName="kube-rbac-proxy-metric"
Sep 16 02:21:09.890206 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:21:09.890078    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="5bf4691b-1501-4d65-9538-52851f45d954" containerName="kube-rbac-proxy-metric"
Sep 16 02:21:09.890206 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:21:09.890085    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5bf4691b-1501-4d65-9538-52851f45d954" containerName="kube-rbac-proxy"
Sep 16 02:21:09.890206 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:21:09.890090    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="5bf4691b-1501-4d65-9538-52851f45d954" containerName="kube-rbac-proxy"
Sep 16 02:21:09.890206 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:21:09.890095    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9cc00559-db17-4390-b926-cb912b947e7b" containerName="thanos-sidecar"
Sep 16 02:21:09.890206 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:21:09.890100    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="9cc00559-db17-4390-b926-cb912b947e7b" containerName="thanos-sidecar"
Sep 16 02:21:09.890206 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:21:09.890106    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5bf4691b-1501-4d65-9538-52851f45d954" containerName="init-config-reloader"
Sep 16 02:21:09.890206 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:21:09.890111    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="5bf4691b-1501-4d65-9538-52851f45d954" containerName="init-config-reloader"
Sep 16 02:21:09.890206 ip-10-0-128-111 kubenswrapper[2583]: E0916 02:21:09.890109    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dc01d49bd843fdcdfc3fe05ac1ebd0d33a9e94874df1f3df550cc51825f83c01\": container with ID starting with dc01d49bd843fdcdfc3fe05ac1ebd0d33a9e94874df1f3df550cc51825f83c01 not found: ID does not exist" containerID="dc01d49bd843fdcdfc3fe05ac1ebd0d33a9e94874df1f3df550cc51825f83c01"
Sep 16 02:21:09.890206 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:21:09.890133    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dc01d49bd843fdcdfc3fe05ac1ebd0d33a9e94874df1f3df550cc51825f83c01"} err="failed to get container status \"dc01d49bd843fdcdfc3fe05ac1ebd0d33a9e94874df1f3df550cc51825f83c01\": rpc error: code = NotFound desc = could not find container \"dc01d49bd843fdcdfc3fe05ac1ebd0d33a9e94874df1f3df550cc51825f83c01\": container with ID starting with dc01d49bd843fdcdfc3fe05ac1ebd0d33a9e94874df1f3df550cc51825f83c01 not found: ID does not exist"
Sep 16 02:21:09.890206 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:21:09.890149    2583 scope.go:117] "RemoveContainer" containerID="b2da4cf4a935141217a1262560b078b359b0162860efdb9647353030e0090590"
Sep 16 02:21:09.890206 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:21:09.890152    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="9cc00559-db17-4390-b926-cb912b947e7b" containerName="thanos-sidecar"
Sep 16 02:21:09.890206 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:21:09.890198    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="9cc00559-db17-4390-b926-cb912b947e7b" containerName="prometheus"
Sep 16 02:21:09.890206 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:21:09.890206    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="5bf4691b-1501-4d65-9538-52851f45d954" containerName="alertmanager"
Sep 16 02:21:09.890206 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:21:09.890212    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="5bf4691b-1501-4d65-9538-52851f45d954" containerName="kube-rbac-proxy"
Sep 16 02:21:09.890206 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:21:09.890219    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="5bf4691b-1501-4d65-9538-52851f45d954" containerName="kube-rbac-proxy-web"
Sep 16 02:21:09.891102 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:21:09.890224    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="9cc00559-db17-4390-b926-cb912b947e7b" containerName="config-reloader"
Sep 16 02:21:09.891102 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:21:09.890230    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="9cc00559-db17-4390-b926-cb912b947e7b" containerName="kube-rbac-proxy"
Sep 16 02:21:09.891102 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:21:09.890236    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="9cc00559-db17-4390-b926-cb912b947e7b" containerName="kube-rbac-proxy-web"
Sep 16 02:21:09.891102 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:21:09.890241    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="9cc00559-db17-4390-b926-cb912b947e7b" containerName="kube-rbac-proxy-thanos"
Sep 16 02:21:09.891102 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:21:09.890250    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="5bf4691b-1501-4d65-9538-52851f45d954" containerName="config-reloader"
Sep 16 02:21:09.891102 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:21:09.890258    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="5bf4691b-1501-4d65-9538-52851f45d954" containerName="kube-rbac-proxy-metric"
Sep 16 02:21:09.891102 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:21:09.890266    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="5bf4691b-1501-4d65-9538-52851f45d954" containerName="prom-label-proxy"
Sep 16 02:21:09.891102 ip-10-0-128-111 kubenswrapper[2583]: E0916 02:21:09.890378    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b2da4cf4a935141217a1262560b078b359b0162860efdb9647353030e0090590\": container with ID starting with b2da4cf4a935141217a1262560b078b359b0162860efdb9647353030e0090590 not found: ID does not exist" containerID="b2da4cf4a935141217a1262560b078b359b0162860efdb9647353030e0090590"
Sep 16 02:21:09.891102 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:21:09.890397    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b2da4cf4a935141217a1262560b078b359b0162860efdb9647353030e0090590"} err="failed to get container status \"b2da4cf4a935141217a1262560b078b359b0162860efdb9647353030e0090590\": rpc error: code = NotFound desc = could not find container \"b2da4cf4a935141217a1262560b078b359b0162860efdb9647353030e0090590\": container with ID starting with b2da4cf4a935141217a1262560b078b359b0162860efdb9647353030e0090590 not found: ID does not exist"
Sep 16 02:21:09.891102 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:21:09.890413    2583 scope.go:117] "RemoveContainer" containerID="501ee12b90e989972c892e33d1b1754695561add0dd08622d66b99b6e2d38f54"
Sep 16 02:21:09.891102 ip-10-0-128-111 kubenswrapper[2583]: E0916 02:21:09.890698    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"501ee12b90e989972c892e33d1b1754695561add0dd08622d66b99b6e2d38f54\": container with ID starting with 501ee12b90e989972c892e33d1b1754695561add0dd08622d66b99b6e2d38f54 not found: ID does not exist" containerID="501ee12b90e989972c892e33d1b1754695561add0dd08622d66b99b6e2d38f54"
Sep 16 02:21:09.891102 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:21:09.890718    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"501ee12b90e989972c892e33d1b1754695561add0dd08622d66b99b6e2d38f54"} err="failed to get container status \"501ee12b90e989972c892e33d1b1754695561add0dd08622d66b99b6e2d38f54\": rpc error: code = NotFound desc = could not find container \"501ee12b90e989972c892e33d1b1754695561add0dd08622d66b99b6e2d38f54\": container with ID starting with 501ee12b90e989972c892e33d1b1754695561add0dd08622d66b99b6e2d38f54 not found: ID does not exist"
Sep 16 02:21:09.891102 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:21:09.890734    2583 scope.go:117] "RemoveContainer" containerID="1421871f4efae41cca1f0a74619eb934782244bf8a909df0351c27932b0b4150"
Sep 16 02:21:09.891102 ip-10-0-128-111 kubenswrapper[2583]: E0916 02:21:09.890962    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1421871f4efae41cca1f0a74619eb934782244bf8a909df0351c27932b0b4150\": container with ID starting with 1421871f4efae41cca1f0a74619eb934782244bf8a909df0351c27932b0b4150 not found: ID does not exist" containerID="1421871f4efae41cca1f0a74619eb934782244bf8a909df0351c27932b0b4150"
Sep 16 02:21:09.891102 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:21:09.890977    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1421871f4efae41cca1f0a74619eb934782244bf8a909df0351c27932b0b4150"} err="failed to get container status \"1421871f4efae41cca1f0a74619eb934782244bf8a909df0351c27932b0b4150\": rpc error: code = NotFound desc = could not find container \"1421871f4efae41cca1f0a74619eb934782244bf8a909df0351c27932b0b4150\": container with ID starting with 1421871f4efae41cca1f0a74619eb934782244bf8a909df0351c27932b0b4150 not found: ID does not exist"
Sep 16 02:21:09.891102 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:21:09.890989    2583 scope.go:117] "RemoveContainer" containerID="36cb8aec3dcb4f882b28557545eb9fd3890739e6141f4e49482d18cadcac7677"
Sep 16 02:21:09.891693 ip-10-0-128-111 kubenswrapper[2583]: E0916 02:21:09.891231    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"36cb8aec3dcb4f882b28557545eb9fd3890739e6141f4e49482d18cadcac7677\": container with ID starting with 36cb8aec3dcb4f882b28557545eb9fd3890739e6141f4e49482d18cadcac7677 not found: ID does not exist" containerID="36cb8aec3dcb4f882b28557545eb9fd3890739e6141f4e49482d18cadcac7677"
Sep 16 02:21:09.891693 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:21:09.891260    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"36cb8aec3dcb4f882b28557545eb9fd3890739e6141f4e49482d18cadcac7677"} err="failed to get container status \"36cb8aec3dcb4f882b28557545eb9fd3890739e6141f4e49482d18cadcac7677\": rpc error: code = NotFound desc = could not find container \"36cb8aec3dcb4f882b28557545eb9fd3890739e6141f4e49482d18cadcac7677\": container with ID starting with 36cb8aec3dcb4f882b28557545eb9fd3890739e6141f4e49482d18cadcac7677 not found: ID does not exist"
Sep 16 02:21:09.891693 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:21:09.891282    2583 scope.go:117] "RemoveContainer" containerID="84b39eb0e85e4cb68e8c22ec7d5ebd8349797f955edd9fa2ad7cc3835eaa4ef3"
Sep 16 02:21:09.891693 ip-10-0-128-111 kubenswrapper[2583]: E0916 02:21:09.891541    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"84b39eb0e85e4cb68e8c22ec7d5ebd8349797f955edd9fa2ad7cc3835eaa4ef3\": container with ID starting with 84b39eb0e85e4cb68e8c22ec7d5ebd8349797f955edd9fa2ad7cc3835eaa4ef3 not found: ID does not exist" containerID="84b39eb0e85e4cb68e8c22ec7d5ebd8349797f955edd9fa2ad7cc3835eaa4ef3"
Sep 16 02:21:09.891693 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:21:09.891581    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"84b39eb0e85e4cb68e8c22ec7d5ebd8349797f955edd9fa2ad7cc3835eaa4ef3"} err="failed to get container status \"84b39eb0e85e4cb68e8c22ec7d5ebd8349797f955edd9fa2ad7cc3835eaa4ef3\": rpc error: code = NotFound desc = could not find container \"84b39eb0e85e4cb68e8c22ec7d5ebd8349797f955edd9fa2ad7cc3835eaa4ef3\": container with ID starting with 84b39eb0e85e4cb68e8c22ec7d5ebd8349797f955edd9fa2ad7cc3835eaa4ef3 not found: ID does not exist"
Sep 16 02:21:09.891693 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:21:09.891595    2583 scope.go:117] "RemoveContainer" containerID="335fb22f91f7fbe2f73e01a471295c3453e09ef84d66a4d843daa5af0f1cba2d"
Sep 16 02:21:09.891959 ip-10-0-128-111 kubenswrapper[2583]: E0916 02:21:09.891866    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"335fb22f91f7fbe2f73e01a471295c3453e09ef84d66a4d843daa5af0f1cba2d\": container with ID starting with 335fb22f91f7fbe2f73e01a471295c3453e09ef84d66a4d843daa5af0f1cba2d not found: ID does not exist" containerID="335fb22f91f7fbe2f73e01a471295c3453e09ef84d66a4d843daa5af0f1cba2d"
Sep 16 02:21:09.891959 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:21:09.891893    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"335fb22f91f7fbe2f73e01a471295c3453e09ef84d66a4d843daa5af0f1cba2d"} err="failed to get container status \"335fb22f91f7fbe2f73e01a471295c3453e09ef84d66a4d843daa5af0f1cba2d\": rpc error: code = NotFound desc = could not find container \"335fb22f91f7fbe2f73e01a471295c3453e09ef84d66a4d843daa5af0f1cba2d\": container with ID starting with 335fb22f91f7fbe2f73e01a471295c3453e09ef84d66a4d843daa5af0f1cba2d not found: ID does not exist"
Sep 16 02:21:09.895370 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:21:09.894415    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 02:21:09.897203 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:21:09.897182    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\""
Sep 16 02:21:09.897306 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:21:09.897266    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-e3djpgrfpc0t\""
Sep 16 02:21:09.897306 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:21:09.897273    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\""
Sep 16 02:21:09.897417 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:21:09.897357    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\""
Sep 16 02:21:09.897417 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:21:09.897412    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\""
Sep 16 02:21:09.897564 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:21:09.897513    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-jjgfm\""
Sep 16 02:21:09.897631 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:21:09.897615    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\""
Sep 16 02:21:09.897683 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:21:09.897644    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\""
Sep 16 02:21:09.897738 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:21:09.897713    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\""
Sep 16 02:21:09.897799 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:21:09.897786    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\""
Sep 16 02:21:09.898078 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:21:09.898060    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\""
Sep 16 02:21:09.898268 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:21:09.898249    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\""
Sep 16 02:21:09.898354 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:21:09.898335    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\""
Sep 16 02:21:09.898477 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:21:09.898457    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\""
Sep 16 02:21:09.901508 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:21:09.901490    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\""
Sep 16 02:21:09.903064 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:21:09.903046    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\""
Sep 16 02:21:09.909950 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:21:09.909930    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 16 02:21:09.938321 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:21:09.936274    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/c29a4a9d-f6a2-446e-b2a9-90292218ae94-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"c29a4a9d-f6a2-446e-b2a9-90292218ae94\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 02:21:09.938321 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:21:09.936315    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/c29a4a9d-f6a2-446e-b2a9-90292218ae94-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"c29a4a9d-f6a2-446e-b2a9-90292218ae94\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 02:21:09.938321 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:21:09.936349    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/c29a4a9d-f6a2-446e-b2a9-90292218ae94-config\") pod \"prometheus-k8s-0\" (UID: \"c29a4a9d-f6a2-446e-b2a9-90292218ae94\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 02:21:09.938321 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:21:09.936372    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/c29a4a9d-f6a2-446e-b2a9-90292218ae94-config-out\") pod \"prometheus-k8s-0\" (UID: \"c29a4a9d-f6a2-446e-b2a9-90292218ae94\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 02:21:09.938321 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:21:09.936395    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/c29a4a9d-f6a2-446e-b2a9-90292218ae94-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"c29a4a9d-f6a2-446e-b2a9-90292218ae94\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 02:21:09.938321 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:21:09.936454    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/c29a4a9d-f6a2-446e-b2a9-90292218ae94-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"c29a4a9d-f6a2-446e-b2a9-90292218ae94\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 02:21:09.938321 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:21:09.936477    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/c29a4a9d-f6a2-446e-b2a9-90292218ae94-web-config\") pod \"prometheus-k8s-0\" (UID: \"c29a4a9d-f6a2-446e-b2a9-90292218ae94\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 02:21:09.938321 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:21:09.936503    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/c29a4a9d-f6a2-446e-b2a9-90292218ae94-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"c29a4a9d-f6a2-446e-b2a9-90292218ae94\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 02:21:09.938321 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:21:09.936523    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c29a4a9d-f6a2-446e-b2a9-90292218ae94-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"c29a4a9d-f6a2-446e-b2a9-90292218ae94\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 02:21:09.938321 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:21:09.936545    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-595wd\" (UniqueName: \"kubernetes.io/projected/c29a4a9d-f6a2-446e-b2a9-90292218ae94-kube-api-access-595wd\") pod \"prometheus-k8s-0\" (UID: \"c29a4a9d-f6a2-446e-b2a9-90292218ae94\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 02:21:09.938321 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:21:09.936589    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/c29a4a9d-f6a2-446e-b2a9-90292218ae94-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"c29a4a9d-f6a2-446e-b2a9-90292218ae94\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 02:21:09.938321 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:21:09.936617    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/c29a4a9d-f6a2-446e-b2a9-90292218ae94-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"c29a4a9d-f6a2-446e-b2a9-90292218ae94\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 02:21:09.938321 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:21:09.936643    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c29a4a9d-f6a2-446e-b2a9-90292218ae94-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"c29a4a9d-f6a2-446e-b2a9-90292218ae94\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 02:21:09.938321 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:21:09.936666    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/c29a4a9d-f6a2-446e-b2a9-90292218ae94-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"c29a4a9d-f6a2-446e-b2a9-90292218ae94\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 02:21:09.938321 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:21:09.936691    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/c29a4a9d-f6a2-446e-b2a9-90292218ae94-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"c29a4a9d-f6a2-446e-b2a9-90292218ae94\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 02:21:09.938321 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:21:09.936714    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/c29a4a9d-f6a2-446e-b2a9-90292218ae94-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"c29a4a9d-f6a2-446e-b2a9-90292218ae94\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 02:21:09.939132 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:21:09.936740    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c29a4a9d-f6a2-446e-b2a9-90292218ae94-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"c29a4a9d-f6a2-446e-b2a9-90292218ae94\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 02:21:09.939132 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:21:09.936761    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/c29a4a9d-f6a2-446e-b2a9-90292218ae94-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"c29a4a9d-f6a2-446e-b2a9-90292218ae94\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 02:21:09.939132 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:21:09.936791    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/c29a4a9d-f6a2-446e-b2a9-90292218ae94-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"c29a4a9d-f6a2-446e-b2a9-90292218ae94\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 02:21:09.939132 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:21:09.936823    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c29a4a9d-f6a2-446e-b2a9-90292218ae94-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"c29a4a9d-f6a2-446e-b2a9-90292218ae94\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 02:21:10.038170 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:21:10.038138    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/c29a4a9d-f6a2-446e-b2a9-90292218ae94-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"c29a4a9d-f6a2-446e-b2a9-90292218ae94\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 02:21:10.038170 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:21:10.038175    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/c29a4a9d-f6a2-446e-b2a9-90292218ae94-web-config\") pod \"prometheus-k8s-0\" (UID: \"c29a4a9d-f6a2-446e-b2a9-90292218ae94\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 02:21:10.038391 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:21:10.038195    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/c29a4a9d-f6a2-446e-b2a9-90292218ae94-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"c29a4a9d-f6a2-446e-b2a9-90292218ae94\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 02:21:10.038391 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:21:10.038212    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c29a4a9d-f6a2-446e-b2a9-90292218ae94-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"c29a4a9d-f6a2-446e-b2a9-90292218ae94\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 02:21:10.038391 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:21:10.038236    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-595wd\" (UniqueName: \"kubernetes.io/projected/c29a4a9d-f6a2-446e-b2a9-90292218ae94-kube-api-access-595wd\") pod \"prometheus-k8s-0\" (UID: \"c29a4a9d-f6a2-446e-b2a9-90292218ae94\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 02:21:10.038391 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:21:10.038260    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/c29a4a9d-f6a2-446e-b2a9-90292218ae94-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"c29a4a9d-f6a2-446e-b2a9-90292218ae94\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 02:21:10.038391 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:21:10.038286    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/c29a4a9d-f6a2-446e-b2a9-90292218ae94-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"c29a4a9d-f6a2-446e-b2a9-90292218ae94\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 02:21:10.038391 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:21:10.038312    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c29a4a9d-f6a2-446e-b2a9-90292218ae94-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"c29a4a9d-f6a2-446e-b2a9-90292218ae94\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 02:21:10.038391 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:21:10.038340    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/c29a4a9d-f6a2-446e-b2a9-90292218ae94-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"c29a4a9d-f6a2-446e-b2a9-90292218ae94\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 02:21:10.038391 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:21:10.038369    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/c29a4a9d-f6a2-446e-b2a9-90292218ae94-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"c29a4a9d-f6a2-446e-b2a9-90292218ae94\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 02:21:10.038776 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:21:10.038395    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/c29a4a9d-f6a2-446e-b2a9-90292218ae94-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"c29a4a9d-f6a2-446e-b2a9-90292218ae94\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 02:21:10.038776 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:21:10.038424    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c29a4a9d-f6a2-446e-b2a9-90292218ae94-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"c29a4a9d-f6a2-446e-b2a9-90292218ae94\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 02:21:10.038776 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:21:10.038447    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/c29a4a9d-f6a2-446e-b2a9-90292218ae94-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"c29a4a9d-f6a2-446e-b2a9-90292218ae94\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 02:21:10.038776 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:21:10.038478    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/c29a4a9d-f6a2-446e-b2a9-90292218ae94-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"c29a4a9d-f6a2-446e-b2a9-90292218ae94\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 02:21:10.038776 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:21:10.038509    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c29a4a9d-f6a2-446e-b2a9-90292218ae94-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"c29a4a9d-f6a2-446e-b2a9-90292218ae94\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 02:21:10.038776 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:21:10.038576    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/c29a4a9d-f6a2-446e-b2a9-90292218ae94-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"c29a4a9d-f6a2-446e-b2a9-90292218ae94\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 02:21:10.038776 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:21:10.038612    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/c29a4a9d-f6a2-446e-b2a9-90292218ae94-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"c29a4a9d-f6a2-446e-b2a9-90292218ae94\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 02:21:10.038776 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:21:10.038651    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/c29a4a9d-f6a2-446e-b2a9-90292218ae94-config\") pod \"prometheus-k8s-0\" (UID: \"c29a4a9d-f6a2-446e-b2a9-90292218ae94\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 02:21:10.038776 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:21:10.038674    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/c29a4a9d-f6a2-446e-b2a9-90292218ae94-config-out\") pod \"prometheus-k8s-0\" (UID: \"c29a4a9d-f6a2-446e-b2a9-90292218ae94\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 02:21:10.038776 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:21:10.038702    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/c29a4a9d-f6a2-446e-b2a9-90292218ae94-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"c29a4a9d-f6a2-446e-b2a9-90292218ae94\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 02:21:10.039242 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:21:10.038878    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/c29a4a9d-f6a2-446e-b2a9-90292218ae94-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"c29a4a9d-f6a2-446e-b2a9-90292218ae94\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 02:21:10.039242 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:21:10.039083    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c29a4a9d-f6a2-446e-b2a9-90292218ae94-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"c29a4a9d-f6a2-446e-b2a9-90292218ae94\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 02:21:10.039662 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:21:10.039636    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c29a4a9d-f6a2-446e-b2a9-90292218ae94-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"c29a4a9d-f6a2-446e-b2a9-90292218ae94\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 02:21:10.039788 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:21:10.039762    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c29a4a9d-f6a2-446e-b2a9-90292218ae94-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"c29a4a9d-f6a2-446e-b2a9-90292218ae94\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 02:21:10.040199 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:21:10.040172    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/c29a4a9d-f6a2-446e-b2a9-90292218ae94-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"c29a4a9d-f6a2-446e-b2a9-90292218ae94\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 02:21:10.040428 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:21:10.040399    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c29a4a9d-f6a2-446e-b2a9-90292218ae94-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"c29a4a9d-f6a2-446e-b2a9-90292218ae94\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 02:21:10.040961 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:21:10.040935    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/c29a4a9d-f6a2-446e-b2a9-90292218ae94-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"c29a4a9d-f6a2-446e-b2a9-90292218ae94\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 02:21:10.041290 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:21:10.041262    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/c29a4a9d-f6a2-446e-b2a9-90292218ae94-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"c29a4a9d-f6a2-446e-b2a9-90292218ae94\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 02:21:10.041452 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:21:10.041428    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/c29a4a9d-f6a2-446e-b2a9-90292218ae94-web-config\") pod \"prometheus-k8s-0\" (UID: \"c29a4a9d-f6a2-446e-b2a9-90292218ae94\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 02:21:10.041713 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:21:10.041690    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/c29a4a9d-f6a2-446e-b2a9-90292218ae94-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"c29a4a9d-f6a2-446e-b2a9-90292218ae94\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 02:21:10.041782 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:21:10.041741    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/c29a4a9d-f6a2-446e-b2a9-90292218ae94-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"c29a4a9d-f6a2-446e-b2a9-90292218ae94\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 02:21:10.042789 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:21:10.042766    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/c29a4a9d-f6a2-446e-b2a9-90292218ae94-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"c29a4a9d-f6a2-446e-b2a9-90292218ae94\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 02:21:10.042917 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:21:10.042817    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/c29a4a9d-f6a2-446e-b2a9-90292218ae94-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"c29a4a9d-f6a2-446e-b2a9-90292218ae94\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 02:21:10.043236 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:21:10.043209    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/c29a4a9d-f6a2-446e-b2a9-90292218ae94-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"c29a4a9d-f6a2-446e-b2a9-90292218ae94\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 02:21:10.043416 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:21:10.043398    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/c29a4a9d-f6a2-446e-b2a9-90292218ae94-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"c29a4a9d-f6a2-446e-b2a9-90292218ae94\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 02:21:10.043856 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:21:10.043837    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/c29a4a9d-f6a2-446e-b2a9-90292218ae94-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"c29a4a9d-f6a2-446e-b2a9-90292218ae94\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 02:21:10.044229 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:21:10.044206    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/c29a4a9d-f6a2-446e-b2a9-90292218ae94-config-out\") pod \"prometheus-k8s-0\" (UID: \"c29a4a9d-f6a2-446e-b2a9-90292218ae94\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 02:21:10.044326 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:21:10.044242    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/c29a4a9d-f6a2-446e-b2a9-90292218ae94-config\") pod \"prometheus-k8s-0\" (UID: \"c29a4a9d-f6a2-446e-b2a9-90292218ae94\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 02:21:10.044932 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:21:10.044916    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/c29a4a9d-f6a2-446e-b2a9-90292218ae94-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"c29a4a9d-f6a2-446e-b2a9-90292218ae94\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 02:21:10.045539 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:21:10.045518    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-595wd\" (UniqueName: \"kubernetes.io/projected/c29a4a9d-f6a2-446e-b2a9-90292218ae94-kube-api-access-595wd\") pod \"prometheus-k8s-0\" (UID: \"c29a4a9d-f6a2-446e-b2a9-90292218ae94\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 02:21:10.047468 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:21:10.047447    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9cc00559-db17-4390-b926-cb912b947e7b" path="/var/lib/kubelet/pods/9cc00559-db17-4390-b926-cb912b947e7b/volumes"
Sep 16 02:21:10.205378 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:21:10.205336    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 02:21:10.481890 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:21:10.481864    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 16 02:21:10.485197 ip-10-0-128-111 kubenswrapper[2583]: W0916 02:21:10.485169    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc29a4a9d_f6a2_446e_b2a9_90292218ae94.slice/crio-1d98cf0674f658990a31fa303ba233dd19c5f0ed5a6846e75822902567b925d6 WatchSource:0}: Error finding container 1d98cf0674f658990a31fa303ba233dd19c5f0ed5a6846e75822902567b925d6: Status 404 returned error can't find the container with id 1d98cf0674f658990a31fa303ba233dd19c5f0ed5a6846e75822902567b925d6
Sep 16 02:21:10.845750 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:21:10.845716    2583 generic.go:356] "Generic (PLEG): container finished" podID="c29a4a9d-f6a2-446e-b2a9-90292218ae94" containerID="b9f70f8899eb78f806399afddd3d26d5ddfd9c54d8c9db185c8e079c84f74b41" exitCode=0
Sep 16 02:21:10.846149 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:21:10.845798    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"c29a4a9d-f6a2-446e-b2a9-90292218ae94","Type":"ContainerDied","Data":"b9f70f8899eb78f806399afddd3d26d5ddfd9c54d8c9db185c8e079c84f74b41"}
Sep 16 02:21:10.846149 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:21:10.845872    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"c29a4a9d-f6a2-446e-b2a9-90292218ae94","Type":"ContainerStarted","Data":"1d98cf0674f658990a31fa303ba233dd19c5f0ed5a6846e75822902567b925d6"}
Sep 16 02:21:11.854760 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:21:11.854727    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"c29a4a9d-f6a2-446e-b2a9-90292218ae94","Type":"ContainerStarted","Data":"c28fa9636cc079bf227b64197102026d72846796c39b9ac81233d219cb8c5316"}
Sep 16 02:21:11.855133 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:21:11.854792    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"c29a4a9d-f6a2-446e-b2a9-90292218ae94","Type":"ContainerStarted","Data":"af51dd43d1f9a721da3dc5c8abfaf39afe57557d573b5580ac4c0cadea860130"}
Sep 16 02:21:11.855133 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:21:11.854820    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"c29a4a9d-f6a2-446e-b2a9-90292218ae94","Type":"ContainerStarted","Data":"55737567738d95acbe70cbf38a9ace8fd12f35d569453bed7529170348c7ae0d"}
Sep 16 02:21:11.855133 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:21:11.854847    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"c29a4a9d-f6a2-446e-b2a9-90292218ae94","Type":"ContainerStarted","Data":"ebea4f669e89a4258a7d37d813d8c2d7823e92795ca1734d2c43e71fb78eda45"}
Sep 16 02:21:11.855133 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:21:11.854869    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"c29a4a9d-f6a2-446e-b2a9-90292218ae94","Type":"ContainerStarted","Data":"57bfae0e132a1038e82cb93f0cfaa6119be17d3544736c77202ed1c4ae12a092"}
Sep 16 02:21:11.855133 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:21:11.854895    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"c29a4a9d-f6a2-446e-b2a9-90292218ae94","Type":"ContainerStarted","Data":"06b4d813443a17d99aa84dd97a1f05853ccc7b022bb007e13afd483ce0ce6d5a"}
Sep 16 02:21:11.883733 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:21:11.883684    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=2.861599528 podStartE2EDuration="2.883663062s" totalImagesPullingTime="0s" totalInitContainerRuntime="22.063534ms" isStatefulPod=false podCreationTimestamp="2026-09-16 02:21:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 02:21:11.881123945 +0000 UTC m=+150.348912768" watchObservedRunningTime="2026-09-16 02:21:11.883663062 +0000 UTC m=+150.351451909"
Sep 16 02:21:15.205495 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:21:15.205457    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 02:22:10.205950 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:22:10.205904    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 02:22:10.221022 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:22:10.220996    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 02:22:11.083877 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:22:11.083848    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 02:22:30.547345 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:22:30.547303    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-znwwb"]
Sep 16 02:22:30.549741 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:22:30.549725    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-559cdd8c87-znwwb"
Sep 16 02:22:30.552128 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:22:30.552106    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\""
Sep 16 02:22:30.553072 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:22:30.553052    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-cainjector-dockercfg-zlhbt\""
Sep 16 02:22:30.553171 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:22:30.553081    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\""
Sep 16 02:22:30.559434 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:22:30.559412    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-znwwb"]
Sep 16 02:22:30.687699 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:22:30.687654    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fj62s\" (UniqueName: \"kubernetes.io/projected/cf04a83e-4eb4-4871-b6ea-7186b0a75b0d-kube-api-access-fj62s\") pod \"cert-manager-cainjector-559cdd8c87-znwwb\" (UID: \"cf04a83e-4eb4-4871-b6ea-7186b0a75b0d\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-znwwb"
Sep 16 02:22:30.687699 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:22:30.687707    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/cf04a83e-4eb4-4871-b6ea-7186b0a75b0d-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-znwwb\" (UID: \"cf04a83e-4eb4-4871-b6ea-7186b0a75b0d\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-znwwb"
Sep 16 02:22:30.788904 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:22:30.788861    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fj62s\" (UniqueName: \"kubernetes.io/projected/cf04a83e-4eb4-4871-b6ea-7186b0a75b0d-kube-api-access-fj62s\") pod \"cert-manager-cainjector-559cdd8c87-znwwb\" (UID: \"cf04a83e-4eb4-4871-b6ea-7186b0a75b0d\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-znwwb"
Sep 16 02:22:30.789046 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:22:30.788915    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/cf04a83e-4eb4-4871-b6ea-7186b0a75b0d-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-znwwb\" (UID: \"cf04a83e-4eb4-4871-b6ea-7186b0a75b0d\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-znwwb"
Sep 16 02:22:30.796504 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:22:30.796482    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/cf04a83e-4eb4-4871-b6ea-7186b0a75b0d-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-znwwb\" (UID: \"cf04a83e-4eb4-4871-b6ea-7186b0a75b0d\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-znwwb"
Sep 16 02:22:30.796599 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:22:30.796541    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fj62s\" (UniqueName: \"kubernetes.io/projected/cf04a83e-4eb4-4871-b6ea-7186b0a75b0d-kube-api-access-fj62s\") pod \"cert-manager-cainjector-559cdd8c87-znwwb\" (UID: \"cf04a83e-4eb4-4871-b6ea-7186b0a75b0d\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-znwwb"
Sep 16 02:22:30.872636 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:22:30.872612    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-559cdd8c87-znwwb"
Sep 16 02:22:31.137280 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:22:31.137255    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-znwwb"]
Sep 16 02:22:31.139962 ip-10-0-128-111 kubenswrapper[2583]: W0916 02:22:31.139934    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podcf04a83e_4eb4_4871_b6ea_7186b0a75b0d.slice/crio-c8b898c235324acf0cad4718423ca2787633ecf572c46cfeb9d3cc42d1e52dfa WatchSource:0}: Error finding container c8b898c235324acf0cad4718423ca2787633ecf572c46cfeb9d3cc42d1e52dfa: Status 404 returned error can't find the container with id c8b898c235324acf0cad4718423ca2787633ecf572c46cfeb9d3cc42d1e52dfa
Sep 16 02:22:31.147098 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:22:31.147074    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-559cdd8c87-znwwb" event={"ID":"cf04a83e-4eb4-4871-b6ea-7186b0a75b0d","Type":"ContainerStarted","Data":"c8b898c235324acf0cad4718423ca2787633ecf572c46cfeb9d3cc42d1e52dfa"}
Sep 16 02:22:35.166693 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:22:35.166656    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-559cdd8c87-znwwb" event={"ID":"cf04a83e-4eb4-4871-b6ea-7186b0a75b0d","Type":"ContainerStarted","Data":"01fb0bc661e2d18bfc35f56a62709bb9d9ba46df86f78dea811308c5d9b1ee5d"}
Sep 16 02:22:35.183301 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:22:35.183247    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-cainjector-559cdd8c87-znwwb" podStartSLOduration=2.0680918090000002 podStartE2EDuration="5.183230289s" totalImagesPullingTime="3.11513848s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 02:22:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 02:22:35.182424101 +0000 UTC m=+233.650212889" watchObservedRunningTime="2026-09-16 02:22:35.183230289 +0000 UTC m=+233.651019070"
Sep 16 02:23:06.745483 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:23:06.745398    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/lws-controller-manager-54ff6bb656-f2fpn"]
Sep 16 02:23:06.751441 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:23:06.751421    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-54ff6bb656-f2fpn"
Sep 16 02:23:06.755520 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:23:06.755493    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"metrics-server-cert\""
Sep 16 02:23:06.755660 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:23:06.755527    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\""
Sep 16 02:23:06.755660 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:23:06.755493    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"lws-controller-manager-dockercfg-5ndtq\""
Sep 16 02:23:06.755660 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:23:06.755502    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"webhook-server-cert\""
Sep 16 02:23:06.755660 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:23:06.755492    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\""
Sep 16 02:23:06.755660 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:23:06.755615    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"lws-manager-config\""
Sep 16 02:23:06.761942 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:23:06.761923    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-54ff6bb656-f2fpn"]
Sep 16 02:23:06.773598 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:23:06.773578    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/28d69534-e2e0-4bda-9eea-f5ff6eb7634a-cert\") pod \"lws-controller-manager-54ff6bb656-f2fpn\" (UID: \"28d69534-e2e0-4bda-9eea-f5ff6eb7634a\") " pod="openshift-lws-operator/lws-controller-manager-54ff6bb656-f2fpn"
Sep 16 02:23:06.773719 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:23:06.773627    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/28d69534-e2e0-4bda-9eea-f5ff6eb7634a-metrics-cert\") pod \"lws-controller-manager-54ff6bb656-f2fpn\" (UID: \"28d69534-e2e0-4bda-9eea-f5ff6eb7634a\") " pod="openshift-lws-operator/lws-controller-manager-54ff6bb656-f2fpn"
Sep 16 02:23:06.773719 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:23:06.773654    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/28d69534-e2e0-4bda-9eea-f5ff6eb7634a-manager-config\") pod \"lws-controller-manager-54ff6bb656-f2fpn\" (UID: \"28d69534-e2e0-4bda-9eea-f5ff6eb7634a\") " pod="openshift-lws-operator/lws-controller-manager-54ff6bb656-f2fpn"
Sep 16 02:23:06.773719 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:23:06.773706    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p9jlr\" (UniqueName: \"kubernetes.io/projected/28d69534-e2e0-4bda-9eea-f5ff6eb7634a-kube-api-access-p9jlr\") pod \"lws-controller-manager-54ff6bb656-f2fpn\" (UID: \"28d69534-e2e0-4bda-9eea-f5ff6eb7634a\") " pod="openshift-lws-operator/lws-controller-manager-54ff6bb656-f2fpn"
Sep 16 02:23:06.874339 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:23:06.874302    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p9jlr\" (UniqueName: \"kubernetes.io/projected/28d69534-e2e0-4bda-9eea-f5ff6eb7634a-kube-api-access-p9jlr\") pod \"lws-controller-manager-54ff6bb656-f2fpn\" (UID: \"28d69534-e2e0-4bda-9eea-f5ff6eb7634a\") " pod="openshift-lws-operator/lws-controller-manager-54ff6bb656-f2fpn"
Sep 16 02:23:06.874582 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:23:06.874391    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/28d69534-e2e0-4bda-9eea-f5ff6eb7634a-cert\") pod \"lws-controller-manager-54ff6bb656-f2fpn\" (UID: \"28d69534-e2e0-4bda-9eea-f5ff6eb7634a\") " pod="openshift-lws-operator/lws-controller-manager-54ff6bb656-f2fpn"
Sep 16 02:23:06.874582 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:23:06.874418    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/28d69534-e2e0-4bda-9eea-f5ff6eb7634a-metrics-cert\") pod \"lws-controller-manager-54ff6bb656-f2fpn\" (UID: \"28d69534-e2e0-4bda-9eea-f5ff6eb7634a\") " pod="openshift-lws-operator/lws-controller-manager-54ff6bb656-f2fpn"
Sep 16 02:23:06.874582 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:23:06.874443    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/28d69534-e2e0-4bda-9eea-f5ff6eb7634a-manager-config\") pod \"lws-controller-manager-54ff6bb656-f2fpn\" (UID: \"28d69534-e2e0-4bda-9eea-f5ff6eb7634a\") " pod="openshift-lws-operator/lws-controller-manager-54ff6bb656-f2fpn"
Sep 16 02:23:06.875180 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:23:06.875156    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/28d69534-e2e0-4bda-9eea-f5ff6eb7634a-manager-config\") pod \"lws-controller-manager-54ff6bb656-f2fpn\" (UID: \"28d69534-e2e0-4bda-9eea-f5ff6eb7634a\") " pod="openshift-lws-operator/lws-controller-manager-54ff6bb656-f2fpn"
Sep 16 02:23:06.877031 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:23:06.877001    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/28d69534-e2e0-4bda-9eea-f5ff6eb7634a-cert\") pod \"lws-controller-manager-54ff6bb656-f2fpn\" (UID: \"28d69534-e2e0-4bda-9eea-f5ff6eb7634a\") " pod="openshift-lws-operator/lws-controller-manager-54ff6bb656-f2fpn"
Sep 16 02:23:06.877031 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:23:06.877026    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/28d69534-e2e0-4bda-9eea-f5ff6eb7634a-metrics-cert\") pod \"lws-controller-manager-54ff6bb656-f2fpn\" (UID: \"28d69534-e2e0-4bda-9eea-f5ff6eb7634a\") " pod="openshift-lws-operator/lws-controller-manager-54ff6bb656-f2fpn"
Sep 16 02:23:06.896958 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:23:06.896923    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-p9jlr\" (UniqueName: \"kubernetes.io/projected/28d69534-e2e0-4bda-9eea-f5ff6eb7634a-kube-api-access-p9jlr\") pod \"lws-controller-manager-54ff6bb656-f2fpn\" (UID: \"28d69534-e2e0-4bda-9eea-f5ff6eb7634a\") " pod="openshift-lws-operator/lws-controller-manager-54ff6bb656-f2fpn"
Sep 16 02:23:07.061501 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:23:07.061418    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-54ff6bb656-f2fpn"
Sep 16 02:23:07.370256 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:23:07.370226    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-54ff6bb656-f2fpn"]
Sep 16 02:23:07.373657 ip-10-0-128-111 kubenswrapper[2583]: W0916 02:23:07.373630    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod28d69534_e2e0_4bda_9eea_f5ff6eb7634a.slice/crio-e57ab9878c0baa0afb16fcc56c040527d5385756b76424b4696e4f39da7fa673 WatchSource:0}: Error finding container e57ab9878c0baa0afb16fcc56c040527d5385756b76424b4696e4f39da7fa673: Status 404 returned error can't find the container with id e57ab9878c0baa0afb16fcc56c040527d5385756b76424b4696e4f39da7fa673
Sep 16 02:23:07.534512 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:23:07.534481    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-599b75f66d-g4c5v"]
Sep 16 02:23:07.560500 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:23:07.560473    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-599b75f66d-g4c5v"]
Sep 16 02:23:07.560676 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:23:07.560591    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-599b75f66d-g4c5v"
Sep 16 02:23:07.564218 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:23:07.564189    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\""
Sep 16 02:23:07.564795 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:23:07.564775    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\""
Sep 16 02:23:07.566393 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:23:07.565305    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\""
Sep 16 02:23:07.566393 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:23:07.566082    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\""
Sep 16 02:23:07.566984 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:23:07.566878    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-tdbgx\""
Sep 16 02:23:07.581445 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:23:07.581424    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/7849eb53-ff7d-4c8e-82a2-982f21b80577-apiservice-cert\") pod \"opendatahub-operator-controller-manager-599b75f66d-g4c5v\" (UID: \"7849eb53-ff7d-4c8e-82a2-982f21b80577\") " pod="opendatahub/opendatahub-operator-controller-manager-599b75f66d-g4c5v"
Sep 16 02:23:07.581541 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:23:07.581463    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/7849eb53-ff7d-4c8e-82a2-982f21b80577-manifests\") pod \"opendatahub-operator-controller-manager-599b75f66d-g4c5v\" (UID: \"7849eb53-ff7d-4c8e-82a2-982f21b80577\") " pod="opendatahub/opendatahub-operator-controller-manager-599b75f66d-g4c5v"
Sep 16 02:23:07.581541 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:23:07.581494    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lsc28\" (UniqueName: \"kubernetes.io/projected/7849eb53-ff7d-4c8e-82a2-982f21b80577-kube-api-access-lsc28\") pod \"opendatahub-operator-controller-manager-599b75f66d-g4c5v\" (UID: \"7849eb53-ff7d-4c8e-82a2-982f21b80577\") " pod="opendatahub/opendatahub-operator-controller-manager-599b75f66d-g4c5v"
Sep 16 02:23:07.581637 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:23:07.581545    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/7849eb53-ff7d-4c8e-82a2-982f21b80577-tmp\") pod \"opendatahub-operator-controller-manager-599b75f66d-g4c5v\" (UID: \"7849eb53-ff7d-4c8e-82a2-982f21b80577\") " pod="opendatahub/opendatahub-operator-controller-manager-599b75f66d-g4c5v"
Sep 16 02:23:07.581637 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:23:07.581588    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/7849eb53-ff7d-4c8e-82a2-982f21b80577-webhook-cert\") pod \"opendatahub-operator-controller-manager-599b75f66d-g4c5v\" (UID: \"7849eb53-ff7d-4c8e-82a2-982f21b80577\") " pod="opendatahub/opendatahub-operator-controller-manager-599b75f66d-g4c5v"
Sep 16 02:23:07.682127 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:23:07.682043    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/7849eb53-ff7d-4c8e-82a2-982f21b80577-apiservice-cert\") pod \"opendatahub-operator-controller-manager-599b75f66d-g4c5v\" (UID: \"7849eb53-ff7d-4c8e-82a2-982f21b80577\") " pod="opendatahub/opendatahub-operator-controller-manager-599b75f66d-g4c5v"
Sep 16 02:23:07.682127 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:23:07.682092    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/7849eb53-ff7d-4c8e-82a2-982f21b80577-manifests\") pod \"opendatahub-operator-controller-manager-599b75f66d-g4c5v\" (UID: \"7849eb53-ff7d-4c8e-82a2-982f21b80577\") " pod="opendatahub/opendatahub-operator-controller-manager-599b75f66d-g4c5v"
Sep 16 02:23:07.682308 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:23:07.682261    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lsc28\" (UniqueName: \"kubernetes.io/projected/7849eb53-ff7d-4c8e-82a2-982f21b80577-kube-api-access-lsc28\") pod \"opendatahub-operator-controller-manager-599b75f66d-g4c5v\" (UID: \"7849eb53-ff7d-4c8e-82a2-982f21b80577\") " pod="opendatahub/opendatahub-operator-controller-manager-599b75f66d-g4c5v"
Sep 16 02:23:07.682368 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:23:07.682317    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/7849eb53-ff7d-4c8e-82a2-982f21b80577-tmp\") pod \"opendatahub-operator-controller-manager-599b75f66d-g4c5v\" (UID: \"7849eb53-ff7d-4c8e-82a2-982f21b80577\") " pod="opendatahub/opendatahub-operator-controller-manager-599b75f66d-g4c5v"
Sep 16 02:23:07.682368 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:23:07.682350    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/7849eb53-ff7d-4c8e-82a2-982f21b80577-webhook-cert\") pod \"opendatahub-operator-controller-manager-599b75f66d-g4c5v\" (UID: \"7849eb53-ff7d-4c8e-82a2-982f21b80577\") " pod="opendatahub/opendatahub-operator-controller-manager-599b75f66d-g4c5v"
Sep 16 02:23:07.682461 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:23:07.682413    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/7849eb53-ff7d-4c8e-82a2-982f21b80577-manifests\") pod \"opendatahub-operator-controller-manager-599b75f66d-g4c5v\" (UID: \"7849eb53-ff7d-4c8e-82a2-982f21b80577\") " pod="opendatahub/opendatahub-operator-controller-manager-599b75f66d-g4c5v"
Sep 16 02:23:07.684474 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:23:07.684447    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/7849eb53-ff7d-4c8e-82a2-982f21b80577-tmp\") pod \"opendatahub-operator-controller-manager-599b75f66d-g4c5v\" (UID: \"7849eb53-ff7d-4c8e-82a2-982f21b80577\") " pod="opendatahub/opendatahub-operator-controller-manager-599b75f66d-g4c5v"
Sep 16 02:23:07.684841 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:23:07.684819    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/7849eb53-ff7d-4c8e-82a2-982f21b80577-webhook-cert\") pod \"opendatahub-operator-controller-manager-599b75f66d-g4c5v\" (UID: \"7849eb53-ff7d-4c8e-82a2-982f21b80577\") " pod="opendatahub/opendatahub-operator-controller-manager-599b75f66d-g4c5v"
Sep 16 02:23:07.684933 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:23:07.684819    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/7849eb53-ff7d-4c8e-82a2-982f21b80577-apiservice-cert\") pod \"opendatahub-operator-controller-manager-599b75f66d-g4c5v\" (UID: \"7849eb53-ff7d-4c8e-82a2-982f21b80577\") " pod="opendatahub/opendatahub-operator-controller-manager-599b75f66d-g4c5v"
Sep 16 02:23:07.699243 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:23:07.699214    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lsc28\" (UniqueName: \"kubernetes.io/projected/7849eb53-ff7d-4c8e-82a2-982f21b80577-kube-api-access-lsc28\") pod \"opendatahub-operator-controller-manager-599b75f66d-g4c5v\" (UID: \"7849eb53-ff7d-4c8e-82a2-982f21b80577\") " pod="opendatahub/opendatahub-operator-controller-manager-599b75f66d-g4c5v"
Sep 16 02:23:07.878738 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:23:07.878693    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-599b75f66d-g4c5v"
Sep 16 02:23:08.172516 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:23:08.172401    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-599b75f66d-g4c5v"]
Sep 16 02:23:08.175589 ip-10-0-128-111 kubenswrapper[2583]: W0916 02:23:08.175412    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7849eb53_ff7d_4c8e_82a2_982f21b80577.slice/crio-ab8552cdfc8879a400afa7414cd9cb3b90f5e7e0d383e9acd6524848f2a59b20 WatchSource:0}: Error finding container ab8552cdfc8879a400afa7414cd9cb3b90f5e7e0d383e9acd6524848f2a59b20: Status 404 returned error can't find the container with id ab8552cdfc8879a400afa7414cd9cb3b90f5e7e0d383e9acd6524848f2a59b20
Sep 16 02:23:08.289477 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:23:08.289389    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-599b75f66d-g4c5v" event={"ID":"7849eb53-ff7d-4c8e-82a2-982f21b80577","Type":"ContainerStarted","Data":"ab8552cdfc8879a400afa7414cd9cb3b90f5e7e0d383e9acd6524848f2a59b20"}
Sep 16 02:23:08.291790 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:23:08.291757    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-54ff6bb656-f2fpn" event={"ID":"28d69534-e2e0-4bda-9eea-f5ff6eb7634a","Type":"ContainerStarted","Data":"e57ab9878c0baa0afb16fcc56c040527d5385756b76424b4696e4f39da7fa673"}
Sep 16 02:23:12.311840 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:23:12.311800    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-599b75f66d-g4c5v" event={"ID":"7849eb53-ff7d-4c8e-82a2-982f21b80577","Type":"ContainerStarted","Data":"47f693de0a2eaa65e119544a8bcb9b64d10b7c4d0b2626a97155a85c3cc2d10b"}
Sep 16 02:23:12.313367 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:23:12.313332    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-54ff6bb656-f2fpn" event={"ID":"28d69534-e2e0-4bda-9eea-f5ff6eb7634a","Type":"ContainerStarted","Data":"8dfb698bf7ec1d5925b838307e1bab6688b57a43aaeb771f3f53b54531eac8b4"}
Sep 16 02:23:12.313528 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:23:12.313417    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-lws-operator/lws-controller-manager-54ff6bb656-f2fpn"
Sep 16 02:23:12.344043 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:23:12.343980    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/lws-controller-manager-54ff6bb656-f2fpn" podStartSLOduration=2.220881594 podStartE2EDuration="6.343963183s" totalImagesPullingTime="4.123081589s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 02:23:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 02:23:12.343077163 +0000 UTC m=+270.810865963" watchObservedRunningTime="2026-09-16 02:23:12.343963183 +0000 UTC m=+270.811751965"
Sep 16 02:23:13.318586 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:23:13.318540    2583 generic.go:356] "Generic (PLEG): container finished" podID="7849eb53-ff7d-4c8e-82a2-982f21b80577" containerID="47f693de0a2eaa65e119544a8bcb9b64d10b7c4d0b2626a97155a85c3cc2d10b" exitCode=0
Sep 16 02:23:13.318981 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:23:13.318646    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-599b75f66d-g4c5v" event={"ID":"7849eb53-ff7d-4c8e-82a2-982f21b80577","Type":"ContainerDied","Data":"47f693de0a2eaa65e119544a8bcb9b64d10b7c4d0b2626a97155a85c3cc2d10b"}
Sep 16 02:23:14.323916 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:23:14.323882    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-599b75f66d-g4c5v" event={"ID":"7849eb53-ff7d-4c8e-82a2-982f21b80577","Type":"ContainerStarted","Data":"5641438b36299cce321a8b65d66afdc09ce275afc3b47a90642a135f61e99fdc"}
Sep 16 02:23:14.324286 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:23:14.323985    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-599b75f66d-g4c5v"
Sep 16 02:23:14.344587 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:23:14.344496    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-599b75f66d-g4c5v" podStartSLOduration=2.6581404429999997 podStartE2EDuration="7.344477358s" totalImagesPullingTime="3.661570316s" totalInitContainerRuntime="1.024766599s" isStatefulPod=false podCreationTimestamp="2026-09-16 02:23:07 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 02:23:14.343872386 +0000 UTC m=+272.811661168" watchObservedRunningTime="2026-09-16 02:23:14.344477358 +0000 UTC m=+272.812266140"
Sep 16 02:23:23.321004 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:23:23.320975    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-lws-operator/lws-controller-manager-54ff6bb656-f2fpn"
Sep 16 02:23:25.332043 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:23:25.332002    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-599b75f66d-g4c5v"
Sep 16 02:23:27.236394 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:23:27.236356    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-6f86d6bc5-hqmmj"]
Sep 16 02:23:27.244878 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:23:27.244856    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-6f86d6bc5-hqmmj"
Sep 16 02:23:27.247933 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:23:27.247911    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
Sep 16 02:23:27.248061 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:23:27.248030    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\""
Sep 16 02:23:27.248177 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:23:27.248113    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
Sep 16 02:23:27.248177 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:23:27.248135    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\""
Sep 16 02:23:27.249021 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:23:27.249007    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-96crj\""
Sep 16 02:23:27.252880 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:23:27.252862    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-6f86d6bc5-hqmmj"]
Sep 16 02:23:27.353204 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:23:27.353164    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/4d013d76-d01d-4ac8-9baf-7809fa1c9ede-tmp\") pod \"kube-auth-proxy-6f86d6bc5-hqmmj\" (UID: \"4d013d76-d01d-4ac8-9baf-7809fa1c9ede\") " pod="openshift-ingress/kube-auth-proxy-6f86d6bc5-hqmmj"
Sep 16 02:23:27.353397 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:23:27.353225    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4d013d76-d01d-4ac8-9baf-7809fa1c9ede-tls-certs\") pod \"kube-auth-proxy-6f86d6bc5-hqmmj\" (UID: \"4d013d76-d01d-4ac8-9baf-7809fa1c9ede\") " pod="openshift-ingress/kube-auth-proxy-6f86d6bc5-hqmmj"
Sep 16 02:23:27.353397 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:23:27.353287    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f75kb\" (UniqueName: \"kubernetes.io/projected/4d013d76-d01d-4ac8-9baf-7809fa1c9ede-kube-api-access-f75kb\") pod \"kube-auth-proxy-6f86d6bc5-hqmmj\" (UID: \"4d013d76-d01d-4ac8-9baf-7809fa1c9ede\") " pod="openshift-ingress/kube-auth-proxy-6f86d6bc5-hqmmj"
Sep 16 02:23:27.454171 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:23:27.454136    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/4d013d76-d01d-4ac8-9baf-7809fa1c9ede-tmp\") pod \"kube-auth-proxy-6f86d6bc5-hqmmj\" (UID: \"4d013d76-d01d-4ac8-9baf-7809fa1c9ede\") " pod="openshift-ingress/kube-auth-proxy-6f86d6bc5-hqmmj"
Sep 16 02:23:27.454171 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:23:27.454174    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4d013d76-d01d-4ac8-9baf-7809fa1c9ede-tls-certs\") pod \"kube-auth-proxy-6f86d6bc5-hqmmj\" (UID: \"4d013d76-d01d-4ac8-9baf-7809fa1c9ede\") " pod="openshift-ingress/kube-auth-proxy-6f86d6bc5-hqmmj"
Sep 16 02:23:27.454418 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:23:27.454193    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f75kb\" (UniqueName: \"kubernetes.io/projected/4d013d76-d01d-4ac8-9baf-7809fa1c9ede-kube-api-access-f75kb\") pod \"kube-auth-proxy-6f86d6bc5-hqmmj\" (UID: \"4d013d76-d01d-4ac8-9baf-7809fa1c9ede\") " pod="openshift-ingress/kube-auth-proxy-6f86d6bc5-hqmmj"
Sep 16 02:23:27.454418 ip-10-0-128-111 kubenswrapper[2583]: E0916 02:23:27.454304    2583 secret.go:189] Couldn't get secret openshift-ingress/kube-auth-proxy-tls: secret "kube-auth-proxy-tls" not found
Sep 16 02:23:27.454418 ip-10-0-128-111 kubenswrapper[2583]: E0916 02:23:27.454387    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4d013d76-d01d-4ac8-9baf-7809fa1c9ede-tls-certs podName:4d013d76-d01d-4ac8-9baf-7809fa1c9ede nodeName:}" failed. No retries permitted until 2026-09-16 02:23:27.954367199 +0000 UTC m=+286.422155962 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/4d013d76-d01d-4ac8-9baf-7809fa1c9ede-tls-certs") pod "kube-auth-proxy-6f86d6bc5-hqmmj" (UID: "4d013d76-d01d-4ac8-9baf-7809fa1c9ede") : secret "kube-auth-proxy-tls" not found
Sep 16 02:23:27.456262 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:23:27.456243    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/4d013d76-d01d-4ac8-9baf-7809fa1c9ede-tmp\") pod \"kube-auth-proxy-6f86d6bc5-hqmmj\" (UID: \"4d013d76-d01d-4ac8-9baf-7809fa1c9ede\") " pod="openshift-ingress/kube-auth-proxy-6f86d6bc5-hqmmj"
Sep 16 02:23:27.463139 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:23:27.463117    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-f75kb\" (UniqueName: \"kubernetes.io/projected/4d013d76-d01d-4ac8-9baf-7809fa1c9ede-kube-api-access-f75kb\") pod \"kube-auth-proxy-6f86d6bc5-hqmmj\" (UID: \"4d013d76-d01d-4ac8-9baf-7809fa1c9ede\") " pod="openshift-ingress/kube-auth-proxy-6f86d6bc5-hqmmj"
Sep 16 02:23:27.958041 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:23:27.958007    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4d013d76-d01d-4ac8-9baf-7809fa1c9ede-tls-certs\") pod \"kube-auth-proxy-6f86d6bc5-hqmmj\" (UID: \"4d013d76-d01d-4ac8-9baf-7809fa1c9ede\") " pod="openshift-ingress/kube-auth-proxy-6f86d6bc5-hqmmj"
Sep 16 02:23:27.960360 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:23:27.960340    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4d013d76-d01d-4ac8-9baf-7809fa1c9ede-tls-certs\") pod \"kube-auth-proxy-6f86d6bc5-hqmmj\" (UID: \"4d013d76-d01d-4ac8-9baf-7809fa1c9ede\") " pod="openshift-ingress/kube-auth-proxy-6f86d6bc5-hqmmj"
Sep 16 02:23:28.155406 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:23:28.155365    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-6f86d6bc5-hqmmj"
Sep 16 02:23:28.426189 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:23:28.426138    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-6f86d6bc5-hqmmj"]
Sep 16 02:23:28.429857 ip-10-0-128-111 kubenswrapper[2583]: W0916 02:23:28.429820    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4d013d76_d01d_4ac8_9baf_7809fa1c9ede.slice/crio-11e0590006b06a8e617b1702d76eef78d3e76274694b352468f0472d4d72c323 WatchSource:0}: Error finding container 11e0590006b06a8e617b1702d76eef78d3e76274694b352468f0472d4d72c323: Status 404 returned error can't find the container with id 11e0590006b06a8e617b1702d76eef78d3e76274694b352468f0472d4d72c323
Sep 16 02:23:29.385821 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:23:29.385787    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-6f86d6bc5-hqmmj" event={"ID":"4d013d76-d01d-4ac8-9baf-7809fa1c9ede","Type":"ContainerStarted","Data":"11e0590006b06a8e617b1702d76eef78d3e76274694b352468f0472d4d72c323"}
Sep 16 02:23:31.396457 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:23:31.396421    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-6f86d6bc5-hqmmj" event={"ID":"4d013d76-d01d-4ac8-9baf-7809fa1c9ede","Type":"ContainerStarted","Data":"449448aaa1c00a97ea21a3684e4623913462d15657fd56a472761b388c509c97"}
Sep 16 02:23:31.411402 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:23:31.411352    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-6f86d6bc5-hqmmj" podStartSLOduration=1.576352996 podStartE2EDuration="4.411335774s" totalImagesPullingTime="2.834982778s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 02:23:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 02:23:31.410320142 +0000 UTC m=+289.878108924" watchObservedRunningTime="2026-09-16 02:23:31.411335774 +0000 UTC m=+289.879124609"
Sep 16 02:23:41.940358 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:23:41.940327    2583 kubelet.go:1648] "Image garbage collection succeeded"
Sep 16 02:23:47.099071 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:23:47.099037    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-clzwd"]
Sep 16 02:23:47.106498 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:23:47.106469    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-d6f4b65b8-clzwd"
Sep 16 02:23:47.109112 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:23:47.109089    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-webhook-server-cert\""
Sep 16 02:23:47.109671 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:23:47.109651    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-controller-manager-dockercfg-hf8gt\""
Sep 16 02:23:47.114527 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:23:47.114505    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-clzwd"]
Sep 16 02:23:47.222301 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:23:47.222259    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lr7bk\" (UniqueName: \"kubernetes.io/projected/fc012f8a-f2bc-4ac6-97ee-1a1604b82c9c-kube-api-access-lr7bk\") pod \"kserve-controller-manager-d6f4b65b8-clzwd\" (UID: \"fc012f8a-f2bc-4ac6-97ee-1a1604b82c9c\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-clzwd"
Sep 16 02:23:47.222488 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:23:47.222318    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/fc012f8a-f2bc-4ac6-97ee-1a1604b82c9c-cert\") pod \"kserve-controller-manager-d6f4b65b8-clzwd\" (UID: \"fc012f8a-f2bc-4ac6-97ee-1a1604b82c9c\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-clzwd"
Sep 16 02:23:47.322784 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:23:47.322743    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lr7bk\" (UniqueName: \"kubernetes.io/projected/fc012f8a-f2bc-4ac6-97ee-1a1604b82c9c-kube-api-access-lr7bk\") pod \"kserve-controller-manager-d6f4b65b8-clzwd\" (UID: \"fc012f8a-f2bc-4ac6-97ee-1a1604b82c9c\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-clzwd"
Sep 16 02:23:47.322983 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:23:47.322800    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/fc012f8a-f2bc-4ac6-97ee-1a1604b82c9c-cert\") pod \"kserve-controller-manager-d6f4b65b8-clzwd\" (UID: \"fc012f8a-f2bc-4ac6-97ee-1a1604b82c9c\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-clzwd"
Sep 16 02:23:47.325259 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:23:47.325233    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/fc012f8a-f2bc-4ac6-97ee-1a1604b82c9c-cert\") pod \"kserve-controller-manager-d6f4b65b8-clzwd\" (UID: \"fc012f8a-f2bc-4ac6-97ee-1a1604b82c9c\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-clzwd"
Sep 16 02:23:47.335072 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:23:47.335040    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lr7bk\" (UniqueName: \"kubernetes.io/projected/fc012f8a-f2bc-4ac6-97ee-1a1604b82c9c-kube-api-access-lr7bk\") pod \"kserve-controller-manager-d6f4b65b8-clzwd\" (UID: \"fc012f8a-f2bc-4ac6-97ee-1a1604b82c9c\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-clzwd"
Sep 16 02:23:47.417854 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:23:47.417760    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-d6f4b65b8-clzwd"
Sep 16 02:23:47.691436 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:23:47.691410    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-clzwd"]
Sep 16 02:23:47.693529 ip-10-0-128-111 kubenswrapper[2583]: W0916 02:23:47.693503    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfc012f8a_f2bc_4ac6_97ee_1a1604b82c9c.slice/crio-0c3ead99f008cfc7ec4415709fa91125a5c8d0721232da5bba36d3af000e4980 WatchSource:0}: Error finding container 0c3ead99f008cfc7ec4415709fa91125a5c8d0721232da5bba36d3af000e4980: Status 404 returned error can't find the container with id 0c3ead99f008cfc7ec4415709fa91125a5c8d0721232da5bba36d3af000e4980
Sep 16 02:23:47.695013 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:23:47.694991    2583 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 16 02:23:48.472959 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:23:48.472921    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-d6f4b65b8-clzwd" event={"ID":"fc012f8a-f2bc-4ac6-97ee-1a1604b82c9c","Type":"ContainerStarted","Data":"0c3ead99f008cfc7ec4415709fa91125a5c8d0721232da5bba36d3af000e4980"}
Sep 16 02:23:50.484432 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:23:50.484389    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-d6f4b65b8-clzwd" event={"ID":"fc012f8a-f2bc-4ac6-97ee-1a1604b82c9c","Type":"ContainerStarted","Data":"8e3a5f97d3b5b8491d4d8cf65baf1c1aae6a9122f0674e4deb525725f29e6beb"}
Sep 16 02:23:50.484838 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:23:50.484463    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/kserve-controller-manager-d6f4b65b8-clzwd"
Sep 16 02:23:50.499116 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:23:50.499066    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/kserve-controller-manager-d6f4b65b8-clzwd" podStartSLOduration=1.134276256 podStartE2EDuration="3.499050962s" totalImagesPullingTime="2.364774706s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 02:23:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 02:23:50.49894882 +0000 UTC m=+308.966737602" watchObservedRunningTime="2026-09-16 02:23:50.499050962 +0000 UTC m=+308.966839744"
Sep 16 02:24:21.495424 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:24:21.495392    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/kserve-controller-manager-d6f4b65b8-clzwd"
Sep 16 02:25:21.087520 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:25:21.087485    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-ntk9n"]
Sep 16 02:25:21.090713 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:25:21.090691    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-ntk9n"
Sep 16 02:25:21.093495 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:25:21.093476    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-vbcj6\""
Sep 16 02:25:21.093623 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:25:21.093515    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\""
Sep 16 02:25:21.093623 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:25:21.093515    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\""
Sep 16 02:25:21.110661 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:25:21.110640    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-ntk9n"]
Sep 16 02:25:21.185544 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:25:21.185511    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/ebe7755a-2856-4656-8c42-84b0549957e4-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-67c5dcb48c-ntk9n\" (UID: \"ebe7755a-2856-4656-8c42-84b0549957e4\") " pod="kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-ntk9n"
Sep 16 02:25:21.185703 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:25:21.185572    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2427p\" (UniqueName: \"kubernetes.io/projected/ebe7755a-2856-4656-8c42-84b0549957e4-kube-api-access-2427p\") pod \"kuadrant-operator-controller-manager-67c5dcb48c-ntk9n\" (UID: \"ebe7755a-2856-4656-8c42-84b0549957e4\") " pod="kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-ntk9n"
Sep 16 02:25:21.286247 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:25:21.286211    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/ebe7755a-2856-4656-8c42-84b0549957e4-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-67c5dcb48c-ntk9n\" (UID: \"ebe7755a-2856-4656-8c42-84b0549957e4\") " pod="kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-ntk9n"
Sep 16 02:25:21.286247 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:25:21.286252    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2427p\" (UniqueName: \"kubernetes.io/projected/ebe7755a-2856-4656-8c42-84b0549957e4-kube-api-access-2427p\") pod \"kuadrant-operator-controller-manager-67c5dcb48c-ntk9n\" (UID: \"ebe7755a-2856-4656-8c42-84b0549957e4\") " pod="kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-ntk9n"
Sep 16 02:25:21.286669 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:25:21.286648    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/ebe7755a-2856-4656-8c42-84b0549957e4-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-67c5dcb48c-ntk9n\" (UID: \"ebe7755a-2856-4656-8c42-84b0549957e4\") " pod="kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-ntk9n"
Sep 16 02:25:21.301965 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:25:21.301936    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2427p\" (UniqueName: \"kubernetes.io/projected/ebe7755a-2856-4656-8c42-84b0549957e4-kube-api-access-2427p\") pod \"kuadrant-operator-controller-manager-67c5dcb48c-ntk9n\" (UID: \"ebe7755a-2856-4656-8c42-84b0549957e4\") " pod="kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-ntk9n"
Sep 16 02:25:21.400097 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:25:21.400021    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-ntk9n"
Sep 16 02:25:21.677077 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:25:21.676985    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-ntk9n"]
Sep 16 02:25:21.680663 ip-10-0-128-111 kubenswrapper[2583]: W0916 02:25:21.680629    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podebe7755a_2856_4656_8c42_84b0549957e4.slice/crio-5e0e230e68074b5ea2de5ce51a0ee32f011f83dacd0c608cfae71fe367a20e68 WatchSource:0}: Error finding container 5e0e230e68074b5ea2de5ce51a0ee32f011f83dacd0c608cfae71fe367a20e68: Status 404 returned error can't find the container with id 5e0e230e68074b5ea2de5ce51a0ee32f011f83dacd0c608cfae71fe367a20e68
Sep 16 02:25:21.858286 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:25:21.858249    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-ntk9n" event={"ID":"ebe7755a-2856-4656-8c42-84b0549957e4","Type":"ContainerStarted","Data":"5e0e230e68074b5ea2de5ce51a0ee32f011f83dacd0c608cfae71fe367a20e68"}
Sep 16 02:25:22.320657 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:25:22.320597    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-ntk9n"]
Sep 16 02:25:22.323485 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:25:22.323450    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-ntk9n"]
Sep 16 02:25:22.345649 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:25:22.344968    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-swlpk"]
Sep 16 02:25:22.359248 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:25:22.358660    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-swlpk"
Sep 16 02:25:22.361441 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:25:22.361413    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-swlpk"]
Sep 16 02:25:22.496605 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:25:22.496545    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/476a0983-a969-4694-b08c-edab91407bb9-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-67c5dcb48c-swlpk\" (UID: \"476a0983-a969-4694-b08c-edab91407bb9\") " pod="kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-swlpk"
Sep 16 02:25:22.496862 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:25:22.496615    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hm8tz\" (UniqueName: \"kubernetes.io/projected/476a0983-a969-4694-b08c-edab91407bb9-kube-api-access-hm8tz\") pod \"kuadrant-operator-controller-manager-67c5dcb48c-swlpk\" (UID: \"476a0983-a969-4694-b08c-edab91407bb9\") " pod="kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-swlpk"
Sep 16 02:25:22.597738 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:25:22.597698    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/476a0983-a969-4694-b08c-edab91407bb9-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-67c5dcb48c-swlpk\" (UID: \"476a0983-a969-4694-b08c-edab91407bb9\") " pod="kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-swlpk"
Sep 16 02:25:22.598035 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:25:22.597747    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hm8tz\" (UniqueName: \"kubernetes.io/projected/476a0983-a969-4694-b08c-edab91407bb9-kube-api-access-hm8tz\") pod \"kuadrant-operator-controller-manager-67c5dcb48c-swlpk\" (UID: \"476a0983-a969-4694-b08c-edab91407bb9\") " pod="kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-swlpk"
Sep 16 02:25:22.598225 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:25:22.598191    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/476a0983-a969-4694-b08c-edab91407bb9-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-67c5dcb48c-swlpk\" (UID: \"476a0983-a969-4694-b08c-edab91407bb9\") " pod="kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-swlpk"
Sep 16 02:25:22.619737 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:25:22.619688    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hm8tz\" (UniqueName: \"kubernetes.io/projected/476a0983-a969-4694-b08c-edab91407bb9-kube-api-access-hm8tz\") pod \"kuadrant-operator-controller-manager-67c5dcb48c-swlpk\" (UID: \"476a0983-a969-4694-b08c-edab91407bb9\") " pod="kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-swlpk"
Sep 16 02:25:22.675326 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:25:22.675192    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-swlpk"
Sep 16 02:25:22.967054 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:25:22.967015    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-swlpk"]
Sep 16 02:25:23.594749 ip-10-0-128-111 kubenswrapper[2583]: W0916 02:25:23.594719    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod476a0983_a969_4694_b08c_edab91407bb9.slice/crio-e3ffdb1f50a9f6637a0ab62cae2500a2819836b4328bc271c1fc8f9bfc03571d WatchSource:0}: Error finding container e3ffdb1f50a9f6637a0ab62cae2500a2819836b4328bc271c1fc8f9bfc03571d: Status 404 returned error can't find the container with id e3ffdb1f50a9f6637a0ab62cae2500a2819836b4328bc271c1fc8f9bfc03571d
Sep 16 02:25:23.870678 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:25:23.870598    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-swlpk" event={"ID":"476a0983-a969-4694-b08c-edab91407bb9","Type":"ContainerStarted","Data":"e3ffdb1f50a9f6637a0ab62cae2500a2819836b4328bc271c1fc8f9bfc03571d"}
Sep 16 02:25:25.881103 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:25:25.881068    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-swlpk" event={"ID":"476a0983-a969-4694-b08c-edab91407bb9","Type":"ContainerStarted","Data":"03fb66fe7afa88c5d2d0bbdb5be1df7323cec81790631902b4ab2c2632882a2e"}
Sep 16 02:25:25.881596 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:25:25.881141    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-swlpk"
Sep 16 02:25:25.882827 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:25:25.882792    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-ntk9n" podUID="ebe7755a-2856-4656-8c42-84b0549957e4" containerName="manager" containerID="cri-o://b2be4cafb739f3062bddd02d813258a47fc94a63cbce9ddfb45025c27d7ace8e" gracePeriod=2
Sep 16 02:25:25.900617 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:25:25.900538    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-swlpk" podStartSLOduration=2.185423386 podStartE2EDuration="3.900521105s" totalImagesPullingTime="1.715097719s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 02:25:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 02:25:25.898541371 +0000 UTC m=+404.366330152" watchObservedRunningTime="2026-09-16 02:25:25.900521105 +0000 UTC m=+404.368309887"
Sep 16 02:25:26.125728 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:25:26.125703    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-ntk9n"
Sep 16 02:25:26.230137 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:25:26.230057    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/ebe7755a-2856-4656-8c42-84b0549957e4-extensions-socket-volume\") pod \"ebe7755a-2856-4656-8c42-84b0549957e4\" (UID: \"ebe7755a-2856-4656-8c42-84b0549957e4\") "
Sep 16 02:25:26.230137 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:25:26.230099    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2427p\" (UniqueName: \"kubernetes.io/projected/ebe7755a-2856-4656-8c42-84b0549957e4-kube-api-access-2427p\") pod \"ebe7755a-2856-4656-8c42-84b0549957e4\" (UID: \"ebe7755a-2856-4656-8c42-84b0549957e4\") "
Sep 16 02:25:26.230410 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:25:26.230360    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ebe7755a-2856-4656-8c42-84b0549957e4-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "ebe7755a-2856-4656-8c42-84b0549957e4" (UID: "ebe7755a-2856-4656-8c42-84b0549957e4"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 02:25:26.232497 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:25:26.232471    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ebe7755a-2856-4656-8c42-84b0549957e4-kube-api-access-2427p" (OuterVolumeSpecName: "kube-api-access-2427p") pod "ebe7755a-2856-4656-8c42-84b0549957e4" (UID: "ebe7755a-2856-4656-8c42-84b0549957e4"). InnerVolumeSpecName "kube-api-access-2427p". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 02:25:26.331064 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:25:26.331031    2583 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/ebe7755a-2856-4656-8c42-84b0549957e4-extensions-socket-volume\") on node \"ip-10-0-128-111.ec2.internal\" DevicePath \"\""
Sep 16 02:25:26.331064 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:25:26.331059    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2427p\" (UniqueName: \"kubernetes.io/projected/ebe7755a-2856-4656-8c42-84b0549957e4-kube-api-access-2427p\") on node \"ip-10-0-128-111.ec2.internal\" DevicePath \"\""
Sep 16 02:25:26.889055 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:25:26.889023    2583 generic.go:356] "Generic (PLEG): container finished" podID="ebe7755a-2856-4656-8c42-84b0549957e4" containerID="b2be4cafb739f3062bddd02d813258a47fc94a63cbce9ddfb45025c27d7ace8e" exitCode=2
Sep 16 02:25:26.889497 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:25:26.889065    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-ntk9n"
Sep 16 02:25:26.889497 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:25:26.889115    2583 scope.go:117] "RemoveContainer" containerID="b2be4cafb739f3062bddd02d813258a47fc94a63cbce9ddfb45025c27d7ace8e"
Sep 16 02:25:26.892144 ip-10-0-128-111 kubenswrapper[2583]: E0916 02:25:26.892115    2583 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-67c5dcb48c-ntk9n\" is forbidden: User \"system:node:ip-10-0-128-111.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-128-111.ec2.internal' and this object" podUID="ebe7755a-2856-4656-8c42-84b0549957e4" pod="kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-ntk9n"
Sep 16 02:25:26.897809 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:25:26.897792    2583 scope.go:117] "RemoveContainer" containerID="b2be4cafb739f3062bddd02d813258a47fc94a63cbce9ddfb45025c27d7ace8e"
Sep 16 02:25:26.898079 ip-10-0-128-111 kubenswrapper[2583]: E0916 02:25:26.898062    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b2be4cafb739f3062bddd02d813258a47fc94a63cbce9ddfb45025c27d7ace8e\": container with ID starting with b2be4cafb739f3062bddd02d813258a47fc94a63cbce9ddfb45025c27d7ace8e not found: ID does not exist" containerID="b2be4cafb739f3062bddd02d813258a47fc94a63cbce9ddfb45025c27d7ace8e"
Sep 16 02:25:26.898129 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:25:26.898087    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b2be4cafb739f3062bddd02d813258a47fc94a63cbce9ddfb45025c27d7ace8e"} err="failed to get container status \"b2be4cafb739f3062bddd02d813258a47fc94a63cbce9ddfb45025c27d7ace8e\": rpc error: code = NotFound desc = could not find container \"b2be4cafb739f3062bddd02d813258a47fc94a63cbce9ddfb45025c27d7ace8e\": container with ID starting with b2be4cafb739f3062bddd02d813258a47fc94a63cbce9ddfb45025c27d7ace8e not found: ID does not exist"
Sep 16 02:25:26.899862 ip-10-0-128-111 kubenswrapper[2583]: E0916 02:25:26.899845    2583 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-67c5dcb48c-ntk9n\" is forbidden: User \"system:node:ip-10-0-128-111.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-128-111.ec2.internal' and this object" podUID="ebe7755a-2856-4656-8c42-84b0549957e4" pod="kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-ntk9n"
Sep 16 02:25:28.044336 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:25:28.044297    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ebe7755a-2856-4656-8c42-84b0549957e4" path="/var/lib/kubelet/pods/ebe7755a-2856-4656-8c42-84b0549957e4/volumes"
Sep 16 02:25:36.892250 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:25:36.892217    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-swlpk"
Sep 16 02:25:49.284602 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:25:49.284569    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-swlpk"]
Sep 16 02:25:49.285049 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:25:49.284940    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-swlpk" podUID="476a0983-a969-4694-b08c-edab91407bb9" containerName="manager" containerID="cri-o://03fb66fe7afa88c5d2d0bbdb5be1df7323cec81790631902b4ab2c2632882a2e" gracePeriod=10
Sep 16 02:25:49.529311 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:25:49.529288    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-swlpk"
Sep 16 02:25:49.722571 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:25:49.722514    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/476a0983-a969-4694-b08c-edab91407bb9-extensions-socket-volume\") pod \"476a0983-a969-4694-b08c-edab91407bb9\" (UID: \"476a0983-a969-4694-b08c-edab91407bb9\") "
Sep 16 02:25:49.722736 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:25:49.722581    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hm8tz\" (UniqueName: \"kubernetes.io/projected/476a0983-a969-4694-b08c-edab91407bb9-kube-api-access-hm8tz\") pod \"476a0983-a969-4694-b08c-edab91407bb9\" (UID: \"476a0983-a969-4694-b08c-edab91407bb9\") "
Sep 16 02:25:49.722968 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:25:49.722942    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/476a0983-a969-4694-b08c-edab91407bb9-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "476a0983-a969-4694-b08c-edab91407bb9" (UID: "476a0983-a969-4694-b08c-edab91407bb9"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 02:25:49.724748 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:25:49.724727    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/476a0983-a969-4694-b08c-edab91407bb9-kube-api-access-hm8tz" (OuterVolumeSpecName: "kube-api-access-hm8tz") pod "476a0983-a969-4694-b08c-edab91407bb9" (UID: "476a0983-a969-4694-b08c-edab91407bb9"). InnerVolumeSpecName "kube-api-access-hm8tz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 02:25:49.823841 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:25:49.823807    2583 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/476a0983-a969-4694-b08c-edab91407bb9-extensions-socket-volume\") on node \"ip-10-0-128-111.ec2.internal\" DevicePath \"\""
Sep 16 02:25:49.823841 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:25:49.823835    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hm8tz\" (UniqueName: \"kubernetes.io/projected/476a0983-a969-4694-b08c-edab91407bb9-kube-api-access-hm8tz\") on node \"ip-10-0-128-111.ec2.internal\" DevicePath \"\""
Sep 16 02:25:49.991957 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:25:49.991876    2583 generic.go:356] "Generic (PLEG): container finished" podID="476a0983-a969-4694-b08c-edab91407bb9" containerID="03fb66fe7afa88c5d2d0bbdb5be1df7323cec81790631902b4ab2c2632882a2e" exitCode=0
Sep 16 02:25:49.991957 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:25:49.991918    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-swlpk"
Sep 16 02:25:49.992147 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:25:49.991959    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-swlpk" event={"ID":"476a0983-a969-4694-b08c-edab91407bb9","Type":"ContainerDied","Data":"03fb66fe7afa88c5d2d0bbdb5be1df7323cec81790631902b4ab2c2632882a2e"}
Sep 16 02:25:49.992147 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:25:49.992011    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-swlpk" event={"ID":"476a0983-a969-4694-b08c-edab91407bb9","Type":"ContainerDied","Data":"e3ffdb1f50a9f6637a0ab62cae2500a2819836b4328bc271c1fc8f9bfc03571d"}
Sep 16 02:25:49.992147 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:25:49.992030    2583 scope.go:117] "RemoveContainer" containerID="03fb66fe7afa88c5d2d0bbdb5be1df7323cec81790631902b4ab2c2632882a2e"
Sep 16 02:25:50.000352 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:25:50.000337    2583 scope.go:117] "RemoveContainer" containerID="03fb66fe7afa88c5d2d0bbdb5be1df7323cec81790631902b4ab2c2632882a2e"
Sep 16 02:25:50.000618 ip-10-0-128-111 kubenswrapper[2583]: E0916 02:25:50.000595    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"03fb66fe7afa88c5d2d0bbdb5be1df7323cec81790631902b4ab2c2632882a2e\": container with ID starting with 03fb66fe7afa88c5d2d0bbdb5be1df7323cec81790631902b4ab2c2632882a2e not found: ID does not exist" containerID="03fb66fe7afa88c5d2d0bbdb5be1df7323cec81790631902b4ab2c2632882a2e"
Sep 16 02:25:50.000712 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:25:50.000626    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"03fb66fe7afa88c5d2d0bbdb5be1df7323cec81790631902b4ab2c2632882a2e"} err="failed to get container status \"03fb66fe7afa88c5d2d0bbdb5be1df7323cec81790631902b4ab2c2632882a2e\": rpc error: code = NotFound desc = could not find container \"03fb66fe7afa88c5d2d0bbdb5be1df7323cec81790631902b4ab2c2632882a2e\": container with ID starting with 03fb66fe7afa88c5d2d0bbdb5be1df7323cec81790631902b4ab2c2632882a2e not found: ID does not exist"
Sep 16 02:25:50.012762 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:25:50.012740    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-swlpk"]
Sep 16 02:25:50.020782 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:25:50.020765    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-swlpk"]
Sep 16 02:25:50.044748 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:25:50.044727    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="476a0983-a969-4694-b08c-edab91407bb9" path="/var/lib/kubelet/pods/476a0983-a969-4694-b08c-edab91407bb9/volumes"
Sep 16 02:25:53.317937 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:25:53.317908    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-xjmsh"]
Sep 16 02:25:53.318408 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:25:53.318248    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="476a0983-a969-4694-b08c-edab91407bb9" containerName="manager"
Sep 16 02:25:53.318408 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:25:53.318261    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="476a0983-a969-4694-b08c-edab91407bb9" containerName="manager"
Sep 16 02:25:53.318408 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:25:53.318270    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ebe7755a-2856-4656-8c42-84b0549957e4" containerName="manager"
Sep 16 02:25:53.318408 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:25:53.318275    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="ebe7755a-2856-4656-8c42-84b0549957e4" containerName="manager"
Sep 16 02:25:53.318408 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:25:53.318357    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="476a0983-a969-4694-b08c-edab91407bb9" containerName="manager"
Sep 16 02:25:53.318408 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:25:53.318373    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="ebe7755a-2856-4656-8c42-84b0549957e4" containerName="manager"
Sep 16 02:25:53.322791 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:25:53.322772    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-xjmsh"
Sep 16 02:25:53.325454 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:25:53.325423    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\""
Sep 16 02:25:53.325583 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:25:53.325520    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-dt7st\""
Sep 16 02:25:53.335328 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:25:53.335306    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-xjmsh"]
Sep 16 02:25:53.356269 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:25:53.356247    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/53ca0d14-d5b8-422c-a044-ab3471e584c9-istio-envoy\") pod \"maas-default-gateway-openshift-default-5ccb968c66-xjmsh\" (UID: \"53ca0d14-d5b8-422c-a044-ab3471e584c9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-xjmsh"
Sep 16 02:25:53.356391 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:25:53.356284    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/53ca0d14-d5b8-422c-a044-ab3471e584c9-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5ccb968c66-xjmsh\" (UID: \"53ca0d14-d5b8-422c-a044-ab3471e584c9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-xjmsh"
Sep 16 02:25:53.356391 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:25:53.356319    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/53ca0d14-d5b8-422c-a044-ab3471e584c9-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5ccb968c66-xjmsh\" (UID: \"53ca0d14-d5b8-422c-a044-ab3471e584c9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-xjmsh"
Sep 16 02:25:53.356513 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:25:53.356418    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/53ca0d14-d5b8-422c-a044-ab3471e584c9-workload-certs\") pod \"maas-default-gateway-openshift-default-5ccb968c66-xjmsh\" (UID: \"53ca0d14-d5b8-422c-a044-ab3471e584c9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-xjmsh"
Sep 16 02:25:53.356513 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:25:53.356451    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/53ca0d14-d5b8-422c-a044-ab3471e584c9-istio-token\") pod \"maas-default-gateway-openshift-default-5ccb968c66-xjmsh\" (UID: \"53ca0d14-d5b8-422c-a044-ab3471e584c9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-xjmsh"
Sep 16 02:25:53.356513 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:25:53.356498    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h9f2w\" (UniqueName: \"kubernetes.io/projected/53ca0d14-d5b8-422c-a044-ab3471e584c9-kube-api-access-h9f2w\") pod \"maas-default-gateway-openshift-default-5ccb968c66-xjmsh\" (UID: \"53ca0d14-d5b8-422c-a044-ab3471e584c9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-xjmsh"
Sep 16 02:25:53.356655 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:25:53.356527    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/53ca0d14-d5b8-422c-a044-ab3471e584c9-istio-data\") pod \"maas-default-gateway-openshift-default-5ccb968c66-xjmsh\" (UID: \"53ca0d14-d5b8-422c-a044-ab3471e584c9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-xjmsh"
Sep 16 02:25:53.356655 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:25:53.356596    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/53ca0d14-d5b8-422c-a044-ab3471e584c9-workload-socket\") pod \"maas-default-gateway-openshift-default-5ccb968c66-xjmsh\" (UID: \"53ca0d14-d5b8-422c-a044-ab3471e584c9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-xjmsh"
Sep 16 02:25:53.356655 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:25:53.356620    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/53ca0d14-d5b8-422c-a044-ab3471e584c9-credential-socket\") pod \"maas-default-gateway-openshift-default-5ccb968c66-xjmsh\" (UID: \"53ca0d14-d5b8-422c-a044-ab3471e584c9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-xjmsh"
Sep 16 02:25:53.457830 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:25:53.457794    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/53ca0d14-d5b8-422c-a044-ab3471e584c9-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5ccb968c66-xjmsh\" (UID: \"53ca0d14-d5b8-422c-a044-ab3471e584c9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-xjmsh"
Sep 16 02:25:53.457975 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:25:53.457858    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/53ca0d14-d5b8-422c-a044-ab3471e584c9-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5ccb968c66-xjmsh\" (UID: \"53ca0d14-d5b8-422c-a044-ab3471e584c9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-xjmsh"
Sep 16 02:25:53.457975 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:25:53.457925    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/53ca0d14-d5b8-422c-a044-ab3471e584c9-workload-certs\") pod \"maas-default-gateway-openshift-default-5ccb968c66-xjmsh\" (UID: \"53ca0d14-d5b8-422c-a044-ab3471e584c9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-xjmsh"
Sep 16 02:25:53.457975 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:25:53.457952    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/53ca0d14-d5b8-422c-a044-ab3471e584c9-istio-token\") pod \"maas-default-gateway-openshift-default-5ccb968c66-xjmsh\" (UID: \"53ca0d14-d5b8-422c-a044-ab3471e584c9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-xjmsh"
Sep 16 02:25:53.458143 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:25:53.457992    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h9f2w\" (UniqueName: \"kubernetes.io/projected/53ca0d14-d5b8-422c-a044-ab3471e584c9-kube-api-access-h9f2w\") pod \"maas-default-gateway-openshift-default-5ccb968c66-xjmsh\" (UID: \"53ca0d14-d5b8-422c-a044-ab3471e584c9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-xjmsh"
Sep 16 02:25:53.458143 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:25:53.458020    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/53ca0d14-d5b8-422c-a044-ab3471e584c9-istio-data\") pod \"maas-default-gateway-openshift-default-5ccb968c66-xjmsh\" (UID: \"53ca0d14-d5b8-422c-a044-ab3471e584c9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-xjmsh"
Sep 16 02:25:53.458143 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:25:53.458063    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/53ca0d14-d5b8-422c-a044-ab3471e584c9-workload-socket\") pod \"maas-default-gateway-openshift-default-5ccb968c66-xjmsh\" (UID: \"53ca0d14-d5b8-422c-a044-ab3471e584c9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-xjmsh"
Sep 16 02:25:53.458143 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:25:53.458104    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/53ca0d14-d5b8-422c-a044-ab3471e584c9-credential-socket\") pod \"maas-default-gateway-openshift-default-5ccb968c66-xjmsh\" (UID: \"53ca0d14-d5b8-422c-a044-ab3471e584c9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-xjmsh"
Sep 16 02:25:53.458349 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:25:53.458148    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/53ca0d14-d5b8-422c-a044-ab3471e584c9-istio-envoy\") pod \"maas-default-gateway-openshift-default-5ccb968c66-xjmsh\" (UID: \"53ca0d14-d5b8-422c-a044-ab3471e584c9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-xjmsh"
Sep 16 02:25:53.458402 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:25:53.458361    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/53ca0d14-d5b8-422c-a044-ab3471e584c9-workload-certs\") pod \"maas-default-gateway-openshift-default-5ccb968c66-xjmsh\" (UID: \"53ca0d14-d5b8-422c-a044-ab3471e584c9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-xjmsh"
Sep 16 02:25:53.458456 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:25:53.458429    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/53ca0d14-d5b8-422c-a044-ab3471e584c9-credential-socket\") pod \"maas-default-gateway-openshift-default-5ccb968c66-xjmsh\" (UID: \"53ca0d14-d5b8-422c-a044-ab3471e584c9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-xjmsh"
Sep 16 02:25:53.458639 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:25:53.458615    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/53ca0d14-d5b8-422c-a044-ab3471e584c9-istio-data\") pod \"maas-default-gateway-openshift-default-5ccb968c66-xjmsh\" (UID: \"53ca0d14-d5b8-422c-a044-ab3471e584c9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-xjmsh"
Sep 16 02:25:53.458639 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:25:53.458631    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/53ca0d14-d5b8-422c-a044-ab3471e584c9-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5ccb968c66-xjmsh\" (UID: \"53ca0d14-d5b8-422c-a044-ab3471e584c9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-xjmsh"
Sep 16 02:25:53.458784 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:25:53.458670    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/53ca0d14-d5b8-422c-a044-ab3471e584c9-workload-socket\") pod \"maas-default-gateway-openshift-default-5ccb968c66-xjmsh\" (UID: \"53ca0d14-d5b8-422c-a044-ab3471e584c9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-xjmsh"
Sep 16 02:25:53.460772 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:25:53.460747    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/53ca0d14-d5b8-422c-a044-ab3471e584c9-istio-envoy\") pod \"maas-default-gateway-openshift-default-5ccb968c66-xjmsh\" (UID: \"53ca0d14-d5b8-422c-a044-ab3471e584c9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-xjmsh"
Sep 16 02:25:53.460937 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:25:53.460917    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/53ca0d14-d5b8-422c-a044-ab3471e584c9-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5ccb968c66-xjmsh\" (UID: \"53ca0d14-d5b8-422c-a044-ab3471e584c9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-xjmsh"
Sep 16 02:25:53.465759 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:25:53.465733    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-h9f2w\" (UniqueName: \"kubernetes.io/projected/53ca0d14-d5b8-422c-a044-ab3471e584c9-kube-api-access-h9f2w\") pod \"maas-default-gateway-openshift-default-5ccb968c66-xjmsh\" (UID: \"53ca0d14-d5b8-422c-a044-ab3471e584c9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-xjmsh"
Sep 16 02:25:53.466054 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:25:53.466033    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/53ca0d14-d5b8-422c-a044-ab3471e584c9-istio-token\") pod \"maas-default-gateway-openshift-default-5ccb968c66-xjmsh\" (UID: \"53ca0d14-d5b8-422c-a044-ab3471e584c9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-xjmsh"
Sep 16 02:25:53.632354 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:25:53.632324    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-xjmsh"
Sep 16 02:25:53.903291 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:25:53.903197    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-xjmsh"]
Sep 16 02:25:53.905368 ip-10-0-128-111 kubenswrapper[2583]: W0916 02:25:53.905343    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod53ca0d14_d5b8_422c_a044_ab3471e584c9.slice/crio-2778bf33d1663d17060df2cd5b95a809863798598d50be65a7183b4c5f7ad500 WatchSource:0}: Error finding container 2778bf33d1663d17060df2cd5b95a809863798598d50be65a7183b4c5f7ad500: Status 404 returned error can't find the container with id 2778bf33d1663d17060df2cd5b95a809863798598d50be65a7183b4c5f7ad500
Sep 16 02:25:54.010851 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:25:54.010818    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-xjmsh" event={"ID":"53ca0d14-d5b8-422c-a044-ab3471e584c9","Type":"ContainerStarted","Data":"2778bf33d1663d17060df2cd5b95a809863798598d50be65a7183b4c5f7ad500"}
Sep 16 02:25:56.579419 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:25:56.579380    2583 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 16 02:25:56.579732 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:25:56.579457    2583 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 16 02:25:56.579732 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:25:56.579488    2583 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 16 02:25:57.026720 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:25:57.026682    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-xjmsh" event={"ID":"53ca0d14-d5b8-422c-a044-ab3471e584c9","Type":"ContainerStarted","Data":"be7eb271b1cf8e744ed79b16198139b2fbc54e24872733e93077b2da24a2671a"}
Sep 16 02:25:57.045910 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:25:57.045833    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-xjmsh" podStartSLOduration=1.374095621 podStartE2EDuration="4.045813265s" totalImagesPullingTime="2.671717644s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 02:25:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 02:25:57.044185078 +0000 UTC m=+435.511973860" watchObservedRunningTime="2026-09-16 02:25:57.045813265 +0000 UTC m=+435.513602050"
Sep 16 02:25:57.632686 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:25:57.632653    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-xjmsh"
Sep 16 02:25:57.637018 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:25:57.636998    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-xjmsh"
Sep 16 02:25:58.031250 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:25:58.031165    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-xjmsh"
Sep 16 02:25:58.032051 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:25:58.032029    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-xjmsh"
Sep 16 02:26:06.733814 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:26:06.733740    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-fzl8t"]
Sep 16 02:26:06.736819 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:26:06.736800    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-84994f7ffc-fzl8t"
Sep 16 02:26:06.739320 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:26:06.739294    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\""
Sep 16 02:26:06.739415 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:26:06.739341    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\""
Sep 16 02:26:06.740365 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:26:06.740345    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-qn5l4\""
Sep 16 02:26:06.740467 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:26:06.740353    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\""
Sep 16 02:26:06.744004 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:26:06.743944    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-fzl8t"]
Sep 16 02:26:06.770841 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:26:06.770819    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/f795b100-3c16-48da-ac11-3899946ca508-config-file\") pod \"limitador-limitador-84994f7ffc-fzl8t\" (UID: \"f795b100-3c16-48da-ac11-3899946ca508\") " pod="kuadrant-system/limitador-limitador-84994f7ffc-fzl8t"
Sep 16 02:26:06.770938 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:26:06.770850    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sfx5s\" (UniqueName: \"kubernetes.io/projected/f795b100-3c16-48da-ac11-3899946ca508-kube-api-access-sfx5s\") pod \"limitador-limitador-84994f7ffc-fzl8t\" (UID: \"f795b100-3c16-48da-ac11-3899946ca508\") " pod="kuadrant-system/limitador-limitador-84994f7ffc-fzl8t"
Sep 16 02:26:06.776420 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:26:06.776398    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-fzl8t"]
Sep 16 02:26:06.871954 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:26:06.871916    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/f795b100-3c16-48da-ac11-3899946ca508-config-file\") pod \"limitador-limitador-84994f7ffc-fzl8t\" (UID: \"f795b100-3c16-48da-ac11-3899946ca508\") " pod="kuadrant-system/limitador-limitador-84994f7ffc-fzl8t"
Sep 16 02:26:06.871954 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:26:06.871957    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sfx5s\" (UniqueName: \"kubernetes.io/projected/f795b100-3c16-48da-ac11-3899946ca508-kube-api-access-sfx5s\") pod \"limitador-limitador-84994f7ffc-fzl8t\" (UID: \"f795b100-3c16-48da-ac11-3899946ca508\") " pod="kuadrant-system/limitador-limitador-84994f7ffc-fzl8t"
Sep 16 02:26:06.872659 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:26:06.872639    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/f795b100-3c16-48da-ac11-3899946ca508-config-file\") pod \"limitador-limitador-84994f7ffc-fzl8t\" (UID: \"f795b100-3c16-48da-ac11-3899946ca508\") " pod="kuadrant-system/limitador-limitador-84994f7ffc-fzl8t"
Sep 16 02:26:06.880291 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:26:06.880268    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sfx5s\" (UniqueName: \"kubernetes.io/projected/f795b100-3c16-48da-ac11-3899946ca508-kube-api-access-sfx5s\") pod \"limitador-limitador-84994f7ffc-fzl8t\" (UID: \"f795b100-3c16-48da-ac11-3899946ca508\") " pod="kuadrant-system/limitador-limitador-84994f7ffc-fzl8t"
Sep 16 02:26:07.047828 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:26:07.047738    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-84994f7ffc-fzl8t"
Sep 16 02:26:07.125179 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:26:07.125148    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-z5sf4"]
Sep 16 02:26:07.128431 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:26:07.128410    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7bcfd8f9cd-z5sf4"
Sep 16 02:26:07.130655 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:26:07.130638    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-xkv7p\""
Sep 16 02:26:07.134729 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:26:07.134707    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-z5sf4"]
Sep 16 02:26:07.173820 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:26:07.173790    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qpdg6\" (UniqueName: \"kubernetes.io/projected/090b38e8-e1a6-46fc-87c1-918347ee30d7-kube-api-access-qpdg6\") pod \"authorino-7bcfd8f9cd-z5sf4\" (UID: \"090b38e8-e1a6-46fc-87c1-918347ee30d7\") " pod="kuadrant-system/authorino-7bcfd8f9cd-z5sf4"
Sep 16 02:26:07.274573 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:26:07.274516    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qpdg6\" (UniqueName: \"kubernetes.io/projected/090b38e8-e1a6-46fc-87c1-918347ee30d7-kube-api-access-qpdg6\") pod \"authorino-7bcfd8f9cd-z5sf4\" (UID: \"090b38e8-e1a6-46fc-87c1-918347ee30d7\") " pod="kuadrant-system/authorino-7bcfd8f9cd-z5sf4"
Sep 16 02:26:07.281894 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:26:07.281865    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qpdg6\" (UniqueName: \"kubernetes.io/projected/090b38e8-e1a6-46fc-87c1-918347ee30d7-kube-api-access-qpdg6\") pod \"authorino-7bcfd8f9cd-z5sf4\" (UID: \"090b38e8-e1a6-46fc-87c1-918347ee30d7\") " pod="kuadrant-system/authorino-7bcfd8f9cd-z5sf4"
Sep 16 02:26:07.318734 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:26:07.318708    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-fzl8t"]
Sep 16 02:26:07.321200 ip-10-0-128-111 kubenswrapper[2583]: W0916 02:26:07.321174    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf795b100_3c16_48da_ac11_3899946ca508.slice/crio-5085f3bcd33a2b0d0f9f5841b4ef4dbe03cb20c0476cb4c784b5c18f4aa47c09 WatchSource:0}: Error finding container 5085f3bcd33a2b0d0f9f5841b4ef4dbe03cb20c0476cb4c784b5c18f4aa47c09: Status 404 returned error can't find the container with id 5085f3bcd33a2b0d0f9f5841b4ef4dbe03cb20c0476cb4c784b5c18f4aa47c09
Sep 16 02:26:07.438901 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:26:07.438866    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7bcfd8f9cd-z5sf4"
Sep 16 02:26:07.705502 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:26:07.705404    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-z5sf4"]
Sep 16 02:26:07.708004 ip-10-0-128-111 kubenswrapper[2583]: W0916 02:26:07.707980    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod090b38e8_e1a6_46fc_87c1_918347ee30d7.slice/crio-bbe36ede5d6a7d6da001b8a16980a44335c212672ba644e3445f81bf07a0f0b4 WatchSource:0}: Error finding container bbe36ede5d6a7d6da001b8a16980a44335c212672ba644e3445f81bf07a0f0b4: Status 404 returned error can't find the container with id bbe36ede5d6a7d6da001b8a16980a44335c212672ba644e3445f81bf07a0f0b4
Sep 16 02:26:08.075190 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:26:08.075103    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7bcfd8f9cd-z5sf4" event={"ID":"090b38e8-e1a6-46fc-87c1-918347ee30d7","Type":"ContainerStarted","Data":"bbe36ede5d6a7d6da001b8a16980a44335c212672ba644e3445f81bf07a0f0b4"}
Sep 16 02:26:08.076451 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:26:08.076427    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-84994f7ffc-fzl8t" event={"ID":"f795b100-3c16-48da-ac11-3899946ca508","Type":"ContainerStarted","Data":"5085f3bcd33a2b0d0f9f5841b4ef4dbe03cb20c0476cb4c784b5c18f4aa47c09"}
Sep 16 02:26:09.082654 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:26:09.082601    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-84994f7ffc-fzl8t" event={"ID":"f795b100-3c16-48da-ac11-3899946ca508","Type":"ContainerStarted","Data":"24b8743206f0597506043940a1c824b59a966565deff702930fecf25f5126e62"}
Sep 16 02:26:09.083056 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:26:09.082791    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-84994f7ffc-fzl8t"
Sep 16 02:26:09.098750 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:26:09.098693    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-84994f7ffc-fzl8t" podStartSLOduration=2.124259825 podStartE2EDuration="3.098673219s" totalImagesPullingTime="974.413394ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 02:26:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 02:26:09.096362418 +0000 UTC m=+447.564151199" watchObservedRunningTime="2026-09-16 02:26:09.098673219 +0000 UTC m=+447.566462002"
Sep 16 02:26:10.088353 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:26:10.088318    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7bcfd8f9cd-z5sf4" event={"ID":"090b38e8-e1a6-46fc-87c1-918347ee30d7","Type":"ContainerStarted","Data":"3df0314f5151e28227514bd87f704ee2847b31d7897d7344fd1ec1c2164f7292"}
Sep 16 02:26:10.102737 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:26:10.102691    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-7bcfd8f9cd-z5sf4" podStartSLOduration=0.998905772 podStartE2EDuration="3.102677793s" totalImagesPullingTime="2.103772021s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 02:26:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 02:26:10.101006733 +0000 UTC m=+448.568795508" watchObservedRunningTime="2026-09-16 02:26:10.102677793 +0000 UTC m=+448.570466618"
Sep 16 02:26:10.426460 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:26:10.426388    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-z5sf4"]
Sep 16 02:26:12.096512 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:26:12.096475    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-7bcfd8f9cd-z5sf4" podUID="090b38e8-e1a6-46fc-87c1-918347ee30d7" containerName="authorino" containerID="cri-o://3df0314f5151e28227514bd87f704ee2847b31d7897d7344fd1ec1c2164f7292" gracePeriod=30
Sep 16 02:26:12.332150 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:26:12.332128    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7bcfd8f9cd-z5sf4"
Sep 16 02:26:12.417368 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:26:12.417270    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qpdg6\" (UniqueName: \"kubernetes.io/projected/090b38e8-e1a6-46fc-87c1-918347ee30d7-kube-api-access-qpdg6\") pod \"090b38e8-e1a6-46fc-87c1-918347ee30d7\" (UID: \"090b38e8-e1a6-46fc-87c1-918347ee30d7\") "
Sep 16 02:26:12.419339 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:26:12.419314    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/090b38e8-e1a6-46fc-87c1-918347ee30d7-kube-api-access-qpdg6" (OuterVolumeSpecName: "kube-api-access-qpdg6") pod "090b38e8-e1a6-46fc-87c1-918347ee30d7" (UID: "090b38e8-e1a6-46fc-87c1-918347ee30d7"). InnerVolumeSpecName "kube-api-access-qpdg6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 02:26:12.518883 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:26:12.518845    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qpdg6\" (UniqueName: \"kubernetes.io/projected/090b38e8-e1a6-46fc-87c1-918347ee30d7-kube-api-access-qpdg6\") on node \"ip-10-0-128-111.ec2.internal\" DevicePath \"\""
Sep 16 02:26:13.101537 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:26:13.101503    2583 generic.go:356] "Generic (PLEG): container finished" podID="090b38e8-e1a6-46fc-87c1-918347ee30d7" containerID="3df0314f5151e28227514bd87f704ee2847b31d7897d7344fd1ec1c2164f7292" exitCode=0
Sep 16 02:26:13.101537 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:26:13.101544    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7bcfd8f9cd-z5sf4"
Sep 16 02:26:13.101985 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:26:13.101598    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7bcfd8f9cd-z5sf4" event={"ID":"090b38e8-e1a6-46fc-87c1-918347ee30d7","Type":"ContainerDied","Data":"3df0314f5151e28227514bd87f704ee2847b31d7897d7344fd1ec1c2164f7292"}
Sep 16 02:26:13.101985 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:26:13.101644    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7bcfd8f9cd-z5sf4" event={"ID":"090b38e8-e1a6-46fc-87c1-918347ee30d7","Type":"ContainerDied","Data":"bbe36ede5d6a7d6da001b8a16980a44335c212672ba644e3445f81bf07a0f0b4"}
Sep 16 02:26:13.101985 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:26:13.101664    2583 scope.go:117] "RemoveContainer" containerID="3df0314f5151e28227514bd87f704ee2847b31d7897d7344fd1ec1c2164f7292"
Sep 16 02:26:13.110020 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:26:13.110004    2583 scope.go:117] "RemoveContainer" containerID="3df0314f5151e28227514bd87f704ee2847b31d7897d7344fd1ec1c2164f7292"
Sep 16 02:26:13.110294 ip-10-0-128-111 kubenswrapper[2583]: E0916 02:26:13.110276    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3df0314f5151e28227514bd87f704ee2847b31d7897d7344fd1ec1c2164f7292\": container with ID starting with 3df0314f5151e28227514bd87f704ee2847b31d7897d7344fd1ec1c2164f7292 not found: ID does not exist" containerID="3df0314f5151e28227514bd87f704ee2847b31d7897d7344fd1ec1c2164f7292"
Sep 16 02:26:13.110338 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:26:13.110303    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3df0314f5151e28227514bd87f704ee2847b31d7897d7344fd1ec1c2164f7292"} err="failed to get container status \"3df0314f5151e28227514bd87f704ee2847b31d7897d7344fd1ec1c2164f7292\": rpc error: code = NotFound desc = could not find container \"3df0314f5151e28227514bd87f704ee2847b31d7897d7344fd1ec1c2164f7292\": container with ID starting with 3df0314f5151e28227514bd87f704ee2847b31d7897d7344fd1ec1c2164f7292 not found: ID does not exist"
Sep 16 02:26:13.120509 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:26:13.120489    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-z5sf4"]
Sep 16 02:26:13.122433 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:26:13.122414    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-z5sf4"]
Sep 16 02:26:14.044523 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:26:14.044490    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="090b38e8-e1a6-46fc-87c1-918347ee30d7" path="/var/lib/kubelet/pods/090b38e8-e1a6-46fc-87c1-918347ee30d7/volumes"
Sep 16 02:26:20.089274 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:26:20.089247    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-84994f7ffc-fzl8t"
Sep 16 02:26:38.119924 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:26:38.119887    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-557f876f69-52hqv"]
Sep 16 02:26:38.120366 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:26:38.120218    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="090b38e8-e1a6-46fc-87c1-918347ee30d7" containerName="authorino"
Sep 16 02:26:38.120366 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:26:38.120229    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="090b38e8-e1a6-46fc-87c1-918347ee30d7" containerName="authorino"
Sep 16 02:26:38.120366 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:26:38.120279    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="090b38e8-e1a6-46fc-87c1-918347ee30d7" containerName="authorino"
Sep 16 02:26:38.123359 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:26:38.123340    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-557f876f69-52hqv"
Sep 16 02:26:38.125582 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:26:38.125564    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-xkv7p\""
Sep 16 02:26:38.128726 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:26:38.128699    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-557f876f69-52hqv"]
Sep 16 02:26:38.241015 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:26:38.240978    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f94s4\" (UniqueName: \"kubernetes.io/projected/ceeeb616-b8b8-4abb-89c2-11779a35b142-kube-api-access-f94s4\") pod \"authorino-557f876f69-52hqv\" (UID: \"ceeeb616-b8b8-4abb-89c2-11779a35b142\") " pod="kuadrant-system/authorino-557f876f69-52hqv"
Sep 16 02:26:38.342177 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:26:38.342140    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f94s4\" (UniqueName: \"kubernetes.io/projected/ceeeb616-b8b8-4abb-89c2-11779a35b142-kube-api-access-f94s4\") pod \"authorino-557f876f69-52hqv\" (UID: \"ceeeb616-b8b8-4abb-89c2-11779a35b142\") " pod="kuadrant-system/authorino-557f876f69-52hqv"
Sep 16 02:26:38.352230 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:26:38.352205    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-f94s4\" (UniqueName: \"kubernetes.io/projected/ceeeb616-b8b8-4abb-89c2-11779a35b142-kube-api-access-f94s4\") pod \"authorino-557f876f69-52hqv\" (UID: \"ceeeb616-b8b8-4abb-89c2-11779a35b142\") " pod="kuadrant-system/authorino-557f876f69-52hqv"
Sep 16 02:26:38.366259 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:26:38.366232    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-557f876f69-52hqv"]
Sep 16 02:26:38.366459 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:26:38.366447    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-557f876f69-52hqv"
Sep 16 02:26:38.387630 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:26:38.387607    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-6b8db695c-l9lnp"]
Sep 16 02:26:38.391122 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:26:38.391102    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6b8db695c-l9lnp"
Sep 16 02:26:38.398357 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:26:38.398335    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-6b8db695c-l9lnp"]
Sep 16 02:26:38.432410 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:26:38.430441    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6b8db695c-l9lnp"]
Sep 16 02:26:38.432410 ip-10-0-128-111 kubenswrapper[2583]: E0916 02:26:38.431113    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kube-api-access-8h4p9], unattached volumes=[], failed to process volumes=[kube-api-access-8h4p9]: context canceled" pod="kuadrant-system/authorino-6b8db695c-l9lnp" podUID="c58f2629-5a7c-4965-958c-a5092997d8a1"
Sep 16 02:26:38.543134 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:26:38.543104    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8h4p9\" (UniqueName: \"kubernetes.io/projected/c58f2629-5a7c-4965-958c-a5092997d8a1-kube-api-access-8h4p9\") pod \"authorino-6b8db695c-l9lnp\" (UID: \"c58f2629-5a7c-4965-958c-a5092997d8a1\") " pod="kuadrant-system/authorino-6b8db695c-l9lnp"
Sep 16 02:26:38.638482 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:26:38.638394    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-557f876f69-52hqv"]
Sep 16 02:26:38.641441 ip-10-0-128-111 kubenswrapper[2583]: W0916 02:26:38.641410    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podceeeb616_b8b8_4abb_89c2_11779a35b142.slice/crio-344b26c958d419ff585b99f642fb66d82b6106f086a7bffaa563587a2b046e87 WatchSource:0}: Error finding container 344b26c958d419ff585b99f642fb66d82b6106f086a7bffaa563587a2b046e87: Status 404 returned error can't find the container with id 344b26c958d419ff585b99f642fb66d82b6106f086a7bffaa563587a2b046e87
Sep 16 02:26:38.644480 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:26:38.644452    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8h4p9\" (UniqueName: \"kubernetes.io/projected/c58f2629-5a7c-4965-958c-a5092997d8a1-kube-api-access-8h4p9\") pod \"authorino-6b8db695c-l9lnp\" (UID: \"c58f2629-5a7c-4965-958c-a5092997d8a1\") " pod="kuadrant-system/authorino-6b8db695c-l9lnp"
Sep 16 02:26:38.651436 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:26:38.651413    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8h4p9\" (UniqueName: \"kubernetes.io/projected/c58f2629-5a7c-4965-958c-a5092997d8a1-kube-api-access-8h4p9\") pod \"authorino-6b8db695c-l9lnp\" (UID: \"c58f2629-5a7c-4965-958c-a5092997d8a1\") " pod="kuadrant-system/authorino-6b8db695c-l9lnp"
Sep 16 02:26:39.214286 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:26:39.214258    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6b8db695c-l9lnp"
Sep 16 02:26:39.214692 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:26:39.214255    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-557f876f69-52hqv" event={"ID":"ceeeb616-b8b8-4abb-89c2-11779a35b142","Type":"ContainerStarted","Data":"344b26c958d419ff585b99f642fb66d82b6106f086a7bffaa563587a2b046e87"}
Sep 16 02:26:39.221515 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:26:39.221495    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6b8db695c-l9lnp"
Sep 16 02:26:39.349880 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:26:39.349857    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8h4p9\" (UniqueName: \"kubernetes.io/projected/c58f2629-5a7c-4965-958c-a5092997d8a1-kube-api-access-8h4p9\") pod \"c58f2629-5a7c-4965-958c-a5092997d8a1\" (UID: \"c58f2629-5a7c-4965-958c-a5092997d8a1\") "
Sep 16 02:26:39.351715 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:26:39.351686    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c58f2629-5a7c-4965-958c-a5092997d8a1-kube-api-access-8h4p9" (OuterVolumeSpecName: "kube-api-access-8h4p9") pod "c58f2629-5a7c-4965-958c-a5092997d8a1" (UID: "c58f2629-5a7c-4965-958c-a5092997d8a1"). InnerVolumeSpecName "kube-api-access-8h4p9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 02:26:39.451030 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:26:39.451002    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8h4p9\" (UniqueName: \"kubernetes.io/projected/c58f2629-5a7c-4965-958c-a5092997d8a1-kube-api-access-8h4p9\") on node \"ip-10-0-128-111.ec2.internal\" DevicePath \"\""
Sep 16 02:26:40.224122 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:26:40.224088    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6b8db695c-l9lnp"
Sep 16 02:26:40.225145 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:26:40.225118    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-557f876f69-52hqv" event={"ID":"ceeeb616-b8b8-4abb-89c2-11779a35b142","Type":"ContainerStarted","Data":"33bb11603fedb4a9de9c87a1b32d8193fe8ed926cd2e4be473491b7b69f6ae05"}
Sep 16 02:26:40.225310 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:26:40.224764    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-557f876f69-52hqv" podUID="ceeeb616-b8b8-4abb-89c2-11779a35b142" containerName="authorino" containerID="cri-o://33bb11603fedb4a9de9c87a1b32d8193fe8ed926cd2e4be473491b7b69f6ae05" gracePeriod=30
Sep 16 02:26:40.241455 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:26:40.241402    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-557f876f69-52hqv" podStartSLOduration=1.5594031149999998 podStartE2EDuration="2.241380977s" totalImagesPullingTime="681.977862ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 02:26:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 02:26:40.239160374 +0000 UTC m=+478.706949157" watchObservedRunningTime="2026-09-16 02:26:40.241380977 +0000 UTC m=+478.709169760"
Sep 16 02:26:40.260948 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:26:40.260918    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6b8db695c-l9lnp"]
Sep 16 02:26:40.264117 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:26:40.264090    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-6b8db695c-l9lnp"]
Sep 16 02:26:40.478977 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:26:40.478952    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-557f876f69-52hqv"
Sep 16 02:26:40.561515 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:26:40.561480    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-f94s4\" (UniqueName: \"kubernetes.io/projected/ceeeb616-b8b8-4abb-89c2-11779a35b142-kube-api-access-f94s4\") pod \"ceeeb616-b8b8-4abb-89c2-11779a35b142\" (UID: \"ceeeb616-b8b8-4abb-89c2-11779a35b142\") "
Sep 16 02:26:40.563499 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:26:40.563470    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ceeeb616-b8b8-4abb-89c2-11779a35b142-kube-api-access-f94s4" (OuterVolumeSpecName: "kube-api-access-f94s4") pod "ceeeb616-b8b8-4abb-89c2-11779a35b142" (UID: "ceeeb616-b8b8-4abb-89c2-11779a35b142"). InnerVolumeSpecName "kube-api-access-f94s4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 02:26:40.662710 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:26:40.662633    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f94s4\" (UniqueName: \"kubernetes.io/projected/ceeeb616-b8b8-4abb-89c2-11779a35b142-kube-api-access-f94s4\") on node \"ip-10-0-128-111.ec2.internal\" DevicePath \"\""
Sep 16 02:26:41.230751 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:26:41.230714    2583 generic.go:356] "Generic (PLEG): container finished" podID="ceeeb616-b8b8-4abb-89c2-11779a35b142" containerID="33bb11603fedb4a9de9c87a1b32d8193fe8ed926cd2e4be473491b7b69f6ae05" exitCode=0
Sep 16 02:26:41.231214 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:26:41.230758    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-557f876f69-52hqv"
Sep 16 02:26:41.231214 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:26:41.230759    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-557f876f69-52hqv" event={"ID":"ceeeb616-b8b8-4abb-89c2-11779a35b142","Type":"ContainerDied","Data":"33bb11603fedb4a9de9c87a1b32d8193fe8ed926cd2e4be473491b7b69f6ae05"}
Sep 16 02:26:41.231214 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:26:41.230811    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-557f876f69-52hqv" event={"ID":"ceeeb616-b8b8-4abb-89c2-11779a35b142","Type":"ContainerDied","Data":"344b26c958d419ff585b99f642fb66d82b6106f086a7bffaa563587a2b046e87"}
Sep 16 02:26:41.231214 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:26:41.230834    2583 scope.go:117] "RemoveContainer" containerID="33bb11603fedb4a9de9c87a1b32d8193fe8ed926cd2e4be473491b7b69f6ae05"
Sep 16 02:26:41.239134 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:26:41.239116    2583 scope.go:117] "RemoveContainer" containerID="33bb11603fedb4a9de9c87a1b32d8193fe8ed926cd2e4be473491b7b69f6ae05"
Sep 16 02:26:41.239397 ip-10-0-128-111 kubenswrapper[2583]: E0916 02:26:41.239380    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"33bb11603fedb4a9de9c87a1b32d8193fe8ed926cd2e4be473491b7b69f6ae05\": container with ID starting with 33bb11603fedb4a9de9c87a1b32d8193fe8ed926cd2e4be473491b7b69f6ae05 not found: ID does not exist" containerID="33bb11603fedb4a9de9c87a1b32d8193fe8ed926cd2e4be473491b7b69f6ae05"
Sep 16 02:26:41.239452 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:26:41.239405    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"33bb11603fedb4a9de9c87a1b32d8193fe8ed926cd2e4be473491b7b69f6ae05"} err="failed to get container status \"33bb11603fedb4a9de9c87a1b32d8193fe8ed926cd2e4be473491b7b69f6ae05\": rpc error: code = NotFound desc = could not find container \"33bb11603fedb4a9de9c87a1b32d8193fe8ed926cd2e4be473491b7b69f6ae05\": container with ID starting with 33bb11603fedb4a9de9c87a1b32d8193fe8ed926cd2e4be473491b7b69f6ae05 not found: ID does not exist"
Sep 16 02:26:41.250821 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:26:41.250800    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-557f876f69-52hqv"]
Sep 16 02:26:41.254093 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:26:41.254071    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-557f876f69-52hqv"]
Sep 16 02:26:42.045565 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:26:42.045521    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c58f2629-5a7c-4965-958c-a5092997d8a1" path="/var/lib/kubelet/pods/c58f2629-5a7c-4965-958c-a5092997d8a1/volumes"
Sep 16 02:26:42.045778 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:26:42.045765    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ceeeb616-b8b8-4abb-89c2-11779a35b142" path="/var/lib/kubelet/pods/ceeeb616-b8b8-4abb-89c2-11779a35b142/volumes"
Sep 16 02:27:19.672934 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:27:19.672897    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-fzl8t"]
Sep 16 02:27:28.759194 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:27:28.759137    2583 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-5ccb968c66-xjmsh container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.24:15021/healthz/ready\": read tcp 10.134.0.2:52546->10.134.0.24:15021: read: connection reset by peer" start-of-body=
Sep 16 02:27:28.759640 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:27:28.759205    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-xjmsh" podUID="53ca0d14-d5b8-422c-a044-ab3471e584c9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.24:15021/healthz/ready\": read tcp 10.134.0.2:52546->10.134.0.24:15021: read: connection reset by peer"
Sep 16 02:27:29.447340 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:27:29.447309    2583 generic.go:356] "Generic (PLEG): container finished" podID="53ca0d14-d5b8-422c-a044-ab3471e584c9" containerID="be7eb271b1cf8e744ed79b16198139b2fbc54e24872733e93077b2da24a2671a" exitCode=0
Sep 16 02:27:29.447515 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:27:29.447383    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-xjmsh" event={"ID":"53ca0d14-d5b8-422c-a044-ab3471e584c9","Type":"ContainerDied","Data":"be7eb271b1cf8e744ed79b16198139b2fbc54e24872733e93077b2da24a2671a"}
Sep 16 02:27:29.447801 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:27:29.447787    2583 scope.go:117] "RemoveContainer" containerID="be7eb271b1cf8e744ed79b16198139b2fbc54e24872733e93077b2da24a2671a"
Sep 16 02:27:29.448789 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:27:29.448764    2583 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 16 02:27:29.448853 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:27:29.448832    2583 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 16 02:27:29.448891 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:27:29.448863    2583 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 16 02:27:30.453338 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:27:30.453301    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-xjmsh" event={"ID":"53ca0d14-d5b8-422c-a044-ab3471e584c9","Type":"ContainerStarted","Data":"981083c48a1589818d66a660e7a946ae3227df6b66f6612d5ba8feb95ed3be8d"}
Sep 16 02:27:30.633613 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:27:30.633504    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-xjmsh"
Sep 16 02:27:30.633929 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:27:30.633907    2583 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-5ccb968c66-xjmsh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.24:15021/healthz/ready\": dial tcp 10.134.0.24:15021: connect: connection refused" start-of-body=
Sep 16 02:27:30.634085 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:27:30.634061    2583 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-xjmsh" podUID="53ca0d14-d5b8-422c-a044-ab3471e584c9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.24:15021/healthz/ready\": dial tcp 10.134.0.24:15021: connect: connection refused"
Sep 16 02:27:31.633262 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:27:31.633220    2583 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-5ccb968c66-xjmsh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.24:15021/healthz/ready\": dial tcp 10.134.0.24:15021: connect: connection refused" start-of-body=
Sep 16 02:27:31.633795 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:27:31.633319    2583 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-xjmsh" podUID="53ca0d14-d5b8-422c-a044-ab3471e584c9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.24:15021/healthz/ready\": dial tcp 10.134.0.24:15021: connect: connection refused"
Sep 16 02:27:32.633789 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:27:32.633743    2583 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-5ccb968c66-xjmsh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.24:15021/healthz/ready\": dial tcp 10.134.0.24:15021: connect: connection refused" start-of-body=
Sep 16 02:27:32.634290 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:27:32.633827    2583 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-xjmsh" podUID="53ca0d14-d5b8-422c-a044-ab3471e584c9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.24:15021/healthz/ready\": dial tcp 10.134.0.24:15021: connect: connection refused"
Sep 16 02:27:33.633859 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:27:33.633395    2583 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-5ccb968c66-xjmsh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.24:15021/healthz/ready\": dial tcp 10.134.0.24:15021: connect: connection refused" start-of-body=
Sep 16 02:27:33.633859 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:27:33.633460    2583 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-xjmsh" podUID="53ca0d14-d5b8-422c-a044-ab3471e584c9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.24:15021/healthz/ready\": dial tcp 10.134.0.24:15021: connect: connection refused"
Sep 16 02:27:34.633190 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:27:34.633142    2583 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-5ccb968c66-xjmsh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.24:15021/healthz/ready\": dial tcp 10.134.0.24:15021: connect: connection refused" start-of-body=
Sep 16 02:27:34.633395 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:27:34.633226    2583 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-xjmsh" podUID="53ca0d14-d5b8-422c-a044-ab3471e584c9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.24:15021/healthz/ready\": dial tcp 10.134.0.24:15021: connect: connection refused"
Sep 16 02:27:35.636664 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:27:35.636628    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-xjmsh"
Sep 16 02:27:35.637031 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:27:35.636999    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-xjmsh"
Sep 16 02:27:35.637734 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:27:35.637714    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-xjmsh"
Sep 16 02:27:37.043141 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:27:37.043107    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-cnzqk"]
Sep 16 02:27:37.043480 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:27:37.043426    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ceeeb616-b8b8-4abb-89c2-11779a35b142" containerName="authorino"
Sep 16 02:27:37.043480 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:27:37.043437    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="ceeeb616-b8b8-4abb-89c2-11779a35b142" containerName="authorino"
Sep 16 02:27:37.043582 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:27:37.043487    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="ceeeb616-b8b8-4abb-89c2-11779a35b142" containerName="authorino"
Sep 16 02:27:37.045337 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:27:37.045321    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-cnzqk"
Sep 16 02:27:37.047621 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:27:37.047601    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\""
Sep 16 02:27:37.048534 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:27:37.048512    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-2-simulated-kserve-self-signed-certs\""
Sep 16 02:27:37.048678 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:27:37.048588    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\""
Sep 16 02:27:37.048678 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:27:37.048605    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-f9flb\""
Sep 16 02:27:37.053691 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:27:37.053669    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-cnzqk"]
Sep 16 02:27:37.130118 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:27:37.130079    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/46ccc07d-2236-434c-95b5-61fb795fdea3-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-cnzqk\" (UID: \"46ccc07d-2236-434c-95b5-61fb795fdea3\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-cnzqk"
Sep 16 02:27:37.130302 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:27:37.130135    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/46ccc07d-2236-434c-95b5-61fb795fdea3-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-cnzqk\" (UID: \"46ccc07d-2236-434c-95b5-61fb795fdea3\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-cnzqk"
Sep 16 02:27:37.130302 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:27:37.130189    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v4g7j\" (UniqueName: \"kubernetes.io/projected/46ccc07d-2236-434c-95b5-61fb795fdea3-kube-api-access-v4g7j\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-cnzqk\" (UID: \"46ccc07d-2236-434c-95b5-61fb795fdea3\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-cnzqk"
Sep 16 02:27:37.130302 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:27:37.130229    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/46ccc07d-2236-434c-95b5-61fb795fdea3-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-cnzqk\" (UID: \"46ccc07d-2236-434c-95b5-61fb795fdea3\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-cnzqk"
Sep 16 02:27:37.130302 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:27:37.130268    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/46ccc07d-2236-434c-95b5-61fb795fdea3-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-cnzqk\" (UID: \"46ccc07d-2236-434c-95b5-61fb795fdea3\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-cnzqk"
Sep 16 02:27:37.130302 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:27:37.130297    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/46ccc07d-2236-434c-95b5-61fb795fdea3-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-cnzqk\" (UID: \"46ccc07d-2236-434c-95b5-61fb795fdea3\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-cnzqk"
Sep 16 02:27:37.231018 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:27:37.230983    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/46ccc07d-2236-434c-95b5-61fb795fdea3-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-cnzqk\" (UID: \"46ccc07d-2236-434c-95b5-61fb795fdea3\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-cnzqk"
Sep 16 02:27:37.231209 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:27:37.231033    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/46ccc07d-2236-434c-95b5-61fb795fdea3-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-cnzqk\" (UID: \"46ccc07d-2236-434c-95b5-61fb795fdea3\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-cnzqk"
Sep 16 02:27:37.231209 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:27:37.231049    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v4g7j\" (UniqueName: \"kubernetes.io/projected/46ccc07d-2236-434c-95b5-61fb795fdea3-kube-api-access-v4g7j\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-cnzqk\" (UID: \"46ccc07d-2236-434c-95b5-61fb795fdea3\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-cnzqk"
Sep 16 02:27:37.231209 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:27:37.231069    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/46ccc07d-2236-434c-95b5-61fb795fdea3-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-cnzqk\" (UID: \"46ccc07d-2236-434c-95b5-61fb795fdea3\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-cnzqk"
Sep 16 02:27:37.231209 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:27:37.231102    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/46ccc07d-2236-434c-95b5-61fb795fdea3-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-cnzqk\" (UID: \"46ccc07d-2236-434c-95b5-61fb795fdea3\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-cnzqk"
Sep 16 02:27:37.231209 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:27:37.231141    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/46ccc07d-2236-434c-95b5-61fb795fdea3-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-cnzqk\" (UID: \"46ccc07d-2236-434c-95b5-61fb795fdea3\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-cnzqk"
Sep 16 02:27:37.231603 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:27:37.231547    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/46ccc07d-2236-434c-95b5-61fb795fdea3-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-cnzqk\" (UID: \"46ccc07d-2236-434c-95b5-61fb795fdea3\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-cnzqk"
Sep 16 02:27:37.231688 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:27:37.231608    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/46ccc07d-2236-434c-95b5-61fb795fdea3-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-cnzqk\" (UID: \"46ccc07d-2236-434c-95b5-61fb795fdea3\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-cnzqk"
Sep 16 02:27:37.231688 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:27:37.231606    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/46ccc07d-2236-434c-95b5-61fb795fdea3-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-cnzqk\" (UID: \"46ccc07d-2236-434c-95b5-61fb795fdea3\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-cnzqk"
Sep 16 02:27:37.233334 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:27:37.233312    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/46ccc07d-2236-434c-95b5-61fb795fdea3-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-cnzqk\" (UID: \"46ccc07d-2236-434c-95b5-61fb795fdea3\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-cnzqk"
Sep 16 02:27:37.233575 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:27:37.233540    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/46ccc07d-2236-434c-95b5-61fb795fdea3-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-cnzqk\" (UID: \"46ccc07d-2236-434c-95b5-61fb795fdea3\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-cnzqk"
Sep 16 02:27:37.238098 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:27:37.238074    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-v4g7j\" (UniqueName: \"kubernetes.io/projected/46ccc07d-2236-434c-95b5-61fb795fdea3-kube-api-access-v4g7j\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-cnzqk\" (UID: \"46ccc07d-2236-434c-95b5-61fb795fdea3\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-cnzqk"
Sep 16 02:27:37.357679 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:27:37.357645    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-cnzqk"
Sep 16 02:27:37.544703 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:27:37.544671    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-fzl8t"]
Sep 16 02:27:37.627784 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:27:37.627759    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-cnzqk"]
Sep 16 02:27:37.630320 ip-10-0-128-111 kubenswrapper[2583]: W0916 02:27:37.630296    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod46ccc07d_2236_434c_95b5_61fb795fdea3.slice/crio-612a79566d30b7fcadd5ac030514d11a06664dc6acfc84c328174855919cb318 WatchSource:0}: Error finding container 612a79566d30b7fcadd5ac030514d11a06664dc6acfc84c328174855919cb318: Status 404 returned error can't find the container with id 612a79566d30b7fcadd5ac030514d11a06664dc6acfc84c328174855919cb318
Sep 16 02:27:38.491731 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:27:38.491679    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-cnzqk" event={"ID":"46ccc07d-2236-434c-95b5-61fb795fdea3","Type":"ContainerStarted","Data":"612a79566d30b7fcadd5ac030514d11a06664dc6acfc84c328174855919cb318"}
Sep 16 02:27:39.498071 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:27:39.498037    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-cnzqk" event={"ID":"46ccc07d-2236-434c-95b5-61fb795fdea3","Type":"ContainerStarted","Data":"ef25e649aba435d2ee58fa0cff9470f6c1516c1d7146072227fa5149c7f16080"}
Sep 16 02:27:39.516008 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:27:39.515959    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-cnzqk" podStartSLOduration=1.56586012 podStartE2EDuration="2.515943827s" totalImagesPullingTime="950.083707ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 02:27:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 02:27:39.514752504 +0000 UTC m=+537.982541286" watchObservedRunningTime="2026-09-16 02:27:39.515943827 +0000 UTC m=+537.983732606"
Sep 16 02:27:47.146388 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:27:47.146352    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-nmdqn"]
Sep 16 02:27:47.148908 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:27:47.148884    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-nmdqn"
Sep 16 02:27:47.151387 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:27:47.151366    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-simulated-kserve-self-signed-certs\""
Sep 16 02:27:47.159662 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:27:47.159637    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-nmdqn"]
Sep 16 02:27:47.216980 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:27:47.216936    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f5676c37-19ac-4cec-b73e-599c0446b965-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-nmdqn\" (UID: \"f5676c37-19ac-4cec-b73e-599c0446b965\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-nmdqn"
Sep 16 02:27:47.217148 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:27:47.216991    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f5676c37-19ac-4cec-b73e-599c0446b965-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-nmdqn\" (UID: \"f5676c37-19ac-4cec-b73e-599c0446b965\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-nmdqn"
Sep 16 02:27:47.217148 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:27:47.217015    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f5676c37-19ac-4cec-b73e-599c0446b965-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-nmdqn\" (UID: \"f5676c37-19ac-4cec-b73e-599c0446b965\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-nmdqn"
Sep 16 02:27:47.217148 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:27:47.217085    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jkm82\" (UniqueName: \"kubernetes.io/projected/f5676c37-19ac-4cec-b73e-599c0446b965-kube-api-access-jkm82\") pod \"e2e-distinct-simulated-kserve-59975d69f9-nmdqn\" (UID: \"f5676c37-19ac-4cec-b73e-599c0446b965\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-nmdqn"
Sep 16 02:27:47.217148 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:27:47.217120    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f5676c37-19ac-4cec-b73e-599c0446b965-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-nmdqn\" (UID: \"f5676c37-19ac-4cec-b73e-599c0446b965\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-nmdqn"
Sep 16 02:27:47.217148 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:27:47.217137    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f5676c37-19ac-4cec-b73e-599c0446b965-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-nmdqn\" (UID: \"f5676c37-19ac-4cec-b73e-599c0446b965\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-nmdqn"
Sep 16 02:27:47.318614 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:27:47.318576    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f5676c37-19ac-4cec-b73e-599c0446b965-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-nmdqn\" (UID: \"f5676c37-19ac-4cec-b73e-599c0446b965\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-nmdqn"
Sep 16 02:27:47.318614 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:27:47.318617    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f5676c37-19ac-4cec-b73e-599c0446b965-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-nmdqn\" (UID: \"f5676c37-19ac-4cec-b73e-599c0446b965\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-nmdqn"
Sep 16 02:27:47.318881 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:27:47.318683    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f5676c37-19ac-4cec-b73e-599c0446b965-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-nmdqn\" (UID: \"f5676c37-19ac-4cec-b73e-599c0446b965\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-nmdqn"
Sep 16 02:27:47.318881 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:27:47.318729    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f5676c37-19ac-4cec-b73e-599c0446b965-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-nmdqn\" (UID: \"f5676c37-19ac-4cec-b73e-599c0446b965\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-nmdqn"
Sep 16 02:27:47.318881 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:27:47.318752    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f5676c37-19ac-4cec-b73e-599c0446b965-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-nmdqn\" (UID: \"f5676c37-19ac-4cec-b73e-599c0446b965\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-nmdqn"
Sep 16 02:27:47.318881 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:27:47.318804    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jkm82\" (UniqueName: \"kubernetes.io/projected/f5676c37-19ac-4cec-b73e-599c0446b965-kube-api-access-jkm82\") pod \"e2e-distinct-simulated-kserve-59975d69f9-nmdqn\" (UID: \"f5676c37-19ac-4cec-b73e-599c0446b965\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-nmdqn"
Sep 16 02:27:47.319082 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:27:47.319013    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f5676c37-19ac-4cec-b73e-599c0446b965-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-nmdqn\" (UID: \"f5676c37-19ac-4cec-b73e-599c0446b965\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-nmdqn"
Sep 16 02:27:47.319136 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:27:47.319094    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f5676c37-19ac-4cec-b73e-599c0446b965-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-nmdqn\" (UID: \"f5676c37-19ac-4cec-b73e-599c0446b965\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-nmdqn"
Sep 16 02:27:47.319258 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:27:47.319234    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f5676c37-19ac-4cec-b73e-599c0446b965-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-nmdqn\" (UID: \"f5676c37-19ac-4cec-b73e-599c0446b965\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-nmdqn"
Sep 16 02:27:47.321052 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:27:47.321031    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f5676c37-19ac-4cec-b73e-599c0446b965-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-nmdqn\" (UID: \"f5676c37-19ac-4cec-b73e-599c0446b965\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-nmdqn"
Sep 16 02:27:47.321466 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:27:47.321445    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f5676c37-19ac-4cec-b73e-599c0446b965-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-nmdqn\" (UID: \"f5676c37-19ac-4cec-b73e-599c0446b965\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-nmdqn"
Sep 16 02:27:47.326155 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:27:47.326132    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jkm82\" (UniqueName: \"kubernetes.io/projected/f5676c37-19ac-4cec-b73e-599c0446b965-kube-api-access-jkm82\") pod \"e2e-distinct-simulated-kserve-59975d69f9-nmdqn\" (UID: \"f5676c37-19ac-4cec-b73e-599c0446b965\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-nmdqn"
Sep 16 02:27:47.358718 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:27:47.358675    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-cnzqk"
Sep 16 02:27:47.358900 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:27:47.358765    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-cnzqk"
Sep 16 02:27:47.371498 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:27:47.371473    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-cnzqk"
Sep 16 02:27:47.460045 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:27:47.459959    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-nmdqn"
Sep 16 02:27:47.544102 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:27:47.544072    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-cnzqk"
Sep 16 02:27:47.730818 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:27:47.728300    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-nmdqn"]
Sep 16 02:27:47.753214 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:27:47.753178    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-fzl8t"]
Sep 16 02:27:48.538798 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:27:48.538762    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-nmdqn" event={"ID":"f5676c37-19ac-4cec-b73e-599c0446b965","Type":"ContainerStarted","Data":"9fa5651d1582a7bdef71fffb8e3e700ce7260f7bd6fd360a89dcda77e3b92bca"}
Sep 16 02:27:48.539178 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:27:48.538821    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-nmdqn" event={"ID":"f5676c37-19ac-4cec-b73e-599c0446b965","Type":"ContainerStarted","Data":"88e77a6b653d355413117179fe4e01399dba3b1421e9618edc88d3e7d630f9c9"}
Sep 16 02:27:48.555504 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:27:48.555431    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-nmdqn" podStartSLOduration=1.391568799 podStartE2EDuration="1.555414468s" totalImagesPullingTime="163.845669ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 02:27:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 02:27:48.553964919 +0000 UTC m=+547.021753700" watchObservedRunningTime="2026-09-16 02:27:48.555414468 +0000 UTC m=+547.023203249"
Sep 16 02:27:51.756936 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:27:51.756902    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-fzl8t"]
Sep 16 02:27:57.460329 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:27:57.460294    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-nmdqn"
Sep 16 02:27:57.460768 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:27:57.460360    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-nmdqn"
Sep 16 02:27:57.472643 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:27:57.472621    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-nmdqn"
Sep 16 02:27:57.589348 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:27:57.589313    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-nmdqn"
Sep 16 02:28:04.344673 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:28:04.344635    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-fzl8t"]
Sep 16 02:28:12.051268 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:28:12.051232    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-fzl8t"]
Sep 16 02:28:18.154092 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:28:18.154054    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-fzl8t"]
Sep 16 02:28:22.753426 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:28:22.753394    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-57fc66f697-rnrfd"]
Sep 16 02:28:22.756902 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:28:22.756881    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-rnrfd"
Sep 16 02:28:22.759405 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:28:22.759381    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"premium-simulated-simulated-premium-kserve-self-signed-certs\""
Sep 16 02:28:22.766413 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:28:22.766388    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-57fc66f697-rnrfd"]
Sep 16 02:28:22.827144 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:28:22.827108    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b0c1dae8-9c6e-40a4-be9b-c7412f7fcb83-model-cache\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-rnrfd\" (UID: \"b0c1dae8-9c6e-40a4-be9b-c7412f7fcb83\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-rnrfd"
Sep 16 02:28:22.827340 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:28:22.827162    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b0c1dae8-9c6e-40a4-be9b-c7412f7fcb83-home\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-rnrfd\" (UID: \"b0c1dae8-9c6e-40a4-be9b-c7412f7fcb83\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-rnrfd"
Sep 16 02:28:22.827340 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:28:22.827208    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6w8t6\" (UniqueName: \"kubernetes.io/projected/b0c1dae8-9c6e-40a4-be9b-c7412f7fcb83-kube-api-access-6w8t6\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-rnrfd\" (UID: \"b0c1dae8-9c6e-40a4-be9b-c7412f7fcb83\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-rnrfd"
Sep 16 02:28:22.827340 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:28:22.827259    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b0c1dae8-9c6e-40a4-be9b-c7412f7fcb83-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-rnrfd\" (UID: \"b0c1dae8-9c6e-40a4-be9b-c7412f7fcb83\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-rnrfd"
Sep 16 02:28:22.827340 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:28:22.827304    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b0c1dae8-9c6e-40a4-be9b-c7412f7fcb83-dshm\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-rnrfd\" (UID: \"b0c1dae8-9c6e-40a4-be9b-c7412f7fcb83\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-rnrfd"
Sep 16 02:28:22.827677 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:28:22.827360    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b0c1dae8-9c6e-40a4-be9b-c7412f7fcb83-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-rnrfd\" (UID: \"b0c1dae8-9c6e-40a4-be9b-c7412f7fcb83\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-rnrfd"
Sep 16 02:28:22.928264 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:28:22.928232    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b0c1dae8-9c6e-40a4-be9b-c7412f7fcb83-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-rnrfd\" (UID: \"b0c1dae8-9c6e-40a4-be9b-c7412f7fcb83\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-rnrfd"
Sep 16 02:28:22.928438 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:28:22.928272    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b0c1dae8-9c6e-40a4-be9b-c7412f7fcb83-model-cache\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-rnrfd\" (UID: \"b0c1dae8-9c6e-40a4-be9b-c7412f7fcb83\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-rnrfd"
Sep 16 02:28:22.928438 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:28:22.928301    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b0c1dae8-9c6e-40a4-be9b-c7412f7fcb83-home\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-rnrfd\" (UID: \"b0c1dae8-9c6e-40a4-be9b-c7412f7fcb83\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-rnrfd"
Sep 16 02:28:22.928438 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:28:22.928343    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6w8t6\" (UniqueName: \"kubernetes.io/projected/b0c1dae8-9c6e-40a4-be9b-c7412f7fcb83-kube-api-access-6w8t6\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-rnrfd\" (UID: \"b0c1dae8-9c6e-40a4-be9b-c7412f7fcb83\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-rnrfd"
Sep 16 02:28:22.928438 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:28:22.928384    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b0c1dae8-9c6e-40a4-be9b-c7412f7fcb83-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-rnrfd\" (UID: \"b0c1dae8-9c6e-40a4-be9b-c7412f7fcb83\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-rnrfd"
Sep 16 02:28:22.928438 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:28:22.928424    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b0c1dae8-9c6e-40a4-be9b-c7412f7fcb83-dshm\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-rnrfd\" (UID: \"b0c1dae8-9c6e-40a4-be9b-c7412f7fcb83\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-rnrfd"
Sep 16 02:28:22.928717 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:28:22.928698    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b0c1dae8-9c6e-40a4-be9b-c7412f7fcb83-model-cache\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-rnrfd\" (UID: \"b0c1dae8-9c6e-40a4-be9b-c7412f7fcb83\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-rnrfd"
Sep 16 02:28:22.928820 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:28:22.928791    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b0c1dae8-9c6e-40a4-be9b-c7412f7fcb83-home\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-rnrfd\" (UID: \"b0c1dae8-9c6e-40a4-be9b-c7412f7fcb83\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-rnrfd"
Sep 16 02:28:22.928820 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:28:22.928803    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b0c1dae8-9c6e-40a4-be9b-c7412f7fcb83-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-rnrfd\" (UID: \"b0c1dae8-9c6e-40a4-be9b-c7412f7fcb83\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-rnrfd"
Sep 16 02:28:22.930646 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:28:22.930624    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b0c1dae8-9c6e-40a4-be9b-c7412f7fcb83-dshm\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-rnrfd\" (UID: \"b0c1dae8-9c6e-40a4-be9b-c7412f7fcb83\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-rnrfd"
Sep 16 02:28:22.930885 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:28:22.930868    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b0c1dae8-9c6e-40a4-be9b-c7412f7fcb83-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-rnrfd\" (UID: \"b0c1dae8-9c6e-40a4-be9b-c7412f7fcb83\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-rnrfd"
Sep 16 02:28:22.935401 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:28:22.935379    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6w8t6\" (UniqueName: \"kubernetes.io/projected/b0c1dae8-9c6e-40a4-be9b-c7412f7fcb83-kube-api-access-6w8t6\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-rnrfd\" (UID: \"b0c1dae8-9c6e-40a4-be9b-c7412f7fcb83\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-rnrfd"
Sep 16 02:28:23.067336 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:28:23.067246    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-rnrfd"
Sep 16 02:28:23.336172 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:28:23.336148    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-57fc66f697-rnrfd"]
Sep 16 02:28:23.338735 ip-10-0-128-111 kubenswrapper[2583]: W0916 02:28:23.338706    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb0c1dae8_9c6e_40a4_be9b_c7412f7fcb83.slice/crio-42542361b94d95c4e1451a92559193d789ef187eb8d4bf49868be0d210746bc3 WatchSource:0}: Error finding container 42542361b94d95c4e1451a92559193d789ef187eb8d4bf49868be0d210746bc3: Status 404 returned error can't find the container with id 42542361b94d95c4e1451a92559193d789ef187eb8d4bf49868be0d210746bc3
Sep 16 02:28:23.694398 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:28:23.694362    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-rnrfd" event={"ID":"b0c1dae8-9c6e-40a4-be9b-c7412f7fcb83","Type":"ContainerStarted","Data":"400483e181f753f4000e296dc8717d402c1ffcf7aff601dec1a9de983ece1b7a"}
Sep 16 02:28:23.694398 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:28:23.694407    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-rnrfd" event={"ID":"b0c1dae8-9c6e-40a4-be9b-c7412f7fcb83","Type":"ContainerStarted","Data":"42542361b94d95c4e1451a92559193d789ef187eb8d4bf49868be0d210746bc3"}
Sep 16 02:28:23.709786 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:28:23.709738    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-rnrfd" podStartSLOduration=1.550558705 podStartE2EDuration="1.709720749s" totalImagesPullingTime="159.162044ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 02:28:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 02:28:23.708789947 +0000 UTC m=+582.176578740" watchObservedRunningTime="2026-09-16 02:28:23.709720749 +0000 UTC m=+582.177509530"
Sep 16 02:28:25.351727 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:28:25.351694    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-fzl8t"]
Sep 16 02:28:33.067795 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:28:33.067760    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-rnrfd"
Sep 16 02:28:33.068184 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:28:33.067834    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-rnrfd"
Sep 16 02:28:33.080329 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:28:33.080304    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-rnrfd"
Sep 16 02:28:33.749316 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:28:33.749280    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-rnrfd"
Sep 16 02:29:37.866342 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:29:37.866308    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-7698c7d5bb-mds65"]
Sep 16 02:29:37.869532 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:29:37.869516    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7698c7d5bb-mds65"
Sep 16 02:29:37.872655 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:29:37.872633    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-xkv7p\""
Sep 16 02:29:37.872780 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:29:37.872634    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-server-cert\""
Sep 16 02:29:37.875304 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:29:37.875272    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-7698c7d5bb-mds65"]
Sep 16 02:29:37.990981 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:29:37.990948    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d9nr6\" (UniqueName: \"kubernetes.io/projected/60642fc2-3eab-4e46-85a9-30bc0e930717-kube-api-access-d9nr6\") pod \"authorino-7698c7d5bb-mds65\" (UID: \"60642fc2-3eab-4e46-85a9-30bc0e930717\") " pod="kuadrant-system/authorino-7698c7d5bb-mds65"
Sep 16 02:29:37.991153 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:29:37.991003    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/60642fc2-3eab-4e46-85a9-30bc0e930717-tls-cert\") pod \"authorino-7698c7d5bb-mds65\" (UID: \"60642fc2-3eab-4e46-85a9-30bc0e930717\") " pod="kuadrant-system/authorino-7698c7d5bb-mds65"
Sep 16 02:29:38.091915 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:29:38.091883    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/60642fc2-3eab-4e46-85a9-30bc0e930717-tls-cert\") pod \"authorino-7698c7d5bb-mds65\" (UID: \"60642fc2-3eab-4e46-85a9-30bc0e930717\") " pod="kuadrant-system/authorino-7698c7d5bb-mds65"
Sep 16 02:29:38.092086 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:29:38.091964    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d9nr6\" (UniqueName: \"kubernetes.io/projected/60642fc2-3eab-4e46-85a9-30bc0e930717-kube-api-access-d9nr6\") pod \"authorino-7698c7d5bb-mds65\" (UID: \"60642fc2-3eab-4e46-85a9-30bc0e930717\") " pod="kuadrant-system/authorino-7698c7d5bb-mds65"
Sep 16 02:29:38.094384 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:29:38.094359    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/60642fc2-3eab-4e46-85a9-30bc0e930717-tls-cert\") pod \"authorino-7698c7d5bb-mds65\" (UID: \"60642fc2-3eab-4e46-85a9-30bc0e930717\") " pod="kuadrant-system/authorino-7698c7d5bb-mds65"
Sep 16 02:29:38.099105 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:29:38.099083    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-d9nr6\" (UniqueName: \"kubernetes.io/projected/60642fc2-3eab-4e46-85a9-30bc0e930717-kube-api-access-d9nr6\") pod \"authorino-7698c7d5bb-mds65\" (UID: \"60642fc2-3eab-4e46-85a9-30bc0e930717\") " pod="kuadrant-system/authorino-7698c7d5bb-mds65"
Sep 16 02:29:38.180118 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:29:38.180050    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7698c7d5bb-mds65"
Sep 16 02:29:38.448331 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:29:38.448302    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-7698c7d5bb-mds65"]
Sep 16 02:29:38.451005 ip-10-0-128-111 kubenswrapper[2583]: W0916 02:29:38.450979    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod60642fc2_3eab_4e46_85a9_30bc0e930717.slice/crio-4fd9fc759f79ed550802a4737e8c4eecf184be2735ede6e6dfad3a2d28706798 WatchSource:0}: Error finding container 4fd9fc759f79ed550802a4737e8c4eecf184be2735ede6e6dfad3a2d28706798: Status 404 returned error can't find the container with id 4fd9fc759f79ed550802a4737e8c4eecf184be2735ede6e6dfad3a2d28706798
Sep 16 02:29:38.452851 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:29:38.452821    2583 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 16 02:29:39.025270 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:29:39.025235    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7698c7d5bb-mds65" event={"ID":"60642fc2-3eab-4e46-85a9-30bc0e930717","Type":"ContainerStarted","Data":"4fd9fc759f79ed550802a4737e8c4eecf184be2735ede6e6dfad3a2d28706798"}
Sep 16 02:29:40.031915 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:29:40.031879    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7698c7d5bb-mds65" event={"ID":"60642fc2-3eab-4e46-85a9-30bc0e930717","Type":"ContainerStarted","Data":"4941d9662a7e06c0a271164710855c0d5c3a239d6e1fc6a2faaa5a03ee546aa2"}
Sep 16 02:29:40.049568 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:29:40.049510    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-7698c7d5bb-mds65" podStartSLOduration=2.328630457 podStartE2EDuration="3.049495808s" totalImagesPullingTime="720.865351ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 02:29:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 02:29:40.04830454 +0000 UTC m=+658.516093325" watchObservedRunningTime="2026-09-16 02:29:40.049495808 +0000 UTC m=+658.517284699"
Sep 16 02:30:47.219059 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:30:47.218974    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-5b59aa-openshift-default-ff449c75d-6ckh5"]
Sep 16 02:30:47.223049 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:30:47.223027    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-5b59aa-openshift-default-ff449c75d-6ckh5"
Sep 16 02:30:47.230122 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:30:47.229927    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w0-5b59aa-openshift-default-dockercfg-z44x6\""
Sep 16 02:30:47.247961 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:30:47.247929    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-5b59aa-openshift-default-ff449c75d-6ckh5"]
Sep 16 02:30:47.305721 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:30:47.305675    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3a21aba3-2483-46a3-8b7f-acac0d646664-istiod-ca-cert\") pod \"e2e-worker-w0-5b59aa-openshift-default-ff449c75d-6ckh5\" (UID: \"3a21aba3-2483-46a3-8b7f-acac0d646664\") " pod="openshift-ingress/e2e-worker-w0-5b59aa-openshift-default-ff449c75d-6ckh5"
Sep 16 02:30:47.305721 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:30:47.305722    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3a21aba3-2483-46a3-8b7f-acac0d646664-istio-envoy\") pod \"e2e-worker-w0-5b59aa-openshift-default-ff449c75d-6ckh5\" (UID: \"3a21aba3-2483-46a3-8b7f-acac0d646664\") " pod="openshift-ingress/e2e-worker-w0-5b59aa-openshift-default-ff449c75d-6ckh5"
Sep 16 02:30:47.305979 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:30:47.305747    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3a21aba3-2483-46a3-8b7f-acac0d646664-istio-token\") pod \"e2e-worker-w0-5b59aa-openshift-default-ff449c75d-6ckh5\" (UID: \"3a21aba3-2483-46a3-8b7f-acac0d646664\") " pod="openshift-ingress/e2e-worker-w0-5b59aa-openshift-default-ff449c75d-6ckh5"
Sep 16 02:30:47.305979 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:30:47.305804    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3a21aba3-2483-46a3-8b7f-acac0d646664-workload-certs\") pod \"e2e-worker-w0-5b59aa-openshift-default-ff449c75d-6ckh5\" (UID: \"3a21aba3-2483-46a3-8b7f-acac0d646664\") " pod="openshift-ingress/e2e-worker-w0-5b59aa-openshift-default-ff449c75d-6ckh5"
Sep 16 02:30:47.305979 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:30:47.305826    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rhn8r\" (UniqueName: \"kubernetes.io/projected/3a21aba3-2483-46a3-8b7f-acac0d646664-kube-api-access-rhn8r\") pod \"e2e-worker-w0-5b59aa-openshift-default-ff449c75d-6ckh5\" (UID: \"3a21aba3-2483-46a3-8b7f-acac0d646664\") " pod="openshift-ingress/e2e-worker-w0-5b59aa-openshift-default-ff449c75d-6ckh5"
Sep 16 02:30:47.305979 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:30:47.305869    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3a21aba3-2483-46a3-8b7f-acac0d646664-workload-socket\") pod \"e2e-worker-w0-5b59aa-openshift-default-ff449c75d-6ckh5\" (UID: \"3a21aba3-2483-46a3-8b7f-acac0d646664\") " pod="openshift-ingress/e2e-worker-w0-5b59aa-openshift-default-ff449c75d-6ckh5"
Sep 16 02:30:47.305979 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:30:47.305888    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3a21aba3-2483-46a3-8b7f-acac0d646664-credential-socket\") pod \"e2e-worker-w0-5b59aa-openshift-default-ff449c75d-6ckh5\" (UID: \"3a21aba3-2483-46a3-8b7f-acac0d646664\") " pod="openshift-ingress/e2e-worker-w0-5b59aa-openshift-default-ff449c75d-6ckh5"
Sep 16 02:30:47.305979 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:30:47.305921    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3a21aba3-2483-46a3-8b7f-acac0d646664-istio-data\") pod \"e2e-worker-w0-5b59aa-openshift-default-ff449c75d-6ckh5\" (UID: \"3a21aba3-2483-46a3-8b7f-acac0d646664\") " pod="openshift-ingress/e2e-worker-w0-5b59aa-openshift-default-ff449c75d-6ckh5"
Sep 16 02:30:47.305979 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:30:47.305962    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3a21aba3-2483-46a3-8b7f-acac0d646664-istio-podinfo\") pod \"e2e-worker-w0-5b59aa-openshift-default-ff449c75d-6ckh5\" (UID: \"3a21aba3-2483-46a3-8b7f-acac0d646664\") " pod="openshift-ingress/e2e-worker-w0-5b59aa-openshift-default-ff449c75d-6ckh5"
Sep 16 02:30:47.406774 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:30:47.406738    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3a21aba3-2483-46a3-8b7f-acac0d646664-istio-envoy\") pod \"e2e-worker-w0-5b59aa-openshift-default-ff449c75d-6ckh5\" (UID: \"3a21aba3-2483-46a3-8b7f-acac0d646664\") " pod="openshift-ingress/e2e-worker-w0-5b59aa-openshift-default-ff449c75d-6ckh5"
Sep 16 02:30:47.406774 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:30:47.406778    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3a21aba3-2483-46a3-8b7f-acac0d646664-istio-token\") pod \"e2e-worker-w0-5b59aa-openshift-default-ff449c75d-6ckh5\" (UID: \"3a21aba3-2483-46a3-8b7f-acac0d646664\") " pod="openshift-ingress/e2e-worker-w0-5b59aa-openshift-default-ff449c75d-6ckh5"
Sep 16 02:30:47.407030 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:30:47.406799    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3a21aba3-2483-46a3-8b7f-acac0d646664-workload-certs\") pod \"e2e-worker-w0-5b59aa-openshift-default-ff449c75d-6ckh5\" (UID: \"3a21aba3-2483-46a3-8b7f-acac0d646664\") " pod="openshift-ingress/e2e-worker-w0-5b59aa-openshift-default-ff449c75d-6ckh5"
Sep 16 02:30:47.407030 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:30:47.406817    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rhn8r\" (UniqueName: \"kubernetes.io/projected/3a21aba3-2483-46a3-8b7f-acac0d646664-kube-api-access-rhn8r\") pod \"e2e-worker-w0-5b59aa-openshift-default-ff449c75d-6ckh5\" (UID: \"3a21aba3-2483-46a3-8b7f-acac0d646664\") " pod="openshift-ingress/e2e-worker-w0-5b59aa-openshift-default-ff449c75d-6ckh5"
Sep 16 02:30:47.407030 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:30:47.406855    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3a21aba3-2483-46a3-8b7f-acac0d646664-workload-socket\") pod \"e2e-worker-w0-5b59aa-openshift-default-ff449c75d-6ckh5\" (UID: \"3a21aba3-2483-46a3-8b7f-acac0d646664\") " pod="openshift-ingress/e2e-worker-w0-5b59aa-openshift-default-ff449c75d-6ckh5"
Sep 16 02:30:47.407191 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:30:47.407041    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3a21aba3-2483-46a3-8b7f-acac0d646664-credential-socket\") pod \"e2e-worker-w0-5b59aa-openshift-default-ff449c75d-6ckh5\" (UID: \"3a21aba3-2483-46a3-8b7f-acac0d646664\") " pod="openshift-ingress/e2e-worker-w0-5b59aa-openshift-default-ff449c75d-6ckh5"
Sep 16 02:30:47.407191 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:30:47.407103    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3a21aba3-2483-46a3-8b7f-acac0d646664-istio-data\") pod \"e2e-worker-w0-5b59aa-openshift-default-ff449c75d-6ckh5\" (UID: \"3a21aba3-2483-46a3-8b7f-acac0d646664\") " pod="openshift-ingress/e2e-worker-w0-5b59aa-openshift-default-ff449c75d-6ckh5"
Sep 16 02:30:47.407304 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:30:47.407269    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3a21aba3-2483-46a3-8b7f-acac0d646664-workload-certs\") pod \"e2e-worker-w0-5b59aa-openshift-default-ff449c75d-6ckh5\" (UID: \"3a21aba3-2483-46a3-8b7f-acac0d646664\") " pod="openshift-ingress/e2e-worker-w0-5b59aa-openshift-default-ff449c75d-6ckh5"
Sep 16 02:30:47.407304 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:30:47.407286    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3a21aba3-2483-46a3-8b7f-acac0d646664-istio-podinfo\") pod \"e2e-worker-w0-5b59aa-openshift-default-ff449c75d-6ckh5\" (UID: \"3a21aba3-2483-46a3-8b7f-acac0d646664\") " pod="openshift-ingress/e2e-worker-w0-5b59aa-openshift-default-ff449c75d-6ckh5"
Sep 16 02:30:47.407406 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:30:47.407307    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3a21aba3-2483-46a3-8b7f-acac0d646664-workload-socket\") pod \"e2e-worker-w0-5b59aa-openshift-default-ff449c75d-6ckh5\" (UID: \"3a21aba3-2483-46a3-8b7f-acac0d646664\") " pod="openshift-ingress/e2e-worker-w0-5b59aa-openshift-default-ff449c75d-6ckh5"
Sep 16 02:30:47.407406 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:30:47.407375    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3a21aba3-2483-46a3-8b7f-acac0d646664-istiod-ca-cert\") pod \"e2e-worker-w0-5b59aa-openshift-default-ff449c75d-6ckh5\" (UID: \"3a21aba3-2483-46a3-8b7f-acac0d646664\") " pod="openshift-ingress/e2e-worker-w0-5b59aa-openshift-default-ff449c75d-6ckh5"
Sep 16 02:30:47.407510 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:30:47.407432    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3a21aba3-2483-46a3-8b7f-acac0d646664-credential-socket\") pod \"e2e-worker-w0-5b59aa-openshift-default-ff449c75d-6ckh5\" (UID: \"3a21aba3-2483-46a3-8b7f-acac0d646664\") " pod="openshift-ingress/e2e-worker-w0-5b59aa-openshift-default-ff449c75d-6ckh5"
Sep 16 02:30:47.407578 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:30:47.407513    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3a21aba3-2483-46a3-8b7f-acac0d646664-istio-data\") pod \"e2e-worker-w0-5b59aa-openshift-default-ff449c75d-6ckh5\" (UID: \"3a21aba3-2483-46a3-8b7f-acac0d646664\") " pod="openshift-ingress/e2e-worker-w0-5b59aa-openshift-default-ff449c75d-6ckh5"
Sep 16 02:30:47.408025 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:30:47.408001    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3a21aba3-2483-46a3-8b7f-acac0d646664-istiod-ca-cert\") pod \"e2e-worker-w0-5b59aa-openshift-default-ff449c75d-6ckh5\" (UID: \"3a21aba3-2483-46a3-8b7f-acac0d646664\") " pod="openshift-ingress/e2e-worker-w0-5b59aa-openshift-default-ff449c75d-6ckh5"
Sep 16 02:30:47.409295 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:30:47.409277    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3a21aba3-2483-46a3-8b7f-acac0d646664-istio-envoy\") pod \"e2e-worker-w0-5b59aa-openshift-default-ff449c75d-6ckh5\" (UID: \"3a21aba3-2483-46a3-8b7f-acac0d646664\") " pod="openshift-ingress/e2e-worker-w0-5b59aa-openshift-default-ff449c75d-6ckh5"
Sep 16 02:30:47.409573 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:30:47.409536    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3a21aba3-2483-46a3-8b7f-acac0d646664-istio-podinfo\") pod \"e2e-worker-w0-5b59aa-openshift-default-ff449c75d-6ckh5\" (UID: \"3a21aba3-2483-46a3-8b7f-acac0d646664\") " pod="openshift-ingress/e2e-worker-w0-5b59aa-openshift-default-ff449c75d-6ckh5"
Sep 16 02:30:47.414454 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:30:47.414401    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rhn8r\" (UniqueName: \"kubernetes.io/projected/3a21aba3-2483-46a3-8b7f-acac0d646664-kube-api-access-rhn8r\") pod \"e2e-worker-w0-5b59aa-openshift-default-ff449c75d-6ckh5\" (UID: \"3a21aba3-2483-46a3-8b7f-acac0d646664\") " pod="openshift-ingress/e2e-worker-w0-5b59aa-openshift-default-ff449c75d-6ckh5"
Sep 16 02:30:47.415174 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:30:47.415151    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3a21aba3-2483-46a3-8b7f-acac0d646664-istio-token\") pod \"e2e-worker-w0-5b59aa-openshift-default-ff449c75d-6ckh5\" (UID: \"3a21aba3-2483-46a3-8b7f-acac0d646664\") " pod="openshift-ingress/e2e-worker-w0-5b59aa-openshift-default-ff449c75d-6ckh5"
Sep 16 02:30:47.533169 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:30:47.533083    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-5b59aa-openshift-default-ff449c75d-6ckh5"
Sep 16 02:30:47.818223 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:30:47.818199    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-5b59aa-openshift-default-ff449c75d-6ckh5"]
Sep 16 02:30:47.820378 ip-10-0-128-111 kubenswrapper[2583]: W0916 02:30:47.820348    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3a21aba3_2483_46a3_8b7f_acac0d646664.slice/crio-27c353be0c77316295e3fade9bdaf85dc2c08dbc5063b5190bea071bf2ecf421 WatchSource:0}: Error finding container 27c353be0c77316295e3fade9bdaf85dc2c08dbc5063b5190bea071bf2ecf421: Status 404 returned error can't find the container with id 27c353be0c77316295e3fade9bdaf85dc2c08dbc5063b5190bea071bf2ecf421
Sep 16 02:30:47.822430 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:30:47.822399    2583 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 16 02:30:47.822503 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:30:47.822459    2583 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 16 02:30:47.822503 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:30:47.822483    2583 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 16 02:30:48.339160 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:30:48.339123    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-5b59aa-openshift-default-ff449c75d-6ckh5" event={"ID":"3a21aba3-2483-46a3-8b7f-acac0d646664","Type":"ContainerStarted","Data":"491b4b4e6dadc31073436a80f16e053c12b47ac8e6b4e1965b8e476f0a458623"}
Sep 16 02:30:48.339670 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:30:48.339173    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-5b59aa-openshift-default-ff449c75d-6ckh5" event={"ID":"3a21aba3-2483-46a3-8b7f-acac0d646664","Type":"ContainerStarted","Data":"27c353be0c77316295e3fade9bdaf85dc2c08dbc5063b5190bea071bf2ecf421"}
Sep 16 02:30:48.365808 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:30:48.360898    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-5b59aa-openshift-default-ff449c75d-6ckh5" podStartSLOduration=1.360878443 podStartE2EDuration="1.360878443s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 02:30:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 02:30:48.35906573 +0000 UTC m=+726.826854518" watchObservedRunningTime="2026-09-16 02:30:48.360878443 +0000 UTC m=+726.828667226"
Sep 16 02:30:48.534134 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:30:48.534102    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-5b59aa-openshift-default-ff449c75d-6ckh5"
Sep 16 02:30:48.539260 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:30:48.539233    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-5b59aa-openshift-default-ff449c75d-6ckh5"
Sep 16 02:30:49.344130 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:30:49.344096    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-5b59aa-openshift-default-ff449c75d-6ckh5"
Sep 16 02:30:49.345061 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:30:49.345042    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-5b59aa-openshift-default-ff449c75d-6ckh5"
Sep 16 02:30:53.801845 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:30:53.801812    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-5b59aa-786d4c9486-fp9xw"]
Sep 16 02:30:53.805559 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:30:53.805527    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-5b59aa-786d4c9486-fp9xw"
Sep 16 02:30:53.807927 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:30:53.807906    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w0-5b59aa\""
Sep 16 02:30:53.808035 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:30:53.807948    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w0-5b59aa-dockercfg-kp9gx\""
Sep 16 02:30:53.814191 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:30:53.814170    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-5b59aa-786d4c9486-fp9xw"]
Sep 16 02:30:53.864967 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:30:53.864928    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/11497a78-de4b-493b-b889-420919a02a69-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w0-5b59aa-786d4c9486-fp9xw\" (UID: \"11497a78-de4b-493b-b889-420919a02a69\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-5b59aa-786d4c9486-fp9xw"
Sep 16 02:30:53.865171 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:30:53.864973    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qsk2z\" (UniqueName: \"kubernetes.io/projected/11497a78-de4b-493b-b889-420919a02a69-kube-api-access-qsk2z\") pod \"payload-pre-processing-e2e-worker-w0-5b59aa-786d4c9486-fp9xw\" (UID: \"11497a78-de4b-493b-b889-420919a02a69\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-5b59aa-786d4c9486-fp9xw"
Sep 16 02:30:53.972313 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:30:53.972274    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/11497a78-de4b-493b-b889-420919a02a69-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w0-5b59aa-786d4c9486-fp9xw\" (UID: \"11497a78-de4b-493b-b889-420919a02a69\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-5b59aa-786d4c9486-fp9xw"
Sep 16 02:30:53.972486 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:30:53.972330    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qsk2z\" (UniqueName: \"kubernetes.io/projected/11497a78-de4b-493b-b889-420919a02a69-kube-api-access-qsk2z\") pod \"payload-pre-processing-e2e-worker-w0-5b59aa-786d4c9486-fp9xw\" (UID: \"11497a78-de4b-493b-b889-420919a02a69\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-5b59aa-786d4c9486-fp9xw"
Sep 16 02:30:53.973001 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:30:53.972981    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/11497a78-de4b-493b-b889-420919a02a69-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w0-5b59aa-786d4c9486-fp9xw\" (UID: \"11497a78-de4b-493b-b889-420919a02a69\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-5b59aa-786d4c9486-fp9xw"
Sep 16 02:30:53.979801 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:30:53.979777    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qsk2z\" (UniqueName: \"kubernetes.io/projected/11497a78-de4b-493b-b889-420919a02a69-kube-api-access-qsk2z\") pod \"payload-pre-processing-e2e-worker-w0-5b59aa-786d4c9486-fp9xw\" (UID: \"11497a78-de4b-493b-b889-420919a02a69\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-5b59aa-786d4c9486-fp9xw"
Sep 16 02:30:54.117581 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:30:54.117534    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-5b59aa-786d4c9486-fp9xw"
Sep 16 02:30:54.387452 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:30:54.387428    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-5b59aa-786d4c9486-fp9xw"]
Sep 16 02:30:54.389688 ip-10-0-128-111 kubenswrapper[2583]: W0916 02:30:54.389665    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod11497a78_de4b_493b_b889_420919a02a69.slice/crio-f1d9d8fe72c03d6667ccb1e7f799e2148b10fb10d627c177acc0582956cef25d WatchSource:0}: Error finding container f1d9d8fe72c03d6667ccb1e7f799e2148b10fb10d627c177acc0582956cef25d: Status 404 returned error can't find the container with id f1d9d8fe72c03d6667ccb1e7f799e2148b10fb10d627c177acc0582956cef25d
Sep 16 02:30:55.372956 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:30:55.372917    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-5b59aa-786d4c9486-fp9xw" event={"ID":"11497a78-de4b-493b-b889-420919a02a69","Type":"ContainerStarted","Data":"f1d9d8fe72c03d6667ccb1e7f799e2148b10fb10d627c177acc0582956cef25d"}
Sep 16 02:30:56.250475 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:30:56.250433    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-fzl8t"]
Sep 16 02:30:56.379635 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:30:56.379590    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-5b59aa-786d4c9486-fp9xw" event={"ID":"11497a78-de4b-493b-b889-420919a02a69","Type":"ContainerStarted","Data":"d0001a11fdf1b930520cc823d425ab6b8697f27d3af5bf5885602a3176bf4dae"}
Sep 16 02:30:56.380091 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:30:56.379668    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-5b59aa-786d4c9486-fp9xw"
Sep 16 02:30:56.395907 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:30:56.395857    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-5b59aa-786d4c9486-fp9xw" podStartSLOduration=1.989333639 podStartE2EDuration="3.395839004s" totalImagesPullingTime="1.406505365s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 02:30:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 02:30:56.395018648 +0000 UTC m=+734.862807432" watchObservedRunningTime="2026-09-16 02:30:56.395839004 +0000 UTC m=+734.863627787"
Sep 16 02:31:05.385750 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:31:05.385706    2583 patch_prober.go:28] interesting pod/e2e-worker-w0-5b59aa-openshift-default-ff449c75d-6ckh5 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.33:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Sep 16 02:31:05.386140 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:31:05.385784    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-5b59aa-openshift-default-ff449c75d-6ckh5" podUID="3a21aba3-2483-46a3-8b7f-acac0d646664" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.33:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Sep 16 02:31:07.386508 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:31:07.386484    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-5b59aa-786d4c9486-fp9xw"
Sep 16 02:31:07.432866 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:31:07.432839    2583 generic.go:356] "Generic (PLEG): container finished" podID="3a21aba3-2483-46a3-8b7f-acac0d646664" containerID="491b4b4e6dadc31073436a80f16e053c12b47ac8e6b4e1965b8e476f0a458623" exitCode=0
Sep 16 02:31:07.433042 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:31:07.432924    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-5b59aa-openshift-default-ff449c75d-6ckh5" event={"ID":"3a21aba3-2483-46a3-8b7f-acac0d646664","Type":"ContainerDied","Data":"491b4b4e6dadc31073436a80f16e053c12b47ac8e6b4e1965b8e476f0a458623"}
Sep 16 02:31:07.433322 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:31:07.433306    2583 scope.go:117] "RemoveContainer" containerID="491b4b4e6dadc31073436a80f16e053c12b47ac8e6b4e1965b8e476f0a458623"
Sep 16 02:31:07.435655 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:31:07.434964    2583 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 16 02:31:07.435655 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:31:07.435129    2583 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 16 02:31:07.435655 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:31:07.435225    2583 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 16 02:31:08.444322 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:31:08.444283    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-5b59aa-openshift-default-ff449c75d-6ckh5" event={"ID":"3a21aba3-2483-46a3-8b7f-acac0d646664","Type":"ContainerStarted","Data":"3c162bd7fdd33b01c85157b6d75034927fa5f72fc3e6bc6938cac244272ad9db"}
Sep 16 02:31:08.534297 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:31:08.534248    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-5b59aa-openshift-default-ff449c75d-6ckh5"
Sep 16 02:31:08.534745 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:31:08.534715    2583 patch_prober.go:28] interesting pod/e2e-worker-w0-5b59aa-openshift-default-ff449c75d-6ckh5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.33:15021/healthz/ready\": dial tcp 10.134.0.33:15021: connect: connection refused" start-of-body=
Sep 16 02:31:08.534860 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:31:08.534784    2583 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-5b59aa-openshift-default-ff449c75d-6ckh5" podUID="3a21aba3-2483-46a3-8b7f-acac0d646664" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.33:15021/healthz/ready\": dial tcp 10.134.0.33:15021: connect: connection refused"
Sep 16 02:31:09.537576 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:31:09.533995    2583 patch_prober.go:28] interesting pod/e2e-worker-w0-5b59aa-openshift-default-ff449c75d-6ckh5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.33:15021/healthz/ready\": dial tcp 10.134.0.33:15021: connect: connection refused" start-of-body=
Sep 16 02:31:09.537576 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:31:09.534072    2583 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-5b59aa-openshift-default-ff449c75d-6ckh5" podUID="3a21aba3-2483-46a3-8b7f-acac0d646664" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.33:15021/healthz/ready\": dial tcp 10.134.0.33:15021: connect: connection refused"
Sep 16 02:31:10.533677 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:31:10.533631    2583 patch_prober.go:28] interesting pod/e2e-worker-w0-5b59aa-openshift-default-ff449c75d-6ckh5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.33:15021/healthz/ready\": dial tcp 10.134.0.33:15021: connect: connection refused" start-of-body=
Sep 16 02:31:10.533888 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:31:10.533706    2583 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-5b59aa-openshift-default-ff449c75d-6ckh5" podUID="3a21aba3-2483-46a3-8b7f-acac0d646664" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.33:15021/healthz/ready\": dial tcp 10.134.0.33:15021: connect: connection refused"
Sep 16 02:31:11.533910 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:31:11.533872    2583 patch_prober.go:28] interesting pod/e2e-worker-w0-5b59aa-openshift-default-ff449c75d-6ckh5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.33:15021/healthz/ready\": dial tcp 10.134.0.33:15021: connect: connection refused" start-of-body=
Sep 16 02:31:11.534517 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:31:11.534482    2583 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-5b59aa-openshift-default-ff449c75d-6ckh5" podUID="3a21aba3-2483-46a3-8b7f-acac0d646664" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.33:15021/healthz/ready\": dial tcp 10.134.0.33:15021: connect: connection refused"
Sep 16 02:31:12.533956 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:31:12.533909    2583 patch_prober.go:28] interesting pod/e2e-worker-w0-5b59aa-openshift-default-ff449c75d-6ckh5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.33:15021/healthz/ready\": dial tcp 10.134.0.33:15021: connect: connection refused" start-of-body=
Sep 16 02:31:12.534465 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:31:12.533995    2583 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-5b59aa-openshift-default-ff449c75d-6ckh5" podUID="3a21aba3-2483-46a3-8b7f-acac0d646664" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.33:15021/healthz/ready\": dial tcp 10.134.0.33:15021: connect: connection refused"
Sep 16 02:31:13.534487 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:31:13.534448    2583 patch_prober.go:28] interesting pod/e2e-worker-w0-5b59aa-openshift-default-ff449c75d-6ckh5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.33:15021/healthz/ready\": dial tcp 10.134.0.33:15021: connect: connection refused" start-of-body=
Sep 16 02:31:13.534901 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:31:13.534519    2583 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-5b59aa-openshift-default-ff449c75d-6ckh5" podUID="3a21aba3-2483-46a3-8b7f-acac0d646664" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.33:15021/healthz/ready\": dial tcp 10.134.0.33:15021: connect: connection refused"
Sep 16 02:31:14.537445 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:31:14.537417    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-5b59aa-openshift-default-ff449c75d-6ckh5"
Sep 16 02:31:14.537875 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:31:14.537783    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-5b59aa-openshift-default-ff449c75d-6ckh5"
Sep 16 02:31:14.538426 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:31:14.538411    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-5b59aa-openshift-default-ff449c75d-6ckh5"
Sep 16 02:31:15.587967 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:31:15.587932    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-da6c4f-openshift-default-7d6598f44-fg5qr"]
Sep 16 02:31:15.592716 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:31:15.592693    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-da6c4f-openshift-default-7d6598f44-fg5qr"
Sep 16 02:31:15.595088 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:31:15.595067    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-da6c4f-openshift-default-dockercfg-fxkn8\""
Sep 16 02:31:15.605717 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:31:15.605687    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-da6c4f-openshift-default-7d6598f44-fg5qr"]
Sep 16 02:31:15.693429 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:31:15.693393    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/88e2b6bf-fe2d-40ea-afdc-a59f8b12f455-workload-socket\") pod \"e2e-inf-b-da6c4f-openshift-default-7d6598f44-fg5qr\" (UID: \"88e2b6bf-fe2d-40ea-afdc-a59f8b12f455\") " pod="openshift-ingress/e2e-inf-b-da6c4f-openshift-default-7d6598f44-fg5qr"
Sep 16 02:31:15.693429 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:31:15.693431    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/88e2b6bf-fe2d-40ea-afdc-a59f8b12f455-workload-certs\") pod \"e2e-inf-b-da6c4f-openshift-default-7d6598f44-fg5qr\" (UID: \"88e2b6bf-fe2d-40ea-afdc-a59f8b12f455\") " pod="openshift-ingress/e2e-inf-b-da6c4f-openshift-default-7d6598f44-fg5qr"
Sep 16 02:31:15.693713 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:31:15.693447    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/88e2b6bf-fe2d-40ea-afdc-a59f8b12f455-istio-token\") pod \"e2e-inf-b-da6c4f-openshift-default-7d6598f44-fg5qr\" (UID: \"88e2b6bf-fe2d-40ea-afdc-a59f8b12f455\") " pod="openshift-ingress/e2e-inf-b-da6c4f-openshift-default-7d6598f44-fg5qr"
Sep 16 02:31:15.693713 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:31:15.693469    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/88e2b6bf-fe2d-40ea-afdc-a59f8b12f455-istio-envoy\") pod \"e2e-inf-b-da6c4f-openshift-default-7d6598f44-fg5qr\" (UID: \"88e2b6bf-fe2d-40ea-afdc-a59f8b12f455\") " pod="openshift-ingress/e2e-inf-b-da6c4f-openshift-default-7d6598f44-fg5qr"
Sep 16 02:31:15.693713 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:31:15.693627    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/88e2b6bf-fe2d-40ea-afdc-a59f8b12f455-credential-socket\") pod \"e2e-inf-b-da6c4f-openshift-default-7d6598f44-fg5qr\" (UID: \"88e2b6bf-fe2d-40ea-afdc-a59f8b12f455\") " pod="openshift-ingress/e2e-inf-b-da6c4f-openshift-default-7d6598f44-fg5qr"
Sep 16 02:31:15.693713 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:31:15.693661    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/88e2b6bf-fe2d-40ea-afdc-a59f8b12f455-istio-data\") pod \"e2e-inf-b-da6c4f-openshift-default-7d6598f44-fg5qr\" (UID: \"88e2b6bf-fe2d-40ea-afdc-a59f8b12f455\") " pod="openshift-ingress/e2e-inf-b-da6c4f-openshift-default-7d6598f44-fg5qr"
Sep 16 02:31:15.693891 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:31:15.693718    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/88e2b6bf-fe2d-40ea-afdc-a59f8b12f455-istio-podinfo\") pod \"e2e-inf-b-da6c4f-openshift-default-7d6598f44-fg5qr\" (UID: \"88e2b6bf-fe2d-40ea-afdc-a59f8b12f455\") " pod="openshift-ingress/e2e-inf-b-da6c4f-openshift-default-7d6598f44-fg5qr"
Sep 16 02:31:15.693891 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:31:15.693768    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tdnxk\" (UniqueName: \"kubernetes.io/projected/88e2b6bf-fe2d-40ea-afdc-a59f8b12f455-kube-api-access-tdnxk\") pod \"e2e-inf-b-da6c4f-openshift-default-7d6598f44-fg5qr\" (UID: \"88e2b6bf-fe2d-40ea-afdc-a59f8b12f455\") " pod="openshift-ingress/e2e-inf-b-da6c4f-openshift-default-7d6598f44-fg5qr"
Sep 16 02:31:15.693891 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:31:15.693791    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/88e2b6bf-fe2d-40ea-afdc-a59f8b12f455-istiod-ca-cert\") pod \"e2e-inf-b-da6c4f-openshift-default-7d6598f44-fg5qr\" (UID: \"88e2b6bf-fe2d-40ea-afdc-a59f8b12f455\") " pod="openshift-ingress/e2e-inf-b-da6c4f-openshift-default-7d6598f44-fg5qr"
Sep 16 02:31:15.794752 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:31:15.794714    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/88e2b6bf-fe2d-40ea-afdc-a59f8b12f455-credential-socket\") pod \"e2e-inf-b-da6c4f-openshift-default-7d6598f44-fg5qr\" (UID: \"88e2b6bf-fe2d-40ea-afdc-a59f8b12f455\") " pod="openshift-ingress/e2e-inf-b-da6c4f-openshift-default-7d6598f44-fg5qr"
Sep 16 02:31:15.794752 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:31:15.794753    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/88e2b6bf-fe2d-40ea-afdc-a59f8b12f455-istio-data\") pod \"e2e-inf-b-da6c4f-openshift-default-7d6598f44-fg5qr\" (UID: \"88e2b6bf-fe2d-40ea-afdc-a59f8b12f455\") " pod="openshift-ingress/e2e-inf-b-da6c4f-openshift-default-7d6598f44-fg5qr"
Sep 16 02:31:15.795012 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:31:15.794771    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/88e2b6bf-fe2d-40ea-afdc-a59f8b12f455-istio-podinfo\") pod \"e2e-inf-b-da6c4f-openshift-default-7d6598f44-fg5qr\" (UID: \"88e2b6bf-fe2d-40ea-afdc-a59f8b12f455\") " pod="openshift-ingress/e2e-inf-b-da6c4f-openshift-default-7d6598f44-fg5qr"
Sep 16 02:31:15.795012 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:31:15.794800    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tdnxk\" (UniqueName: \"kubernetes.io/projected/88e2b6bf-fe2d-40ea-afdc-a59f8b12f455-kube-api-access-tdnxk\") pod \"e2e-inf-b-da6c4f-openshift-default-7d6598f44-fg5qr\" (UID: \"88e2b6bf-fe2d-40ea-afdc-a59f8b12f455\") " pod="openshift-ingress/e2e-inf-b-da6c4f-openshift-default-7d6598f44-fg5qr"
Sep 16 02:31:15.795012 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:31:15.794824    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/88e2b6bf-fe2d-40ea-afdc-a59f8b12f455-istiod-ca-cert\") pod \"e2e-inf-b-da6c4f-openshift-default-7d6598f44-fg5qr\" (UID: \"88e2b6bf-fe2d-40ea-afdc-a59f8b12f455\") " pod="openshift-ingress/e2e-inf-b-da6c4f-openshift-default-7d6598f44-fg5qr"
Sep 16 02:31:15.795012 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:31:15.794887    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/88e2b6bf-fe2d-40ea-afdc-a59f8b12f455-workload-socket\") pod \"e2e-inf-b-da6c4f-openshift-default-7d6598f44-fg5qr\" (UID: \"88e2b6bf-fe2d-40ea-afdc-a59f8b12f455\") " pod="openshift-ingress/e2e-inf-b-da6c4f-openshift-default-7d6598f44-fg5qr"
Sep 16 02:31:15.795012 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:31:15.794908    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/88e2b6bf-fe2d-40ea-afdc-a59f8b12f455-workload-certs\") pod \"e2e-inf-b-da6c4f-openshift-default-7d6598f44-fg5qr\" (UID: \"88e2b6bf-fe2d-40ea-afdc-a59f8b12f455\") " pod="openshift-ingress/e2e-inf-b-da6c4f-openshift-default-7d6598f44-fg5qr"
Sep 16 02:31:15.795012 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:31:15.794930    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/88e2b6bf-fe2d-40ea-afdc-a59f8b12f455-istio-token\") pod \"e2e-inf-b-da6c4f-openshift-default-7d6598f44-fg5qr\" (UID: \"88e2b6bf-fe2d-40ea-afdc-a59f8b12f455\") " pod="openshift-ingress/e2e-inf-b-da6c4f-openshift-default-7d6598f44-fg5qr"
Sep 16 02:31:15.795012 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:31:15.794964    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/88e2b6bf-fe2d-40ea-afdc-a59f8b12f455-istio-envoy\") pod \"e2e-inf-b-da6c4f-openshift-default-7d6598f44-fg5qr\" (UID: \"88e2b6bf-fe2d-40ea-afdc-a59f8b12f455\") " pod="openshift-ingress/e2e-inf-b-da6c4f-openshift-default-7d6598f44-fg5qr"
Sep 16 02:31:15.795355 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:31:15.795214    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/88e2b6bf-fe2d-40ea-afdc-a59f8b12f455-credential-socket\") pod \"e2e-inf-b-da6c4f-openshift-default-7d6598f44-fg5qr\" (UID: \"88e2b6bf-fe2d-40ea-afdc-a59f8b12f455\") " pod="openshift-ingress/e2e-inf-b-da6c4f-openshift-default-7d6598f44-fg5qr"
Sep 16 02:31:15.795355 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:31:15.795262    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/88e2b6bf-fe2d-40ea-afdc-a59f8b12f455-istio-data\") pod \"e2e-inf-b-da6c4f-openshift-default-7d6598f44-fg5qr\" (UID: \"88e2b6bf-fe2d-40ea-afdc-a59f8b12f455\") " pod="openshift-ingress/e2e-inf-b-da6c4f-openshift-default-7d6598f44-fg5qr"
Sep 16 02:31:15.795600 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:31:15.795537    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/88e2b6bf-fe2d-40ea-afdc-a59f8b12f455-workload-certs\") pod \"e2e-inf-b-da6c4f-openshift-default-7d6598f44-fg5qr\" (UID: \"88e2b6bf-fe2d-40ea-afdc-a59f8b12f455\") " pod="openshift-ingress/e2e-inf-b-da6c4f-openshift-default-7d6598f44-fg5qr"
Sep 16 02:31:15.795600 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:31:15.795576    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/88e2b6bf-fe2d-40ea-afdc-a59f8b12f455-workload-socket\") pod \"e2e-inf-b-da6c4f-openshift-default-7d6598f44-fg5qr\" (UID: \"88e2b6bf-fe2d-40ea-afdc-a59f8b12f455\") " pod="openshift-ingress/e2e-inf-b-da6c4f-openshift-default-7d6598f44-fg5qr"
Sep 16 02:31:15.795823 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:31:15.795802    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/88e2b6bf-fe2d-40ea-afdc-a59f8b12f455-istiod-ca-cert\") pod \"e2e-inf-b-da6c4f-openshift-default-7d6598f44-fg5qr\" (UID: \"88e2b6bf-fe2d-40ea-afdc-a59f8b12f455\") " pod="openshift-ingress/e2e-inf-b-da6c4f-openshift-default-7d6598f44-fg5qr"
Sep 16 02:31:15.797175 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:31:15.797149    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/88e2b6bf-fe2d-40ea-afdc-a59f8b12f455-istio-envoy\") pod \"e2e-inf-b-da6c4f-openshift-default-7d6598f44-fg5qr\" (UID: \"88e2b6bf-fe2d-40ea-afdc-a59f8b12f455\") " pod="openshift-ingress/e2e-inf-b-da6c4f-openshift-default-7d6598f44-fg5qr"
Sep 16 02:31:15.797314 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:31:15.797298    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/88e2b6bf-fe2d-40ea-afdc-a59f8b12f455-istio-podinfo\") pod \"e2e-inf-b-da6c4f-openshift-default-7d6598f44-fg5qr\" (UID: \"88e2b6bf-fe2d-40ea-afdc-a59f8b12f455\") " pod="openshift-ingress/e2e-inf-b-da6c4f-openshift-default-7d6598f44-fg5qr"
Sep 16 02:31:15.802022 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:31:15.801996    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/88e2b6bf-fe2d-40ea-afdc-a59f8b12f455-istio-token\") pod \"e2e-inf-b-da6c4f-openshift-default-7d6598f44-fg5qr\" (UID: \"88e2b6bf-fe2d-40ea-afdc-a59f8b12f455\") " pod="openshift-ingress/e2e-inf-b-da6c4f-openshift-default-7d6598f44-fg5qr"
Sep 16 02:31:15.802271 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:31:15.802254    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tdnxk\" (UniqueName: \"kubernetes.io/projected/88e2b6bf-fe2d-40ea-afdc-a59f8b12f455-kube-api-access-tdnxk\") pod \"e2e-inf-b-da6c4f-openshift-default-7d6598f44-fg5qr\" (UID: \"88e2b6bf-fe2d-40ea-afdc-a59f8b12f455\") " pod="openshift-ingress/e2e-inf-b-da6c4f-openshift-default-7d6598f44-fg5qr"
Sep 16 02:31:15.903409 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:31:15.903325    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-da6c4f-openshift-default-7d6598f44-fg5qr"
Sep 16 02:31:16.182591 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:31:16.182545    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-da6c4f-openshift-default-7d6598f44-fg5qr"]
Sep 16 02:31:16.184277 ip-10-0-128-111 kubenswrapper[2583]: W0916 02:31:16.184247    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod88e2b6bf_fe2d_40ea_afdc_a59f8b12f455.slice/crio-0d4656d1034889bc5e47d4457fadc45cc4e856dee2c5a10af72f4de03c1d261d WatchSource:0}: Error finding container 0d4656d1034889bc5e47d4457fadc45cc4e856dee2c5a10af72f4de03c1d261d: Status 404 returned error can't find the container with id 0d4656d1034889bc5e47d4457fadc45cc4e856dee2c5a10af72f4de03c1d261d
Sep 16 02:31:16.186583 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:31:16.186533    2583 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 16 02:31:16.186678 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:31:16.186617    2583 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 16 02:31:16.186678 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:31:16.186666    2583 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 16 02:31:16.481154 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:31:16.481055    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-da6c4f-openshift-default-7d6598f44-fg5qr" event={"ID":"88e2b6bf-fe2d-40ea-afdc-a59f8b12f455","Type":"ContainerStarted","Data":"bc8eb03c5367c491ba7df4dc61d8f9fcd2a15dbaaf1e12194c2e632beb67c50b"}
Sep 16 02:31:16.481154 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:31:16.481146    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-da6c4f-openshift-default-7d6598f44-fg5qr" event={"ID":"88e2b6bf-fe2d-40ea-afdc-a59f8b12f455","Type":"ContainerStarted","Data":"0d4656d1034889bc5e47d4457fadc45cc4e856dee2c5a10af72f4de03c1d261d"}
Sep 16 02:31:16.500296 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:31:16.500242    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-da6c4f-openshift-default-7d6598f44-fg5qr" podStartSLOduration=1.5002260010000001 podStartE2EDuration="1.500226001s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 02:31:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 02:31:16.498722563 +0000 UTC m=+754.966511347" watchObservedRunningTime="2026-09-16 02:31:16.500226001 +0000 UTC m=+754.968014783"
Sep 16 02:31:16.903662 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:31:16.903625    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-da6c4f-openshift-default-7d6598f44-fg5qr"
Sep 16 02:31:17.908789 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:31:17.908760    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-da6c4f-openshift-default-7d6598f44-fg5qr"
Sep 16 02:31:18.490472 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:31:18.490418    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-da6c4f-openshift-default-7d6598f44-fg5qr"
Sep 16 02:31:18.491528 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:31:18.491512    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-da6c4f-openshift-default-7d6598f44-fg5qr"
Sep 16 02:31:21.461780 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:31:21.461746    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-da6c4f-79c4c854b-vkd9b"]
Sep 16 02:31:21.465558 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:31:21.465531    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-da6c4f-79c4c854b-vkd9b"
Sep 16 02:31:21.468002 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:31:21.467978    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-da6c4f\""
Sep 16 02:31:21.473875 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:31:21.473853    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-da6c4f-79c4c854b-vkd9b"]
Sep 16 02:31:21.550493 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:31:21.550464    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8795109a-cd57-4312-9988-01f98259eb7f-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-da6c4f-79c4c854b-vkd9b\" (UID: \"8795109a-cd57-4312-9988-01f98259eb7f\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-da6c4f-79c4c854b-vkd9b"
Sep 16 02:31:21.550493 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:31:21.550496    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vk7mj\" (UniqueName: \"kubernetes.io/projected/8795109a-cd57-4312-9988-01f98259eb7f-kube-api-access-vk7mj\") pod \"payload-pre-processing-e2e-inf-b-da6c4f-79c4c854b-vkd9b\" (UID: \"8795109a-cd57-4312-9988-01f98259eb7f\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-da6c4f-79c4c854b-vkd9b"
Sep 16 02:31:21.603308 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:31:21.596822    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-da6c4f-7d7768c479-nc9hn"]
Sep 16 02:31:21.605172 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:31:21.605131    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-da6c4f-7d7768c479-nc9hn"
Sep 16 02:31:21.607009 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:31:21.606052    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-da6c4f-7d7768c479-nc9hn"]
Sep 16 02:31:21.608934 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:31:21.608908    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-b-da6c4f-dockercfg-7np7h\""
Sep 16 02:31:21.651919 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:31:21.651884    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kqwmk\" (UniqueName: \"kubernetes.io/projected/a80a9d20-dc4b-4020-96f8-8f1c9e0be8bf-kube-api-access-kqwmk\") pod \"payload-processing-e2e-inf-b-da6c4f-7d7768c479-nc9hn\" (UID: \"a80a9d20-dc4b-4020-96f8-8f1c9e0be8bf\") " pod="openshift-ingress/payload-processing-e2e-inf-b-da6c4f-7d7768c479-nc9hn"
Sep 16 02:31:21.652094 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:31:21.651966    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8795109a-cd57-4312-9988-01f98259eb7f-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-da6c4f-79c4c854b-vkd9b\" (UID: \"8795109a-cd57-4312-9988-01f98259eb7f\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-da6c4f-79c4c854b-vkd9b"
Sep 16 02:31:21.652094 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:31:21.651985    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vk7mj\" (UniqueName: \"kubernetes.io/projected/8795109a-cd57-4312-9988-01f98259eb7f-kube-api-access-vk7mj\") pod \"payload-pre-processing-e2e-inf-b-da6c4f-79c4c854b-vkd9b\" (UID: \"8795109a-cd57-4312-9988-01f98259eb7f\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-da6c4f-79c4c854b-vkd9b"
Sep 16 02:31:21.652094 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:31:21.652022    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a80a9d20-dc4b-4020-96f8-8f1c9e0be8bf-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-da6c4f-7d7768c479-nc9hn\" (UID: \"a80a9d20-dc4b-4020-96f8-8f1c9e0be8bf\") " pod="openshift-ingress/payload-processing-e2e-inf-b-da6c4f-7d7768c479-nc9hn"
Sep 16 02:31:21.652642 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:31:21.652621    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8795109a-cd57-4312-9988-01f98259eb7f-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-da6c4f-79c4c854b-vkd9b\" (UID: \"8795109a-cd57-4312-9988-01f98259eb7f\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-da6c4f-79c4c854b-vkd9b"
Sep 16 02:31:21.659815 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:31:21.659789    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vk7mj\" (UniqueName: \"kubernetes.io/projected/8795109a-cd57-4312-9988-01f98259eb7f-kube-api-access-vk7mj\") pod \"payload-pre-processing-e2e-inf-b-da6c4f-79c4c854b-vkd9b\" (UID: \"8795109a-cd57-4312-9988-01f98259eb7f\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-da6c4f-79c4c854b-vkd9b"
Sep 16 02:31:21.752872 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:31:21.752616    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a80a9d20-dc4b-4020-96f8-8f1c9e0be8bf-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-da6c4f-7d7768c479-nc9hn\" (UID: \"a80a9d20-dc4b-4020-96f8-8f1c9e0be8bf\") " pod="openshift-ingress/payload-processing-e2e-inf-b-da6c4f-7d7768c479-nc9hn"
Sep 16 02:31:21.752872 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:31:21.752721    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kqwmk\" (UniqueName: \"kubernetes.io/projected/a80a9d20-dc4b-4020-96f8-8f1c9e0be8bf-kube-api-access-kqwmk\") pod \"payload-processing-e2e-inf-b-da6c4f-7d7768c479-nc9hn\" (UID: \"a80a9d20-dc4b-4020-96f8-8f1c9e0be8bf\") " pod="openshift-ingress/payload-processing-e2e-inf-b-da6c4f-7d7768c479-nc9hn"
Sep 16 02:31:21.753468 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:31:21.753436    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a80a9d20-dc4b-4020-96f8-8f1c9e0be8bf-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-da6c4f-7d7768c479-nc9hn\" (UID: \"a80a9d20-dc4b-4020-96f8-8f1c9e0be8bf\") " pod="openshift-ingress/payload-processing-e2e-inf-b-da6c4f-7d7768c479-nc9hn"
Sep 16 02:31:21.760609 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:31:21.760589    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kqwmk\" (UniqueName: \"kubernetes.io/projected/a80a9d20-dc4b-4020-96f8-8f1c9e0be8bf-kube-api-access-kqwmk\") pod \"payload-processing-e2e-inf-b-da6c4f-7d7768c479-nc9hn\" (UID: \"a80a9d20-dc4b-4020-96f8-8f1c9e0be8bf\") " pod="openshift-ingress/payload-processing-e2e-inf-b-da6c4f-7d7768c479-nc9hn"
Sep 16 02:31:21.776942 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:31:21.776922    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-da6c4f-79c4c854b-vkd9b"
Sep 16 02:31:21.918623 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:31:21.918591    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-da6c4f-7d7768c479-nc9hn"
Sep 16 02:31:22.051502 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:31:22.051420    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-da6c4f-79c4c854b-vkd9b"]
Sep 16 02:31:22.054296 ip-10-0-128-111 kubenswrapper[2583]: W0916 02:31:22.054269    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8795109a_cd57_4312_9988_01f98259eb7f.slice/crio-c54a9a27005aa91b5f98d41c6bfd9422bb8d7853bc03d5e12de9648893229845 WatchSource:0}: Error finding container c54a9a27005aa91b5f98d41c6bfd9422bb8d7853bc03d5e12de9648893229845: Status 404 returned error can't find the container with id c54a9a27005aa91b5f98d41c6bfd9422bb8d7853bc03d5e12de9648893229845
Sep 16 02:31:22.188204 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:31:22.188181    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-da6c4f-7d7768c479-nc9hn"]
Sep 16 02:31:22.189789 ip-10-0-128-111 kubenswrapper[2583]: W0916 02:31:22.189761    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda80a9d20_dc4b_4020_96f8_8f1c9e0be8bf.slice/crio-25364ccaa8a7e6aaefd383f147b02178275bb7f23537c0827e678b1dfcef4eb3 WatchSource:0}: Error finding container 25364ccaa8a7e6aaefd383f147b02178275bb7f23537c0827e678b1dfcef4eb3: Status 404 returned error can't find the container with id 25364ccaa8a7e6aaefd383f147b02178275bb7f23537c0827e678b1dfcef4eb3
Sep 16 02:31:22.510008 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:31:22.509974    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-da6c4f-79c4c854b-vkd9b" event={"ID":"8795109a-cd57-4312-9988-01f98259eb7f","Type":"ContainerStarted","Data":"d32d962539aaab85b45ee5f62ac89798878915afe8449b31be45addd10ccbb34"}
Sep 16 02:31:22.510493 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:31:22.510027    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-da6c4f-79c4c854b-vkd9b" event={"ID":"8795109a-cd57-4312-9988-01f98259eb7f","Type":"ContainerStarted","Data":"c54a9a27005aa91b5f98d41c6bfd9422bb8d7853bc03d5e12de9648893229845"}
Sep 16 02:31:22.510493 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:31:22.510148    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-da6c4f-79c4c854b-vkd9b"
Sep 16 02:31:22.511690 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:31:22.511664    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-da6c4f-7d7768c479-nc9hn" event={"ID":"a80a9d20-dc4b-4020-96f8-8f1c9e0be8bf","Type":"ContainerStarted","Data":"00ae671ee092da03ffcb8c3766dcd9ac37e3d941b394fb522d1539a1703fa7ee"}
Sep 16 02:31:22.511823 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:31:22.511714    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-b-da6c4f-7d7768c479-nc9hn"
Sep 16 02:31:22.511823 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:31:22.511738    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-da6c4f-7d7768c479-nc9hn" event={"ID":"a80a9d20-dc4b-4020-96f8-8f1c9e0be8bf","Type":"ContainerStarted","Data":"25364ccaa8a7e6aaefd383f147b02178275bb7f23537c0827e678b1dfcef4eb3"}
Sep 16 02:31:22.525395 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:31:22.525335    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-da6c4f-79c4c854b-vkd9b" podStartSLOduration=1.5253204839999999 podStartE2EDuration="1.525320484s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 02:31:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 02:31:22.523945522 +0000 UTC m=+760.991734304" watchObservedRunningTime="2026-09-16 02:31:22.525320484 +0000 UTC m=+760.993109265"
Sep 16 02:31:22.540418 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:31:22.540371    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-b-da6c4f-7d7768c479-nc9hn" podStartSLOduration=1.540353585 podStartE2EDuration="1.540353585s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 02:31:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 02:31:22.537517721 +0000 UTC m=+761.005306507" watchObservedRunningTime="2026-09-16 02:31:22.540353585 +0000 UTC m=+761.008142370"
Sep 16 02:31:25.053133 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:31:25.053090    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-fzl8t"]
Sep 16 02:31:25.751356 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:31:25.751327    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w3-a3d7eb/simulator-w3-a3d7eb-kserve-7bcf597646-zj6qt"]
Sep 16 02:31:25.755200 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:31:25.755176    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-a3d7eb/simulator-w3-a3d7eb-kserve-7bcf597646-zj6qt"
Sep 16 02:31:25.758869 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:31:25.758840    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w3-a3d7eb\"/\"openshift-service-ca.crt\""
Sep 16 02:31:25.759040 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:31:25.758896    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w3-a3d7eb\"/\"kube-root-ca.crt\""
Sep 16 02:31:25.759040 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:31:25.758922    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-a3d7eb\"/\"default-dockercfg-klnxc\""
Sep 16 02:31:25.759040 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:31:25.758894    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-a3d7eb\"/\"simulator-w3-a3d7eb-kserve-self-signed-certs\""
Sep 16 02:31:25.767148 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:31:25.767125    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-a3d7eb/simulator-w3-a3d7eb-kserve-7bcf597646-zj6qt"]
Sep 16 02:31:25.895231 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:31:25.895193    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cc9488c9-649c-4992-9c1e-fa311434f6d5-dshm\") pod \"simulator-w3-a3d7eb-kserve-7bcf597646-zj6qt\" (UID: \"cc9488c9-649c-4992-9c1e-fa311434f6d5\") " pod="e2e-models-e2e-worker-w3-a3d7eb/simulator-w3-a3d7eb-kserve-7bcf597646-zj6qt"
Sep 16 02:31:25.895408 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:31:25.895244    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cc9488c9-649c-4992-9c1e-fa311434f6d5-model-cache\") pod \"simulator-w3-a3d7eb-kserve-7bcf597646-zj6qt\" (UID: \"cc9488c9-649c-4992-9c1e-fa311434f6d5\") " pod="e2e-models-e2e-worker-w3-a3d7eb/simulator-w3-a3d7eb-kserve-7bcf597646-zj6qt"
Sep 16 02:31:25.895408 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:31:25.895275    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cc9488c9-649c-4992-9c1e-fa311434f6d5-home\") pod \"simulator-w3-a3d7eb-kserve-7bcf597646-zj6qt\" (UID: \"cc9488c9-649c-4992-9c1e-fa311434f6d5\") " pod="e2e-models-e2e-worker-w3-a3d7eb/simulator-w3-a3d7eb-kserve-7bcf597646-zj6qt"
Sep 16 02:31:25.895408 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:31:25.895299    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d2zqt\" (UniqueName: \"kubernetes.io/projected/cc9488c9-649c-4992-9c1e-fa311434f6d5-kube-api-access-d2zqt\") pod \"simulator-w3-a3d7eb-kserve-7bcf597646-zj6qt\" (UID: \"cc9488c9-649c-4992-9c1e-fa311434f6d5\") " pod="e2e-models-e2e-worker-w3-a3d7eb/simulator-w3-a3d7eb-kserve-7bcf597646-zj6qt"
Sep 16 02:31:25.895408 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:31:25.895326    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cc9488c9-649c-4992-9c1e-fa311434f6d5-tmp-dir\") pod \"simulator-w3-a3d7eb-kserve-7bcf597646-zj6qt\" (UID: \"cc9488c9-649c-4992-9c1e-fa311434f6d5\") " pod="e2e-models-e2e-worker-w3-a3d7eb/simulator-w3-a3d7eb-kserve-7bcf597646-zj6qt"
Sep 16 02:31:25.895574 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:31:25.895446    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cc9488c9-649c-4992-9c1e-fa311434f6d5-tls-certs\") pod \"simulator-w3-a3d7eb-kserve-7bcf597646-zj6qt\" (UID: \"cc9488c9-649c-4992-9c1e-fa311434f6d5\") " pod="e2e-models-e2e-worker-w3-a3d7eb/simulator-w3-a3d7eb-kserve-7bcf597646-zj6qt"
Sep 16 02:31:25.996702 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:31:25.996659    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cc9488c9-649c-4992-9c1e-fa311434f6d5-home\") pod \"simulator-w3-a3d7eb-kserve-7bcf597646-zj6qt\" (UID: \"cc9488c9-649c-4992-9c1e-fa311434f6d5\") " pod="e2e-models-e2e-worker-w3-a3d7eb/simulator-w3-a3d7eb-kserve-7bcf597646-zj6qt"
Sep 16 02:31:25.996910 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:31:25.996710    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d2zqt\" (UniqueName: \"kubernetes.io/projected/cc9488c9-649c-4992-9c1e-fa311434f6d5-kube-api-access-d2zqt\") pod \"simulator-w3-a3d7eb-kserve-7bcf597646-zj6qt\" (UID: \"cc9488c9-649c-4992-9c1e-fa311434f6d5\") " pod="e2e-models-e2e-worker-w3-a3d7eb/simulator-w3-a3d7eb-kserve-7bcf597646-zj6qt"
Sep 16 02:31:25.996910 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:31:25.996742    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cc9488c9-649c-4992-9c1e-fa311434f6d5-tmp-dir\") pod \"simulator-w3-a3d7eb-kserve-7bcf597646-zj6qt\" (UID: \"cc9488c9-649c-4992-9c1e-fa311434f6d5\") " pod="e2e-models-e2e-worker-w3-a3d7eb/simulator-w3-a3d7eb-kserve-7bcf597646-zj6qt"
Sep 16 02:31:25.996910 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:31:25.996790    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cc9488c9-649c-4992-9c1e-fa311434f6d5-tls-certs\") pod \"simulator-w3-a3d7eb-kserve-7bcf597646-zj6qt\" (UID: \"cc9488c9-649c-4992-9c1e-fa311434f6d5\") " pod="e2e-models-e2e-worker-w3-a3d7eb/simulator-w3-a3d7eb-kserve-7bcf597646-zj6qt"
Sep 16 02:31:25.996910 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:31:25.996874    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cc9488c9-649c-4992-9c1e-fa311434f6d5-dshm\") pod \"simulator-w3-a3d7eb-kserve-7bcf597646-zj6qt\" (UID: \"cc9488c9-649c-4992-9c1e-fa311434f6d5\") " pod="e2e-models-e2e-worker-w3-a3d7eb/simulator-w3-a3d7eb-kserve-7bcf597646-zj6qt"
Sep 16 02:31:25.997123 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:31:25.996914    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cc9488c9-649c-4992-9c1e-fa311434f6d5-model-cache\") pod \"simulator-w3-a3d7eb-kserve-7bcf597646-zj6qt\" (UID: \"cc9488c9-649c-4992-9c1e-fa311434f6d5\") " pod="e2e-models-e2e-worker-w3-a3d7eb/simulator-w3-a3d7eb-kserve-7bcf597646-zj6qt"
Sep 16 02:31:25.997191 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:31:25.997155    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cc9488c9-649c-4992-9c1e-fa311434f6d5-home\") pod \"simulator-w3-a3d7eb-kserve-7bcf597646-zj6qt\" (UID: \"cc9488c9-649c-4992-9c1e-fa311434f6d5\") " pod="e2e-models-e2e-worker-w3-a3d7eb/simulator-w3-a3d7eb-kserve-7bcf597646-zj6qt"
Sep 16 02:31:25.997191 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:31:25.997178    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cc9488c9-649c-4992-9c1e-fa311434f6d5-tmp-dir\") pod \"simulator-w3-a3d7eb-kserve-7bcf597646-zj6qt\" (UID: \"cc9488c9-649c-4992-9c1e-fa311434f6d5\") " pod="e2e-models-e2e-worker-w3-a3d7eb/simulator-w3-a3d7eb-kserve-7bcf597646-zj6qt"
Sep 16 02:31:25.997322 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:31:25.997291    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cc9488c9-649c-4992-9c1e-fa311434f6d5-model-cache\") pod \"simulator-w3-a3d7eb-kserve-7bcf597646-zj6qt\" (UID: \"cc9488c9-649c-4992-9c1e-fa311434f6d5\") " pod="e2e-models-e2e-worker-w3-a3d7eb/simulator-w3-a3d7eb-kserve-7bcf597646-zj6qt"
Sep 16 02:31:26.000003 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:31:25.999978    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cc9488c9-649c-4992-9c1e-fa311434f6d5-tls-certs\") pod \"simulator-w3-a3d7eb-kserve-7bcf597646-zj6qt\" (UID: \"cc9488c9-649c-4992-9c1e-fa311434f6d5\") " pod="e2e-models-e2e-worker-w3-a3d7eb/simulator-w3-a3d7eb-kserve-7bcf597646-zj6qt"
Sep 16 02:31:26.000242 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:31:26.000222    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cc9488c9-649c-4992-9c1e-fa311434f6d5-dshm\") pod \"simulator-w3-a3d7eb-kserve-7bcf597646-zj6qt\" (UID: \"cc9488c9-649c-4992-9c1e-fa311434f6d5\") " pod="e2e-models-e2e-worker-w3-a3d7eb/simulator-w3-a3d7eb-kserve-7bcf597646-zj6qt"
Sep 16 02:31:26.004505 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:31:26.004447    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-d2zqt\" (UniqueName: \"kubernetes.io/projected/cc9488c9-649c-4992-9c1e-fa311434f6d5-kube-api-access-d2zqt\") pod \"simulator-w3-a3d7eb-kserve-7bcf597646-zj6qt\" (UID: \"cc9488c9-649c-4992-9c1e-fa311434f6d5\") " pod="e2e-models-e2e-worker-w3-a3d7eb/simulator-w3-a3d7eb-kserve-7bcf597646-zj6qt"
Sep 16 02:31:26.067195 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:31:26.067163    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-a3d7eb/simulator-w3-a3d7eb-kserve-7bcf597646-zj6qt"
Sep 16 02:31:26.353747 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:31:26.353710    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-a3d7eb/simulator-w3-a3d7eb-kserve-7bcf597646-zj6qt"]
Sep 16 02:31:26.355681 ip-10-0-128-111 kubenswrapper[2583]: W0916 02:31:26.355656    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podcc9488c9_649c_4992_9c1e_fa311434f6d5.slice/crio-adab4e626ea014bb2d443fcdd54f7147954b8dc560d89c83aaf31dd2e51e7e32 WatchSource:0}: Error finding container adab4e626ea014bb2d443fcdd54f7147954b8dc560d89c83aaf31dd2e51e7e32: Status 404 returned error can't find the container with id adab4e626ea014bb2d443fcdd54f7147954b8dc560d89c83aaf31dd2e51e7e32
Sep 16 02:31:26.536913 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:31:26.536862    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-a3d7eb/simulator-w3-a3d7eb-kserve-7bcf597646-zj6qt" event={"ID":"cc9488c9-649c-4992-9c1e-fa311434f6d5","Type":"ContainerStarted","Data":"00b3a10f8f77a6594565222cf05f2e6a5c46c5c6d787fa20b60f2968449e0690"}
Sep 16 02:31:26.537123 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:31:26.536925    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-a3d7eb/simulator-w3-a3d7eb-kserve-7bcf597646-zj6qt" event={"ID":"cc9488c9-649c-4992-9c1e-fa311434f6d5","Type":"ContainerStarted","Data":"adab4e626ea014bb2d443fcdd54f7147954b8dc560d89c83aaf31dd2e51e7e32"}
Sep 16 02:31:26.553433 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:31:26.553383    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-models-e2e-worker-w3-a3d7eb/simulator-w3-a3d7eb-kserve-7bcf597646-zj6qt" podStartSLOduration=1.553366341 podStartE2EDuration="1.553366341s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 02:31:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 02:31:26.55169208 +0000 UTC m=+765.019480872" watchObservedRunningTime="2026-09-16 02:31:26.553366341 +0000 UTC m=+765.021155139"
Sep 16 02:31:33.519501 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:31:33.519471    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-da6c4f-79c4c854b-vkd9b"
Sep 16 02:31:33.519914 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:31:33.519652    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-b-da6c4f-7d7768c479-nc9hn"
Sep 16 02:31:34.532723 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:31:34.532680    2583 patch_prober.go:28] interesting pod/e2e-inf-b-da6c4f-openshift-default-7d6598f44-fg5qr container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.35:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Sep 16 02:31:34.533106 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:31:34.532760    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-da6c4f-openshift-default-7d6598f44-fg5qr" podUID="88e2b6bf-fe2d-40ea-afdc-a59f8b12f455" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.35:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Sep 16 02:31:35.591021 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:31:35.590989    2583 generic.go:356] "Generic (PLEG): container finished" podID="88e2b6bf-fe2d-40ea-afdc-a59f8b12f455" containerID="bc8eb03c5367c491ba7df4dc61d8f9fcd2a15dbaaf1e12194c2e632beb67c50b" exitCode=0
Sep 16 02:31:35.591414 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:31:35.591069    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-da6c4f-openshift-default-7d6598f44-fg5qr" event={"ID":"88e2b6bf-fe2d-40ea-afdc-a59f8b12f455","Type":"ContainerDied","Data":"bc8eb03c5367c491ba7df4dc61d8f9fcd2a15dbaaf1e12194c2e632beb67c50b"}
Sep 16 02:31:35.591653 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:31:35.591635    2583 scope.go:117] "RemoveContainer" containerID="bc8eb03c5367c491ba7df4dc61d8f9fcd2a15dbaaf1e12194c2e632beb67c50b"
Sep 16 02:31:35.592993 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:31:35.592964    2583 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 16 02:31:35.593081 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:31:35.593065    2583 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 16 02:31:35.593128 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:31:35.593113    2583 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 16 02:31:36.067797 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:31:36.067714    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w3-a3d7eb/simulator-w3-a3d7eb-kserve-7bcf597646-zj6qt"
Sep 16 02:31:36.067797 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:31:36.067762    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-a3d7eb/simulator-w3-a3d7eb-kserve-7bcf597646-zj6qt"
Sep 16 02:31:36.080709 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:31:36.080682    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w3-a3d7eb/simulator-w3-a3d7eb-kserve-7bcf597646-zj6qt"
Sep 16 02:31:36.597701 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:31:36.597666    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-da6c4f-openshift-default-7d6598f44-fg5qr" event={"ID":"88e2b6bf-fe2d-40ea-afdc-a59f8b12f455","Type":"ContainerStarted","Data":"b0609a92ed841e0fa33f06b093579e6c313e8a41d41194eda4adae82ed8e4e46"}
Sep 16 02:31:36.608974 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:31:36.608950    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-a3d7eb/simulator-w3-a3d7eb-kserve-7bcf597646-zj6qt"
Sep 16 02:31:36.855101 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:31:36.855015    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-fzl8t"]
Sep 16 02:31:36.904513 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:31:36.904483    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-da6c4f-openshift-default-7d6598f44-fg5qr"
Sep 16 02:31:36.904700 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:31:36.904677    2583 patch_prober.go:28] interesting pod/e2e-inf-b-da6c4f-openshift-default-7d6598f44-fg5qr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.35:15021/healthz/ready\": dial tcp 10.134.0.35:15021: connect: connection refused" start-of-body=
Sep 16 02:31:36.904749 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:31:36.904735    2583 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-da6c4f-openshift-default-7d6598f44-fg5qr" podUID="88e2b6bf-fe2d-40ea-afdc-a59f8b12f455" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.35:15021/healthz/ready\": dial tcp 10.134.0.35:15021: connect: connection refused"
Sep 16 02:31:37.904441 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:31:37.904401    2583 patch_prober.go:28] interesting pod/e2e-inf-b-da6c4f-openshift-default-7d6598f44-fg5qr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.35:15021/healthz/ready\": dial tcp 10.134.0.35:15021: connect: connection refused" start-of-body=
Sep 16 02:31:37.904920 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:31:37.904466    2583 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-da6c4f-openshift-default-7d6598f44-fg5qr" podUID="88e2b6bf-fe2d-40ea-afdc-a59f8b12f455" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.35:15021/healthz/ready\": dial tcp 10.134.0.35:15021: connect: connection refused"
Sep 16 02:31:38.903972 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:31:38.903937    2583 patch_prober.go:28] interesting pod/e2e-inf-b-da6c4f-openshift-default-7d6598f44-fg5qr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.35:15021/healthz/ready\": dial tcp 10.134.0.35:15021: connect: connection refused" start-of-body=
Sep 16 02:31:38.904153 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:31:38.904003    2583 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-da6c4f-openshift-default-7d6598f44-fg5qr" podUID="88e2b6bf-fe2d-40ea-afdc-a59f8b12f455" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.35:15021/healthz/ready\": dial tcp 10.134.0.35:15021: connect: connection refused"
Sep 16 02:31:39.904661 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:31:39.904622    2583 patch_prober.go:28] interesting pod/e2e-inf-b-da6c4f-openshift-default-7d6598f44-fg5qr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.35:15021/healthz/ready\": dial tcp 10.134.0.35:15021: connect: connection refused" start-of-body=
Sep 16 02:31:39.905036 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:31:39.904682    2583 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-da6c4f-openshift-default-7d6598f44-fg5qr" podUID="88e2b6bf-fe2d-40ea-afdc-a59f8b12f455" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.35:15021/healthz/ready\": dial tcp 10.134.0.35:15021: connect: connection refused"
Sep 16 02:31:40.904447 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:31:40.904413    2583 patch_prober.go:28] interesting pod/e2e-inf-b-da6c4f-openshift-default-7d6598f44-fg5qr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.35:15021/healthz/ready\": dial tcp 10.134.0.35:15021: connect: connection refused" start-of-body=
Sep 16 02:31:40.904651 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:31:40.904465    2583 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-da6c4f-openshift-default-7d6598f44-fg5qr" podUID="88e2b6bf-fe2d-40ea-afdc-a59f8b12f455" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.35:15021/healthz/ready\": dial tcp 10.134.0.35:15021: connect: connection refused"
Sep 16 02:31:41.903685 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:31:41.903648    2583 patch_prober.go:28] interesting pod/e2e-inf-b-da6c4f-openshift-default-7d6598f44-fg5qr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.35:15021/healthz/ready\": dial tcp 10.134.0.35:15021: connect: connection refused" start-of-body=
Sep 16 02:31:41.904064 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:31:41.903698    2583 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-da6c4f-openshift-default-7d6598f44-fg5qr" podUID="88e2b6bf-fe2d-40ea-afdc-a59f8b12f455" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.35:15021/healthz/ready\": dial tcp 10.134.0.35:15021: connect: connection refused"
Sep 16 02:31:42.907473 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:31:42.907437    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-da6c4f-openshift-default-7d6598f44-fg5qr"
Sep 16 02:31:42.907882 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:31:42.907781    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-da6c4f-openshift-default-7d6598f44-fg5qr"
Sep 16 02:31:42.908448 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:31:42.908427    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-da6c4f-openshift-default-7d6598f44-fg5qr"
Sep 16 02:32:03.451762 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:32:03.451722    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-a-6dfb47/test-model-6dfb47-kserve-c848c5767-nxqxt"]
Sep 16 02:32:03.458213 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:32:03.458184    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-6dfb47/test-model-6dfb47-kserve-c848c5767-nxqxt"
Sep 16 02:32:03.461530 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:32:03.461501    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-6dfb47\"/\"kube-root-ca.crt\""
Sep 16 02:32:03.462538 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:32:03.462513    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-6dfb47\"/\"openshift-service-ca.crt\""
Sep 16 02:32:03.462538 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:32:03.462539    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-6dfb47\"/\"test-model-6dfb47-kserve-self-signed-certs\""
Sep 16 02:32:03.463233 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:32:03.462803    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-6dfb47\"/\"default-dockercfg-hhw9v\""
Sep 16 02:32:03.464572 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:32:03.464523    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-6dfb47/test-model-6dfb47-kserve-c848c5767-nxqxt"]
Sep 16 02:32:03.575020 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:32:03.574989    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/13c0a1db-a190-492e-bd52-5dc20e0b5ec6-tmp-dir\") pod \"test-model-6dfb47-kserve-c848c5767-nxqxt\" (UID: \"13c0a1db-a190-492e-bd52-5dc20e0b5ec6\") " pod="ai-tenant-e2e-inf-a-6dfb47/test-model-6dfb47-kserve-c848c5767-nxqxt"
Sep 16 02:32:03.575198 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:32:03.575031    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dpzx8\" (UniqueName: \"kubernetes.io/projected/13c0a1db-a190-492e-bd52-5dc20e0b5ec6-kube-api-access-dpzx8\") pod \"test-model-6dfb47-kserve-c848c5767-nxqxt\" (UID: \"13c0a1db-a190-492e-bd52-5dc20e0b5ec6\") " pod="ai-tenant-e2e-inf-a-6dfb47/test-model-6dfb47-kserve-c848c5767-nxqxt"
Sep 16 02:32:03.575198 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:32:03.575058    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/13c0a1db-a190-492e-bd52-5dc20e0b5ec6-dshm\") pod \"test-model-6dfb47-kserve-c848c5767-nxqxt\" (UID: \"13c0a1db-a190-492e-bd52-5dc20e0b5ec6\") " pod="ai-tenant-e2e-inf-a-6dfb47/test-model-6dfb47-kserve-c848c5767-nxqxt"
Sep 16 02:32:03.575198 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:32:03.575112    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/13c0a1db-a190-492e-bd52-5dc20e0b5ec6-home\") pod \"test-model-6dfb47-kserve-c848c5767-nxqxt\" (UID: \"13c0a1db-a190-492e-bd52-5dc20e0b5ec6\") " pod="ai-tenant-e2e-inf-a-6dfb47/test-model-6dfb47-kserve-c848c5767-nxqxt"
Sep 16 02:32:03.575198 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:32:03.575129    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/13c0a1db-a190-492e-bd52-5dc20e0b5ec6-model-cache\") pod \"test-model-6dfb47-kserve-c848c5767-nxqxt\" (UID: \"13c0a1db-a190-492e-bd52-5dc20e0b5ec6\") " pod="ai-tenant-e2e-inf-a-6dfb47/test-model-6dfb47-kserve-c848c5767-nxqxt"
Sep 16 02:32:03.575198 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:32:03.575144    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/13c0a1db-a190-492e-bd52-5dc20e0b5ec6-tls-certs\") pod \"test-model-6dfb47-kserve-c848c5767-nxqxt\" (UID: \"13c0a1db-a190-492e-bd52-5dc20e0b5ec6\") " pod="ai-tenant-e2e-inf-a-6dfb47/test-model-6dfb47-kserve-c848c5767-nxqxt"
Sep 16 02:32:03.676127 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:32:03.676092    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/13c0a1db-a190-492e-bd52-5dc20e0b5ec6-home\") pod \"test-model-6dfb47-kserve-c848c5767-nxqxt\" (UID: \"13c0a1db-a190-492e-bd52-5dc20e0b5ec6\") " pod="ai-tenant-e2e-inf-a-6dfb47/test-model-6dfb47-kserve-c848c5767-nxqxt"
Sep 16 02:32:03.676310 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:32:03.676139    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/13c0a1db-a190-492e-bd52-5dc20e0b5ec6-model-cache\") pod \"test-model-6dfb47-kserve-c848c5767-nxqxt\" (UID: \"13c0a1db-a190-492e-bd52-5dc20e0b5ec6\") " pod="ai-tenant-e2e-inf-a-6dfb47/test-model-6dfb47-kserve-c848c5767-nxqxt"
Sep 16 02:32:03.676310 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:32:03.676168    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/13c0a1db-a190-492e-bd52-5dc20e0b5ec6-tls-certs\") pod \"test-model-6dfb47-kserve-c848c5767-nxqxt\" (UID: \"13c0a1db-a190-492e-bd52-5dc20e0b5ec6\") " pod="ai-tenant-e2e-inf-a-6dfb47/test-model-6dfb47-kserve-c848c5767-nxqxt"
Sep 16 02:32:03.676310 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:32:03.676252    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/13c0a1db-a190-492e-bd52-5dc20e0b5ec6-tmp-dir\") pod \"test-model-6dfb47-kserve-c848c5767-nxqxt\" (UID: \"13c0a1db-a190-492e-bd52-5dc20e0b5ec6\") " pod="ai-tenant-e2e-inf-a-6dfb47/test-model-6dfb47-kserve-c848c5767-nxqxt"
Sep 16 02:32:03.676310 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:32:03.676307    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dpzx8\" (UniqueName: \"kubernetes.io/projected/13c0a1db-a190-492e-bd52-5dc20e0b5ec6-kube-api-access-dpzx8\") pod \"test-model-6dfb47-kserve-c848c5767-nxqxt\" (UID: \"13c0a1db-a190-492e-bd52-5dc20e0b5ec6\") " pod="ai-tenant-e2e-inf-a-6dfb47/test-model-6dfb47-kserve-c848c5767-nxqxt"
Sep 16 02:32:03.676885 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:32:03.676352    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/13c0a1db-a190-492e-bd52-5dc20e0b5ec6-dshm\") pod \"test-model-6dfb47-kserve-c848c5767-nxqxt\" (UID: \"13c0a1db-a190-492e-bd52-5dc20e0b5ec6\") " pod="ai-tenant-e2e-inf-a-6dfb47/test-model-6dfb47-kserve-c848c5767-nxqxt"
Sep 16 02:32:03.676885 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:32:03.676567    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/13c0a1db-a190-492e-bd52-5dc20e0b5ec6-home\") pod \"test-model-6dfb47-kserve-c848c5767-nxqxt\" (UID: \"13c0a1db-a190-492e-bd52-5dc20e0b5ec6\") " pod="ai-tenant-e2e-inf-a-6dfb47/test-model-6dfb47-kserve-c848c5767-nxqxt"
Sep 16 02:32:03.676885 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:32:03.676604    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/13c0a1db-a190-492e-bd52-5dc20e0b5ec6-model-cache\") pod \"test-model-6dfb47-kserve-c848c5767-nxqxt\" (UID: \"13c0a1db-a190-492e-bd52-5dc20e0b5ec6\") " pod="ai-tenant-e2e-inf-a-6dfb47/test-model-6dfb47-kserve-c848c5767-nxqxt"
Sep 16 02:32:03.676885 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:32:03.676644    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/13c0a1db-a190-492e-bd52-5dc20e0b5ec6-tmp-dir\") pod \"test-model-6dfb47-kserve-c848c5767-nxqxt\" (UID: \"13c0a1db-a190-492e-bd52-5dc20e0b5ec6\") " pod="ai-tenant-e2e-inf-a-6dfb47/test-model-6dfb47-kserve-c848c5767-nxqxt"
Sep 16 02:32:03.678464 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:32:03.678442    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/13c0a1db-a190-492e-bd52-5dc20e0b5ec6-dshm\") pod \"test-model-6dfb47-kserve-c848c5767-nxqxt\" (UID: \"13c0a1db-a190-492e-bd52-5dc20e0b5ec6\") " pod="ai-tenant-e2e-inf-a-6dfb47/test-model-6dfb47-kserve-c848c5767-nxqxt"
Sep 16 02:32:03.678768 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:32:03.678750    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/13c0a1db-a190-492e-bd52-5dc20e0b5ec6-tls-certs\") pod \"test-model-6dfb47-kserve-c848c5767-nxqxt\" (UID: \"13c0a1db-a190-492e-bd52-5dc20e0b5ec6\") " pod="ai-tenant-e2e-inf-a-6dfb47/test-model-6dfb47-kserve-c848c5767-nxqxt"
Sep 16 02:32:03.684120 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:32:03.684100    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dpzx8\" (UniqueName: \"kubernetes.io/projected/13c0a1db-a190-492e-bd52-5dc20e0b5ec6-kube-api-access-dpzx8\") pod \"test-model-6dfb47-kserve-c848c5767-nxqxt\" (UID: \"13c0a1db-a190-492e-bd52-5dc20e0b5ec6\") " pod="ai-tenant-e2e-inf-a-6dfb47/test-model-6dfb47-kserve-c848c5767-nxqxt"
Sep 16 02:32:03.770698 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:32:03.770613    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-6dfb47/test-model-6dfb47-kserve-c848c5767-nxqxt"
Sep 16 02:32:04.042537 ip-10-0-128-111 kubenswrapper[2583]: W0916 02:32:04.042508    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod13c0a1db_a190_492e_bd52_5dc20e0b5ec6.slice/crio-51517f4c04843fe5eefd9dca91cc4c881170c61383644427c23374d3b5581d0c WatchSource:0}: Error finding container 51517f4c04843fe5eefd9dca91cc4c881170c61383644427c23374d3b5581d0c: Status 404 returned error can't find the container with id 51517f4c04843fe5eefd9dca91cc4c881170c61383644427c23374d3b5581d0c
Sep 16 02:32:04.046154 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:32:04.046126    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-6dfb47/test-model-6dfb47-kserve-c848c5767-nxqxt"]
Sep 16 02:32:04.736853 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:32:04.736813    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-6dfb47/test-model-6dfb47-kserve-c848c5767-nxqxt" event={"ID":"13c0a1db-a190-492e-bd52-5dc20e0b5ec6","Type":"ContainerStarted","Data":"f7f0524204ffca58dd5888e573b9b63eb27cb30ff1dab09a6a8b19573d73caa5"}
Sep 16 02:32:04.737227 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:32:04.736859    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-6dfb47/test-model-6dfb47-kserve-c848c5767-nxqxt" event={"ID":"13c0a1db-a190-492e-bd52-5dc20e0b5ec6","Type":"ContainerStarted","Data":"51517f4c04843fe5eefd9dca91cc4c881170c61383644427c23374d3b5581d0c"}
Sep 16 02:32:04.754148 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:32:04.754099    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-inf-a-6dfb47/test-model-6dfb47-kserve-c848c5767-nxqxt" podStartSLOduration=1.7540836419999999 podStartE2EDuration="1.754083642s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 02:32:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 02:32:04.751970894 +0000 UTC m=+803.219759677" watchObservedRunningTime="2026-09-16 02:32:04.754083642 +0000 UTC m=+803.221872759"
Sep 16 02:32:08.455991 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:32:08.455953    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-fzl8t"]
Sep 16 02:32:13.771794 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:32:13.771750    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-a-6dfb47/test-model-6dfb47-kserve-c848c5767-nxqxt"
Sep 16 02:32:13.772174 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:32:13.771811    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-6dfb47/test-model-6dfb47-kserve-c848c5767-nxqxt"
Sep 16 02:32:13.784525 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:32:13.784501    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-a-6dfb47/test-model-6dfb47-kserve-c848c5767-nxqxt"
Sep 16 02:32:13.796271 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:32:13.796249    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-6dfb47/test-model-6dfb47-kserve-c848c5767-nxqxt"
Sep 16 02:32:27.753864 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:32:27.753830    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-fzl8t"]
Sep 16 02:32:43.156959 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:32:43.156923    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-fzl8t"]
Sep 16 02:33:10.753680 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:33:10.753642    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-fzl8t"]
Sep 16 02:33:28.051467 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:33:28.051431    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-fzl8t"]
Sep 16 02:33:36.754806 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:33:36.754718    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w3-a3d7eb/distinct-w3-a3d7eb-kserve-79d99dc9d6-mt2wx"]
Sep 16 02:33:36.759120 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:33:36.759095    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-a3d7eb/distinct-w3-a3d7eb-kserve-79d99dc9d6-mt2wx"
Sep 16 02:33:36.761489 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:33:36.761469    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-a3d7eb\"/\"distinct-w3-a3d7eb-kserve-self-signed-certs\""
Sep 16 02:33:36.765328 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:33:36.765307    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-a3d7eb/distinct-w3-a3d7eb-kserve-79d99dc9d6-mt2wx"]
Sep 16 02:33:36.879780 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:33:36.879741    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8f08f681-a03c-49b8-a118-f7101aaeb69c-model-cache\") pod \"distinct-w3-a3d7eb-kserve-79d99dc9d6-mt2wx\" (UID: \"8f08f681-a03c-49b8-a118-f7101aaeb69c\") " pod="e2e-models-e2e-worker-w3-a3d7eb/distinct-w3-a3d7eb-kserve-79d99dc9d6-mt2wx"
Sep 16 02:33:36.879780 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:33:36.879787    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8f08f681-a03c-49b8-a118-f7101aaeb69c-tls-certs\") pod \"distinct-w3-a3d7eb-kserve-79d99dc9d6-mt2wx\" (UID: \"8f08f681-a03c-49b8-a118-f7101aaeb69c\") " pod="e2e-models-e2e-worker-w3-a3d7eb/distinct-w3-a3d7eb-kserve-79d99dc9d6-mt2wx"
Sep 16 02:33:36.879974 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:33:36.879867    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8f08f681-a03c-49b8-a118-f7101aaeb69c-dshm\") pod \"distinct-w3-a3d7eb-kserve-79d99dc9d6-mt2wx\" (UID: \"8f08f681-a03c-49b8-a118-f7101aaeb69c\") " pod="e2e-models-e2e-worker-w3-a3d7eb/distinct-w3-a3d7eb-kserve-79d99dc9d6-mt2wx"
Sep 16 02:33:36.879974 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:33:36.879958    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ncqf5\" (UniqueName: \"kubernetes.io/projected/8f08f681-a03c-49b8-a118-f7101aaeb69c-kube-api-access-ncqf5\") pod \"distinct-w3-a3d7eb-kserve-79d99dc9d6-mt2wx\" (UID: \"8f08f681-a03c-49b8-a118-f7101aaeb69c\") " pod="e2e-models-e2e-worker-w3-a3d7eb/distinct-w3-a3d7eb-kserve-79d99dc9d6-mt2wx"
Sep 16 02:33:36.880048 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:33:36.880007    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8f08f681-a03c-49b8-a118-f7101aaeb69c-tmp-dir\") pod \"distinct-w3-a3d7eb-kserve-79d99dc9d6-mt2wx\" (UID: \"8f08f681-a03c-49b8-a118-f7101aaeb69c\") " pod="e2e-models-e2e-worker-w3-a3d7eb/distinct-w3-a3d7eb-kserve-79d99dc9d6-mt2wx"
Sep 16 02:33:36.880082 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:33:36.880058    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8f08f681-a03c-49b8-a118-f7101aaeb69c-home\") pod \"distinct-w3-a3d7eb-kserve-79d99dc9d6-mt2wx\" (UID: \"8f08f681-a03c-49b8-a118-f7101aaeb69c\") " pod="e2e-models-e2e-worker-w3-a3d7eb/distinct-w3-a3d7eb-kserve-79d99dc9d6-mt2wx"
Sep 16 02:33:36.981464 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:33:36.981425    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8f08f681-a03c-49b8-a118-f7101aaeb69c-dshm\") pod \"distinct-w3-a3d7eb-kserve-79d99dc9d6-mt2wx\" (UID: \"8f08f681-a03c-49b8-a118-f7101aaeb69c\") " pod="e2e-models-e2e-worker-w3-a3d7eb/distinct-w3-a3d7eb-kserve-79d99dc9d6-mt2wx"
Sep 16 02:33:36.981658 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:33:36.981496    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ncqf5\" (UniqueName: \"kubernetes.io/projected/8f08f681-a03c-49b8-a118-f7101aaeb69c-kube-api-access-ncqf5\") pod \"distinct-w3-a3d7eb-kserve-79d99dc9d6-mt2wx\" (UID: \"8f08f681-a03c-49b8-a118-f7101aaeb69c\") " pod="e2e-models-e2e-worker-w3-a3d7eb/distinct-w3-a3d7eb-kserve-79d99dc9d6-mt2wx"
Sep 16 02:33:36.981658 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:33:36.981543    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8f08f681-a03c-49b8-a118-f7101aaeb69c-tmp-dir\") pod \"distinct-w3-a3d7eb-kserve-79d99dc9d6-mt2wx\" (UID: \"8f08f681-a03c-49b8-a118-f7101aaeb69c\") " pod="e2e-models-e2e-worker-w3-a3d7eb/distinct-w3-a3d7eb-kserve-79d99dc9d6-mt2wx"
Sep 16 02:33:36.981658 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:33:36.981594    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8f08f681-a03c-49b8-a118-f7101aaeb69c-home\") pod \"distinct-w3-a3d7eb-kserve-79d99dc9d6-mt2wx\" (UID: \"8f08f681-a03c-49b8-a118-f7101aaeb69c\") " pod="e2e-models-e2e-worker-w3-a3d7eb/distinct-w3-a3d7eb-kserve-79d99dc9d6-mt2wx"
Sep 16 02:33:36.981658 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:33:36.981639    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8f08f681-a03c-49b8-a118-f7101aaeb69c-model-cache\") pod \"distinct-w3-a3d7eb-kserve-79d99dc9d6-mt2wx\" (UID: \"8f08f681-a03c-49b8-a118-f7101aaeb69c\") " pod="e2e-models-e2e-worker-w3-a3d7eb/distinct-w3-a3d7eb-kserve-79d99dc9d6-mt2wx"
Sep 16 02:33:36.981867 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:33:36.981664    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8f08f681-a03c-49b8-a118-f7101aaeb69c-tls-certs\") pod \"distinct-w3-a3d7eb-kserve-79d99dc9d6-mt2wx\" (UID: \"8f08f681-a03c-49b8-a118-f7101aaeb69c\") " pod="e2e-models-e2e-worker-w3-a3d7eb/distinct-w3-a3d7eb-kserve-79d99dc9d6-mt2wx"
Sep 16 02:33:36.981962 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:33:36.981943    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8f08f681-a03c-49b8-a118-f7101aaeb69c-tmp-dir\") pod \"distinct-w3-a3d7eb-kserve-79d99dc9d6-mt2wx\" (UID: \"8f08f681-a03c-49b8-a118-f7101aaeb69c\") " pod="e2e-models-e2e-worker-w3-a3d7eb/distinct-w3-a3d7eb-kserve-79d99dc9d6-mt2wx"
Sep 16 02:33:36.982053 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:33:36.982015    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8f08f681-a03c-49b8-a118-f7101aaeb69c-model-cache\") pod \"distinct-w3-a3d7eb-kserve-79d99dc9d6-mt2wx\" (UID: \"8f08f681-a03c-49b8-a118-f7101aaeb69c\") " pod="e2e-models-e2e-worker-w3-a3d7eb/distinct-w3-a3d7eb-kserve-79d99dc9d6-mt2wx"
Sep 16 02:33:36.982116 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:33:36.982056    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8f08f681-a03c-49b8-a118-f7101aaeb69c-home\") pod \"distinct-w3-a3d7eb-kserve-79d99dc9d6-mt2wx\" (UID: \"8f08f681-a03c-49b8-a118-f7101aaeb69c\") " pod="e2e-models-e2e-worker-w3-a3d7eb/distinct-w3-a3d7eb-kserve-79d99dc9d6-mt2wx"
Sep 16 02:33:36.983903 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:33:36.983882    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8f08f681-a03c-49b8-a118-f7101aaeb69c-dshm\") pod \"distinct-w3-a3d7eb-kserve-79d99dc9d6-mt2wx\" (UID: \"8f08f681-a03c-49b8-a118-f7101aaeb69c\") " pod="e2e-models-e2e-worker-w3-a3d7eb/distinct-w3-a3d7eb-kserve-79d99dc9d6-mt2wx"
Sep 16 02:33:36.984155 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:33:36.984137    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8f08f681-a03c-49b8-a118-f7101aaeb69c-tls-certs\") pod \"distinct-w3-a3d7eb-kserve-79d99dc9d6-mt2wx\" (UID: \"8f08f681-a03c-49b8-a118-f7101aaeb69c\") " pod="e2e-models-e2e-worker-w3-a3d7eb/distinct-w3-a3d7eb-kserve-79d99dc9d6-mt2wx"
Sep 16 02:33:36.988477 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:33:36.988451    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ncqf5\" (UniqueName: \"kubernetes.io/projected/8f08f681-a03c-49b8-a118-f7101aaeb69c-kube-api-access-ncqf5\") pod \"distinct-w3-a3d7eb-kserve-79d99dc9d6-mt2wx\" (UID: \"8f08f681-a03c-49b8-a118-f7101aaeb69c\") " pod="e2e-models-e2e-worker-w3-a3d7eb/distinct-w3-a3d7eb-kserve-79d99dc9d6-mt2wx"
Sep 16 02:33:37.085508 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:33:37.085428    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-a3d7eb/distinct-w3-a3d7eb-kserve-79d99dc9d6-mt2wx"
Sep 16 02:33:37.356679 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:33:37.356654    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-a3d7eb/distinct-w3-a3d7eb-kserve-79d99dc9d6-mt2wx"]
Sep 16 02:33:37.358594 ip-10-0-128-111 kubenswrapper[2583]: W0916 02:33:37.358546    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod8f08f681_a03c_49b8_a118_f7101aaeb69c.slice/crio-13869204e7c859c103a9186a6c1c90678cfe0e946c7d48f72e60f53f4a2ecff3 WatchSource:0}: Error finding container 13869204e7c859c103a9186a6c1c90678cfe0e946c7d48f72e60f53f4a2ecff3: Status 404 returned error can't find the container with id 13869204e7c859c103a9186a6c1c90678cfe0e946c7d48f72e60f53f4a2ecff3
Sep 16 02:33:38.197162 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:33:38.197120    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-a3d7eb/distinct-w3-a3d7eb-kserve-79d99dc9d6-mt2wx" event={"ID":"8f08f681-a03c-49b8-a118-f7101aaeb69c","Type":"ContainerStarted","Data":"eb619e491b6b4e343d5c2c90000de4c82e79eba46adce04bbb7a5c546158c33f"}
Sep 16 02:33:38.197162 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:33:38.197165    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-a3d7eb/distinct-w3-a3d7eb-kserve-79d99dc9d6-mt2wx" event={"ID":"8f08f681-a03c-49b8-a118-f7101aaeb69c","Type":"ContainerStarted","Data":"13869204e7c859c103a9186a6c1c90678cfe0e946c7d48f72e60f53f4a2ecff3"}
Sep 16 02:33:38.215077 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:33:38.215032    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-models-e2e-worker-w3-a3d7eb/distinct-w3-a3d7eb-kserve-79d99dc9d6-mt2wx" podStartSLOduration=2.21501679 podStartE2EDuration="2.21501679s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 02:33:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 02:33:38.213022258 +0000 UTC m=+896.680811033" watchObservedRunningTime="2026-09-16 02:33:38.21501679 +0000 UTC m=+896.682805572"
Sep 16 02:33:47.086244 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:33:47.086172    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w3-a3d7eb/distinct-w3-a3d7eb-kserve-79d99dc9d6-mt2wx"
Sep 16 02:33:47.086741 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:33:47.086360    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-a3d7eb/distinct-w3-a3d7eb-kserve-79d99dc9d6-mt2wx"
Sep 16 02:33:47.105530 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:33:47.105340    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w3-a3d7eb/distinct-w3-a3d7eb-kserve-79d99dc9d6-mt2wx"
Sep 16 02:33:47.253909 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:33:47.253877    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-a3d7eb/distinct-w3-a3d7eb-kserve-79d99dc9d6-mt2wx"
Sep 16 02:34:00.448759 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:34:00.448719    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-fzl8t"]
Sep 16 02:34:09.209775 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:34:09.209743    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w3-a3d7eb/distinct-2-w3-a3d7eb-kserve-7b76c4fc77-pg8fv"]
Sep 16 02:34:09.213352 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:34:09.213332    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-a3d7eb/distinct-2-w3-a3d7eb-kserve-7b76c4fc77-pg8fv"
Sep 16 02:34:09.216106 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:34:09.216088    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-a3d7eb\"/\"distinct-2-w3-a3d7eb-kserve-self-signed-certs\""
Sep 16 02:34:09.226718 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:34:09.226695    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-a3d7eb/distinct-2-w3-a3d7eb-kserve-7b76c4fc77-pg8fv"]
Sep 16 02:34:09.312315 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:34:09.312278    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/02f506a2-c921-462d-afc7-5b47c3b30d54-tls-certs\") pod \"distinct-2-w3-a3d7eb-kserve-7b76c4fc77-pg8fv\" (UID: \"02f506a2-c921-462d-afc7-5b47c3b30d54\") " pod="e2e-models-e2e-worker-w3-a3d7eb/distinct-2-w3-a3d7eb-kserve-7b76c4fc77-pg8fv"
Sep 16 02:34:09.312497 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:34:09.312326    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j7p2f\" (UniqueName: \"kubernetes.io/projected/02f506a2-c921-462d-afc7-5b47c3b30d54-kube-api-access-j7p2f\") pod \"distinct-2-w3-a3d7eb-kserve-7b76c4fc77-pg8fv\" (UID: \"02f506a2-c921-462d-afc7-5b47c3b30d54\") " pod="e2e-models-e2e-worker-w3-a3d7eb/distinct-2-w3-a3d7eb-kserve-7b76c4fc77-pg8fv"
Sep 16 02:34:09.312497 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:34:09.312369    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/02f506a2-c921-462d-afc7-5b47c3b30d54-dshm\") pod \"distinct-2-w3-a3d7eb-kserve-7b76c4fc77-pg8fv\" (UID: \"02f506a2-c921-462d-afc7-5b47c3b30d54\") " pod="e2e-models-e2e-worker-w3-a3d7eb/distinct-2-w3-a3d7eb-kserve-7b76c4fc77-pg8fv"
Sep 16 02:34:09.312497 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:34:09.312391    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/02f506a2-c921-462d-afc7-5b47c3b30d54-model-cache\") pod \"distinct-2-w3-a3d7eb-kserve-7b76c4fc77-pg8fv\" (UID: \"02f506a2-c921-462d-afc7-5b47c3b30d54\") " pod="e2e-models-e2e-worker-w3-a3d7eb/distinct-2-w3-a3d7eb-kserve-7b76c4fc77-pg8fv"
Sep 16 02:34:09.312497 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:34:09.312471    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/02f506a2-c921-462d-afc7-5b47c3b30d54-tmp-dir\") pod \"distinct-2-w3-a3d7eb-kserve-7b76c4fc77-pg8fv\" (UID: \"02f506a2-c921-462d-afc7-5b47c3b30d54\") " pod="e2e-models-e2e-worker-w3-a3d7eb/distinct-2-w3-a3d7eb-kserve-7b76c4fc77-pg8fv"
Sep 16 02:34:09.312726 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:34:09.312592    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/02f506a2-c921-462d-afc7-5b47c3b30d54-home\") pod \"distinct-2-w3-a3d7eb-kserve-7b76c4fc77-pg8fv\" (UID: \"02f506a2-c921-462d-afc7-5b47c3b30d54\") " pod="e2e-models-e2e-worker-w3-a3d7eb/distinct-2-w3-a3d7eb-kserve-7b76c4fc77-pg8fv"
Sep 16 02:34:09.413139 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:34:09.413103    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/02f506a2-c921-462d-afc7-5b47c3b30d54-model-cache\") pod \"distinct-2-w3-a3d7eb-kserve-7b76c4fc77-pg8fv\" (UID: \"02f506a2-c921-462d-afc7-5b47c3b30d54\") " pod="e2e-models-e2e-worker-w3-a3d7eb/distinct-2-w3-a3d7eb-kserve-7b76c4fc77-pg8fv"
Sep 16 02:34:09.413139 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:34:09.413152    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/02f506a2-c921-462d-afc7-5b47c3b30d54-tmp-dir\") pod \"distinct-2-w3-a3d7eb-kserve-7b76c4fc77-pg8fv\" (UID: \"02f506a2-c921-462d-afc7-5b47c3b30d54\") " pod="e2e-models-e2e-worker-w3-a3d7eb/distinct-2-w3-a3d7eb-kserve-7b76c4fc77-pg8fv"
Sep 16 02:34:09.413418 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:34:09.413217    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/02f506a2-c921-462d-afc7-5b47c3b30d54-home\") pod \"distinct-2-w3-a3d7eb-kserve-7b76c4fc77-pg8fv\" (UID: \"02f506a2-c921-462d-afc7-5b47c3b30d54\") " pod="e2e-models-e2e-worker-w3-a3d7eb/distinct-2-w3-a3d7eb-kserve-7b76c4fc77-pg8fv"
Sep 16 02:34:09.413418 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:34:09.413306    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/02f506a2-c921-462d-afc7-5b47c3b30d54-tls-certs\") pod \"distinct-2-w3-a3d7eb-kserve-7b76c4fc77-pg8fv\" (UID: \"02f506a2-c921-462d-afc7-5b47c3b30d54\") " pod="e2e-models-e2e-worker-w3-a3d7eb/distinct-2-w3-a3d7eb-kserve-7b76c4fc77-pg8fv"
Sep 16 02:34:09.413418 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:34:09.413353    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j7p2f\" (UniqueName: \"kubernetes.io/projected/02f506a2-c921-462d-afc7-5b47c3b30d54-kube-api-access-j7p2f\") pod \"distinct-2-w3-a3d7eb-kserve-7b76c4fc77-pg8fv\" (UID: \"02f506a2-c921-462d-afc7-5b47c3b30d54\") " pod="e2e-models-e2e-worker-w3-a3d7eb/distinct-2-w3-a3d7eb-kserve-7b76c4fc77-pg8fv"
Sep 16 02:34:09.413418 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:34:09.413386    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/02f506a2-c921-462d-afc7-5b47c3b30d54-dshm\") pod \"distinct-2-w3-a3d7eb-kserve-7b76c4fc77-pg8fv\" (UID: \"02f506a2-c921-462d-afc7-5b47c3b30d54\") " pod="e2e-models-e2e-worker-w3-a3d7eb/distinct-2-w3-a3d7eb-kserve-7b76c4fc77-pg8fv"
Sep 16 02:34:09.413653 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:34:09.413625    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/02f506a2-c921-462d-afc7-5b47c3b30d54-model-cache\") pod \"distinct-2-w3-a3d7eb-kserve-7b76c4fc77-pg8fv\" (UID: \"02f506a2-c921-462d-afc7-5b47c3b30d54\") " pod="e2e-models-e2e-worker-w3-a3d7eb/distinct-2-w3-a3d7eb-kserve-7b76c4fc77-pg8fv"
Sep 16 02:34:09.413709 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:34:09.413690    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/02f506a2-c921-462d-afc7-5b47c3b30d54-home\") pod \"distinct-2-w3-a3d7eb-kserve-7b76c4fc77-pg8fv\" (UID: \"02f506a2-c921-462d-afc7-5b47c3b30d54\") " pod="e2e-models-e2e-worker-w3-a3d7eb/distinct-2-w3-a3d7eb-kserve-7b76c4fc77-pg8fv"
Sep 16 02:34:09.413786 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:34:09.413770    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/02f506a2-c921-462d-afc7-5b47c3b30d54-tmp-dir\") pod \"distinct-2-w3-a3d7eb-kserve-7b76c4fc77-pg8fv\" (UID: \"02f506a2-c921-462d-afc7-5b47c3b30d54\") " pod="e2e-models-e2e-worker-w3-a3d7eb/distinct-2-w3-a3d7eb-kserve-7b76c4fc77-pg8fv"
Sep 16 02:34:09.415658 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:34:09.415627    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/02f506a2-c921-462d-afc7-5b47c3b30d54-dshm\") pod \"distinct-2-w3-a3d7eb-kserve-7b76c4fc77-pg8fv\" (UID: \"02f506a2-c921-462d-afc7-5b47c3b30d54\") " pod="e2e-models-e2e-worker-w3-a3d7eb/distinct-2-w3-a3d7eb-kserve-7b76c4fc77-pg8fv"
Sep 16 02:34:09.416085 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:34:09.416064    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/02f506a2-c921-462d-afc7-5b47c3b30d54-tls-certs\") pod \"distinct-2-w3-a3d7eb-kserve-7b76c4fc77-pg8fv\" (UID: \"02f506a2-c921-462d-afc7-5b47c3b30d54\") " pod="e2e-models-e2e-worker-w3-a3d7eb/distinct-2-w3-a3d7eb-kserve-7b76c4fc77-pg8fv"
Sep 16 02:34:09.420498 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:34:09.420475    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-j7p2f\" (UniqueName: \"kubernetes.io/projected/02f506a2-c921-462d-afc7-5b47c3b30d54-kube-api-access-j7p2f\") pod \"distinct-2-w3-a3d7eb-kserve-7b76c4fc77-pg8fv\" (UID: \"02f506a2-c921-462d-afc7-5b47c3b30d54\") " pod="e2e-models-e2e-worker-w3-a3d7eb/distinct-2-w3-a3d7eb-kserve-7b76c4fc77-pg8fv"
Sep 16 02:34:09.522332 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:34:09.522248    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-a3d7eb/distinct-2-w3-a3d7eb-kserve-7b76c4fc77-pg8fv"
Sep 16 02:34:09.798174 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:34:09.798146    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-a3d7eb/distinct-2-w3-a3d7eb-kserve-7b76c4fc77-pg8fv"]
Sep 16 02:34:09.800114 ip-10-0-128-111 kubenswrapper[2583]: W0916 02:34:09.800080    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod02f506a2_c921_462d_afc7_5b47c3b30d54.slice/crio-3ff118d28adfff41990e0f12e715e72e28b5364592e3b287c6bc524562ddbbfd WatchSource:0}: Error finding container 3ff118d28adfff41990e0f12e715e72e28b5364592e3b287c6bc524562ddbbfd: Status 404 returned error can't find the container with id 3ff118d28adfff41990e0f12e715e72e28b5364592e3b287c6bc524562ddbbfd
Sep 16 02:34:10.365890 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:34:10.365850    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-a3d7eb/distinct-2-w3-a3d7eb-kserve-7b76c4fc77-pg8fv" event={"ID":"02f506a2-c921-462d-afc7-5b47c3b30d54","Type":"ContainerStarted","Data":"863217e97f1eccace04ca34e2287861a1a0fb5a39df7ed20866b1097ea95de7f"}
Sep 16 02:34:10.366269 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:34:10.365906    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-a3d7eb/distinct-2-w3-a3d7eb-kserve-7b76c4fc77-pg8fv" event={"ID":"02f506a2-c921-462d-afc7-5b47c3b30d54","Type":"ContainerStarted","Data":"3ff118d28adfff41990e0f12e715e72e28b5364592e3b287c6bc524562ddbbfd"}
Sep 16 02:34:10.382856 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:34:10.382811    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-models-e2e-worker-w3-a3d7eb/distinct-2-w3-a3d7eb-kserve-7b76c4fc77-pg8fv" podStartSLOduration=1.382795387 podStartE2EDuration="1.382795387s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 02:34:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 02:34:10.380912757 +0000 UTC m=+928.848701539" watchObservedRunningTime="2026-09-16 02:34:10.382795387 +0000 UTC m=+928.850584168"
Sep 16 02:34:19.523053 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:34:19.523016    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-a3d7eb/distinct-2-w3-a3d7eb-kserve-7b76c4fc77-pg8fv"
Sep 16 02:34:19.523515 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:34:19.523068    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w3-a3d7eb/distinct-2-w3-a3d7eb-kserve-7b76c4fc77-pg8fv"
Sep 16 02:34:19.535841 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:34:19.535812    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w3-a3d7eb/distinct-2-w3-a3d7eb-kserve-7b76c4fc77-pg8fv"
Sep 16 02:34:20.427615 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:34:20.427588    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-a3d7eb/distinct-2-w3-a3d7eb-kserve-7b76c4fc77-pg8fv"
Sep 16 02:34:33.348682 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:34:33.348645    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-fzl8t"]
Sep 16 02:34:42.430235 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:34:42.430202    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-6dfb47/test-model-6dfb47-kserve-c848c5767-nxqxt"]
Sep 16 02:34:57.449886 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:34:57.449814    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-inf-a-6dfb47/test-model-6dfb47-kserve-c848c5767-nxqxt" podUID="13c0a1db-a190-492e-bd52-5dc20e0b5ec6" containerName="main" containerID="cri-o://f7f0524204ffca58dd5888e573b9b63eb27cb30ff1dab09a6a8b19573d73caa5" gracePeriod=45
Sep 16 02:34:57.604702 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:34:57.604669    2583 generic.go:356] "Generic (PLEG): container finished" podID="13c0a1db-a190-492e-bd52-5dc20e0b5ec6" containerID="f7f0524204ffca58dd5888e573b9b63eb27cb30ff1dab09a6a8b19573d73caa5" exitCode=0
Sep 16 02:34:57.604866 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:34:57.604740    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-6dfb47/test-model-6dfb47-kserve-c848c5767-nxqxt" event={"ID":"13c0a1db-a190-492e-bd52-5dc20e0b5ec6","Type":"ContainerDied","Data":"f7f0524204ffca58dd5888e573b9b63eb27cb30ff1dab09a6a8b19573d73caa5"}
Sep 16 02:34:57.701920 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:34:57.701852    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-6dfb47/test-model-6dfb47-kserve-c848c5767-nxqxt"
Sep 16 02:34:57.780362 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:34:57.780329    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/13c0a1db-a190-492e-bd52-5dc20e0b5ec6-model-cache\") pod \"13c0a1db-a190-492e-bd52-5dc20e0b5ec6\" (UID: \"13c0a1db-a190-492e-bd52-5dc20e0b5ec6\") "
Sep 16 02:34:57.780579 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:34:57.780386    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/13c0a1db-a190-492e-bd52-5dc20e0b5ec6-dshm\") pod \"13c0a1db-a190-492e-bd52-5dc20e0b5ec6\" (UID: \"13c0a1db-a190-492e-bd52-5dc20e0b5ec6\") "
Sep 16 02:34:57.780579 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:34:57.780439    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/13c0a1db-a190-492e-bd52-5dc20e0b5ec6-home\") pod \"13c0a1db-a190-492e-bd52-5dc20e0b5ec6\" (UID: \"13c0a1db-a190-492e-bd52-5dc20e0b5ec6\") "
Sep 16 02:34:57.780579 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:34:57.780486    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/13c0a1db-a190-492e-bd52-5dc20e0b5ec6-tmp-dir\") pod \"13c0a1db-a190-492e-bd52-5dc20e0b5ec6\" (UID: \"13c0a1db-a190-492e-bd52-5dc20e0b5ec6\") "
Sep 16 02:34:57.780579 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:34:57.780544    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/13c0a1db-a190-492e-bd52-5dc20e0b5ec6-tls-certs\") pod \"13c0a1db-a190-492e-bd52-5dc20e0b5ec6\" (UID: \"13c0a1db-a190-492e-bd52-5dc20e0b5ec6\") "
Sep 16 02:34:57.780809 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:34:57.780605    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dpzx8\" (UniqueName: \"kubernetes.io/projected/13c0a1db-a190-492e-bd52-5dc20e0b5ec6-kube-api-access-dpzx8\") pod \"13c0a1db-a190-492e-bd52-5dc20e0b5ec6\" (UID: \"13c0a1db-a190-492e-bd52-5dc20e0b5ec6\") "
Sep 16 02:34:57.780809 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:34:57.780638    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/13c0a1db-a190-492e-bd52-5dc20e0b5ec6-model-cache" (OuterVolumeSpecName: "model-cache") pod "13c0a1db-a190-492e-bd52-5dc20e0b5ec6" (UID: "13c0a1db-a190-492e-bd52-5dc20e0b5ec6"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 02:34:57.780809 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:34:57.780755    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/13c0a1db-a190-492e-bd52-5dc20e0b5ec6-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "13c0a1db-a190-492e-bd52-5dc20e0b5ec6" (UID: "13c0a1db-a190-492e-bd52-5dc20e0b5ec6"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 02:34:57.780809 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:34:57.780767    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/13c0a1db-a190-492e-bd52-5dc20e0b5ec6-home" (OuterVolumeSpecName: "home") pod "13c0a1db-a190-492e-bd52-5dc20e0b5ec6" (UID: "13c0a1db-a190-492e-bd52-5dc20e0b5ec6"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 02:34:57.781032 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:34:57.781014    2583 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/13c0a1db-a190-492e-bd52-5dc20e0b5ec6-home\") on node \"ip-10-0-128-111.ec2.internal\" DevicePath \"\""
Sep 16 02:34:57.781094 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:34:57.781050    2583 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/13c0a1db-a190-492e-bd52-5dc20e0b5ec6-tmp-dir\") on node \"ip-10-0-128-111.ec2.internal\" DevicePath \"\""
Sep 16 02:34:57.781094 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:34:57.781061    2583 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/13c0a1db-a190-492e-bd52-5dc20e0b5ec6-model-cache\") on node \"ip-10-0-128-111.ec2.internal\" DevicePath \"\""
Sep 16 02:34:57.782739 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:34:57.782713    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/13c0a1db-a190-492e-bd52-5dc20e0b5ec6-dshm" (OuterVolumeSpecName: "dshm") pod "13c0a1db-a190-492e-bd52-5dc20e0b5ec6" (UID: "13c0a1db-a190-492e-bd52-5dc20e0b5ec6"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 02:34:57.782856 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:34:57.782798    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/13c0a1db-a190-492e-bd52-5dc20e0b5ec6-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "13c0a1db-a190-492e-bd52-5dc20e0b5ec6" (UID: "13c0a1db-a190-492e-bd52-5dc20e0b5ec6"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 02:34:57.782856 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:34:57.782808    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/13c0a1db-a190-492e-bd52-5dc20e0b5ec6-kube-api-access-dpzx8" (OuterVolumeSpecName: "kube-api-access-dpzx8") pod "13c0a1db-a190-492e-bd52-5dc20e0b5ec6" (UID: "13c0a1db-a190-492e-bd52-5dc20e0b5ec6"). InnerVolumeSpecName "kube-api-access-dpzx8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 02:34:57.882375 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:34:57.882338    2583 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/13c0a1db-a190-492e-bd52-5dc20e0b5ec6-dshm\") on node \"ip-10-0-128-111.ec2.internal\" DevicePath \"\""
Sep 16 02:34:57.882375 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:34:57.882368    2583 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/13c0a1db-a190-492e-bd52-5dc20e0b5ec6-tls-certs\") on node \"ip-10-0-128-111.ec2.internal\" DevicePath \"\""
Sep 16 02:34:57.882375 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:34:57.882379    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dpzx8\" (UniqueName: \"kubernetes.io/projected/13c0a1db-a190-492e-bd52-5dc20e0b5ec6-kube-api-access-dpzx8\") on node \"ip-10-0-128-111.ec2.internal\" DevicePath \"\""
Sep 16 02:34:58.613267 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:34:58.613219    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-6dfb47/test-model-6dfb47-kserve-c848c5767-nxqxt"
Sep 16 02:34:58.613901 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:34:58.613227    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-6dfb47/test-model-6dfb47-kserve-c848c5767-nxqxt" event={"ID":"13c0a1db-a190-492e-bd52-5dc20e0b5ec6","Type":"ContainerDied","Data":"51517f4c04843fe5eefd9dca91cc4c881170c61383644427c23374d3b5581d0c"}
Sep 16 02:34:58.613901 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:34:58.613339    2583 scope.go:117] "RemoveContainer" containerID="f7f0524204ffca58dd5888e573b9b63eb27cb30ff1dab09a6a8b19573d73caa5"
Sep 16 02:34:58.629181 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:34:58.629159    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-6dfb47/test-model-6dfb47-kserve-c848c5767-nxqxt"]
Sep 16 02:34:58.632840 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:34:58.632821    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-a-6dfb47/test-model-6dfb47-kserve-c848c5767-nxqxt"]
Sep 16 02:34:59.046509 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:34:59.046417    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-06e75986-6c9fc5c47d-86q2r"]
Sep 16 02:34:59.046960 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:34:59.046941    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="13c0a1db-a190-492e-bd52-5dc20e0b5ec6" containerName="main"
Sep 16 02:34:59.047048 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:34:59.046962    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="13c0a1db-a190-492e-bd52-5dc20e0b5ec6" containerName="main"
Sep 16 02:34:59.047104 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:34:59.047055    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="13c0a1db-a190-492e-bd52-5dc20e0b5ec6" containerName="main"
Sep 16 02:34:59.051477 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:34:59.051456    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-06e75986-6c9fc5c47d-86q2r"
Sep 16 02:34:59.053801 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:34:59.053782    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-06e75986-dockercfg-tsjs2\""
Sep 16 02:34:59.053902 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:34:59.053791    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-06e75986\""
Sep 16 02:34:59.060464 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:34:59.060439    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-06e75986-6c9fc5c47d-86q2r"]
Sep 16 02:34:59.086000 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:34:59.085975    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-06e75986-5f9fc5bb8c-8k7rg"]
Sep 16 02:34:59.089407 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:34:59.089389    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-06e75986-5f9fc5bb8c-8k7rg"
Sep 16 02:34:59.094710 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:34:59.094689    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c80f7420-af2e-49bb-8757-a718e86e1e48-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-06e75986-6c9fc5c47d-86q2r\" (UID: \"c80f7420-af2e-49bb-8757-a718e86e1e48\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-06e75986-6c9fc5c47d-86q2r"
Sep 16 02:34:59.094815 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:34:59.094783    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c54xp\" (UniqueName: \"kubernetes.io/projected/c80f7420-af2e-49bb-8757-a718e86e1e48-kube-api-access-c54xp\") pod \"payload-pre-processing-e2e-ait-06e75986-6c9fc5c47d-86q2r\" (UID: \"c80f7420-af2e-49bb-8757-a718e86e1e48\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-06e75986-6c9fc5c47d-86q2r"
Sep 16 02:34:59.098671 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:34:59.098648    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-06e75986-5f9fc5bb8c-8k7rg"]
Sep 16 02:34:59.195863 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:34:59.195830    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c80f7420-af2e-49bb-8757-a718e86e1e48-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-06e75986-6c9fc5c47d-86q2r\" (UID: \"c80f7420-af2e-49bb-8757-a718e86e1e48\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-06e75986-6c9fc5c47d-86q2r"
Sep 16 02:34:59.196050 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:34:59.195883    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c54xp\" (UniqueName: \"kubernetes.io/projected/c80f7420-af2e-49bb-8757-a718e86e1e48-kube-api-access-c54xp\") pod \"payload-pre-processing-e2e-ait-06e75986-6c9fc5c47d-86q2r\" (UID: \"c80f7420-af2e-49bb-8757-a718e86e1e48\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-06e75986-6c9fc5c47d-86q2r"
Sep 16 02:34:59.196050 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:34:59.196009    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4q45g\" (UniqueName: \"kubernetes.io/projected/fd155d51-af92-4587-9a27-7b42dd217a83-kube-api-access-4q45g\") pod \"payload-processing-e2e-ait-06e75986-5f9fc5bb8c-8k7rg\" (UID: \"fd155d51-af92-4587-9a27-7b42dd217a83\") " pod="openshift-ingress/payload-processing-e2e-ait-06e75986-5f9fc5bb8c-8k7rg"
Sep 16 02:34:59.196155 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:34:59.196057    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fd155d51-af92-4587-9a27-7b42dd217a83-plugins-config-volume\") pod \"payload-processing-e2e-ait-06e75986-5f9fc5bb8c-8k7rg\" (UID: \"fd155d51-af92-4587-9a27-7b42dd217a83\") " pod="openshift-ingress/payload-processing-e2e-ait-06e75986-5f9fc5bb8c-8k7rg"
Sep 16 02:34:59.196498 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:34:59.196480    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c80f7420-af2e-49bb-8757-a718e86e1e48-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-06e75986-6c9fc5c47d-86q2r\" (UID: \"c80f7420-af2e-49bb-8757-a718e86e1e48\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-06e75986-6c9fc5c47d-86q2r"
Sep 16 02:34:59.203303 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:34:59.203277    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-c54xp\" (UniqueName: \"kubernetes.io/projected/c80f7420-af2e-49bb-8757-a718e86e1e48-kube-api-access-c54xp\") pod \"payload-pre-processing-e2e-ait-06e75986-6c9fc5c47d-86q2r\" (UID: \"c80f7420-af2e-49bb-8757-a718e86e1e48\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-06e75986-6c9fc5c47d-86q2r"
Sep 16 02:34:59.297016 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:34:59.296916    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4q45g\" (UniqueName: \"kubernetes.io/projected/fd155d51-af92-4587-9a27-7b42dd217a83-kube-api-access-4q45g\") pod \"payload-processing-e2e-ait-06e75986-5f9fc5bb8c-8k7rg\" (UID: \"fd155d51-af92-4587-9a27-7b42dd217a83\") " pod="openshift-ingress/payload-processing-e2e-ait-06e75986-5f9fc5bb8c-8k7rg"
Sep 16 02:34:59.297016 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:34:59.296957    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fd155d51-af92-4587-9a27-7b42dd217a83-plugins-config-volume\") pod \"payload-processing-e2e-ait-06e75986-5f9fc5bb8c-8k7rg\" (UID: \"fd155d51-af92-4587-9a27-7b42dd217a83\") " pod="openshift-ingress/payload-processing-e2e-ait-06e75986-5f9fc5bb8c-8k7rg"
Sep 16 02:34:59.297672 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:34:59.297648    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fd155d51-af92-4587-9a27-7b42dd217a83-plugins-config-volume\") pod \"payload-processing-e2e-ait-06e75986-5f9fc5bb8c-8k7rg\" (UID: \"fd155d51-af92-4587-9a27-7b42dd217a83\") " pod="openshift-ingress/payload-processing-e2e-ait-06e75986-5f9fc5bb8c-8k7rg"
Sep 16 02:34:59.308136 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:34:59.308102    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4q45g\" (UniqueName: \"kubernetes.io/projected/fd155d51-af92-4587-9a27-7b42dd217a83-kube-api-access-4q45g\") pod \"payload-processing-e2e-ait-06e75986-5f9fc5bb8c-8k7rg\" (UID: \"fd155d51-af92-4587-9a27-7b42dd217a83\") " pod="openshift-ingress/payload-processing-e2e-ait-06e75986-5f9fc5bb8c-8k7rg"
Sep 16 02:34:59.362734 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:34:59.362706    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-06e75986-6c9fc5c47d-86q2r"
Sep 16 02:34:59.400783 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:34:59.400737    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-06e75986-5f9fc5bb8c-8k7rg"
Sep 16 02:34:59.643106 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:34:59.643079    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-06e75986-6c9fc5c47d-86q2r"]
Sep 16 02:34:59.644627 ip-10-0-128-111 kubenswrapper[2583]: W0916 02:34:59.644602    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc80f7420_af2e_49bb_8757_a718e86e1e48.slice/crio-4efa2854143cfaaa447e23b69cd631bcdb3f4a52d0392fde394401b9351eefe9 WatchSource:0}: Error finding container 4efa2854143cfaaa447e23b69cd631bcdb3f4a52d0392fde394401b9351eefe9: Status 404 returned error can't find the container with id 4efa2854143cfaaa447e23b69cd631bcdb3f4a52d0392fde394401b9351eefe9
Sep 16 02:34:59.689576 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:34:59.689535    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-06e75986-5f9fc5bb8c-8k7rg"]
Sep 16 02:34:59.695572 ip-10-0-128-111 kubenswrapper[2583]: W0916 02:34:59.695531    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfd155d51_af92_4587_9a27_7b42dd217a83.slice/crio-6943ad23c4ac8106aac1bb4ce65529a23253346f9fcfe6fdf812415d18ba53ce WatchSource:0}: Error finding container 6943ad23c4ac8106aac1bb4ce65529a23253346f9fcfe6fdf812415d18ba53ce: Status 404 returned error can't find the container with id 6943ad23c4ac8106aac1bb4ce65529a23253346f9fcfe6fdf812415d18ba53ce
Sep 16 02:35:00.046745 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:35:00.046657    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="13c0a1db-a190-492e-bd52-5dc20e0b5ec6" path="/var/lib/kubelet/pods/13c0a1db-a190-492e-bd52-5dc20e0b5ec6/volumes"
Sep 16 02:35:00.626068 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:35:00.626025    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-06e75986-6c9fc5c47d-86q2r" event={"ID":"c80f7420-af2e-49bb-8757-a718e86e1e48","Type":"ContainerStarted","Data":"e82f938a4a69531c039ddafe7cb8694e36bf5b54e5261dc3e529a41d912321e9"}
Sep 16 02:35:00.626276 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:35:00.626250    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-06e75986-6c9fc5c47d-86q2r"
Sep 16 02:35:00.626432 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:35:00.626293    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-06e75986-6c9fc5c47d-86q2r" event={"ID":"c80f7420-af2e-49bb-8757-a718e86e1e48","Type":"ContainerStarted","Data":"4efa2854143cfaaa447e23b69cd631bcdb3f4a52d0392fde394401b9351eefe9"}
Sep 16 02:35:00.627837 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:35:00.627812    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-06e75986-5f9fc5bb8c-8k7rg" event={"ID":"fd155d51-af92-4587-9a27-7b42dd217a83","Type":"ContainerStarted","Data":"0c129409219e6a00c4fc521daf2e7165f7c25393f6ac69b97ffc6c319cf16c36"}
Sep 16 02:35:00.627935 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:35:00.627859    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-06e75986-5f9fc5bb8c-8k7rg" event={"ID":"fd155d51-af92-4587-9a27-7b42dd217a83","Type":"ContainerStarted","Data":"6943ad23c4ac8106aac1bb4ce65529a23253346f9fcfe6fdf812415d18ba53ce"}
Sep 16 02:35:00.627935 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:35:00.627903    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-06e75986-5f9fc5bb8c-8k7rg"
Sep 16 02:35:00.643421 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:35:00.643371    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-06e75986-6c9fc5c47d-86q2r" podStartSLOduration=1.64335628 podStartE2EDuration="1.64335628s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 02:34:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 02:35:00.640400219 +0000 UTC m=+979.108189001" watchObservedRunningTime="2026-09-16 02:35:00.64335628 +0000 UTC m=+979.111145064"
Sep 16 02:35:00.655749 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:35:00.655705    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-06e75986-5f9fc5bb8c-8k7rg" podStartSLOduration=1.655692874 podStartE2EDuration="1.655692874s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 02:34:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 02:35:00.653932275 +0000 UTC m=+979.121721058" watchObservedRunningTime="2026-09-16 02:35:00.655692874 +0000 UTC m=+979.123481656"
Sep 16 02:35:07.453360 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:35:07.453192    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-fzl8t"]
Sep 16 02:35:11.634208 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:35:11.634172    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-06e75986-5f9fc5bb8c-8k7rg"
Sep 16 02:35:11.634703 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:35:11.634436    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-06e75986-6c9fc5c47d-86q2r"
Sep 16 02:35:21.725228 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:35:21.725189    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-06e75986-5f9fc5bb8c-8k7rg"]
Sep 16 02:35:21.725806 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:35:21.725495    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-06e75986-5f9fc5bb8c-8k7rg" podUID="fd155d51-af92-4587-9a27-7b42dd217a83" containerName="payload-processing" containerID="cri-o://0c129409219e6a00c4fc521daf2e7165f7c25393f6ac69b97ffc6c319cf16c36" gracePeriod=30
Sep 16 02:35:21.731401 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:35:21.731371    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-06e75986-6c9fc5c47d-86q2r"]
Sep 16 02:35:21.732033 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:35:21.731984    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-06e75986-6c9fc5c47d-86q2r" podUID="c80f7420-af2e-49bb-8757-a718e86e1e48" containerName="payload-pre-processing" containerID="cri-o://e82f938a4a69531c039ddafe7cb8694e36bf5b54e5261dc3e529a41d912321e9" gracePeriod=30
Sep 16 02:35:21.980465 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:35:21.980408    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-06e75986-5f9fc5bb8c-8k7rg"
Sep 16 02:35:22.017003 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:35:22.016797    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-06e75986-6c9fc5c47d-86q2r"
Sep 16 02:35:22.120943 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:35:22.120905    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fd155d51-af92-4587-9a27-7b42dd217a83-plugins-config-volume\") pod \"fd155d51-af92-4587-9a27-7b42dd217a83\" (UID: \"fd155d51-af92-4587-9a27-7b42dd217a83\") "
Sep 16 02:35:22.121134 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:35:22.120956    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4q45g\" (UniqueName: \"kubernetes.io/projected/fd155d51-af92-4587-9a27-7b42dd217a83-kube-api-access-4q45g\") pod \"fd155d51-af92-4587-9a27-7b42dd217a83\" (UID: \"fd155d51-af92-4587-9a27-7b42dd217a83\") "
Sep 16 02:35:22.121134 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:35:22.120987    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c80f7420-af2e-49bb-8757-a718e86e1e48-plugins-config-volume\") pod \"c80f7420-af2e-49bb-8757-a718e86e1e48\" (UID: \"c80f7420-af2e-49bb-8757-a718e86e1e48\") "
Sep 16 02:35:22.121134 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:35:22.121024    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-c54xp\" (UniqueName: \"kubernetes.io/projected/c80f7420-af2e-49bb-8757-a718e86e1e48-kube-api-access-c54xp\") pod \"c80f7420-af2e-49bb-8757-a718e86e1e48\" (UID: \"c80f7420-af2e-49bb-8757-a718e86e1e48\") "
Sep 16 02:35:22.121481 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:35:22.121367    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c80f7420-af2e-49bb-8757-a718e86e1e48-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "c80f7420-af2e-49bb-8757-a718e86e1e48" (UID: "c80f7420-af2e-49bb-8757-a718e86e1e48"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 02:35:22.121655 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:35:22.121636    2583 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c80f7420-af2e-49bb-8757-a718e86e1e48-plugins-config-volume\") on node \"ip-10-0-128-111.ec2.internal\" DevicePath \"\""
Sep 16 02:35:22.122126 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:35:22.122062    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fd155d51-af92-4587-9a27-7b42dd217a83-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "fd155d51-af92-4587-9a27-7b42dd217a83" (UID: "fd155d51-af92-4587-9a27-7b42dd217a83"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 02:35:22.123450 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:35:22.123420    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fd155d51-af92-4587-9a27-7b42dd217a83-kube-api-access-4q45g" (OuterVolumeSpecName: "kube-api-access-4q45g") pod "fd155d51-af92-4587-9a27-7b42dd217a83" (UID: "fd155d51-af92-4587-9a27-7b42dd217a83"). InnerVolumeSpecName "kube-api-access-4q45g". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 02:35:22.124174 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:35:22.124146    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c80f7420-af2e-49bb-8757-a718e86e1e48-kube-api-access-c54xp" (OuterVolumeSpecName: "kube-api-access-c54xp") pod "c80f7420-af2e-49bb-8757-a718e86e1e48" (UID: "c80f7420-af2e-49bb-8757-a718e86e1e48"). InnerVolumeSpecName "kube-api-access-c54xp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 02:35:22.222284 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:35:22.222243    2583 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fd155d51-af92-4587-9a27-7b42dd217a83-plugins-config-volume\") on node \"ip-10-0-128-111.ec2.internal\" DevicePath \"\""
Sep 16 02:35:22.222284 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:35:22.222276    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4q45g\" (UniqueName: \"kubernetes.io/projected/fd155d51-af92-4587-9a27-7b42dd217a83-kube-api-access-4q45g\") on node \"ip-10-0-128-111.ec2.internal\" DevicePath \"\""
Sep 16 02:35:22.222284 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:35:22.222291    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c54xp\" (UniqueName: \"kubernetes.io/projected/c80f7420-af2e-49bb-8757-a718e86e1e48-kube-api-access-c54xp\") on node \"ip-10-0-128-111.ec2.internal\" DevicePath \"\""
Sep 16 02:35:22.744425 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:35:22.744239    2583 generic.go:356] "Generic (PLEG): container finished" podID="fd155d51-af92-4587-9a27-7b42dd217a83" containerID="0c129409219e6a00c4fc521daf2e7165f7c25393f6ac69b97ffc6c319cf16c36" exitCode=0
Sep 16 02:35:22.744425 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:35:22.744285    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-06e75986-5f9fc5bb8c-8k7rg"
Sep 16 02:35:22.744425 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:35:22.744323    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-06e75986-5f9fc5bb8c-8k7rg" event={"ID":"fd155d51-af92-4587-9a27-7b42dd217a83","Type":"ContainerDied","Data":"0c129409219e6a00c4fc521daf2e7165f7c25393f6ac69b97ffc6c319cf16c36"}
Sep 16 02:35:22.744425 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:35:22.744378    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-06e75986-5f9fc5bb8c-8k7rg" event={"ID":"fd155d51-af92-4587-9a27-7b42dd217a83","Type":"ContainerDied","Data":"6943ad23c4ac8106aac1bb4ce65529a23253346f9fcfe6fdf812415d18ba53ce"}
Sep 16 02:35:22.744425 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:35:22.744410    2583 scope.go:117] "RemoveContainer" containerID="0c129409219e6a00c4fc521daf2e7165f7c25393f6ac69b97ffc6c319cf16c36"
Sep 16 02:35:22.745974 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:35:22.745929    2583 generic.go:356] "Generic (PLEG): container finished" podID="c80f7420-af2e-49bb-8757-a718e86e1e48" containerID="e82f938a4a69531c039ddafe7cb8694e36bf5b54e5261dc3e529a41d912321e9" exitCode=0
Sep 16 02:35:22.745974 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:35:22.745964    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-06e75986-6c9fc5c47d-86q2r" event={"ID":"c80f7420-af2e-49bb-8757-a718e86e1e48","Type":"ContainerDied","Data":"e82f938a4a69531c039ddafe7cb8694e36bf5b54e5261dc3e529a41d912321e9"}
Sep 16 02:35:22.746185 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:35:22.746024    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-06e75986-6c9fc5c47d-86q2r" event={"ID":"c80f7420-af2e-49bb-8757-a718e86e1e48","Type":"ContainerDied","Data":"4efa2854143cfaaa447e23b69cd631bcdb3f4a52d0392fde394401b9351eefe9"}
Sep 16 02:35:22.746185 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:35:22.745973    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-06e75986-6c9fc5c47d-86q2r"
Sep 16 02:35:22.755384 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:35:22.755352    2583 scope.go:117] "RemoveContainer" containerID="0c129409219e6a00c4fc521daf2e7165f7c25393f6ac69b97ffc6c319cf16c36"
Sep 16 02:35:22.755686 ip-10-0-128-111 kubenswrapper[2583]: E0916 02:35:22.755667    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0c129409219e6a00c4fc521daf2e7165f7c25393f6ac69b97ffc6c319cf16c36\": container with ID starting with 0c129409219e6a00c4fc521daf2e7165f7c25393f6ac69b97ffc6c319cf16c36 not found: ID does not exist" containerID="0c129409219e6a00c4fc521daf2e7165f7c25393f6ac69b97ffc6c319cf16c36"
Sep 16 02:35:22.755784 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:35:22.755693    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0c129409219e6a00c4fc521daf2e7165f7c25393f6ac69b97ffc6c319cf16c36"} err="failed to get container status \"0c129409219e6a00c4fc521daf2e7165f7c25393f6ac69b97ffc6c319cf16c36\": rpc error: code = NotFound desc = could not find container \"0c129409219e6a00c4fc521daf2e7165f7c25393f6ac69b97ffc6c319cf16c36\": container with ID starting with 0c129409219e6a00c4fc521daf2e7165f7c25393f6ac69b97ffc6c319cf16c36 not found: ID does not exist"
Sep 16 02:35:22.755784 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:35:22.755709    2583 scope.go:117] "RemoveContainer" containerID="e82f938a4a69531c039ddafe7cb8694e36bf5b54e5261dc3e529a41d912321e9"
Sep 16 02:35:22.763374 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:35:22.763357    2583 scope.go:117] "RemoveContainer" containerID="e82f938a4a69531c039ddafe7cb8694e36bf5b54e5261dc3e529a41d912321e9"
Sep 16 02:35:22.763637 ip-10-0-128-111 kubenswrapper[2583]: E0916 02:35:22.763622    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e82f938a4a69531c039ddafe7cb8694e36bf5b54e5261dc3e529a41d912321e9\": container with ID starting with e82f938a4a69531c039ddafe7cb8694e36bf5b54e5261dc3e529a41d912321e9 not found: ID does not exist" containerID="e82f938a4a69531c039ddafe7cb8694e36bf5b54e5261dc3e529a41d912321e9"
Sep 16 02:35:22.763688 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:35:22.763641    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e82f938a4a69531c039ddafe7cb8694e36bf5b54e5261dc3e529a41d912321e9"} err="failed to get container status \"e82f938a4a69531c039ddafe7cb8694e36bf5b54e5261dc3e529a41d912321e9\": rpc error: code = NotFound desc = could not find container \"e82f938a4a69531c039ddafe7cb8694e36bf5b54e5261dc3e529a41d912321e9\": container with ID starting with e82f938a4a69531c039ddafe7cb8694e36bf5b54e5261dc3e529a41d912321e9 not found: ID does not exist"
Sep 16 02:35:22.775239 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:35:22.775151    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-06e75986-5f9fc5bb8c-8k7rg"]
Sep 16 02:35:22.778732 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:35:22.778710    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-06e75986-5f9fc5bb8c-8k7rg"]
Sep 16 02:35:22.792423 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:35:22.792401    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-06e75986-6c9fc5c47d-86q2r"]
Sep 16 02:35:22.797465 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:35:22.797441    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-06e75986-6c9fc5c47d-86q2r"]
Sep 16 02:35:23.410267 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:35:23.410235    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-da6c4f-zffkz"]
Sep 16 02:35:23.410652 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:35:23.410639    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fd155d51-af92-4587-9a27-7b42dd217a83" containerName="payload-processing"
Sep 16 02:35:23.410703 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:35:23.410656    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="fd155d51-af92-4587-9a27-7b42dd217a83" containerName="payload-processing"
Sep 16 02:35:23.410703 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:35:23.410672    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c80f7420-af2e-49bb-8757-a718e86e1e48" containerName="payload-pre-processing"
Sep 16 02:35:23.410703 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:35:23.410677    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="c80f7420-af2e-49bb-8757-a718e86e1e48" containerName="payload-pre-processing"
Sep 16 02:35:23.410806 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:35:23.410737    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="c80f7420-af2e-49bb-8757-a718e86e1e48" containerName="payload-pre-processing"
Sep 16 02:35:23.410806 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:35:23.410747    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="fd155d51-af92-4587-9a27-7b42dd217a83" containerName="payload-processing"
Sep 16 02:35:23.415205 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:35:23.415188    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-da6c4f-zffkz"
Sep 16 02:35:23.417443 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:35:23.417421    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 16 02:35:23.417608 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:35:23.417422    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-w8hnp\""
Sep 16 02:35:23.426570 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:35:23.422577    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-da6c4f-zffkz"]
Sep 16 02:35:23.536417 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:35:23.536374    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e1e04d6f-7bea-4a89-ba47-0fb7682746a2-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-da6c4f-zffkz\" (UID: \"e1e04d6f-7bea-4a89-ba47-0fb7682746a2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-da6c4f-zffkz"
Sep 16 02:35:23.637424 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:35:23.637383    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e1e04d6f-7bea-4a89-ba47-0fb7682746a2-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-da6c4f-zffkz\" (UID: \"e1e04d6f-7bea-4a89-ba47-0fb7682746a2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-da6c4f-zffkz"
Sep 16 02:35:23.637981 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:35:23.637962    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e1e04d6f-7bea-4a89-ba47-0fb7682746a2-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-da6c4f-zffkz\" (UID: \"e1e04d6f-7bea-4a89-ba47-0fb7682746a2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-da6c4f-zffkz"
Sep 16 02:35:23.737007 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:35:23.736923    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-da6c4f-zffkz"
Sep 16 02:35:24.009575 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:35:24.009537    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-da6c4f-zffkz"]
Sep 16 02:35:24.011590 ip-10-0-128-111 kubenswrapper[2583]: W0916 02:35:24.011542    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode1e04d6f_7bea_4a89_ba47_0fb7682746a2.slice/crio-0753d4641c234048c1dc61f5c3d77698a31071fdd710898aacba02a10fec7e3a WatchSource:0}: Error finding container 0753d4641c234048c1dc61f5c3d77698a31071fdd710898aacba02a10fec7e3a: Status 404 returned error can't find the container with id 0753d4641c234048c1dc61f5c3d77698a31071fdd710898aacba02a10fec7e3a
Sep 16 02:35:24.013413 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:35:24.013395    2583 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 16 02:35:24.045577 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:35:24.045535    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c80f7420-af2e-49bb-8757-a718e86e1e48" path="/var/lib/kubelet/pods/c80f7420-af2e-49bb-8757-a718e86e1e48/volumes"
Sep 16 02:35:24.045922 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:35:24.045909    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fd155d51-af92-4587-9a27-7b42dd217a83" path="/var/lib/kubelet/pods/fd155d51-af92-4587-9a27-7b42dd217a83/volumes"
Sep 16 02:35:24.762817 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:35:24.762770    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-da6c4f-zffkz" event={"ID":"e1e04d6f-7bea-4a89-ba47-0fb7682746a2","Type":"ContainerStarted","Data":"0753d4641c234048c1dc61f5c3d77698a31071fdd710898aacba02a10fec7e3a"}
Sep 16 02:35:26.776375 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:35:26.776338    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-da6c4f-zffkz" event={"ID":"e1e04d6f-7bea-4a89-ba47-0fb7682746a2","Type":"ContainerStarted","Data":"76c349fdc50fcd8b425c0ef05e86f2966e1c751f61869bbc86f9d15714363793"}
Sep 16 02:35:26.795269 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:35:26.795222    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-da6c4f-zffkz" podStartSLOduration=1.7495756949999999 podStartE2EDuration="3.795205123s" totalImagesPullingTime="2.045629428s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 02:35:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 02:35:26.79344093 +0000 UTC m=+1005.261229723" watchObservedRunningTime="2026-09-16 02:35:26.795205123 +0000 UTC m=+1005.262993904"
Sep 16 02:35:27.221619 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:35:27.221587    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-c40df7fa-7bdb8b9b86-4mnld"]
Sep 16 02:35:27.225235 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:35:27.225217    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-c40df7fa-7bdb8b9b86-4mnld"
Sep 16 02:35:27.227602 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:35:27.227583    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-migrate-c40df7fa\""
Sep 16 02:35:27.227710 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:35:27.227589    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-migrate-c40df7fa-dockercfg-d5w66\""
Sep 16 02:35:27.234149 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:35:27.234123    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-c40df7fa-7bdb8b9b86-4mnld"]
Sep 16 02:35:27.375073 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:35:27.375037    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/00c0c4bc-820c-4822-b7bb-2e3bbf9cbab4-plugins-config-volume\") pod \"payload-processing-e2e-migrate-c40df7fa-7bdb8b9b86-4mnld\" (UID: \"00c0c4bc-820c-4822-b7bb-2e3bbf9cbab4\") " pod="openshift-ingress/payload-processing-e2e-migrate-c40df7fa-7bdb8b9b86-4mnld"
Sep 16 02:35:27.375259 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:35:27.375130    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rj4hk\" (UniqueName: \"kubernetes.io/projected/00c0c4bc-820c-4822-b7bb-2e3bbf9cbab4-kube-api-access-rj4hk\") pod \"payload-processing-e2e-migrate-c40df7fa-7bdb8b9b86-4mnld\" (UID: \"00c0c4bc-820c-4822-b7bb-2e3bbf9cbab4\") " pod="openshift-ingress/payload-processing-e2e-migrate-c40df7fa-7bdb8b9b86-4mnld"
Sep 16 02:35:27.475901 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:35:27.475806    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/00c0c4bc-820c-4822-b7bb-2e3bbf9cbab4-plugins-config-volume\") pod \"payload-processing-e2e-migrate-c40df7fa-7bdb8b9b86-4mnld\" (UID: \"00c0c4bc-820c-4822-b7bb-2e3bbf9cbab4\") " pod="openshift-ingress/payload-processing-e2e-migrate-c40df7fa-7bdb8b9b86-4mnld"
Sep 16 02:35:27.476059 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:35:27.475903    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rj4hk\" (UniqueName: \"kubernetes.io/projected/00c0c4bc-820c-4822-b7bb-2e3bbf9cbab4-kube-api-access-rj4hk\") pod \"payload-processing-e2e-migrate-c40df7fa-7bdb8b9b86-4mnld\" (UID: \"00c0c4bc-820c-4822-b7bb-2e3bbf9cbab4\") " pod="openshift-ingress/payload-processing-e2e-migrate-c40df7fa-7bdb8b9b86-4mnld"
Sep 16 02:35:27.476522 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:35:27.476501    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/00c0c4bc-820c-4822-b7bb-2e3bbf9cbab4-plugins-config-volume\") pod \"payload-processing-e2e-migrate-c40df7fa-7bdb8b9b86-4mnld\" (UID: \"00c0c4bc-820c-4822-b7bb-2e3bbf9cbab4\") " pod="openshift-ingress/payload-processing-e2e-migrate-c40df7fa-7bdb8b9b86-4mnld"
Sep 16 02:35:27.483543 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:35:27.483516    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rj4hk\" (UniqueName: \"kubernetes.io/projected/00c0c4bc-820c-4822-b7bb-2e3bbf9cbab4-kube-api-access-rj4hk\") pod \"payload-processing-e2e-migrate-c40df7fa-7bdb8b9b86-4mnld\" (UID: \"00c0c4bc-820c-4822-b7bb-2e3bbf9cbab4\") " pod="openshift-ingress/payload-processing-e2e-migrate-c40df7fa-7bdb8b9b86-4mnld"
Sep 16 02:35:27.537286 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:35:27.537254    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-c40df7fa-7bdb8b9b86-4mnld"
Sep 16 02:35:27.783143 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:35:27.783059    2583 generic.go:356] "Generic (PLEG): container finished" podID="e1e04d6f-7bea-4a89-ba47-0fb7682746a2" containerID="76c349fdc50fcd8b425c0ef05e86f2966e1c751f61869bbc86f9d15714363793" exitCode=0
Sep 16 02:35:27.783501 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:35:27.783141    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-da6c4f-zffkz" event={"ID":"e1e04d6f-7bea-4a89-ba47-0fb7682746a2","Type":"ContainerDied","Data":"76c349fdc50fcd8b425c0ef05e86f2966e1c751f61869bbc86f9d15714363793"}
Sep 16 02:35:27.813485 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:35:27.813220    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-c40df7fa-7bdb8b9b86-4mnld"]
Sep 16 02:35:28.789353 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:35:28.789314    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-c40df7fa-7bdb8b9b86-4mnld" event={"ID":"00c0c4bc-820c-4822-b7bb-2e3bbf9cbab4","Type":"ContainerStarted","Data":"133ab3082e29a715875a6a706adc77ced63ac61607bb99375cb2645929c3663f"}
Sep 16 02:35:28.789910 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:35:28.789387    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-c40df7fa-7bdb8b9b86-4mnld" event={"ID":"00c0c4bc-820c-4822-b7bb-2e3bbf9cbab4","Type":"ContainerStarted","Data":"50c9aa347cdb7ae2a6f8299ebea8b93e20c878923cded62636c5d19eeb871bb2"}
Sep 16 02:35:28.808987 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:35:28.808930    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-migrate-c40df7fa-7bdb8b9b86-4mnld" podStartSLOduration=1.808907968 podStartE2EDuration="1.808907968s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 02:35:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 02:35:28.808355759 +0000 UTC m=+1007.276144541" watchObservedRunningTime="2026-09-16 02:35:28.808907968 +0000 UTC m=+1007.276696753"
Sep 16 02:35:28.935481 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:35:28.935456    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-da6c4f-zffkz"
Sep 16 02:35:28.990414 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:35:28.990385    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e1e04d6f-7bea-4a89-ba47-0fb7682746a2-maas-api-service-ca\") pod \"e1e04d6f-7bea-4a89-ba47-0fb7682746a2\" (UID: \"e1e04d6f-7bea-4a89-ba47-0fb7682746a2\") "
Sep 16 02:35:28.990811 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:35:28.990787    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e1e04d6f-7bea-4a89-ba47-0fb7682746a2-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "e1e04d6f-7bea-4a89-ba47-0fb7682746a2" (UID: "e1e04d6f-7bea-4a89-ba47-0fb7682746a2"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 02:35:29.092350 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:35:29.092318    2583 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e1e04d6f-7bea-4a89-ba47-0fb7682746a2-maas-api-service-ca\") on node \"ip-10-0-128-111.ec2.internal\" DevicePath \"\""
Sep 16 02:35:29.796065 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:35:29.796036    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-da6c4f-zffkz"
Sep 16 02:35:29.796611 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:35:29.796075    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-da6c4f-zffkz" event={"ID":"e1e04d6f-7bea-4a89-ba47-0fb7682746a2","Type":"ContainerDied","Data":"0753d4641c234048c1dc61f5c3d77698a31071fdd710898aacba02a10fec7e3a"}
Sep 16 02:35:29.796611 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:35:29.796121    2583 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0753d4641c234048c1dc61f5c3d77698a31071fdd710898aacba02a10fec7e3a"
Sep 16 02:35:29.796611 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:35:29.796334    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-migrate-c40df7fa-7bdb8b9b86-4mnld"
Sep 16 02:35:34.270890 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:35:34.270856    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-da6c4f-7d7768c479-nc9hn"]
Sep 16 02:35:34.271291 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:35:34.271067    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-b-da6c4f-7d7768c479-nc9hn" podUID="a80a9d20-dc4b-4020-96f8-8f1c9e0be8bf" containerName="payload-processing" containerID="cri-o://00ae671ee092da03ffcb8c3766dcd9ac37e3d941b394fb522d1539a1703fa7ee" gracePeriod=30
Sep 16 02:35:34.276482 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:35:34.276156    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-da6c4f-79c4c854b-vkd9b"]
Sep 16 02:35:34.276787 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:35:34.276745    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-da6c4f-79c4c854b-vkd9b" podUID="8795109a-cd57-4312-9988-01f98259eb7f" containerName="payload-pre-processing" containerID="cri-o://d32d962539aaab85b45ee5f62ac89798878915afe8449b31be45addd10ccbb34" gracePeriod=30
Sep 16 02:35:34.415506 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:35:34.415470    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-da6c4f-zffkz"]
Sep 16 02:35:34.419623 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:35:34.419590    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-da6c4f-zffkz"]
Sep 16 02:35:34.545345 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:35:34.545308    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-da6c4f-7d7768c479-nc9hn"
Sep 16 02:35:34.561235 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:35:34.561211    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-da6c4f-79c4c854b-vkd9b"
Sep 16 02:35:34.647788 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:35:34.647748    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a80a9d20-dc4b-4020-96f8-8f1c9e0be8bf-plugins-config-volume\") pod \"a80a9d20-dc4b-4020-96f8-8f1c9e0be8bf\" (UID: \"a80a9d20-dc4b-4020-96f8-8f1c9e0be8bf\") "
Sep 16 02:35:34.647788 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:35:34.647803    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kqwmk\" (UniqueName: \"kubernetes.io/projected/a80a9d20-dc4b-4020-96f8-8f1c9e0be8bf-kube-api-access-kqwmk\") pod \"a80a9d20-dc4b-4020-96f8-8f1c9e0be8bf\" (UID: \"a80a9d20-dc4b-4020-96f8-8f1c9e0be8bf\") "
Sep 16 02:35:34.648037 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:35:34.647847    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8795109a-cd57-4312-9988-01f98259eb7f-plugins-config-volume\") pod \"8795109a-cd57-4312-9988-01f98259eb7f\" (UID: \"8795109a-cd57-4312-9988-01f98259eb7f\") "
Sep 16 02:35:34.648037 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:35:34.647929    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vk7mj\" (UniqueName: \"kubernetes.io/projected/8795109a-cd57-4312-9988-01f98259eb7f-kube-api-access-vk7mj\") pod \"8795109a-cd57-4312-9988-01f98259eb7f\" (UID: \"8795109a-cd57-4312-9988-01f98259eb7f\") "
Sep 16 02:35:34.648270 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:35:34.648249    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a80a9d20-dc4b-4020-96f8-8f1c9e0be8bf-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "a80a9d20-dc4b-4020-96f8-8f1c9e0be8bf" (UID: "a80a9d20-dc4b-4020-96f8-8f1c9e0be8bf"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 02:35:34.648333 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:35:34.648265    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8795109a-cd57-4312-9988-01f98259eb7f-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "8795109a-cd57-4312-9988-01f98259eb7f" (UID: "8795109a-cd57-4312-9988-01f98259eb7f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 02:35:34.650082 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:35:34.650063    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a80a9d20-dc4b-4020-96f8-8f1c9e0be8bf-kube-api-access-kqwmk" (OuterVolumeSpecName: "kube-api-access-kqwmk") pod "a80a9d20-dc4b-4020-96f8-8f1c9e0be8bf" (UID: "a80a9d20-dc4b-4020-96f8-8f1c9e0be8bf"). InnerVolumeSpecName "kube-api-access-kqwmk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 02:35:34.650156 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:35:34.650098    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8795109a-cd57-4312-9988-01f98259eb7f-kube-api-access-vk7mj" (OuterVolumeSpecName: "kube-api-access-vk7mj") pod "8795109a-cd57-4312-9988-01f98259eb7f" (UID: "8795109a-cd57-4312-9988-01f98259eb7f"). InnerVolumeSpecName "kube-api-access-vk7mj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 02:35:34.748702 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:35:34.748657    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vk7mj\" (UniqueName: \"kubernetes.io/projected/8795109a-cd57-4312-9988-01f98259eb7f-kube-api-access-vk7mj\") on node \"ip-10-0-128-111.ec2.internal\" DevicePath \"\""
Sep 16 02:35:34.748702 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:35:34.748694    2583 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a80a9d20-dc4b-4020-96f8-8f1c9e0be8bf-plugins-config-volume\") on node \"ip-10-0-128-111.ec2.internal\" DevicePath \"\""
Sep 16 02:35:34.748702 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:35:34.748705    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kqwmk\" (UniqueName: \"kubernetes.io/projected/a80a9d20-dc4b-4020-96f8-8f1c9e0be8bf-kube-api-access-kqwmk\") on node \"ip-10-0-128-111.ec2.internal\" DevicePath \"\""
Sep 16 02:35:34.748702 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:35:34.748713    2583 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8795109a-cd57-4312-9988-01f98259eb7f-plugins-config-volume\") on node \"ip-10-0-128-111.ec2.internal\" DevicePath \"\""
Sep 16 02:35:34.822872 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:35:34.822778    2583 generic.go:356] "Generic (PLEG): container finished" podID="a80a9d20-dc4b-4020-96f8-8f1c9e0be8bf" containerID="00ae671ee092da03ffcb8c3766dcd9ac37e3d941b394fb522d1539a1703fa7ee" exitCode=0
Sep 16 02:35:34.822872 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:35:34.822822    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-da6c4f-7d7768c479-nc9hn"
Sep 16 02:35:34.822872 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:35:34.822864    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-da6c4f-7d7768c479-nc9hn" event={"ID":"a80a9d20-dc4b-4020-96f8-8f1c9e0be8bf","Type":"ContainerDied","Data":"00ae671ee092da03ffcb8c3766dcd9ac37e3d941b394fb522d1539a1703fa7ee"}
Sep 16 02:35:34.823111 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:35:34.822911    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-da6c4f-7d7768c479-nc9hn" event={"ID":"a80a9d20-dc4b-4020-96f8-8f1c9e0be8bf","Type":"ContainerDied","Data":"25364ccaa8a7e6aaefd383f147b02178275bb7f23537c0827e678b1dfcef4eb3"}
Sep 16 02:35:34.823111 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:35:34.822938    2583 scope.go:117] "RemoveContainer" containerID="00ae671ee092da03ffcb8c3766dcd9ac37e3d941b394fb522d1539a1703fa7ee"
Sep 16 02:35:34.824262 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:35:34.824240    2583 generic.go:356] "Generic (PLEG): container finished" podID="8795109a-cd57-4312-9988-01f98259eb7f" containerID="d32d962539aaab85b45ee5f62ac89798878915afe8449b31be45addd10ccbb34" exitCode=0
Sep 16 02:35:34.824344 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:35:34.824285    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-da6c4f-79c4c854b-vkd9b" event={"ID":"8795109a-cd57-4312-9988-01f98259eb7f","Type":"ContainerDied","Data":"d32d962539aaab85b45ee5f62ac89798878915afe8449b31be45addd10ccbb34"}
Sep 16 02:35:34.824344 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:35:34.824321    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-da6c4f-79c4c854b-vkd9b" event={"ID":"8795109a-cd57-4312-9988-01f98259eb7f","Type":"ContainerDied","Data":"c54a9a27005aa91b5f98d41c6bfd9422bb8d7853bc03d5e12de9648893229845"}
Sep 16 02:35:34.824344 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:35:34.824289    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-da6c4f-79c4c854b-vkd9b"
Sep 16 02:35:34.832184 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:35:34.832164    2583 scope.go:117] "RemoveContainer" containerID="00ae671ee092da03ffcb8c3766dcd9ac37e3d941b394fb522d1539a1703fa7ee"
Sep 16 02:35:34.832595 ip-10-0-128-111 kubenswrapper[2583]: E0916 02:35:34.832576    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"00ae671ee092da03ffcb8c3766dcd9ac37e3d941b394fb522d1539a1703fa7ee\": container with ID starting with 00ae671ee092da03ffcb8c3766dcd9ac37e3d941b394fb522d1539a1703fa7ee not found: ID does not exist" containerID="00ae671ee092da03ffcb8c3766dcd9ac37e3d941b394fb522d1539a1703fa7ee"
Sep 16 02:35:34.832673 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:35:34.832602    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"00ae671ee092da03ffcb8c3766dcd9ac37e3d941b394fb522d1539a1703fa7ee"} err="failed to get container status \"00ae671ee092da03ffcb8c3766dcd9ac37e3d941b394fb522d1539a1703fa7ee\": rpc error: code = NotFound desc = could not find container \"00ae671ee092da03ffcb8c3766dcd9ac37e3d941b394fb522d1539a1703fa7ee\": container with ID starting with 00ae671ee092da03ffcb8c3766dcd9ac37e3d941b394fb522d1539a1703fa7ee not found: ID does not exist"
Sep 16 02:35:34.832673 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:35:34.832617    2583 scope.go:117] "RemoveContainer" containerID="d32d962539aaab85b45ee5f62ac89798878915afe8449b31be45addd10ccbb34"
Sep 16 02:35:34.840358 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:35:34.840342    2583 scope.go:117] "RemoveContainer" containerID="d32d962539aaab85b45ee5f62ac89798878915afe8449b31be45addd10ccbb34"
Sep 16 02:35:34.840646 ip-10-0-128-111 kubenswrapper[2583]: E0916 02:35:34.840625    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d32d962539aaab85b45ee5f62ac89798878915afe8449b31be45addd10ccbb34\": container with ID starting with d32d962539aaab85b45ee5f62ac89798878915afe8449b31be45addd10ccbb34 not found: ID does not exist" containerID="d32d962539aaab85b45ee5f62ac89798878915afe8449b31be45addd10ccbb34"
Sep 16 02:35:34.840714 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:35:34.840651    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d32d962539aaab85b45ee5f62ac89798878915afe8449b31be45addd10ccbb34"} err="failed to get container status \"d32d962539aaab85b45ee5f62ac89798878915afe8449b31be45addd10ccbb34\": rpc error: code = NotFound desc = could not find container \"d32d962539aaab85b45ee5f62ac89798878915afe8449b31be45addd10ccbb34\": container with ID starting with d32d962539aaab85b45ee5f62ac89798878915afe8449b31be45addd10ccbb34 not found: ID does not exist"
Sep 16 02:35:34.844917 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:35:34.844897    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-da6c4f-7d7768c479-nc9hn"]
Sep 16 02:35:34.848502 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:35:34.848484    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-da6c4f-7d7768c479-nc9hn"]
Sep 16 02:35:34.858151 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:35:34.858133    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-da6c4f-79c4c854b-vkd9b"]
Sep 16 02:35:34.863199 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:35:34.863174    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-da6c4f-79c4c854b-vkd9b"]
Sep 16 02:35:36.045179 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:35:36.045145    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8795109a-cd57-4312-9988-01f98259eb7f" path="/var/lib/kubelet/pods/8795109a-cd57-4312-9988-01f98259eb7f/volumes"
Sep 16 02:35:36.045542 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:35:36.045520    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a80a9d20-dc4b-4020-96f8-8f1c9e0be8bf" path="/var/lib/kubelet/pods/a80a9d20-dc4b-4020-96f8-8f1c9e0be8bf/volumes"
Sep 16 02:35:36.045873 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:35:36.045855    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e1e04d6f-7bea-4a89-ba47-0fb7682746a2" path="/var/lib/kubelet/pods/e1e04d6f-7bea-4a89-ba47-0fb7682746a2/volumes"
Sep 16 02:35:40.802984 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:35:40.802947    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-migrate-c40df7fa-7bdb8b9b86-4mnld"
Sep 16 02:35:41.228736 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:35:41.228701    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-da6c4f-openshift-default-7d6598f44-fg5qr"]
Sep 16 02:35:41.228959 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:35:41.228931    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-da6c4f-openshift-default-7d6598f44-fg5qr" podUID="88e2b6bf-fe2d-40ea-afdc-a59f8b12f455" containerName="istio-proxy" containerID="cri-o://b0609a92ed841e0fa33f06b093579e6c313e8a41d41194eda4adae82ed8e4e46" gracePeriod=30
Sep 16 02:35:42.910054 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:35:42.910014    2583 patch_prober.go:28] interesting pod/e2e-inf-b-da6c4f-openshift-default-7d6598f44-fg5qr container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Sep 16 02:35:42.910418 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:35:42.910075    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-da6c4f-openshift-default-7d6598f44-fg5qr" podUID="88e2b6bf-fe2d-40ea-afdc-a59f8b12f455" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 16 02:35:46.478526 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:35:46.478502    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-da6c4f-openshift-default-7d6598f44-fg5qr"
Sep 16 02:35:46.567104 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:35:46.567014    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/88e2b6bf-fe2d-40ea-afdc-a59f8b12f455-istio-envoy\") pod \"88e2b6bf-fe2d-40ea-afdc-a59f8b12f455\" (UID: \"88e2b6bf-fe2d-40ea-afdc-a59f8b12f455\") "
Sep 16 02:35:46.567104 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:35:46.567097    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/88e2b6bf-fe2d-40ea-afdc-a59f8b12f455-istiod-ca-cert\") pod \"88e2b6bf-fe2d-40ea-afdc-a59f8b12f455\" (UID: \"88e2b6bf-fe2d-40ea-afdc-a59f8b12f455\") "
Sep 16 02:35:46.567339 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:35:46.567120    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/88e2b6bf-fe2d-40ea-afdc-a59f8b12f455-istio-token\") pod \"88e2b6bf-fe2d-40ea-afdc-a59f8b12f455\" (UID: \"88e2b6bf-fe2d-40ea-afdc-a59f8b12f455\") "
Sep 16 02:35:46.567339 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:35:46.567155    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tdnxk\" (UniqueName: \"kubernetes.io/projected/88e2b6bf-fe2d-40ea-afdc-a59f8b12f455-kube-api-access-tdnxk\") pod \"88e2b6bf-fe2d-40ea-afdc-a59f8b12f455\" (UID: \"88e2b6bf-fe2d-40ea-afdc-a59f8b12f455\") "
Sep 16 02:35:46.567339 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:35:46.567187    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/88e2b6bf-fe2d-40ea-afdc-a59f8b12f455-istio-data\") pod \"88e2b6bf-fe2d-40ea-afdc-a59f8b12f455\" (UID: \"88e2b6bf-fe2d-40ea-afdc-a59f8b12f455\") "
Sep 16 02:35:46.567502 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:35:46.567340    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/88e2b6bf-fe2d-40ea-afdc-a59f8b12f455-credential-socket\") pod \"88e2b6bf-fe2d-40ea-afdc-a59f8b12f455\" (UID: \"88e2b6bf-fe2d-40ea-afdc-a59f8b12f455\") "
Sep 16 02:35:46.567502 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:35:46.567409    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/88e2b6bf-fe2d-40ea-afdc-a59f8b12f455-workload-certs\") pod \"88e2b6bf-fe2d-40ea-afdc-a59f8b12f455\" (UID: \"88e2b6bf-fe2d-40ea-afdc-a59f8b12f455\") "
Sep 16 02:35:46.567502 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:35:46.567442    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/88e2b6bf-fe2d-40ea-afdc-a59f8b12f455-workload-socket\") pod \"88e2b6bf-fe2d-40ea-afdc-a59f8b12f455\" (UID: \"88e2b6bf-fe2d-40ea-afdc-a59f8b12f455\") "
Sep 16 02:35:46.567502 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:35:46.567469    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/88e2b6bf-fe2d-40ea-afdc-a59f8b12f455-istio-data" (OuterVolumeSpecName: "istio-data") pod "88e2b6bf-fe2d-40ea-afdc-a59f8b12f455" (UID: "88e2b6bf-fe2d-40ea-afdc-a59f8b12f455"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 02:35:46.567722 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:35:46.567476    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/88e2b6bf-fe2d-40ea-afdc-a59f8b12f455-istio-podinfo\") pod \"88e2b6bf-fe2d-40ea-afdc-a59f8b12f455\" (UID: \"88e2b6bf-fe2d-40ea-afdc-a59f8b12f455\") "
Sep 16 02:35:46.567722 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:35:46.567601    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/88e2b6bf-fe2d-40ea-afdc-a59f8b12f455-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "88e2b6bf-fe2d-40ea-afdc-a59f8b12f455" (UID: "88e2b6bf-fe2d-40ea-afdc-a59f8b12f455"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 02:35:46.568079 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:35:46.567853    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/88e2b6bf-fe2d-40ea-afdc-a59f8b12f455-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "88e2b6bf-fe2d-40ea-afdc-a59f8b12f455" (UID: "88e2b6bf-fe2d-40ea-afdc-a59f8b12f455"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 02:35:46.568079 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:35:46.567949    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/88e2b6bf-fe2d-40ea-afdc-a59f8b12f455-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "88e2b6bf-fe2d-40ea-afdc-a59f8b12f455" (UID: "88e2b6bf-fe2d-40ea-afdc-a59f8b12f455"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 02:35:46.568079 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:35:46.567968    2583 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/88e2b6bf-fe2d-40ea-afdc-a59f8b12f455-istio-data\") on node \"ip-10-0-128-111.ec2.internal\" DevicePath \"\""
Sep 16 02:35:46.568079 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:35:46.567988    2583 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/88e2b6bf-fe2d-40ea-afdc-a59f8b12f455-workload-certs\") on node \"ip-10-0-128-111.ec2.internal\" DevicePath \"\""
Sep 16 02:35:46.568079 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:35:46.568001    2583 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/88e2b6bf-fe2d-40ea-afdc-a59f8b12f455-istiod-ca-cert\") on node \"ip-10-0-128-111.ec2.internal\" DevicePath \"\""
Sep 16 02:35:46.568079 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:35:46.568043    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/88e2b6bf-fe2d-40ea-afdc-a59f8b12f455-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "88e2b6bf-fe2d-40ea-afdc-a59f8b12f455" (UID: "88e2b6bf-fe2d-40ea-afdc-a59f8b12f455"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 02:35:46.569530 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:35:46.569502    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/88e2b6bf-fe2d-40ea-afdc-a59f8b12f455-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "88e2b6bf-fe2d-40ea-afdc-a59f8b12f455" (UID: "88e2b6bf-fe2d-40ea-afdc-a59f8b12f455"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 02:35:46.569530 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:35:46.569509    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/88e2b6bf-fe2d-40ea-afdc-a59f8b12f455-istio-token" (OuterVolumeSpecName: "istio-token") pod "88e2b6bf-fe2d-40ea-afdc-a59f8b12f455" (UID: "88e2b6bf-fe2d-40ea-afdc-a59f8b12f455"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 02:35:46.569772 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:35:46.569756    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/88e2b6bf-fe2d-40ea-afdc-a59f8b12f455-kube-api-access-tdnxk" (OuterVolumeSpecName: "kube-api-access-tdnxk") pod "88e2b6bf-fe2d-40ea-afdc-a59f8b12f455" (UID: "88e2b6bf-fe2d-40ea-afdc-a59f8b12f455"). InnerVolumeSpecName "kube-api-access-tdnxk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 02:35:46.569817 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:35:46.569782    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/88e2b6bf-fe2d-40ea-afdc-a59f8b12f455-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "88e2b6bf-fe2d-40ea-afdc-a59f8b12f455" (UID: "88e2b6bf-fe2d-40ea-afdc-a59f8b12f455"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 16 02:35:46.659498 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:35:46.659459    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-fzl8t"]
Sep 16 02:35:46.669317 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:35:46.669291    2583 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/88e2b6bf-fe2d-40ea-afdc-a59f8b12f455-istio-envoy\") on node \"ip-10-0-128-111.ec2.internal\" DevicePath \"\""
Sep 16 02:35:46.669317 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:35:46.669313    2583 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/88e2b6bf-fe2d-40ea-afdc-a59f8b12f455-istio-token\") on node \"ip-10-0-128-111.ec2.internal\" DevicePath \"\""
Sep 16 02:35:46.669317 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:35:46.669323    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tdnxk\" (UniqueName: \"kubernetes.io/projected/88e2b6bf-fe2d-40ea-afdc-a59f8b12f455-kube-api-access-tdnxk\") on node \"ip-10-0-128-111.ec2.internal\" DevicePath \"\""
Sep 16 02:35:46.669520 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:35:46.669332    2583 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/88e2b6bf-fe2d-40ea-afdc-a59f8b12f455-credential-socket\") on node \"ip-10-0-128-111.ec2.internal\" DevicePath \"\""
Sep 16 02:35:46.669520 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:35:46.669343    2583 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/88e2b6bf-fe2d-40ea-afdc-a59f8b12f455-workload-socket\") on node \"ip-10-0-128-111.ec2.internal\" DevicePath \"\""
Sep 16 02:35:46.669520 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:35:46.669351    2583 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/88e2b6bf-fe2d-40ea-afdc-a59f8b12f455-istio-podinfo\") on node \"ip-10-0-128-111.ec2.internal\" DevicePath \"\""
Sep 16 02:35:46.888868 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:35:46.888827    2583 generic.go:356] "Generic (PLEG): container finished" podID="88e2b6bf-fe2d-40ea-afdc-a59f8b12f455" containerID="b0609a92ed841e0fa33f06b093579e6c313e8a41d41194eda4adae82ed8e4e46" exitCode=0
Sep 16 02:35:46.889029 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:35:46.888878    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-da6c4f-openshift-default-7d6598f44-fg5qr"
Sep 16 02:35:46.889029 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:35:46.888875    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-da6c4f-openshift-default-7d6598f44-fg5qr" event={"ID":"88e2b6bf-fe2d-40ea-afdc-a59f8b12f455","Type":"ContainerDied","Data":"b0609a92ed841e0fa33f06b093579e6c313e8a41d41194eda4adae82ed8e4e46"}
Sep 16 02:35:46.889029 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:35:46.889005    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-da6c4f-openshift-default-7d6598f44-fg5qr" event={"ID":"88e2b6bf-fe2d-40ea-afdc-a59f8b12f455","Type":"ContainerDied","Data":"0d4656d1034889bc5e47d4457fadc45cc4e856dee2c5a10af72f4de03c1d261d"}
Sep 16 02:35:46.889140 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:35:46.889036    2583 scope.go:117] "RemoveContainer" containerID="b0609a92ed841e0fa33f06b093579e6c313e8a41d41194eda4adae82ed8e4e46"
Sep 16 02:35:46.900821 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:35:46.900774    2583 scope.go:117] "RemoveContainer" containerID="bc8eb03c5367c491ba7df4dc61d8f9fcd2a15dbaaf1e12194c2e632beb67c50b"
Sep 16 02:35:46.916426 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:35:46.909918    2583 scope.go:117] "RemoveContainer" containerID="b0609a92ed841e0fa33f06b093579e6c313e8a41d41194eda4adae82ed8e4e46"
Sep 16 02:35:46.916426 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:35:46.913479    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-da6c4f-openshift-default-7d6598f44-fg5qr"]
Sep 16 02:35:46.917698 ip-10-0-128-111 kubenswrapper[2583]: E0916 02:35:46.917662    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b0609a92ed841e0fa33f06b093579e6c313e8a41d41194eda4adae82ed8e4e46\": container with ID starting with b0609a92ed841e0fa33f06b093579e6c313e8a41d41194eda4adae82ed8e4e46 not found: ID does not exist" containerID="b0609a92ed841e0fa33f06b093579e6c313e8a41d41194eda4adae82ed8e4e46"
Sep 16 02:35:46.917878 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:35:46.917711    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b0609a92ed841e0fa33f06b093579e6c313e8a41d41194eda4adae82ed8e4e46"} err="failed to get container status \"b0609a92ed841e0fa33f06b093579e6c313e8a41d41194eda4adae82ed8e4e46\": rpc error: code = NotFound desc = could not find container \"b0609a92ed841e0fa33f06b093579e6c313e8a41d41194eda4adae82ed8e4e46\": container with ID starting with b0609a92ed841e0fa33f06b093579e6c313e8a41d41194eda4adae82ed8e4e46 not found: ID does not exist"
Sep 16 02:35:46.917990 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:35:46.917886    2583 scope.go:117] "RemoveContainer" containerID="bc8eb03c5367c491ba7df4dc61d8f9fcd2a15dbaaf1e12194c2e632beb67c50b"
Sep 16 02:35:46.918288 ip-10-0-128-111 kubenswrapper[2583]: E0916 02:35:46.918262    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bc8eb03c5367c491ba7df4dc61d8f9fcd2a15dbaaf1e12194c2e632beb67c50b\": container with ID starting with bc8eb03c5367c491ba7df4dc61d8f9fcd2a15dbaaf1e12194c2e632beb67c50b not found: ID does not exist" containerID="bc8eb03c5367c491ba7df4dc61d8f9fcd2a15dbaaf1e12194c2e632beb67c50b"
Sep 16 02:35:46.918366 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:35:46.918296    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bc8eb03c5367c491ba7df4dc61d8f9fcd2a15dbaaf1e12194c2e632beb67c50b"} err="failed to get container status \"bc8eb03c5367c491ba7df4dc61d8f9fcd2a15dbaaf1e12194c2e632beb67c50b\": rpc error: code = NotFound desc = could not find container \"bc8eb03c5367c491ba7df4dc61d8f9fcd2a15dbaaf1e12194c2e632beb67c50b\": container with ID starting with bc8eb03c5367c491ba7df4dc61d8f9fcd2a15dbaaf1e12194c2e632beb67c50b not found: ID does not exist"
Sep 16 02:35:46.919950 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:35:46.919874    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-da6c4f-openshift-default-7d6598f44-fg5qr"]
Sep 16 02:35:47.993375 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:35:47.993342    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-db1ed7-57b4fc7f46-6jjzx"]
Sep 16 02:35:47.993762 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:35:47.993741    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="88e2b6bf-fe2d-40ea-afdc-a59f8b12f455" containerName="istio-proxy"
Sep 16 02:35:47.993762 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:35:47.993753    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="88e2b6bf-fe2d-40ea-afdc-a59f8b12f455" containerName="istio-proxy"
Sep 16 02:35:47.993762 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:35:47.993760    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="88e2b6bf-fe2d-40ea-afdc-a59f8b12f455" containerName="istio-proxy"
Sep 16 02:35:47.993872 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:35:47.993767    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="88e2b6bf-fe2d-40ea-afdc-a59f8b12f455" containerName="istio-proxy"
Sep 16 02:35:47.993872 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:35:47.993774    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e1e04d6f-7bea-4a89-ba47-0fb7682746a2" containerName="revoke-keys"
Sep 16 02:35:47.993872 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:35:47.993779    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="e1e04d6f-7bea-4a89-ba47-0fb7682746a2" containerName="revoke-keys"
Sep 16 02:35:47.993872 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:35:47.993794    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8795109a-cd57-4312-9988-01f98259eb7f" containerName="payload-pre-processing"
Sep 16 02:35:47.993872 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:35:47.993799    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="8795109a-cd57-4312-9988-01f98259eb7f" containerName="payload-pre-processing"
Sep 16 02:35:47.993872 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:35:47.993812    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a80a9d20-dc4b-4020-96f8-8f1c9e0be8bf" containerName="payload-processing"
Sep 16 02:35:47.993872 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:35:47.993818    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="a80a9d20-dc4b-4020-96f8-8f1c9e0be8bf" containerName="payload-processing"
Sep 16 02:35:47.994086 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:35:47.993876    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="88e2b6bf-fe2d-40ea-afdc-a59f8b12f455" containerName="istio-proxy"
Sep 16 02:35:47.994086 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:35:47.993884    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="e1e04d6f-7bea-4a89-ba47-0fb7682746a2" containerName="revoke-keys"
Sep 16 02:35:47.994086 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:35:47.993891    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="a80a9d20-dc4b-4020-96f8-8f1c9e0be8bf" containerName="payload-processing"
Sep 16 02:35:47.994086 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:35:47.993898    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="88e2b6bf-fe2d-40ea-afdc-a59f8b12f455" containerName="istio-proxy"
Sep 16 02:35:47.994086 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:35:47.993905    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="8795109a-cd57-4312-9988-01f98259eb7f" containerName="payload-pre-processing"
Sep 16 02:35:47.998284 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:35:47.998267    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-db1ed7-57b4fc7f46-6jjzx"
Sep 16 02:35:48.000747 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:35:48.000727    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w2-db1ed7-dockercfg-xdllf\""
Sep 16 02:35:48.000857 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:35:48.000746    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w2-db1ed7\""
Sep 16 02:35:48.006123 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:35:48.006094    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-db1ed7-57b4fc7f46-6jjzx"]
Sep 16 02:35:48.045931 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:35:48.045897    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="88e2b6bf-fe2d-40ea-afdc-a59f8b12f455" path="/var/lib/kubelet/pods/88e2b6bf-fe2d-40ea-afdc-a59f8b12f455/volumes"
Sep 16 02:35:48.083125 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:35:48.083088    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a9250574-d9e5-4188-b8db-0fb7a9cf1f09-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w2-db1ed7-57b4fc7f46-6jjzx\" (UID: \"a9250574-d9e5-4188-b8db-0fb7a9cf1f09\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-db1ed7-57b4fc7f46-6jjzx"
Sep 16 02:35:48.083298 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:35:48.083155    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lkbfq\" (UniqueName: \"kubernetes.io/projected/a9250574-d9e5-4188-b8db-0fb7a9cf1f09-kube-api-access-lkbfq\") pod \"payload-pre-processing-e2e-shared-a-w2-db1ed7-57b4fc7f46-6jjzx\" (UID: \"a9250574-d9e5-4188-b8db-0fb7a9cf1f09\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-db1ed7-57b4fc7f46-6jjzx"
Sep 16 02:35:48.183962 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:35:48.183926    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a9250574-d9e5-4188-b8db-0fb7a9cf1f09-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w2-db1ed7-57b4fc7f46-6jjzx\" (UID: \"a9250574-d9e5-4188-b8db-0fb7a9cf1f09\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-db1ed7-57b4fc7f46-6jjzx"
Sep 16 02:35:48.184160 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:35:48.183973    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lkbfq\" (UniqueName: \"kubernetes.io/projected/a9250574-d9e5-4188-b8db-0fb7a9cf1f09-kube-api-access-lkbfq\") pod \"payload-pre-processing-e2e-shared-a-w2-db1ed7-57b4fc7f46-6jjzx\" (UID: \"a9250574-d9e5-4188-b8db-0fb7a9cf1f09\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-db1ed7-57b4fc7f46-6jjzx"
Sep 16 02:35:48.184669 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:35:48.184643    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a9250574-d9e5-4188-b8db-0fb7a9cf1f09-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w2-db1ed7-57b4fc7f46-6jjzx\" (UID: \"a9250574-d9e5-4188-b8db-0fb7a9cf1f09\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-db1ed7-57b4fc7f46-6jjzx"
Sep 16 02:35:48.191659 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:35:48.191638    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lkbfq\" (UniqueName: \"kubernetes.io/projected/a9250574-d9e5-4188-b8db-0fb7a9cf1f09-kube-api-access-lkbfq\") pod \"payload-pre-processing-e2e-shared-a-w2-db1ed7-57b4fc7f46-6jjzx\" (UID: \"a9250574-d9e5-4188-b8db-0fb7a9cf1f09\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-db1ed7-57b4fc7f46-6jjzx"
Sep 16 02:35:48.310003 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:35:48.309924    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-db1ed7-57b4fc7f46-6jjzx"
Sep 16 02:35:48.578623 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:35:48.578597    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-db1ed7-57b4fc7f46-6jjzx"]
Sep 16 02:35:48.581633 ip-10-0-128-111 kubenswrapper[2583]: W0916 02:35:48.581597    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda9250574_d9e5_4188_b8db_0fb7a9cf1f09.slice/crio-ee8cfdc1023748743df27e5e264809f78f45ab85b3cc7ce5864f0c0f9d1b56b6 WatchSource:0}: Error finding container ee8cfdc1023748743df27e5e264809f78f45ab85b3cc7ce5864f0c0f9d1b56b6: Status 404 returned error can't find the container with id ee8cfdc1023748743df27e5e264809f78f45ab85b3cc7ce5864f0c0f9d1b56b6
Sep 16 02:35:48.902007 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:35:48.901969    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-db1ed7-57b4fc7f46-6jjzx" event={"ID":"a9250574-d9e5-4188-b8db-0fb7a9cf1f09","Type":"ContainerStarted","Data":"a2a42f5a989940fdd33a897fecba063c1d32e61d20711e40a477ff3061724454"}
Sep 16 02:35:48.902219 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:35:48.902021    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-db1ed7-57b4fc7f46-6jjzx" event={"ID":"a9250574-d9e5-4188-b8db-0fb7a9cf1f09","Type":"ContainerStarted","Data":"ee8cfdc1023748743df27e5e264809f78f45ab85b3cc7ce5864f0c0f9d1b56b6"}
Sep 16 02:35:48.902219 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:35:48.902087    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-db1ed7-57b4fc7f46-6jjzx"
Sep 16 02:35:48.920699 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:35:48.920649    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-db1ed7-57b4fc7f46-6jjzx" podStartSLOduration=1.920631475 podStartE2EDuration="1.920631475s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 02:35:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 02:35:48.918940712 +0000 UTC m=+1027.386729495" watchObservedRunningTime="2026-09-16 02:35:48.920631475 +0000 UTC m=+1027.388420258"
Sep 16 02:35:50.480631 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:35:50.480602    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-c40df7fa-7bdb8b9b86-4mnld"]
Sep 16 02:35:50.481080 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:35:50.480806    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-migrate-c40df7fa-7bdb8b9b86-4mnld" podUID="00c0c4bc-820c-4822-b7bb-2e3bbf9cbab4" containerName="payload-processing" containerID="cri-o://133ab3082e29a715875a6a706adc77ced63ac61607bb99375cb2645929c3663f" gracePeriod=30
Sep 16 02:35:50.736933 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:35:50.736876    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-c40df7fa-7bdb8b9b86-4mnld"
Sep 16 02:35:50.811236 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:35:50.811207    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rj4hk\" (UniqueName: \"kubernetes.io/projected/00c0c4bc-820c-4822-b7bb-2e3bbf9cbab4-kube-api-access-rj4hk\") pod \"00c0c4bc-820c-4822-b7bb-2e3bbf9cbab4\" (UID: \"00c0c4bc-820c-4822-b7bb-2e3bbf9cbab4\") "
Sep 16 02:35:50.811394 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:35:50.811257    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/00c0c4bc-820c-4822-b7bb-2e3bbf9cbab4-plugins-config-volume\") pod \"00c0c4bc-820c-4822-b7bb-2e3bbf9cbab4\" (UID: \"00c0c4bc-820c-4822-b7bb-2e3bbf9cbab4\") "
Sep 16 02:35:50.811695 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:35:50.811674    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/00c0c4bc-820c-4822-b7bb-2e3bbf9cbab4-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "00c0c4bc-820c-4822-b7bb-2e3bbf9cbab4" (UID: "00c0c4bc-820c-4822-b7bb-2e3bbf9cbab4"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 02:35:50.813506 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:35:50.813451    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/00c0c4bc-820c-4822-b7bb-2e3bbf9cbab4-kube-api-access-rj4hk" (OuterVolumeSpecName: "kube-api-access-rj4hk") pod "00c0c4bc-820c-4822-b7bb-2e3bbf9cbab4" (UID: "00c0c4bc-820c-4822-b7bb-2e3bbf9cbab4"). InnerVolumeSpecName "kube-api-access-rj4hk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 02:35:50.912063 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:35:50.912030    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rj4hk\" (UniqueName: \"kubernetes.io/projected/00c0c4bc-820c-4822-b7bb-2e3bbf9cbab4-kube-api-access-rj4hk\") on node \"ip-10-0-128-111.ec2.internal\" DevicePath \"\""
Sep 16 02:35:50.912063 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:35:50.912059    2583 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/00c0c4bc-820c-4822-b7bb-2e3bbf9cbab4-plugins-config-volume\") on node \"ip-10-0-128-111.ec2.internal\" DevicePath \"\""
Sep 16 02:35:50.912632 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:35:50.912609    2583 generic.go:356] "Generic (PLEG): container finished" podID="00c0c4bc-820c-4822-b7bb-2e3bbf9cbab4" containerID="133ab3082e29a715875a6a706adc77ced63ac61607bb99375cb2645929c3663f" exitCode=0
Sep 16 02:35:50.912725 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:35:50.912662    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-c40df7fa-7bdb8b9b86-4mnld"
Sep 16 02:35:50.912725 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:35:50.912696    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-c40df7fa-7bdb8b9b86-4mnld" event={"ID":"00c0c4bc-820c-4822-b7bb-2e3bbf9cbab4","Type":"ContainerDied","Data":"133ab3082e29a715875a6a706adc77ced63ac61607bb99375cb2645929c3663f"}
Sep 16 02:35:50.912829 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:35:50.912750    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-c40df7fa-7bdb8b9b86-4mnld" event={"ID":"00c0c4bc-820c-4822-b7bb-2e3bbf9cbab4","Type":"ContainerDied","Data":"50c9aa347cdb7ae2a6f8299ebea8b93e20c878923cded62636c5d19eeb871bb2"}
Sep 16 02:35:50.912829 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:35:50.912776    2583 scope.go:117] "RemoveContainer" containerID="133ab3082e29a715875a6a706adc77ced63ac61607bb99375cb2645929c3663f"
Sep 16 02:35:50.921423 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:35:50.921407    2583 scope.go:117] "RemoveContainer" containerID="133ab3082e29a715875a6a706adc77ced63ac61607bb99375cb2645929c3663f"
Sep 16 02:35:50.921697 ip-10-0-128-111 kubenswrapper[2583]: E0916 02:35:50.921679    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"133ab3082e29a715875a6a706adc77ced63ac61607bb99375cb2645929c3663f\": container with ID starting with 133ab3082e29a715875a6a706adc77ced63ac61607bb99375cb2645929c3663f not found: ID does not exist" containerID="133ab3082e29a715875a6a706adc77ced63ac61607bb99375cb2645929c3663f"
Sep 16 02:35:50.921759 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:35:50.921703    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"133ab3082e29a715875a6a706adc77ced63ac61607bb99375cb2645929c3663f"} err="failed to get container status \"133ab3082e29a715875a6a706adc77ced63ac61607bb99375cb2645929c3663f\": rpc error: code = NotFound desc = could not find container \"133ab3082e29a715875a6a706adc77ced63ac61607bb99375cb2645929c3663f\": container with ID starting with 133ab3082e29a715875a6a706adc77ced63ac61607bb99375cb2645929c3663f not found: ID does not exist"
Sep 16 02:35:50.941029 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:35:50.939533    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-c40df7fa-7bdb8b9b86-4mnld"]
Sep 16 02:35:50.941029 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:35:50.939597    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-c40df7fa-7bdb8b9b86-4mnld"]
Sep 16 02:35:51.437665 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:35:51.437633    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-b099417d-openshift-default-7c4dc99d7b-plbvp"]
Sep 16 02:35:51.438053 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:35:51.438040    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="00c0c4bc-820c-4822-b7bb-2e3bbf9cbab4" containerName="payload-processing"
Sep 16 02:35:51.438113 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:35:51.438055    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="00c0c4bc-820c-4822-b7bb-2e3bbf9cbab4" containerName="payload-processing"
Sep 16 02:35:51.438113 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:35:51.438110    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="00c0c4bc-820c-4822-b7bb-2e3bbf9cbab4" containerName="payload-processing"
Sep 16 02:35:51.442439 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:35:51.442418    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-b099417d-openshift-default-7c4dc99d7b-plbvp"
Sep 16 02:35:51.444799 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:35:51.444779    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-b099417d-openshift-default-dockercfg-jxnk9\""
Sep 16 02:35:51.449213 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:35:51.449192    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-b099417d-openshift-default-7c4dc99d7b-plbvp"]
Sep 16 02:35:51.517570 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:35:51.517516    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8e4a50d5-d5fb-47fa-b1f2-4f51cdc42aaf-istiod-ca-cert\") pod \"e2e-ait-b099417d-openshift-default-7c4dc99d7b-plbvp\" (UID: \"8e4a50d5-d5fb-47fa-b1f2-4f51cdc42aaf\") " pod="openshift-ingress/e2e-ait-b099417d-openshift-default-7c4dc99d7b-plbvp"
Sep 16 02:35:51.517949 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:35:51.517579    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8e4a50d5-d5fb-47fa-b1f2-4f51cdc42aaf-istio-token\") pod \"e2e-ait-b099417d-openshift-default-7c4dc99d7b-plbvp\" (UID: \"8e4a50d5-d5fb-47fa-b1f2-4f51cdc42aaf\") " pod="openshift-ingress/e2e-ait-b099417d-openshift-default-7c4dc99d7b-plbvp"
Sep 16 02:35:51.517949 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:35:51.517699    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8e4a50d5-d5fb-47fa-b1f2-4f51cdc42aaf-istio-podinfo\") pod \"e2e-ait-b099417d-openshift-default-7c4dc99d7b-plbvp\" (UID: \"8e4a50d5-d5fb-47fa-b1f2-4f51cdc42aaf\") " pod="openshift-ingress/e2e-ait-b099417d-openshift-default-7c4dc99d7b-plbvp"
Sep 16 02:35:51.517949 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:35:51.517745    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rtts2\" (UniqueName: \"kubernetes.io/projected/8e4a50d5-d5fb-47fa-b1f2-4f51cdc42aaf-kube-api-access-rtts2\") pod \"e2e-ait-b099417d-openshift-default-7c4dc99d7b-plbvp\" (UID: \"8e4a50d5-d5fb-47fa-b1f2-4f51cdc42aaf\") " pod="openshift-ingress/e2e-ait-b099417d-openshift-default-7c4dc99d7b-plbvp"
Sep 16 02:35:51.517949 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:35:51.517789    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e4a50d5-d5fb-47fa-b1f2-4f51cdc42aaf-workload-socket\") pod \"e2e-ait-b099417d-openshift-default-7c4dc99d7b-plbvp\" (UID: \"8e4a50d5-d5fb-47fa-b1f2-4f51cdc42aaf\") " pod="openshift-ingress/e2e-ait-b099417d-openshift-default-7c4dc99d7b-plbvp"
Sep 16 02:35:51.517949 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:35:51.517810    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e4a50d5-d5fb-47fa-b1f2-4f51cdc42aaf-credential-socket\") pod \"e2e-ait-b099417d-openshift-default-7c4dc99d7b-plbvp\" (UID: \"8e4a50d5-d5fb-47fa-b1f2-4f51cdc42aaf\") " pod="openshift-ingress/e2e-ait-b099417d-openshift-default-7c4dc99d7b-plbvp"
Sep 16 02:35:51.517949 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:35:51.517836    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8e4a50d5-d5fb-47fa-b1f2-4f51cdc42aaf-istio-data\") pod \"e2e-ait-b099417d-openshift-default-7c4dc99d7b-plbvp\" (UID: \"8e4a50d5-d5fb-47fa-b1f2-4f51cdc42aaf\") " pod="openshift-ingress/e2e-ait-b099417d-openshift-default-7c4dc99d7b-plbvp"
Sep 16 02:35:51.517949 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:35:51.517857    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8e4a50d5-d5fb-47fa-b1f2-4f51cdc42aaf-workload-certs\") pod \"e2e-ait-b099417d-openshift-default-7c4dc99d7b-plbvp\" (UID: \"8e4a50d5-d5fb-47fa-b1f2-4f51cdc42aaf\") " pod="openshift-ingress/e2e-ait-b099417d-openshift-default-7c4dc99d7b-plbvp"
Sep 16 02:35:51.517949 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:35:51.517920    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8e4a50d5-d5fb-47fa-b1f2-4f51cdc42aaf-istio-envoy\") pod \"e2e-ait-b099417d-openshift-default-7c4dc99d7b-plbvp\" (UID: \"8e4a50d5-d5fb-47fa-b1f2-4f51cdc42aaf\") " pod="openshift-ingress/e2e-ait-b099417d-openshift-default-7c4dc99d7b-plbvp"
Sep 16 02:35:51.618966 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:35:51.618925    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8e4a50d5-d5fb-47fa-b1f2-4f51cdc42aaf-istio-token\") pod \"e2e-ait-b099417d-openshift-default-7c4dc99d7b-plbvp\" (UID: \"8e4a50d5-d5fb-47fa-b1f2-4f51cdc42aaf\") " pod="openshift-ingress/e2e-ait-b099417d-openshift-default-7c4dc99d7b-plbvp"
Sep 16 02:35:51.619177 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:35:51.619025    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8e4a50d5-d5fb-47fa-b1f2-4f51cdc42aaf-istio-podinfo\") pod \"e2e-ait-b099417d-openshift-default-7c4dc99d7b-plbvp\" (UID: \"8e4a50d5-d5fb-47fa-b1f2-4f51cdc42aaf\") " pod="openshift-ingress/e2e-ait-b099417d-openshift-default-7c4dc99d7b-plbvp"
Sep 16 02:35:51.619177 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:35:51.619071    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rtts2\" (UniqueName: \"kubernetes.io/projected/8e4a50d5-d5fb-47fa-b1f2-4f51cdc42aaf-kube-api-access-rtts2\") pod \"e2e-ait-b099417d-openshift-default-7c4dc99d7b-plbvp\" (UID: \"8e4a50d5-d5fb-47fa-b1f2-4f51cdc42aaf\") " pod="openshift-ingress/e2e-ait-b099417d-openshift-default-7c4dc99d7b-plbvp"
Sep 16 02:35:51.619177 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:35:51.619100    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e4a50d5-d5fb-47fa-b1f2-4f51cdc42aaf-workload-socket\") pod \"e2e-ait-b099417d-openshift-default-7c4dc99d7b-plbvp\" (UID: \"8e4a50d5-d5fb-47fa-b1f2-4f51cdc42aaf\") " pod="openshift-ingress/e2e-ait-b099417d-openshift-default-7c4dc99d7b-plbvp"
Sep 16 02:35:51.619177 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:35:51.619126    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e4a50d5-d5fb-47fa-b1f2-4f51cdc42aaf-credential-socket\") pod \"e2e-ait-b099417d-openshift-default-7c4dc99d7b-plbvp\" (UID: \"8e4a50d5-d5fb-47fa-b1f2-4f51cdc42aaf\") " pod="openshift-ingress/e2e-ait-b099417d-openshift-default-7c4dc99d7b-plbvp"
Sep 16 02:35:51.619177 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:35:51.619165    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8e4a50d5-d5fb-47fa-b1f2-4f51cdc42aaf-istio-data\") pod \"e2e-ait-b099417d-openshift-default-7c4dc99d7b-plbvp\" (UID: \"8e4a50d5-d5fb-47fa-b1f2-4f51cdc42aaf\") " pod="openshift-ingress/e2e-ait-b099417d-openshift-default-7c4dc99d7b-plbvp"
Sep 16 02:35:51.619446 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:35:51.619199    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8e4a50d5-d5fb-47fa-b1f2-4f51cdc42aaf-workload-certs\") pod \"e2e-ait-b099417d-openshift-default-7c4dc99d7b-plbvp\" (UID: \"8e4a50d5-d5fb-47fa-b1f2-4f51cdc42aaf\") " pod="openshift-ingress/e2e-ait-b099417d-openshift-default-7c4dc99d7b-plbvp"
Sep 16 02:35:51.619446 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:35:51.619237    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8e4a50d5-d5fb-47fa-b1f2-4f51cdc42aaf-istio-envoy\") pod \"e2e-ait-b099417d-openshift-default-7c4dc99d7b-plbvp\" (UID: \"8e4a50d5-d5fb-47fa-b1f2-4f51cdc42aaf\") " pod="openshift-ingress/e2e-ait-b099417d-openshift-default-7c4dc99d7b-plbvp"
Sep 16 02:35:51.619446 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:35:51.619283    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8e4a50d5-d5fb-47fa-b1f2-4f51cdc42aaf-istiod-ca-cert\") pod \"e2e-ait-b099417d-openshift-default-7c4dc99d7b-plbvp\" (UID: \"8e4a50d5-d5fb-47fa-b1f2-4f51cdc42aaf\") " pod="openshift-ingress/e2e-ait-b099417d-openshift-default-7c4dc99d7b-plbvp"
Sep 16 02:35:51.619657 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:35:51.619635    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e4a50d5-d5fb-47fa-b1f2-4f51cdc42aaf-workload-socket\") pod \"e2e-ait-b099417d-openshift-default-7c4dc99d7b-plbvp\" (UID: \"8e4a50d5-d5fb-47fa-b1f2-4f51cdc42aaf\") " pod="openshift-ingress/e2e-ait-b099417d-openshift-default-7c4dc99d7b-plbvp"
Sep 16 02:35:51.619717 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:35:51.619653    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e4a50d5-d5fb-47fa-b1f2-4f51cdc42aaf-credential-socket\") pod \"e2e-ait-b099417d-openshift-default-7c4dc99d7b-plbvp\" (UID: \"8e4a50d5-d5fb-47fa-b1f2-4f51cdc42aaf\") " pod="openshift-ingress/e2e-ait-b099417d-openshift-default-7c4dc99d7b-plbvp"
Sep 16 02:35:51.619773 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:35:51.619743    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8e4a50d5-d5fb-47fa-b1f2-4f51cdc42aaf-istio-data\") pod \"e2e-ait-b099417d-openshift-default-7c4dc99d7b-plbvp\" (UID: \"8e4a50d5-d5fb-47fa-b1f2-4f51cdc42aaf\") " pod="openshift-ingress/e2e-ait-b099417d-openshift-default-7c4dc99d7b-plbvp"
Sep 16 02:35:51.619898 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:35:51.619877    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8e4a50d5-d5fb-47fa-b1f2-4f51cdc42aaf-workload-certs\") pod \"e2e-ait-b099417d-openshift-default-7c4dc99d7b-plbvp\" (UID: \"8e4a50d5-d5fb-47fa-b1f2-4f51cdc42aaf\") " pod="openshift-ingress/e2e-ait-b099417d-openshift-default-7c4dc99d7b-plbvp"
Sep 16 02:35:51.620045 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:35:51.620025    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8e4a50d5-d5fb-47fa-b1f2-4f51cdc42aaf-istiod-ca-cert\") pod \"e2e-ait-b099417d-openshift-default-7c4dc99d7b-plbvp\" (UID: \"8e4a50d5-d5fb-47fa-b1f2-4f51cdc42aaf\") " pod="openshift-ingress/e2e-ait-b099417d-openshift-default-7c4dc99d7b-plbvp"
Sep 16 02:35:51.621596 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:35:51.621576    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8e4a50d5-d5fb-47fa-b1f2-4f51cdc42aaf-istio-envoy\") pod \"e2e-ait-b099417d-openshift-default-7c4dc99d7b-plbvp\" (UID: \"8e4a50d5-d5fb-47fa-b1f2-4f51cdc42aaf\") " pod="openshift-ingress/e2e-ait-b099417d-openshift-default-7c4dc99d7b-plbvp"
Sep 16 02:35:51.621741 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:35:51.621723    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8e4a50d5-d5fb-47fa-b1f2-4f51cdc42aaf-istio-podinfo\") pod \"e2e-ait-b099417d-openshift-default-7c4dc99d7b-plbvp\" (UID: \"8e4a50d5-d5fb-47fa-b1f2-4f51cdc42aaf\") " pod="openshift-ingress/e2e-ait-b099417d-openshift-default-7c4dc99d7b-plbvp"
Sep 16 02:35:51.626718 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:35:51.626697    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rtts2\" (UniqueName: \"kubernetes.io/projected/8e4a50d5-d5fb-47fa-b1f2-4f51cdc42aaf-kube-api-access-rtts2\") pod \"e2e-ait-b099417d-openshift-default-7c4dc99d7b-plbvp\" (UID: \"8e4a50d5-d5fb-47fa-b1f2-4f51cdc42aaf\") " pod="openshift-ingress/e2e-ait-b099417d-openshift-default-7c4dc99d7b-plbvp"
Sep 16 02:35:51.626810 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:35:51.626764    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8e4a50d5-d5fb-47fa-b1f2-4f51cdc42aaf-istio-token\") pod \"e2e-ait-b099417d-openshift-default-7c4dc99d7b-plbvp\" (UID: \"8e4a50d5-d5fb-47fa-b1f2-4f51cdc42aaf\") " pod="openshift-ingress/e2e-ait-b099417d-openshift-default-7c4dc99d7b-plbvp"
Sep 16 02:35:51.753855 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:35:51.753757    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-b099417d-openshift-default-7c4dc99d7b-plbvp"
Sep 16 02:35:52.029408 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:35:52.029369    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-b099417d-openshift-default-7c4dc99d7b-plbvp"]
Sep 16 02:35:52.031772 ip-10-0-128-111 kubenswrapper[2583]: W0916 02:35:52.031747    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8e4a50d5_d5fb_47fa_b1f2_4f51cdc42aaf.slice/crio-719d2614f5573cbd99d8c4532fa2a1e1d4329e8837dd1c00e89588b354011245 WatchSource:0}: Error finding container 719d2614f5573cbd99d8c4532fa2a1e1d4329e8837dd1c00e89588b354011245: Status 404 returned error can't find the container with id 719d2614f5573cbd99d8c4532fa2a1e1d4329e8837dd1c00e89588b354011245
Sep 16 02:35:52.034515 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:35:52.034480    2583 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 16 02:35:52.034611 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:35:52.034573    2583 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 16 02:35:52.034611 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:35:52.034607    2583 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 16 02:35:52.045998 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:35:52.045972    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="00c0c4bc-820c-4822-b7bb-2e3bbf9cbab4" path="/var/lib/kubelet/pods/00c0c4bc-820c-4822-b7bb-2e3bbf9cbab4/volumes"
Sep 16 02:35:52.924924 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:35:52.924891    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-b099417d-openshift-default-7c4dc99d7b-plbvp" event={"ID":"8e4a50d5-d5fb-47fa-b1f2-4f51cdc42aaf","Type":"ContainerStarted","Data":"718e4605f8aaf01bd9ebf1e2af38af0aa7a7360322ce690269327ac53aa37779"}
Sep 16 02:35:52.925339 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:35:52.924944    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-b099417d-openshift-default-7c4dc99d7b-plbvp" event={"ID":"8e4a50d5-d5fb-47fa-b1f2-4f51cdc42aaf","Type":"ContainerStarted","Data":"719d2614f5573cbd99d8c4532fa2a1e1d4329e8837dd1c00e89588b354011245"}
Sep 16 02:35:52.942651 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:35:52.942595    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-b099417d-openshift-default-7c4dc99d7b-plbvp" podStartSLOduration=1.9425450340000001 podStartE2EDuration="1.942545034s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 02:35:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 02:35:52.942260459 +0000 UTC m=+1031.410049243" watchObservedRunningTime="2026-09-16 02:35:52.942545034 +0000 UTC m=+1031.410333818"
Sep 16 02:35:53.753911 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:35:53.753879    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-b099417d-openshift-default-7c4dc99d7b-plbvp"
Sep 16 02:35:53.758997 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:35:53.758973    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-b099417d-openshift-default-7c4dc99d7b-plbvp"
Sep 16 02:35:53.930003 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:35:53.929973    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-b099417d-openshift-default-7c4dc99d7b-plbvp"
Sep 16 02:35:53.931046 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:35:53.931025    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-b099417d-openshift-default-7c4dc99d7b-plbvp"
Sep 16 02:35:56.913787 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:35:56.913753    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-b099417d-7744cc6bc8-zdfcd"]
Sep 16 02:35:56.917266 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:35:56.917249    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-b099417d-7744cc6bc8-zdfcd"
Sep 16 02:35:56.921036 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:35:56.921015    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-b099417d-dockercfg-wplbt\""
Sep 16 02:35:56.921036 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:35:56.921034    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-b099417d\""
Sep 16 02:35:56.928835 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:35:56.928813    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-b099417d-7744cc6bc8-zdfcd"]
Sep 16 02:35:56.948425 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:35:56.948398    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-b099417d-5d6fb6b6c5-q2hzp"]
Sep 16 02:35:56.951957 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:35:56.951942    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-b099417d-5d6fb6b6c5-q2hzp"
Sep 16 02:35:56.960304 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:35:56.960281    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-b099417d-5d6fb6b6c5-q2hzp"]
Sep 16 02:35:57.072245 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:35:57.072207    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d8c85be6-e00e-4437-8d5c-ee02d6ccc25c-plugins-config-volume\") pod \"payload-processing-e2e-ait-b099417d-5d6fb6b6c5-q2hzp\" (UID: \"d8c85be6-e00e-4437-8d5c-ee02d6ccc25c\") " pod="openshift-ingress/payload-processing-e2e-ait-b099417d-5d6fb6b6c5-q2hzp"
Sep 16 02:35:57.072430 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:35:57.072314    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/97d943e2-e2b0-44b9-933a-b1223c1f0774-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-b099417d-7744cc6bc8-zdfcd\" (UID: \"97d943e2-e2b0-44b9-933a-b1223c1f0774\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-b099417d-7744cc6bc8-zdfcd"
Sep 16 02:35:57.072430 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:35:57.072360    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9njbp\" (UniqueName: \"kubernetes.io/projected/d8c85be6-e00e-4437-8d5c-ee02d6ccc25c-kube-api-access-9njbp\") pod \"payload-processing-e2e-ait-b099417d-5d6fb6b6c5-q2hzp\" (UID: \"d8c85be6-e00e-4437-8d5c-ee02d6ccc25c\") " pod="openshift-ingress/payload-processing-e2e-ait-b099417d-5d6fb6b6c5-q2hzp"
Sep 16 02:35:57.072430 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:35:57.072380    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jlbf7\" (UniqueName: \"kubernetes.io/projected/97d943e2-e2b0-44b9-933a-b1223c1f0774-kube-api-access-jlbf7\") pod \"payload-pre-processing-e2e-ait-b099417d-7744cc6bc8-zdfcd\" (UID: \"97d943e2-e2b0-44b9-933a-b1223c1f0774\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-b099417d-7744cc6bc8-zdfcd"
Sep 16 02:35:57.173480 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:35:57.173390    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d8c85be6-e00e-4437-8d5c-ee02d6ccc25c-plugins-config-volume\") pod \"payload-processing-e2e-ait-b099417d-5d6fb6b6c5-q2hzp\" (UID: \"d8c85be6-e00e-4437-8d5c-ee02d6ccc25c\") " pod="openshift-ingress/payload-processing-e2e-ait-b099417d-5d6fb6b6c5-q2hzp"
Sep 16 02:35:57.173480 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:35:57.173459    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/97d943e2-e2b0-44b9-933a-b1223c1f0774-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-b099417d-7744cc6bc8-zdfcd\" (UID: \"97d943e2-e2b0-44b9-933a-b1223c1f0774\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-b099417d-7744cc6bc8-zdfcd"
Sep 16 02:35:57.173764 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:35:57.173489    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9njbp\" (UniqueName: \"kubernetes.io/projected/d8c85be6-e00e-4437-8d5c-ee02d6ccc25c-kube-api-access-9njbp\") pod \"payload-processing-e2e-ait-b099417d-5d6fb6b6c5-q2hzp\" (UID: \"d8c85be6-e00e-4437-8d5c-ee02d6ccc25c\") " pod="openshift-ingress/payload-processing-e2e-ait-b099417d-5d6fb6b6c5-q2hzp"
Sep 16 02:35:57.173764 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:35:57.173507    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jlbf7\" (UniqueName: \"kubernetes.io/projected/97d943e2-e2b0-44b9-933a-b1223c1f0774-kube-api-access-jlbf7\") pod \"payload-pre-processing-e2e-ait-b099417d-7744cc6bc8-zdfcd\" (UID: \"97d943e2-e2b0-44b9-933a-b1223c1f0774\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-b099417d-7744cc6bc8-zdfcd"
Sep 16 02:35:57.174216 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:35:57.174191    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/97d943e2-e2b0-44b9-933a-b1223c1f0774-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-b099417d-7744cc6bc8-zdfcd\" (UID: \"97d943e2-e2b0-44b9-933a-b1223c1f0774\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-b099417d-7744cc6bc8-zdfcd"
Sep 16 02:35:57.174308 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:35:57.174199    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d8c85be6-e00e-4437-8d5c-ee02d6ccc25c-plugins-config-volume\") pod \"payload-processing-e2e-ait-b099417d-5d6fb6b6c5-q2hzp\" (UID: \"d8c85be6-e00e-4437-8d5c-ee02d6ccc25c\") " pod="openshift-ingress/payload-processing-e2e-ait-b099417d-5d6fb6b6c5-q2hzp"
Sep 16 02:35:57.183903 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:35:57.183881    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9njbp\" (UniqueName: \"kubernetes.io/projected/d8c85be6-e00e-4437-8d5c-ee02d6ccc25c-kube-api-access-9njbp\") pod \"payload-processing-e2e-ait-b099417d-5d6fb6b6c5-q2hzp\" (UID: \"d8c85be6-e00e-4437-8d5c-ee02d6ccc25c\") " pod="openshift-ingress/payload-processing-e2e-ait-b099417d-5d6fb6b6c5-q2hzp"
Sep 16 02:35:57.184163 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:35:57.184144    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jlbf7\" (UniqueName: \"kubernetes.io/projected/97d943e2-e2b0-44b9-933a-b1223c1f0774-kube-api-access-jlbf7\") pod \"payload-pre-processing-e2e-ait-b099417d-7744cc6bc8-zdfcd\" (UID: \"97d943e2-e2b0-44b9-933a-b1223c1f0774\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-b099417d-7744cc6bc8-zdfcd"
Sep 16 02:35:57.227738 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:35:57.227699    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-b099417d-7744cc6bc8-zdfcd"
Sep 16 02:35:57.262581 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:35:57.262525    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-b099417d-5d6fb6b6c5-q2hzp"
Sep 16 02:35:57.509679 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:35:57.509654    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-b099417d-7744cc6bc8-zdfcd"]
Sep 16 02:35:57.511391 ip-10-0-128-111 kubenswrapper[2583]: W0916 02:35:57.511356    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod97d943e2_e2b0_44b9_933a_b1223c1f0774.slice/crio-d80d17885d55e426457816b4c359cc2a9c02257e2a0aa1b94fe75e81efda4509 WatchSource:0}: Error finding container d80d17885d55e426457816b4c359cc2a9c02257e2a0aa1b94fe75e81efda4509: Status 404 returned error can't find the container with id d80d17885d55e426457816b4c359cc2a9c02257e2a0aa1b94fe75e81efda4509
Sep 16 02:35:57.548411 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:35:57.548387    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-b099417d-5d6fb6b6c5-q2hzp"]
Sep 16 02:35:57.550947 ip-10-0-128-111 kubenswrapper[2583]: W0916 02:35:57.550919    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd8c85be6_e00e_4437_8d5c_ee02d6ccc25c.slice/crio-4b529e88f0f56bdaa6c73d9e4803e093cb8f1ae0edce5693163aa89ce2c418d4 WatchSource:0}: Error finding container 4b529e88f0f56bdaa6c73d9e4803e093cb8f1ae0edce5693163aa89ce2c418d4: Status 404 returned error can't find the container with id 4b529e88f0f56bdaa6c73d9e4803e093cb8f1ae0edce5693163aa89ce2c418d4
Sep 16 02:35:57.954347 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:35:57.954313    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-b099417d-5d6fb6b6c5-q2hzp" event={"ID":"d8c85be6-e00e-4437-8d5c-ee02d6ccc25c","Type":"ContainerStarted","Data":"75b4f25d99986e0021e8b18d6ca63ba53287ac01019c20a245aeb792f0f2079f"}
Sep 16 02:35:57.954760 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:35:57.954373    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-b099417d-5d6fb6b6c5-q2hzp" event={"ID":"d8c85be6-e00e-4437-8d5c-ee02d6ccc25c","Type":"ContainerStarted","Data":"4b529e88f0f56bdaa6c73d9e4803e093cb8f1ae0edce5693163aa89ce2c418d4"}
Sep 16 02:35:57.954760 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:35:57.954403    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-b099417d-5d6fb6b6c5-q2hzp"
Sep 16 02:35:57.956397 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:35:57.956372    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-b099417d-7744cc6bc8-zdfcd" event={"ID":"97d943e2-e2b0-44b9-933a-b1223c1f0774","Type":"ContainerStarted","Data":"ac93400f2f4d6aae5702f7503b502da16289595aa9788d3b039a9008ed1a0263"}
Sep 16 02:35:57.956521 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:35:57.956416    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-b099417d-7744cc6bc8-zdfcd" event={"ID":"97d943e2-e2b0-44b9-933a-b1223c1f0774","Type":"ContainerStarted","Data":"d80d17885d55e426457816b4c359cc2a9c02257e2a0aa1b94fe75e81efda4509"}
Sep 16 02:35:57.956521 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:35:57.956446    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-b099417d-7744cc6bc8-zdfcd"
Sep 16 02:35:57.969864 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:35:57.969822    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-b099417d-5d6fb6b6c5-q2hzp" podStartSLOduration=1.969807751 podStartE2EDuration="1.969807751s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 02:35:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 02:35:57.968220577 +0000 UTC m=+1036.436009386" watchObservedRunningTime="2026-09-16 02:35:57.969807751 +0000 UTC m=+1036.437596533"
Sep 16 02:35:57.983315 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:35:57.983271    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-b099417d-7744cc6bc8-zdfcd" podStartSLOduration=1.983257088 podStartE2EDuration="1.983257088s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 02:35:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 02:35:57.980936577 +0000 UTC m=+1036.448725360" watchObservedRunningTime="2026-09-16 02:35:57.983257088 +0000 UTC m=+1036.451045960"
Sep 16 02:35:59.909044 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:35:59.909010    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-db1ed7-57b4fc7f46-6jjzx"
Sep 16 02:36:05.792299 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:36:05.792257    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-eee708-78849d699f-lm2ks"]
Sep 16 02:36:05.800361 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:36:05.800335    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-eee708-78849d699f-lm2ks"
Sep 16 02:36:05.803854 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:36:05.803828    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w2-eee708-dockercfg-mhbzv\""
Sep 16 02:36:05.804106 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:36:05.804087    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w2-eee708\""
Sep 16 02:36:05.805695 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:36:05.805669    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-eee708-78849d699f-lm2ks"]
Sep 16 02:36:05.859272 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:36:05.859068    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hjn9s\" (UniqueName: \"kubernetes.io/projected/be3b30cb-484b-4f99-a52c-0567667b31c3-kube-api-access-hjn9s\") pod \"payload-processing-e2e-shared-b-w2-eee708-78849d699f-lm2ks\" (UID: \"be3b30cb-484b-4f99-a52c-0567667b31c3\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-eee708-78849d699f-lm2ks"
Sep 16 02:36:05.859272 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:36:05.859166    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/be3b30cb-484b-4f99-a52c-0567667b31c3-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w2-eee708-78849d699f-lm2ks\" (UID: \"be3b30cb-484b-4f99-a52c-0567667b31c3\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-eee708-78849d699f-lm2ks"
Sep 16 02:36:05.959712 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:36:05.959668    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hjn9s\" (UniqueName: \"kubernetes.io/projected/be3b30cb-484b-4f99-a52c-0567667b31c3-kube-api-access-hjn9s\") pod \"payload-processing-e2e-shared-b-w2-eee708-78849d699f-lm2ks\" (UID: \"be3b30cb-484b-4f99-a52c-0567667b31c3\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-eee708-78849d699f-lm2ks"
Sep 16 02:36:05.959869 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:36:05.959837    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/be3b30cb-484b-4f99-a52c-0567667b31c3-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w2-eee708-78849d699f-lm2ks\" (UID: \"be3b30cb-484b-4f99-a52c-0567667b31c3\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-eee708-78849d699f-lm2ks"
Sep 16 02:36:05.960535 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:36:05.960510    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/be3b30cb-484b-4f99-a52c-0567667b31c3-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w2-eee708-78849d699f-lm2ks\" (UID: \"be3b30cb-484b-4f99-a52c-0567667b31c3\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-eee708-78849d699f-lm2ks"
Sep 16 02:36:05.966963 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:36:05.966941    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hjn9s\" (UniqueName: \"kubernetes.io/projected/be3b30cb-484b-4f99-a52c-0567667b31c3-kube-api-access-hjn9s\") pod \"payload-processing-e2e-shared-b-w2-eee708-78849d699f-lm2ks\" (UID: \"be3b30cb-484b-4f99-a52c-0567667b31c3\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-eee708-78849d699f-lm2ks"
Sep 16 02:36:06.113227 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:36:06.113200    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-eee708-78849d699f-lm2ks"
Sep 16 02:36:06.382467 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:36:06.382442    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-eee708-78849d699f-lm2ks"]
Sep 16 02:36:06.384122 ip-10-0-128-111 kubenswrapper[2583]: W0916 02:36:06.384094    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbe3b30cb_484b_4f99_a52c_0567667b31c3.slice/crio-c5859a4606964bda20d6f16fe9793c321716f7cf1ea7a1f87c0c33c500284f08 WatchSource:0}: Error finding container c5859a4606964bda20d6f16fe9793c321716f7cf1ea7a1f87c0c33c500284f08: Status 404 returned error can't find the container with id c5859a4606964bda20d6f16fe9793c321716f7cf1ea7a1f87c0c33c500284f08
Sep 16 02:36:07.002810 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:36:07.002777    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-eee708-78849d699f-lm2ks" event={"ID":"be3b30cb-484b-4f99-a52c-0567667b31c3","Type":"ContainerStarted","Data":"5b81837183304191ee7ebd4cd247a66c05bb565772c3a97a81e753858a390afe"}
Sep 16 02:36:07.003259 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:36:07.002836    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-eee708-78849d699f-lm2ks" event={"ID":"be3b30cb-484b-4f99-a52c-0567667b31c3","Type":"ContainerStarted","Data":"c5859a4606964bda20d6f16fe9793c321716f7cf1ea7a1f87c0c33c500284f08"}
Sep 16 02:36:07.003259 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:36:07.002879    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-eee708-78849d699f-lm2ks"
Sep 16 02:36:08.963174 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:36:08.963130    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-b099417d-5d6fb6b6c5-q2hzp"
Sep 16 02:36:08.963532 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:36:08.963214    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-b099417d-7744cc6bc8-zdfcd"
Sep 16 02:36:08.978613 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:36:08.978564    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-eee708-78849d699f-lm2ks" podStartSLOduration=3.978534503 podStartE2EDuration="3.978534503s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 02:36:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 02:36:07.020598041 +0000 UTC m=+1045.488386824" watchObservedRunningTime="2026-09-16 02:36:08.978534503 +0000 UTC m=+1047.446323284"
Sep 16 02:36:16.083099 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:36:16.083065    2583 generic.go:356] "Generic (PLEG): container finished" podID="8e4a50d5-d5fb-47fa-b1f2-4f51cdc42aaf" containerID="718e4605f8aaf01bd9ebf1e2af38af0aa7a7360322ce690269327ac53aa37779" exitCode=0
Sep 16 02:36:16.083648 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:36:16.083132    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-b099417d-openshift-default-7c4dc99d7b-plbvp" event={"ID":"8e4a50d5-d5fb-47fa-b1f2-4f51cdc42aaf","Type":"ContainerDied","Data":"718e4605f8aaf01bd9ebf1e2af38af0aa7a7360322ce690269327ac53aa37779"}
Sep 16 02:36:16.083648 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:36:16.083568    2583 scope.go:117] "RemoveContainer" containerID="718e4605f8aaf01bd9ebf1e2af38af0aa7a7360322ce690269327ac53aa37779"
Sep 16 02:36:16.085176 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:36:16.085146    2583 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 16 02:36:16.085302 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:36:16.085228    2583 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 16 02:36:16.085302 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:36:16.085275    2583 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 16 02:36:17.091732 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:36:17.091699    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-b099417d-openshift-default-7c4dc99d7b-plbvp" event={"ID":"8e4a50d5-d5fb-47fa-b1f2-4f51cdc42aaf","Type":"ContainerStarted","Data":"dc177105fc47b3941081a980fdbb6de107cc64dd6c13f43ce395fd6fa477c1df"}
Sep 16 02:36:17.754367 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:36:17.754330    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-b099417d-openshift-default-7c4dc99d7b-plbvp"
Sep 16 02:36:17.754597 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:36:17.754538    2583 patch_prober.go:28] interesting pod/e2e-ait-b099417d-openshift-default-7c4dc99d7b-plbvp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.47:15021/healthz/ready\": dial tcp 10.134.0.47:15021: connect: connection refused" start-of-body=
Sep 16 02:36:17.754676 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:36:17.754597    2583 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-b099417d-openshift-default-7c4dc99d7b-plbvp" podUID="8e4a50d5-d5fb-47fa-b1f2-4f51cdc42aaf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.47:15021/healthz/ready\": dial tcp 10.134.0.47:15021: connect: connection refused"
Sep 16 02:36:18.011361 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:36:18.011275    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-eee708-78849d699f-lm2ks"
Sep 16 02:36:18.754789 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:36:18.754750    2583 patch_prober.go:28] interesting pod/e2e-ait-b099417d-openshift-default-7c4dc99d7b-plbvp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.47:15021/healthz/ready\": dial tcp 10.134.0.47:15021: connect: connection refused" start-of-body=
Sep 16 02:36:18.755197 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:36:18.754824    2583 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-b099417d-openshift-default-7c4dc99d7b-plbvp" podUID="8e4a50d5-d5fb-47fa-b1f2-4f51cdc42aaf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.47:15021/healthz/ready\": dial tcp 10.134.0.47:15021: connect: connection refused"
Sep 16 02:36:19.754390 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:36:19.754352    2583 patch_prober.go:28] interesting pod/e2e-ait-b099417d-openshift-default-7c4dc99d7b-plbvp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.47:15021/healthz/ready\": dial tcp 10.134.0.47:15021: connect: connection refused" start-of-body=
Sep 16 02:36:19.754596 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:36:19.754415    2583 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-b099417d-openshift-default-7c4dc99d7b-plbvp" podUID="8e4a50d5-d5fb-47fa-b1f2-4f51cdc42aaf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.47:15021/healthz/ready\": dial tcp 10.134.0.47:15021: connect: connection refused"
Sep 16 02:36:20.754226 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:36:20.754189    2583 patch_prober.go:28] interesting pod/e2e-ait-b099417d-openshift-default-7c4dc99d7b-plbvp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.47:15021/healthz/ready\": dial tcp 10.134.0.47:15021: connect: connection refused" start-of-body=
Sep 16 02:36:20.754651 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:36:20.754255    2583 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-b099417d-openshift-default-7c4dc99d7b-plbvp" podUID="8e4a50d5-d5fb-47fa-b1f2-4f51cdc42aaf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.47:15021/healthz/ready\": dial tcp 10.134.0.47:15021: connect: connection refused"
Sep 16 02:36:21.754779 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:36:21.754739    2583 patch_prober.go:28] interesting pod/e2e-ait-b099417d-openshift-default-7c4dc99d7b-plbvp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.47:15021/healthz/ready\": dial tcp 10.134.0.47:15021: connect: connection refused" start-of-body=
Sep 16 02:36:21.755270 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:36:21.754813    2583 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-b099417d-openshift-default-7c4dc99d7b-plbvp" podUID="8e4a50d5-d5fb-47fa-b1f2-4f51cdc42aaf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.47:15021/healthz/ready\": dial tcp 10.134.0.47:15021: connect: connection refused"
Sep 16 02:36:22.256038 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:36:22.255992    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-fzl8t"]
Sep 16 02:36:22.758499 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:36:22.758467    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-b099417d-openshift-default-7c4dc99d7b-plbvp"
Sep 16 02:36:22.758972 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:36:22.758788    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-b099417d-openshift-default-7c4dc99d7b-plbvp"
Sep 16 02:36:22.759616 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:36:22.759596    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-b099417d-openshift-default-7c4dc99d7b-plbvp"
Sep 16 02:36:26.251214 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:36:26.251172    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-b099417d-5d6fb6b6c5-q2hzp"]
Sep 16 02:36:26.251744 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:36:26.251455    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-b099417d-5d6fb6b6c5-q2hzp" podUID="d8c85be6-e00e-4437-8d5c-ee02d6ccc25c" containerName="payload-processing" containerID="cri-o://75b4f25d99986e0021e8b18d6ca63ba53287ac01019c20a245aeb792f0f2079f" gracePeriod=30
Sep 16 02:36:26.253698 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:36:26.253673    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-b099417d-7744cc6bc8-zdfcd"]
Sep 16 02:36:26.253893 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:36:26.253871    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-b099417d-7744cc6bc8-zdfcd" podUID="97d943e2-e2b0-44b9-933a-b1223c1f0774" containerName="payload-pre-processing" containerID="cri-o://ac93400f2f4d6aae5702f7503b502da16289595aa9788d3b039a9008ed1a0263" gracePeriod=30
Sep 16 02:36:26.535243 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:36:26.535217    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-b099417d-5d6fb6b6c5-q2hzp"
Sep 16 02:36:26.537301 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:36:26.537284    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-b099417d-7744cc6bc8-zdfcd"
Sep 16 02:36:26.668593 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:36:26.668532    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/97d943e2-e2b0-44b9-933a-b1223c1f0774-plugins-config-volume\") pod \"97d943e2-e2b0-44b9-933a-b1223c1f0774\" (UID: \"97d943e2-e2b0-44b9-933a-b1223c1f0774\") "
Sep 16 02:36:26.668800 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:36:26.668724    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d8c85be6-e00e-4437-8d5c-ee02d6ccc25c-plugins-config-volume\") pod \"d8c85be6-e00e-4437-8d5c-ee02d6ccc25c\" (UID: \"d8c85be6-e00e-4437-8d5c-ee02d6ccc25c\") "
Sep 16 02:36:26.668800 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:36:26.668761    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9njbp\" (UniqueName: \"kubernetes.io/projected/d8c85be6-e00e-4437-8d5c-ee02d6ccc25c-kube-api-access-9njbp\") pod \"d8c85be6-e00e-4437-8d5c-ee02d6ccc25c\" (UID: \"d8c85be6-e00e-4437-8d5c-ee02d6ccc25c\") "
Sep 16 02:36:26.668929 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:36:26.668819    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jlbf7\" (UniqueName: \"kubernetes.io/projected/97d943e2-e2b0-44b9-933a-b1223c1f0774-kube-api-access-jlbf7\") pod \"97d943e2-e2b0-44b9-933a-b1223c1f0774\" (UID: \"97d943e2-e2b0-44b9-933a-b1223c1f0774\") "
Sep 16 02:36:26.668929 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:36:26.668878    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/97d943e2-e2b0-44b9-933a-b1223c1f0774-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "97d943e2-e2b0-44b9-933a-b1223c1f0774" (UID: "97d943e2-e2b0-44b9-933a-b1223c1f0774"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 02:36:26.669128 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:36:26.669100    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d8c85be6-e00e-4437-8d5c-ee02d6ccc25c-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "d8c85be6-e00e-4437-8d5c-ee02d6ccc25c" (UID: "d8c85be6-e00e-4437-8d5c-ee02d6ccc25c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 02:36:26.669190 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:36:26.669175    2583 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/97d943e2-e2b0-44b9-933a-b1223c1f0774-plugins-config-volume\") on node \"ip-10-0-128-111.ec2.internal\" DevicePath \"\""
Sep 16 02:36:26.670919 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:36:26.670898    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/97d943e2-e2b0-44b9-933a-b1223c1f0774-kube-api-access-jlbf7" (OuterVolumeSpecName: "kube-api-access-jlbf7") pod "97d943e2-e2b0-44b9-933a-b1223c1f0774" (UID: "97d943e2-e2b0-44b9-933a-b1223c1f0774"). InnerVolumeSpecName "kube-api-access-jlbf7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 02:36:26.671028 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:36:26.671008    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d8c85be6-e00e-4437-8d5c-ee02d6ccc25c-kube-api-access-9njbp" (OuterVolumeSpecName: "kube-api-access-9njbp") pod "d8c85be6-e00e-4437-8d5c-ee02d6ccc25c" (UID: "d8c85be6-e00e-4437-8d5c-ee02d6ccc25c"). InnerVolumeSpecName "kube-api-access-9njbp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 02:36:26.770338 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:36:26.770258    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jlbf7\" (UniqueName: \"kubernetes.io/projected/97d943e2-e2b0-44b9-933a-b1223c1f0774-kube-api-access-jlbf7\") on node \"ip-10-0-128-111.ec2.internal\" DevicePath \"\""
Sep 16 02:36:26.770338 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:36:26.770287    2583 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d8c85be6-e00e-4437-8d5c-ee02d6ccc25c-plugins-config-volume\") on node \"ip-10-0-128-111.ec2.internal\" DevicePath \"\""
Sep 16 02:36:26.770338 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:36:26.770297    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9njbp\" (UniqueName: \"kubernetes.io/projected/d8c85be6-e00e-4437-8d5c-ee02d6ccc25c-kube-api-access-9njbp\") on node \"ip-10-0-128-111.ec2.internal\" DevicePath \"\""
Sep 16 02:36:27.145056 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:36:27.145021    2583 generic.go:356] "Generic (PLEG): container finished" podID="d8c85be6-e00e-4437-8d5c-ee02d6ccc25c" containerID="75b4f25d99986e0021e8b18d6ca63ba53287ac01019c20a245aeb792f0f2079f" exitCode=0
Sep 16 02:36:27.145247 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:36:27.145061    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-b099417d-5d6fb6b6c5-q2hzp"
Sep 16 02:36:27.145247 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:36:27.145110    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-b099417d-5d6fb6b6c5-q2hzp" event={"ID":"d8c85be6-e00e-4437-8d5c-ee02d6ccc25c","Type":"ContainerDied","Data":"75b4f25d99986e0021e8b18d6ca63ba53287ac01019c20a245aeb792f0f2079f"}
Sep 16 02:36:27.145247 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:36:27.145159    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-b099417d-5d6fb6b6c5-q2hzp" event={"ID":"d8c85be6-e00e-4437-8d5c-ee02d6ccc25c","Type":"ContainerDied","Data":"4b529e88f0f56bdaa6c73d9e4803e093cb8f1ae0edce5693163aa89ce2c418d4"}
Sep 16 02:36:27.145247 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:36:27.145183    2583 scope.go:117] "RemoveContainer" containerID="75b4f25d99986e0021e8b18d6ca63ba53287ac01019c20a245aeb792f0f2079f"
Sep 16 02:36:27.146444 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:36:27.146410    2583 generic.go:356] "Generic (PLEG): container finished" podID="97d943e2-e2b0-44b9-933a-b1223c1f0774" containerID="ac93400f2f4d6aae5702f7503b502da16289595aa9788d3b039a9008ed1a0263" exitCode=0
Sep 16 02:36:27.146581 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:36:27.146452    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-b099417d-7744cc6bc8-zdfcd"
Sep 16 02:36:27.146581 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:36:27.146501    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-b099417d-7744cc6bc8-zdfcd" event={"ID":"97d943e2-e2b0-44b9-933a-b1223c1f0774","Type":"ContainerDied","Data":"ac93400f2f4d6aae5702f7503b502da16289595aa9788d3b039a9008ed1a0263"}
Sep 16 02:36:27.146581 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:36:27.146571    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-b099417d-7744cc6bc8-zdfcd" event={"ID":"97d943e2-e2b0-44b9-933a-b1223c1f0774","Type":"ContainerDied","Data":"d80d17885d55e426457816b4c359cc2a9c02257e2a0aa1b94fe75e81efda4509"}
Sep 16 02:36:27.155402 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:36:27.155391    2583 scope.go:117] "RemoveContainer" containerID="75b4f25d99986e0021e8b18d6ca63ba53287ac01019c20a245aeb792f0f2079f"
Sep 16 02:36:27.155763 ip-10-0-128-111 kubenswrapper[2583]: E0916 02:36:27.155736    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"75b4f25d99986e0021e8b18d6ca63ba53287ac01019c20a245aeb792f0f2079f\": container with ID starting with 75b4f25d99986e0021e8b18d6ca63ba53287ac01019c20a245aeb792f0f2079f not found: ID does not exist" containerID="75b4f25d99986e0021e8b18d6ca63ba53287ac01019c20a245aeb792f0f2079f"
Sep 16 02:36:27.155870 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:36:27.155775    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"75b4f25d99986e0021e8b18d6ca63ba53287ac01019c20a245aeb792f0f2079f"} err="failed to get container status \"75b4f25d99986e0021e8b18d6ca63ba53287ac01019c20a245aeb792f0f2079f\": rpc error: code = NotFound desc = could not find container \"75b4f25d99986e0021e8b18d6ca63ba53287ac01019c20a245aeb792f0f2079f\": container with ID starting with 75b4f25d99986e0021e8b18d6ca63ba53287ac01019c20a245aeb792f0f2079f not found: ID does not exist"
Sep 16 02:36:27.155870 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:36:27.155798    2583 scope.go:117] "RemoveContainer" containerID="ac93400f2f4d6aae5702f7503b502da16289595aa9788d3b039a9008ed1a0263"
Sep 16 02:36:27.163863 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:36:27.163846    2583 scope.go:117] "RemoveContainer" containerID="ac93400f2f4d6aae5702f7503b502da16289595aa9788d3b039a9008ed1a0263"
Sep 16 02:36:27.164123 ip-10-0-128-111 kubenswrapper[2583]: E0916 02:36:27.164107    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ac93400f2f4d6aae5702f7503b502da16289595aa9788d3b039a9008ed1a0263\": container with ID starting with ac93400f2f4d6aae5702f7503b502da16289595aa9788d3b039a9008ed1a0263 not found: ID does not exist" containerID="ac93400f2f4d6aae5702f7503b502da16289595aa9788d3b039a9008ed1a0263"
Sep 16 02:36:27.164185 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:36:27.164134    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ac93400f2f4d6aae5702f7503b502da16289595aa9788d3b039a9008ed1a0263"} err="failed to get container status \"ac93400f2f4d6aae5702f7503b502da16289595aa9788d3b039a9008ed1a0263\": rpc error: code = NotFound desc = could not find container \"ac93400f2f4d6aae5702f7503b502da16289595aa9788d3b039a9008ed1a0263\": container with ID starting with ac93400f2f4d6aae5702f7503b502da16289595aa9788d3b039a9008ed1a0263 not found: ID does not exist"
Sep 16 02:36:27.171130 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:36:27.171108    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-b099417d-7744cc6bc8-zdfcd"]
Sep 16 02:36:27.172932 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:36:27.172915    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-b099417d-7744cc6bc8-zdfcd"]
Sep 16 02:36:27.182429 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:36:27.182409    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-b099417d-5d6fb6b6c5-q2hzp"]
Sep 16 02:36:27.186749 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:36:27.186731    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-b099417d-5d6fb6b6c5-q2hzp"]
Sep 16 02:36:27.547687 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:36:27.547603    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-b099417d-openshift-default-7c4dc99d7b-plbvp"]
Sep 16 02:36:27.548163 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:36:27.547818    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-b099417d-openshift-default-7c4dc99d7b-plbvp" podUID="8e4a50d5-d5fb-47fa-b1f2-4f51cdc42aaf" containerName="istio-proxy" containerID="cri-o://dc177105fc47b3941081a980fdbb6de107cc64dd6c13f43ce395fd6fa477c1df" gracePeriod=30
Sep 16 02:36:28.046159 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:36:28.046125    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="97d943e2-e2b0-44b9-933a-b1223c1f0774" path="/var/lib/kubelet/pods/97d943e2-e2b0-44b9-933a-b1223c1f0774/volumes"
Sep 16 02:36:28.046514 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:36:28.046500    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d8c85be6-e00e-4437-8d5c-ee02d6ccc25c" path="/var/lib/kubelet/pods/d8c85be6-e00e-4437-8d5c-ee02d6ccc25c/volumes"
Sep 16 02:36:32.813853 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:36:32.813789    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-b099417d-openshift-default-7c4dc99d7b-plbvp"
Sep 16 02:36:32.926929 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:36:32.926876    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8e4a50d5-d5fb-47fa-b1f2-4f51cdc42aaf-istio-token\") pod \"8e4a50d5-d5fb-47fa-b1f2-4f51cdc42aaf\" (UID: \"8e4a50d5-d5fb-47fa-b1f2-4f51cdc42aaf\") "
Sep 16 02:36:32.927121 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:36:32.926943    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8e4a50d5-d5fb-47fa-b1f2-4f51cdc42aaf-istio-envoy\") pod \"8e4a50d5-d5fb-47fa-b1f2-4f51cdc42aaf\" (UID: \"8e4a50d5-d5fb-47fa-b1f2-4f51cdc42aaf\") "
Sep 16 02:36:32.927121 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:36:32.927067    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rtts2\" (UniqueName: \"kubernetes.io/projected/8e4a50d5-d5fb-47fa-b1f2-4f51cdc42aaf-kube-api-access-rtts2\") pod \"8e4a50d5-d5fb-47fa-b1f2-4f51cdc42aaf\" (UID: \"8e4a50d5-d5fb-47fa-b1f2-4f51cdc42aaf\") "
Sep 16 02:36:32.927121 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:36:32.927118    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8e4a50d5-d5fb-47fa-b1f2-4f51cdc42aaf-istio-data\") pod \"8e4a50d5-d5fb-47fa-b1f2-4f51cdc42aaf\" (UID: \"8e4a50d5-d5fb-47fa-b1f2-4f51cdc42aaf\") "
Sep 16 02:36:32.927299 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:36:32.927140    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8e4a50d5-d5fb-47fa-b1f2-4f51cdc42aaf-workload-certs\") pod \"8e4a50d5-d5fb-47fa-b1f2-4f51cdc42aaf\" (UID: \"8e4a50d5-d5fb-47fa-b1f2-4f51cdc42aaf\") "
Sep 16 02:36:32.927299 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:36:32.927161    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e4a50d5-d5fb-47fa-b1f2-4f51cdc42aaf-credential-socket\") pod \"8e4a50d5-d5fb-47fa-b1f2-4f51cdc42aaf\" (UID: \"8e4a50d5-d5fb-47fa-b1f2-4f51cdc42aaf\") "
Sep 16 02:36:32.927299 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:36:32.927190    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e4a50d5-d5fb-47fa-b1f2-4f51cdc42aaf-workload-socket\") pod \"8e4a50d5-d5fb-47fa-b1f2-4f51cdc42aaf\" (UID: \"8e4a50d5-d5fb-47fa-b1f2-4f51cdc42aaf\") "
Sep 16 02:36:32.927299 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:36:32.927242    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8e4a50d5-d5fb-47fa-b1f2-4f51cdc42aaf-istiod-ca-cert\") pod \"8e4a50d5-d5fb-47fa-b1f2-4f51cdc42aaf\" (UID: \"8e4a50d5-d5fb-47fa-b1f2-4f51cdc42aaf\") "
Sep 16 02:36:32.927299 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:36:32.927274    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8e4a50d5-d5fb-47fa-b1f2-4f51cdc42aaf-istio-podinfo\") pod \"8e4a50d5-d5fb-47fa-b1f2-4f51cdc42aaf\" (UID: \"8e4a50d5-d5fb-47fa-b1f2-4f51cdc42aaf\") "
Sep 16 02:36:32.927584 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:36:32.927405    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8e4a50d5-d5fb-47fa-b1f2-4f51cdc42aaf-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "8e4a50d5-d5fb-47fa-b1f2-4f51cdc42aaf" (UID: "8e4a50d5-d5fb-47fa-b1f2-4f51cdc42aaf"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 02:36:32.927584 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:36:32.927399    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8e4a50d5-d5fb-47fa-b1f2-4f51cdc42aaf-istio-data" (OuterVolumeSpecName: "istio-data") pod "8e4a50d5-d5fb-47fa-b1f2-4f51cdc42aaf" (UID: "8e4a50d5-d5fb-47fa-b1f2-4f51cdc42aaf"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 02:36:32.927584 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:36:32.927432    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8e4a50d5-d5fb-47fa-b1f2-4f51cdc42aaf-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "8e4a50d5-d5fb-47fa-b1f2-4f51cdc42aaf" (UID: "8e4a50d5-d5fb-47fa-b1f2-4f51cdc42aaf"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 02:36:32.927761 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:36:32.927609    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8e4a50d5-d5fb-47fa-b1f2-4f51cdc42aaf-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "8e4a50d5-d5fb-47fa-b1f2-4f51cdc42aaf" (UID: "8e4a50d5-d5fb-47fa-b1f2-4f51cdc42aaf"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 02:36:32.927761 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:36:32.927709    2583 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8e4a50d5-d5fb-47fa-b1f2-4f51cdc42aaf-istio-data\") on node \"ip-10-0-128-111.ec2.internal\" DevicePath \"\""
Sep 16 02:36:32.927761 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:36:32.927728    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8e4a50d5-d5fb-47fa-b1f2-4f51cdc42aaf-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "8e4a50d5-d5fb-47fa-b1f2-4f51cdc42aaf" (UID: "8e4a50d5-d5fb-47fa-b1f2-4f51cdc42aaf"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 02:36:32.927761 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:36:32.927733    2583 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8e4a50d5-d5fb-47fa-b1f2-4f51cdc42aaf-workload-certs\") on node \"ip-10-0-128-111.ec2.internal\" DevicePath \"\""
Sep 16 02:36:32.927761 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:36:32.927748    2583 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e4a50d5-d5fb-47fa-b1f2-4f51cdc42aaf-credential-socket\") on node \"ip-10-0-128-111.ec2.internal\" DevicePath \"\""
Sep 16 02:36:32.928022 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:36:32.927763    2583 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e4a50d5-d5fb-47fa-b1f2-4f51cdc42aaf-workload-socket\") on node \"ip-10-0-128-111.ec2.internal\" DevicePath \"\""
Sep 16 02:36:32.929285 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:36:32.929255    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8e4a50d5-d5fb-47fa-b1f2-4f51cdc42aaf-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "8e4a50d5-d5fb-47fa-b1f2-4f51cdc42aaf" (UID: "8e4a50d5-d5fb-47fa-b1f2-4f51cdc42aaf"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 02:36:32.929285 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:36:32.929268    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8e4a50d5-d5fb-47fa-b1f2-4f51cdc42aaf-istio-token" (OuterVolumeSpecName: "istio-token") pod "8e4a50d5-d5fb-47fa-b1f2-4f51cdc42aaf" (UID: "8e4a50d5-d5fb-47fa-b1f2-4f51cdc42aaf"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 02:36:32.929434 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:36:32.929259    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8e4a50d5-d5fb-47fa-b1f2-4f51cdc42aaf-kube-api-access-rtts2" (OuterVolumeSpecName: "kube-api-access-rtts2") pod "8e4a50d5-d5fb-47fa-b1f2-4f51cdc42aaf" (UID: "8e4a50d5-d5fb-47fa-b1f2-4f51cdc42aaf"). InnerVolumeSpecName "kube-api-access-rtts2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 02:36:32.929434 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:36:32.929311    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/8e4a50d5-d5fb-47fa-b1f2-4f51cdc42aaf-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "8e4a50d5-d5fb-47fa-b1f2-4f51cdc42aaf" (UID: "8e4a50d5-d5fb-47fa-b1f2-4f51cdc42aaf"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 16 02:36:33.028845 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:36:33.028815    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rtts2\" (UniqueName: \"kubernetes.io/projected/8e4a50d5-d5fb-47fa-b1f2-4f51cdc42aaf-kube-api-access-rtts2\") on node \"ip-10-0-128-111.ec2.internal\" DevicePath \"\""
Sep 16 02:36:33.028845 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:36:33.028839    2583 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8e4a50d5-d5fb-47fa-b1f2-4f51cdc42aaf-istiod-ca-cert\") on node \"ip-10-0-128-111.ec2.internal\" DevicePath \"\""
Sep 16 02:36:33.028845 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:36:33.028850    2583 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8e4a50d5-d5fb-47fa-b1f2-4f51cdc42aaf-istio-podinfo\") on node \"ip-10-0-128-111.ec2.internal\" DevicePath \"\""
Sep 16 02:36:33.029055 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:36:33.028857    2583 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8e4a50d5-d5fb-47fa-b1f2-4f51cdc42aaf-istio-token\") on node \"ip-10-0-128-111.ec2.internal\" DevicePath \"\""
Sep 16 02:36:33.029055 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:36:33.028865    2583 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8e4a50d5-d5fb-47fa-b1f2-4f51cdc42aaf-istio-envoy\") on node \"ip-10-0-128-111.ec2.internal\" DevicePath \"\""
Sep 16 02:36:33.181261 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:36:33.181229    2583 generic.go:356] "Generic (PLEG): container finished" podID="8e4a50d5-d5fb-47fa-b1f2-4f51cdc42aaf" containerID="dc177105fc47b3941081a980fdbb6de107cc64dd6c13f43ce395fd6fa477c1df" exitCode=0
Sep 16 02:36:33.181437 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:36:33.181280    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-b099417d-openshift-default-7c4dc99d7b-plbvp"
Sep 16 02:36:33.181437 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:36:33.181305    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-b099417d-openshift-default-7c4dc99d7b-plbvp" event={"ID":"8e4a50d5-d5fb-47fa-b1f2-4f51cdc42aaf","Type":"ContainerDied","Data":"dc177105fc47b3941081a980fdbb6de107cc64dd6c13f43ce395fd6fa477c1df"}
Sep 16 02:36:33.181437 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:36:33.181356    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-b099417d-openshift-default-7c4dc99d7b-plbvp" event={"ID":"8e4a50d5-d5fb-47fa-b1f2-4f51cdc42aaf","Type":"ContainerDied","Data":"719d2614f5573cbd99d8c4532fa2a1e1d4329e8837dd1c00e89588b354011245"}
Sep 16 02:36:33.181437 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:36:33.181378    2583 scope.go:117] "RemoveContainer" containerID="dc177105fc47b3941081a980fdbb6de107cc64dd6c13f43ce395fd6fa477c1df"
Sep 16 02:36:33.190928 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:36:33.190911    2583 scope.go:117] "RemoveContainer" containerID="718e4605f8aaf01bd9ebf1e2af38af0aa7a7360322ce690269327ac53aa37779"
Sep 16 02:36:33.198735 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:36:33.198717    2583 scope.go:117] "RemoveContainer" containerID="dc177105fc47b3941081a980fdbb6de107cc64dd6c13f43ce395fd6fa477c1df"
Sep 16 02:36:33.198996 ip-10-0-128-111 kubenswrapper[2583]: E0916 02:36:33.198970    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dc177105fc47b3941081a980fdbb6de107cc64dd6c13f43ce395fd6fa477c1df\": container with ID starting with dc177105fc47b3941081a980fdbb6de107cc64dd6c13f43ce395fd6fa477c1df not found: ID does not exist" containerID="dc177105fc47b3941081a980fdbb6de107cc64dd6c13f43ce395fd6fa477c1df"
Sep 16 02:36:33.199062 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:36:33.199004    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dc177105fc47b3941081a980fdbb6de107cc64dd6c13f43ce395fd6fa477c1df"} err="failed to get container status \"dc177105fc47b3941081a980fdbb6de107cc64dd6c13f43ce395fd6fa477c1df\": rpc error: code = NotFound desc = could not find container \"dc177105fc47b3941081a980fdbb6de107cc64dd6c13f43ce395fd6fa477c1df\": container with ID starting with dc177105fc47b3941081a980fdbb6de107cc64dd6c13f43ce395fd6fa477c1df not found: ID does not exist"
Sep 16 02:36:33.199062 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:36:33.199020    2583 scope.go:117] "RemoveContainer" containerID="718e4605f8aaf01bd9ebf1e2af38af0aa7a7360322ce690269327ac53aa37779"
Sep 16 02:36:33.199243 ip-10-0-128-111 kubenswrapper[2583]: E0916 02:36:33.199229    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"718e4605f8aaf01bd9ebf1e2af38af0aa7a7360322ce690269327ac53aa37779\": container with ID starting with 718e4605f8aaf01bd9ebf1e2af38af0aa7a7360322ce690269327ac53aa37779 not found: ID does not exist" containerID="718e4605f8aaf01bd9ebf1e2af38af0aa7a7360322ce690269327ac53aa37779"
Sep 16 02:36:33.199284 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:36:33.199246    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"718e4605f8aaf01bd9ebf1e2af38af0aa7a7360322ce690269327ac53aa37779"} err="failed to get container status \"718e4605f8aaf01bd9ebf1e2af38af0aa7a7360322ce690269327ac53aa37779\": rpc error: code = NotFound desc = could not find container \"718e4605f8aaf01bd9ebf1e2af38af0aa7a7360322ce690269327ac53aa37779\": container with ID starting with 718e4605f8aaf01bd9ebf1e2af38af0aa7a7360322ce690269327ac53aa37779 not found: ID does not exist"
Sep 16 02:36:33.203827 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:36:33.203803    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-b099417d-openshift-default-7c4dc99d7b-plbvp"]
Sep 16 02:36:33.207900 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:36:33.207878    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-b099417d-openshift-default-7c4dc99d7b-plbvp"]
Sep 16 02:36:34.045717 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:36:34.045640    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8e4a50d5-d5fb-47fa-b1f2-4f51cdc42aaf" path="/var/lib/kubelet/pods/8e4a50d5-d5fb-47fa-b1f2-4f51cdc42aaf/volumes"
Sep 16 02:36:34.378438 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:36:34.378406    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-642a8155-67d6745478-9lllv"]
Sep 16 02:36:34.378793 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:36:34.378780    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8e4a50d5-d5fb-47fa-b1f2-4f51cdc42aaf" containerName="istio-proxy"
Sep 16 02:36:34.378850 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:36:34.378794    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="8e4a50d5-d5fb-47fa-b1f2-4f51cdc42aaf" containerName="istio-proxy"
Sep 16 02:36:34.378850 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:36:34.378803    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d8c85be6-e00e-4437-8d5c-ee02d6ccc25c" containerName="payload-processing"
Sep 16 02:36:34.378850 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:36:34.378810    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="d8c85be6-e00e-4437-8d5c-ee02d6ccc25c" containerName="payload-processing"
Sep 16 02:36:34.378850 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:36:34.378827    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="97d943e2-e2b0-44b9-933a-b1223c1f0774" containerName="payload-pre-processing"
Sep 16 02:36:34.378850 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:36:34.378832    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="97d943e2-e2b0-44b9-933a-b1223c1f0774" containerName="payload-pre-processing"
Sep 16 02:36:34.379013 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:36:34.378906    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="97d943e2-e2b0-44b9-933a-b1223c1f0774" containerName="payload-pre-processing"
Sep 16 02:36:34.379013 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:36:34.378916    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="8e4a50d5-d5fb-47fa-b1f2-4f51cdc42aaf" containerName="istio-proxy"
Sep 16 02:36:34.379013 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:36:34.378927    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="8e4a50d5-d5fb-47fa-b1f2-4f51cdc42aaf" containerName="istio-proxy"
Sep 16 02:36:34.379013 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:36:34.378937    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="d8c85be6-e00e-4437-8d5c-ee02d6ccc25c" containerName="payload-processing"
Sep 16 02:36:34.383538 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:36:34.383522    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-642a8155-67d6745478-9lllv"
Sep 16 02:36:34.386086 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:36:34.386062    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-derive-642a8155\""
Sep 16 02:36:34.386176 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:36:34.386155    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-derive-642a8155-dockercfg-r2pnf\""
Sep 16 02:36:34.391605 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:36:34.391581    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-642a8155-67d6745478-9lllv"]
Sep 16 02:36:34.414377 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:36:34.414343    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-642a8155-756f57f47c-gwjv8"]
Sep 16 02:36:34.414737 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:36:34.414724    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8e4a50d5-d5fb-47fa-b1f2-4f51cdc42aaf" containerName="istio-proxy"
Sep 16 02:36:34.414786 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:36:34.414739    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="8e4a50d5-d5fb-47fa-b1f2-4f51cdc42aaf" containerName="istio-proxy"
Sep 16 02:36:34.417739 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:36:34.417726    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-642a8155-756f57f47c-gwjv8"
Sep 16 02:36:34.430567 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:36:34.427796    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-642a8155-756f57f47c-gwjv8"]
Sep 16 02:36:34.546158 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:36:34.546122    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9929c\" (UniqueName: \"kubernetes.io/projected/a006042d-eddd-4564-923e-04e0f548d58c-kube-api-access-9929c\") pod \"payload-pre-processing-e2e-derive-642a8155-67d6745478-9lllv\" (UID: \"a006042d-eddd-4564-923e-04e0f548d58c\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-642a8155-67d6745478-9lllv"
Sep 16 02:36:34.546158 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:36:34.546162    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mdg6c\" (UniqueName: \"kubernetes.io/projected/02336b88-6308-4c8c-aa5f-e3e77e950b33-kube-api-access-mdg6c\") pod \"payload-processing-e2e-derive-642a8155-756f57f47c-gwjv8\" (UID: \"02336b88-6308-4c8c-aa5f-e3e77e950b33\") " pod="openshift-ingress/payload-processing-e2e-derive-642a8155-756f57f47c-gwjv8"
Sep 16 02:36:34.546371 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:36:34.546200    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/02336b88-6308-4c8c-aa5f-e3e77e950b33-plugins-config-volume\") pod \"payload-processing-e2e-derive-642a8155-756f57f47c-gwjv8\" (UID: \"02336b88-6308-4c8c-aa5f-e3e77e950b33\") " pod="openshift-ingress/payload-processing-e2e-derive-642a8155-756f57f47c-gwjv8"
Sep 16 02:36:34.546371 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:36:34.546274    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a006042d-eddd-4564-923e-04e0f548d58c-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-642a8155-67d6745478-9lllv\" (UID: \"a006042d-eddd-4564-923e-04e0f548d58c\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-642a8155-67d6745478-9lllv"
Sep 16 02:36:34.647066 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:36:34.646961    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a006042d-eddd-4564-923e-04e0f548d58c-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-642a8155-67d6745478-9lllv\" (UID: \"a006042d-eddd-4564-923e-04e0f548d58c\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-642a8155-67d6745478-9lllv"
Sep 16 02:36:34.647066 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:36:34.647067    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9929c\" (UniqueName: \"kubernetes.io/projected/a006042d-eddd-4564-923e-04e0f548d58c-kube-api-access-9929c\") pod \"payload-pre-processing-e2e-derive-642a8155-67d6745478-9lllv\" (UID: \"a006042d-eddd-4564-923e-04e0f548d58c\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-642a8155-67d6745478-9lllv"
Sep 16 02:36:34.647303 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:36:34.647096    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mdg6c\" (UniqueName: \"kubernetes.io/projected/02336b88-6308-4c8c-aa5f-e3e77e950b33-kube-api-access-mdg6c\") pod \"payload-processing-e2e-derive-642a8155-756f57f47c-gwjv8\" (UID: \"02336b88-6308-4c8c-aa5f-e3e77e950b33\") " pod="openshift-ingress/payload-processing-e2e-derive-642a8155-756f57f47c-gwjv8"
Sep 16 02:36:34.647303 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:36:34.647128    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/02336b88-6308-4c8c-aa5f-e3e77e950b33-plugins-config-volume\") pod \"payload-processing-e2e-derive-642a8155-756f57f47c-gwjv8\" (UID: \"02336b88-6308-4c8c-aa5f-e3e77e950b33\") " pod="openshift-ingress/payload-processing-e2e-derive-642a8155-756f57f47c-gwjv8"
Sep 16 02:36:34.647818 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:36:34.647793    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a006042d-eddd-4564-923e-04e0f548d58c-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-642a8155-67d6745478-9lllv\" (UID: \"a006042d-eddd-4564-923e-04e0f548d58c\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-642a8155-67d6745478-9lllv"
Sep 16 02:36:34.647922 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:36:34.647865    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/02336b88-6308-4c8c-aa5f-e3e77e950b33-plugins-config-volume\") pod \"payload-processing-e2e-derive-642a8155-756f57f47c-gwjv8\" (UID: \"02336b88-6308-4c8c-aa5f-e3e77e950b33\") " pod="openshift-ingress/payload-processing-e2e-derive-642a8155-756f57f47c-gwjv8"
Sep 16 02:36:34.654646 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:36:34.654625    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9929c\" (UniqueName: \"kubernetes.io/projected/a006042d-eddd-4564-923e-04e0f548d58c-kube-api-access-9929c\") pod \"payload-pre-processing-e2e-derive-642a8155-67d6745478-9lllv\" (UID: \"a006042d-eddd-4564-923e-04e0f548d58c\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-642a8155-67d6745478-9lllv"
Sep 16 02:36:34.654734 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:36:34.654681    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mdg6c\" (UniqueName: \"kubernetes.io/projected/02336b88-6308-4c8c-aa5f-e3e77e950b33-kube-api-access-mdg6c\") pod \"payload-processing-e2e-derive-642a8155-756f57f47c-gwjv8\" (UID: \"02336b88-6308-4c8c-aa5f-e3e77e950b33\") " pod="openshift-ingress/payload-processing-e2e-derive-642a8155-756f57f47c-gwjv8"
Sep 16 02:36:34.694646 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:36:34.694609    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-642a8155-67d6745478-9lllv"
Sep 16 02:36:34.728083 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:36:34.728049    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-642a8155-756f57f47c-gwjv8"
Sep 16 02:36:34.976414 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:36:34.976389    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-642a8155-67d6745478-9lllv"]
Sep 16 02:36:34.978607 ip-10-0-128-111 kubenswrapper[2583]: W0916 02:36:34.978585    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda006042d_eddd_4564_923e_04e0f548d58c.slice/crio-d97d14fe15b3cea170724a21b4f497f0c7314a0f2fdec450282484ca92e2c690 WatchSource:0}: Error finding container d97d14fe15b3cea170724a21b4f497f0c7314a0f2fdec450282484ca92e2c690: Status 404 returned error can't find the container with id d97d14fe15b3cea170724a21b4f497f0c7314a0f2fdec450282484ca92e2c690
Sep 16 02:36:35.006308 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:36:35.006277    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-642a8155-756f57f47c-gwjv8"]
Sep 16 02:36:35.013683 ip-10-0-128-111 kubenswrapper[2583]: W0916 02:36:35.013653    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod02336b88_6308_4c8c_aa5f_e3e77e950b33.slice/crio-8a7a60fc8e03cf74bb8736d0911f10058d2ecd60a15dba6d65031f2a05926422 WatchSource:0}: Error finding container 8a7a60fc8e03cf74bb8736d0911f10058d2ecd60a15dba6d65031f2a05926422: Status 404 returned error can't find the container with id 8a7a60fc8e03cf74bb8736d0911f10058d2ecd60a15dba6d65031f2a05926422
Sep 16 02:36:35.196322 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:36:35.196269    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-642a8155-67d6745478-9lllv" event={"ID":"a006042d-eddd-4564-923e-04e0f548d58c","Type":"ContainerStarted","Data":"17478b9e67021c618bdb78652a9441454bafc11a91f8ef1c2cc0068b00eb1ce7"}
Sep 16 02:36:35.196786 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:36:35.196449    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-derive-642a8155-67d6745478-9lllv"
Sep 16 02:36:35.196786 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:36:35.196485    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-642a8155-67d6745478-9lllv" event={"ID":"a006042d-eddd-4564-923e-04e0f548d58c","Type":"ContainerStarted","Data":"d97d14fe15b3cea170724a21b4f497f0c7314a0f2fdec450282484ca92e2c690"}
Sep 16 02:36:35.197878 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:36:35.197857    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-642a8155-756f57f47c-gwjv8" event={"ID":"02336b88-6308-4c8c-aa5f-e3e77e950b33","Type":"ContainerStarted","Data":"33fa1376f0f074b966acee471bd0ec27560f281c011a142ff4e5d7d3413f4e35"}
Sep 16 02:36:35.198010 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:36:35.197896    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-642a8155-756f57f47c-gwjv8" event={"ID":"02336b88-6308-4c8c-aa5f-e3e77e950b33","Type":"ContainerStarted","Data":"8a7a60fc8e03cf74bb8736d0911f10058d2ecd60a15dba6d65031f2a05926422"}
Sep 16 02:36:35.198010 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:36:35.197937    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-derive-642a8155-756f57f47c-gwjv8"
Sep 16 02:36:35.210394 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:36:35.210351    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-derive-642a8155-67d6745478-9lllv" podStartSLOduration=1.210333225 podStartE2EDuration="1.210333225s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 02:36:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 02:36:35.210007048 +0000 UTC m=+1073.677795831" watchObservedRunningTime="2026-09-16 02:36:35.210333225 +0000 UTC m=+1073.678122012"
Sep 16 02:36:35.225714 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:36:35.225670    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-derive-642a8155-756f57f47c-gwjv8" podStartSLOduration=1.225654368 podStartE2EDuration="1.225654368s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 02:36:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 02:36:35.224120801 +0000 UTC m=+1073.691909584" watchObservedRunningTime="2026-09-16 02:36:35.225654368 +0000 UTC m=+1073.693443171"
Sep 16 02:36:40.335861 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:36:40.335832    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w2-db1ed7/auth-test-model-db1ed7-kserve-5474d84c75-q8j7z"]
Sep 16 02:36:40.339276 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:36:40.339257    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w2-db1ed7/auth-test-model-db1ed7-kserve-5474d84c75-q8j7z"
Sep 16 02:36:40.342703 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:36:40.342675    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w2-db1ed7\"/\"openshift-service-ca.crt\""
Sep 16 02:36:40.342703 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:36:40.342688    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w2-db1ed7\"/\"auth-test-model-db1ed7-kserve-self-signed-certs\""
Sep 16 02:36:40.342876 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:36:40.342690    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w2-db1ed7\"/\"kube-root-ca.crt\""
Sep 16 02:36:40.342876 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:36:40.342683    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w2-db1ed7\"/\"default-dockercfg-jjm52\""
Sep 16 02:36:40.346983 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:36:40.346957    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w2-db1ed7/auth-test-model-db1ed7-kserve-5474d84c75-q8j7z"]
Sep 16 02:36:40.505801 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:36:40.505767    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d073b71f-8e1d-4624-9192-08021795bf50-dshm\") pod \"auth-test-model-db1ed7-kserve-5474d84c75-q8j7z\" (UID: \"d073b71f-8e1d-4624-9192-08021795bf50\") " pod="ai-tenant-e2e-shared-a-w2-db1ed7/auth-test-model-db1ed7-kserve-5474d84c75-q8j7z"
Sep 16 02:36:40.505801 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:36:40.505810    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d073b71f-8e1d-4624-9192-08021795bf50-model-cache\") pod \"auth-test-model-db1ed7-kserve-5474d84c75-q8j7z\" (UID: \"d073b71f-8e1d-4624-9192-08021795bf50\") " pod="ai-tenant-e2e-shared-a-w2-db1ed7/auth-test-model-db1ed7-kserve-5474d84c75-q8j7z"
Sep 16 02:36:40.506054 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:36:40.505828    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d073b71f-8e1d-4624-9192-08021795bf50-home\") pod \"auth-test-model-db1ed7-kserve-5474d84c75-q8j7z\" (UID: \"d073b71f-8e1d-4624-9192-08021795bf50\") " pod="ai-tenant-e2e-shared-a-w2-db1ed7/auth-test-model-db1ed7-kserve-5474d84c75-q8j7z"
Sep 16 02:36:40.506054 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:36:40.505955    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d073b71f-8e1d-4624-9192-08021795bf50-tmp-dir\") pod \"auth-test-model-db1ed7-kserve-5474d84c75-q8j7z\" (UID: \"d073b71f-8e1d-4624-9192-08021795bf50\") " pod="ai-tenant-e2e-shared-a-w2-db1ed7/auth-test-model-db1ed7-kserve-5474d84c75-q8j7z"
Sep 16 02:36:40.506054 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:36:40.506016    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d073b71f-8e1d-4624-9192-08021795bf50-tls-certs\") pod \"auth-test-model-db1ed7-kserve-5474d84c75-q8j7z\" (UID: \"d073b71f-8e1d-4624-9192-08021795bf50\") " pod="ai-tenant-e2e-shared-a-w2-db1ed7/auth-test-model-db1ed7-kserve-5474d84c75-q8j7z"
Sep 16 02:36:40.506234 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:36:40.506076    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mkx7d\" (UniqueName: \"kubernetes.io/projected/d073b71f-8e1d-4624-9192-08021795bf50-kube-api-access-mkx7d\") pod \"auth-test-model-db1ed7-kserve-5474d84c75-q8j7z\" (UID: \"d073b71f-8e1d-4624-9192-08021795bf50\") " pod="ai-tenant-e2e-shared-a-w2-db1ed7/auth-test-model-db1ed7-kserve-5474d84c75-q8j7z"
Sep 16 02:36:40.607639 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:36:40.607605    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d073b71f-8e1d-4624-9192-08021795bf50-tmp-dir\") pod \"auth-test-model-db1ed7-kserve-5474d84c75-q8j7z\" (UID: \"d073b71f-8e1d-4624-9192-08021795bf50\") " pod="ai-tenant-e2e-shared-a-w2-db1ed7/auth-test-model-db1ed7-kserve-5474d84c75-q8j7z"
Sep 16 02:36:40.607820 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:36:40.607669    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d073b71f-8e1d-4624-9192-08021795bf50-tls-certs\") pod \"auth-test-model-db1ed7-kserve-5474d84c75-q8j7z\" (UID: \"d073b71f-8e1d-4624-9192-08021795bf50\") " pod="ai-tenant-e2e-shared-a-w2-db1ed7/auth-test-model-db1ed7-kserve-5474d84c75-q8j7z"
Sep 16 02:36:40.607820 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:36:40.607722    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mkx7d\" (UniqueName: \"kubernetes.io/projected/d073b71f-8e1d-4624-9192-08021795bf50-kube-api-access-mkx7d\") pod \"auth-test-model-db1ed7-kserve-5474d84c75-q8j7z\" (UID: \"d073b71f-8e1d-4624-9192-08021795bf50\") " pod="ai-tenant-e2e-shared-a-w2-db1ed7/auth-test-model-db1ed7-kserve-5474d84c75-q8j7z"
Sep 16 02:36:40.607820 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:36:40.607789    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d073b71f-8e1d-4624-9192-08021795bf50-dshm\") pod \"auth-test-model-db1ed7-kserve-5474d84c75-q8j7z\" (UID: \"d073b71f-8e1d-4624-9192-08021795bf50\") " pod="ai-tenant-e2e-shared-a-w2-db1ed7/auth-test-model-db1ed7-kserve-5474d84c75-q8j7z"
Sep 16 02:36:40.608000 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:36:40.607821    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d073b71f-8e1d-4624-9192-08021795bf50-model-cache\") pod \"auth-test-model-db1ed7-kserve-5474d84c75-q8j7z\" (UID: \"d073b71f-8e1d-4624-9192-08021795bf50\") " pod="ai-tenant-e2e-shared-a-w2-db1ed7/auth-test-model-db1ed7-kserve-5474d84c75-q8j7z"
Sep 16 02:36:40.608000 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:36:40.607844    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d073b71f-8e1d-4624-9192-08021795bf50-home\") pod \"auth-test-model-db1ed7-kserve-5474d84c75-q8j7z\" (UID: \"d073b71f-8e1d-4624-9192-08021795bf50\") " pod="ai-tenant-e2e-shared-a-w2-db1ed7/auth-test-model-db1ed7-kserve-5474d84c75-q8j7z"
Sep 16 02:36:40.608099 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:36:40.608070    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d073b71f-8e1d-4624-9192-08021795bf50-tmp-dir\") pod \"auth-test-model-db1ed7-kserve-5474d84c75-q8j7z\" (UID: \"d073b71f-8e1d-4624-9192-08021795bf50\") " pod="ai-tenant-e2e-shared-a-w2-db1ed7/auth-test-model-db1ed7-kserve-5474d84c75-q8j7z"
Sep 16 02:36:40.608217 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:36:40.608199    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d073b71f-8e1d-4624-9192-08021795bf50-home\") pod \"auth-test-model-db1ed7-kserve-5474d84c75-q8j7z\" (UID: \"d073b71f-8e1d-4624-9192-08021795bf50\") " pod="ai-tenant-e2e-shared-a-w2-db1ed7/auth-test-model-db1ed7-kserve-5474d84c75-q8j7z"
Sep 16 02:36:40.608297 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:36:40.608225    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d073b71f-8e1d-4624-9192-08021795bf50-model-cache\") pod \"auth-test-model-db1ed7-kserve-5474d84c75-q8j7z\" (UID: \"d073b71f-8e1d-4624-9192-08021795bf50\") " pod="ai-tenant-e2e-shared-a-w2-db1ed7/auth-test-model-db1ed7-kserve-5474d84c75-q8j7z"
Sep 16 02:36:40.610004 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:36:40.609983    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d073b71f-8e1d-4624-9192-08021795bf50-dshm\") pod \"auth-test-model-db1ed7-kserve-5474d84c75-q8j7z\" (UID: \"d073b71f-8e1d-4624-9192-08021795bf50\") " pod="ai-tenant-e2e-shared-a-w2-db1ed7/auth-test-model-db1ed7-kserve-5474d84c75-q8j7z"
Sep 16 02:36:40.610182 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:36:40.610166    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d073b71f-8e1d-4624-9192-08021795bf50-tls-certs\") pod \"auth-test-model-db1ed7-kserve-5474d84c75-q8j7z\" (UID: \"d073b71f-8e1d-4624-9192-08021795bf50\") " pod="ai-tenant-e2e-shared-a-w2-db1ed7/auth-test-model-db1ed7-kserve-5474d84c75-q8j7z"
Sep 16 02:36:40.617064 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:36:40.617044    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mkx7d\" (UniqueName: \"kubernetes.io/projected/d073b71f-8e1d-4624-9192-08021795bf50-kube-api-access-mkx7d\") pod \"auth-test-model-db1ed7-kserve-5474d84c75-q8j7z\" (UID: \"d073b71f-8e1d-4624-9192-08021795bf50\") " pod="ai-tenant-e2e-shared-a-w2-db1ed7/auth-test-model-db1ed7-kserve-5474d84c75-q8j7z"
Sep 16 02:36:40.650365 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:36:40.650328    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w2-db1ed7/auth-test-model-db1ed7-kserve-5474d84c75-q8j7z"
Sep 16 02:36:40.926761 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:36:40.926730    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w2-db1ed7/auth-test-model-db1ed7-kserve-5474d84c75-q8j7z"]
Sep 16 02:36:40.929998 ip-10-0-128-111 kubenswrapper[2583]: W0916 02:36:40.929971    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd073b71f_8e1d_4624_9192_08021795bf50.slice/crio-4cb8fbbe070f381984bc3d763ffac6d3037989de6595f50cab4d70af214b284e WatchSource:0}: Error finding container 4cb8fbbe070f381984bc3d763ffac6d3037989de6595f50cab4d70af214b284e: Status 404 returned error can't find the container with id 4cb8fbbe070f381984bc3d763ffac6d3037989de6595f50cab4d70af214b284e
Sep 16 02:36:41.232048 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:36:41.231956    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w2-db1ed7/auth-test-model-db1ed7-kserve-5474d84c75-q8j7z" event={"ID":"d073b71f-8e1d-4624-9192-08021795bf50","Type":"ContainerStarted","Data":"768e589cfbdbab02561853d1f594974fdf7d08e0a071c351a81e250102a48fbd"}
Sep 16 02:36:41.232048 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:36:41.232010    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w2-db1ed7/auth-test-model-db1ed7-kserve-5474d84c75-q8j7z" event={"ID":"d073b71f-8e1d-4624-9192-08021795bf50","Type":"ContainerStarted","Data":"4cb8fbbe070f381984bc3d763ffac6d3037989de6595f50cab4d70af214b284e"}
Sep 16 02:36:41.247211 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:36:41.247158    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w2-db1ed7/auth-test-model-db1ed7-kserve-5474d84c75-q8j7z" podStartSLOduration=1.24714307 podStartE2EDuration="1.24714307s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 02:36:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 02:36:41.245908806 +0000 UTC m=+1079.713697602" watchObservedRunningTime="2026-09-16 02:36:41.24714307 +0000 UTC m=+1079.714931851"
Sep 16 02:36:46.206546 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:36:46.206513    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-derive-642a8155-756f57f47c-gwjv8"
Sep 16 02:36:46.206964 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:36:46.206609    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-derive-642a8155-67d6745478-9lllv"
Sep 16 02:36:50.651022 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:36:50.650986    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w2-db1ed7/auth-test-model-db1ed7-kserve-5474d84c75-q8j7z"
Sep 16 02:36:50.651421 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:36:50.651043    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w2-db1ed7/auth-test-model-db1ed7-kserve-5474d84c75-q8j7z"
Sep 16 02:36:50.663239 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:36:50.663215    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w2-db1ed7/auth-test-model-db1ed7-kserve-5474d84c75-q8j7z"
Sep 16 02:36:51.293894 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:36:51.293865    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w2-db1ed7/auth-test-model-db1ed7-kserve-5474d84c75-q8j7z"
Sep 16 02:36:58.064344 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:36:58.064308    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-fzl8t"]
Sep 16 02:37:00.527513 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:37:00.527475    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-642a8155-6m6qp"]
Sep 16 02:37:00.532896 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:37:00.532875    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-642a8155-6m6qp"
Sep 16 02:37:00.535654 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:37:00.535631    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 16 02:37:00.535991 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:37:00.535681    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-w8hnp\""
Sep 16 02:37:00.537431 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:37:00.537410    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-642a8155-6m6qp"]
Sep 16 02:37:00.598013 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:37:00.597976    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6e7e4add-97c8-4b9b-a793-d18322a78879-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-642a8155-6m6qp\" (UID: \"6e7e4add-97c8-4b9b-a793-d18322a78879\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-642a8155-6m6qp"
Sep 16 02:37:00.698758 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:37:00.698724    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6e7e4add-97c8-4b9b-a793-d18322a78879-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-642a8155-6m6qp\" (UID: \"6e7e4add-97c8-4b9b-a793-d18322a78879\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-642a8155-6m6qp"
Sep 16 02:37:00.699407 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:37:00.699383    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6e7e4add-97c8-4b9b-a793-d18322a78879-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-642a8155-6m6qp\" (UID: \"6e7e4add-97c8-4b9b-a793-d18322a78879\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-642a8155-6m6qp"
Sep 16 02:37:00.845102 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:37:00.845071    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-642a8155-6m6qp"
Sep 16 02:37:01.117122 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:37:01.117100    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-642a8155-6m6qp"]
Sep 16 02:37:01.118952 ip-10-0-128-111 kubenswrapper[2583]: W0916 02:37:01.118922    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6e7e4add_97c8_4b9b_a793_d18322a78879.slice/crio-e78e6ede952fd3036fb963f8f1a162ff54d8598cc7e49cf2b8ae0045e2c87946 WatchSource:0}: Error finding container e78e6ede952fd3036fb963f8f1a162ff54d8598cc7e49cf2b8ae0045e2c87946: Status 404 returned error can't find the container with id e78e6ede952fd3036fb963f8f1a162ff54d8598cc7e49cf2b8ae0045e2c87946
Sep 16 02:37:01.337018 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:37:01.336986    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-642a8155-6m6qp" event={"ID":"6e7e4add-97c8-4b9b-a793-d18322a78879","Type":"ContainerStarted","Data":"e78e6ede952fd3036fb963f8f1a162ff54d8598cc7e49cf2b8ae0045e2c87946"}
Sep 16 02:37:02.343971 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:37:02.343936    2583 generic.go:356] "Generic (PLEG): container finished" podID="6e7e4add-97c8-4b9b-a793-d18322a78879" containerID="5d21b9b4fe65eb21b660ddb39982857e6d1fb02b63984a3609980fafd4b3ea97" exitCode=0
Sep 16 02:37:02.344350 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:37:02.344022    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-642a8155-6m6qp" event={"ID":"6e7e4add-97c8-4b9b-a793-d18322a78879","Type":"ContainerDied","Data":"5d21b9b4fe65eb21b660ddb39982857e6d1fb02b63984a3609980fafd4b3ea97"}
Sep 16 02:37:03.481702 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:37:03.481673    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-642a8155-6m6qp"
Sep 16 02:37:03.528598 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:37:03.528563    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6e7e4add-97c8-4b9b-a793-d18322a78879-maas-api-service-ca\") pod \"6e7e4add-97c8-4b9b-a793-d18322a78879\" (UID: \"6e7e4add-97c8-4b9b-a793-d18322a78879\") "
Sep 16 02:37:03.528937 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:37:03.528912    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6e7e4add-97c8-4b9b-a793-d18322a78879-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "6e7e4add-97c8-4b9b-a793-d18322a78879" (UID: "6e7e4add-97c8-4b9b-a793-d18322a78879"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 02:37:03.629953 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:37:03.629871    2583 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6e7e4add-97c8-4b9b-a793-d18322a78879-maas-api-service-ca\") on node \"ip-10-0-128-111.ec2.internal\" DevicePath \"\""
Sep 16 02:37:04.098119 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:37:04.098089    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-a3d7eb-qdfz2"]
Sep 16 02:37:04.098546 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:37:04.098525    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6e7e4add-97c8-4b9b-a793-d18322a78879" containerName="revoke-keys"
Sep 16 02:37:04.098716 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:37:04.098604    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="6e7e4add-97c8-4b9b-a793-d18322a78879" containerName="revoke-keys"
Sep 16 02:37:04.098813 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:37:04.098795    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="6e7e4add-97c8-4b9b-a793-d18322a78879" containerName="revoke-keys"
Sep 16 02:37:04.102023 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:37:04.102005    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-a3d7eb-qdfz2"
Sep 16 02:37:04.108124 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:37:04.108089    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-a3d7eb-qdfz2"]
Sep 16 02:37:04.236096 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:37:04.236058    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a5bebfdc-6ac7-4ce6-b12d-7bff79424333-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-worker-w3-a3d7eb-qdfz2\" (UID: \"a5bebfdc-6ac7-4ce6-b12d-7bff79424333\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-a3d7eb-qdfz2"
Sep 16 02:37:04.337580 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:37:04.337541    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a5bebfdc-6ac7-4ce6-b12d-7bff79424333-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-worker-w3-a3d7eb-qdfz2\" (UID: \"a5bebfdc-6ac7-4ce6-b12d-7bff79424333\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-a3d7eb-qdfz2"
Sep 16 02:37:04.338209 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:37:04.338185    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a5bebfdc-6ac7-4ce6-b12d-7bff79424333-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-worker-w3-a3d7eb-qdfz2\" (UID: \"a5bebfdc-6ac7-4ce6-b12d-7bff79424333\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-a3d7eb-qdfz2"
Sep 16 02:37:04.355101 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:37:04.355074    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-642a8155-6m6qp"
Sep 16 02:37:04.355101 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:37:04.355089    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-642a8155-6m6qp" event={"ID":"6e7e4add-97c8-4b9b-a793-d18322a78879","Type":"ContainerDied","Data":"e78e6ede952fd3036fb963f8f1a162ff54d8598cc7e49cf2b8ae0045e2c87946"}
Sep 16 02:37:04.355278 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:37:04.355129    2583 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e78e6ede952fd3036fb963f8f1a162ff54d8598cc7e49cf2b8ae0045e2c87946"
Sep 16 02:37:04.414313 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:37:04.414286    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-a3d7eb-qdfz2"
Sep 16 02:37:04.683597 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:37:04.683546    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-a3d7eb-qdfz2"]
Sep 16 02:37:04.685859 ip-10-0-128-111 kubenswrapper[2583]: W0916 02:37:04.685834    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda5bebfdc_6ac7_4ce6_b12d_7bff79424333.slice/crio-0b7bda05f72d6092b77c51b2408c4f48dad5cdbc6b9d5666d71edb9e0a917aab WatchSource:0}: Error finding container 0b7bda05f72d6092b77c51b2408c4f48dad5cdbc6b9d5666d71edb9e0a917aab: Status 404 returned error can't find the container with id 0b7bda05f72d6092b77c51b2408c4f48dad5cdbc6b9d5666d71edb9e0a917aab
Sep 16 02:37:05.361441 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:37:05.361407    2583 generic.go:356] "Generic (PLEG): container finished" podID="a5bebfdc-6ac7-4ce6-b12d-7bff79424333" containerID="371fbbe45fe526b8488b386f6b98ceb5c8f63618aae0469cad60eb2fc634a5d4" exitCode=0
Sep 16 02:37:05.361606 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:37:05.361456    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-a3d7eb-qdfz2" event={"ID":"a5bebfdc-6ac7-4ce6-b12d-7bff79424333","Type":"ContainerDied","Data":"371fbbe45fe526b8488b386f6b98ceb5c8f63618aae0469cad60eb2fc634a5d4"}
Sep 16 02:37:05.361606 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:37:05.361514    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-a3d7eb-qdfz2" event={"ID":"a5bebfdc-6ac7-4ce6-b12d-7bff79424333","Type":"ContainerStarted","Data":"0b7bda05f72d6092b77c51b2408c4f48dad5cdbc6b9d5666d71edb9e0a917aab"}
Sep 16 02:37:06.497754 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:37:06.497730    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-a3d7eb-qdfz2"
Sep 16 02:37:06.560385 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:37:06.560351    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a5bebfdc-6ac7-4ce6-b12d-7bff79424333-maas-api-service-ca\") pod \"a5bebfdc-6ac7-4ce6-b12d-7bff79424333\" (UID: \"a5bebfdc-6ac7-4ce6-b12d-7bff79424333\") "
Sep 16 02:37:06.560679 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:37:06.560658    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a5bebfdc-6ac7-4ce6-b12d-7bff79424333-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "a5bebfdc-6ac7-4ce6-b12d-7bff79424333" (UID: "a5bebfdc-6ac7-4ce6-b12d-7bff79424333"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 02:37:06.661314 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:37:06.661224    2583 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a5bebfdc-6ac7-4ce6-b12d-7bff79424333-maas-api-service-ca\") on node \"ip-10-0-128-111.ec2.internal\" DevicePath \"\""
Sep 16 02:37:07.372764 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:37:07.372735    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-a3d7eb-qdfz2"
Sep 16 02:37:07.372919 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:37:07.372772    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-a3d7eb-qdfz2" event={"ID":"a5bebfdc-6ac7-4ce6-b12d-7bff79424333","Type":"ContainerDied","Data":"0b7bda05f72d6092b77c51b2408c4f48dad5cdbc6b9d5666d71edb9e0a917aab"}
Sep 16 02:37:07.372919 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:37:07.372816    2583 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0b7bda05f72d6092b77c51b2408c4f48dad5cdbc6b9d5666d71edb9e0a917aab"
Sep 16 02:37:08.951160 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:37:08.951124    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-a3d7eb/distinct-2-w3-a3d7eb-kserve-7b76c4fc77-pg8fv"]
Sep 16 02:37:08.955889 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:37:08.955861    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-a3d7eb/distinct-w3-a3d7eb-kserve-79d99dc9d6-mt2wx"]
Sep 16 02:37:08.976070 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:37:08.976041    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-a3d7eb/simulator-w3-a3d7eb-kserve-7bcf597646-zj6qt"]
Sep 16 02:37:12.366948 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:37:12.366916    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-642a8155-756f57f47c-gwjv8"]
Sep 16 02:37:12.367366 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:37:12.367178    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-derive-642a8155-756f57f47c-gwjv8" podUID="02336b88-6308-4c8c-aa5f-e3e77e950b33" containerName="payload-processing" containerID="cri-o://33fa1376f0f074b966acee471bd0ec27560f281c011a142ff4e5d7d3413f4e35" gracePeriod=30
Sep 16 02:37:12.391176 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:37:12.391141    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-642a8155-67d6745478-9lllv"]
Sep 16 02:37:12.391956 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:37:12.391929    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-derive-642a8155-67d6745478-9lllv" podUID="a006042d-eddd-4564-923e-04e0f548d58c" containerName="payload-pre-processing" containerID="cri-o://17478b9e67021c618bdb78652a9441454bafc11a91f8ef1c2cc0068b00eb1ce7" gracePeriod=30
Sep 16 02:37:12.512377 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:37:12.512350    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-642a8155-6m6qp"]
Sep 16 02:37:12.517333 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:37:12.517312    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-642a8155-6m6qp"]
Sep 16 02:37:12.693744 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:37:12.693716    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-642a8155-756f57f47c-gwjv8"
Sep 16 02:37:12.696643 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:37:12.696624    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-642a8155-67d6745478-9lllv"
Sep 16 02:37:12.825076 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:37:12.825041    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a006042d-eddd-4564-923e-04e0f548d58c-plugins-config-volume\") pod \"a006042d-eddd-4564-923e-04e0f548d58c\" (UID: \"a006042d-eddd-4564-923e-04e0f548d58c\") "
Sep 16 02:37:12.825076 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:37:12.825082    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mdg6c\" (UniqueName: \"kubernetes.io/projected/02336b88-6308-4c8c-aa5f-e3e77e950b33-kube-api-access-mdg6c\") pod \"02336b88-6308-4c8c-aa5f-e3e77e950b33\" (UID: \"02336b88-6308-4c8c-aa5f-e3e77e950b33\") "
Sep 16 02:37:12.825328 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:37:12.825122    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9929c\" (UniqueName: \"kubernetes.io/projected/a006042d-eddd-4564-923e-04e0f548d58c-kube-api-access-9929c\") pod \"a006042d-eddd-4564-923e-04e0f548d58c\" (UID: \"a006042d-eddd-4564-923e-04e0f548d58c\") "
Sep 16 02:37:12.825328 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:37:12.825199    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/02336b88-6308-4c8c-aa5f-e3e77e950b33-plugins-config-volume\") pod \"02336b88-6308-4c8c-aa5f-e3e77e950b33\" (UID: \"02336b88-6308-4c8c-aa5f-e3e77e950b33\") "
Sep 16 02:37:12.825542 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:37:12.825516    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a006042d-eddd-4564-923e-04e0f548d58c-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "a006042d-eddd-4564-923e-04e0f548d58c" (UID: "a006042d-eddd-4564-923e-04e0f548d58c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 02:37:12.825668 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:37:12.825607    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/02336b88-6308-4c8c-aa5f-e3e77e950b33-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "02336b88-6308-4c8c-aa5f-e3e77e950b33" (UID: "02336b88-6308-4c8c-aa5f-e3e77e950b33"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 02:37:12.827337 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:37:12.827315    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/02336b88-6308-4c8c-aa5f-e3e77e950b33-kube-api-access-mdg6c" (OuterVolumeSpecName: "kube-api-access-mdg6c") pod "02336b88-6308-4c8c-aa5f-e3e77e950b33" (UID: "02336b88-6308-4c8c-aa5f-e3e77e950b33"). InnerVolumeSpecName "kube-api-access-mdg6c". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 02:37:12.827427 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:37:12.827407    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a006042d-eddd-4564-923e-04e0f548d58c-kube-api-access-9929c" (OuterVolumeSpecName: "kube-api-access-9929c") pod "a006042d-eddd-4564-923e-04e0f548d58c" (UID: "a006042d-eddd-4564-923e-04e0f548d58c"). InnerVolumeSpecName "kube-api-access-9929c". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 02:37:12.926348 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:37:12.926262    2583 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a006042d-eddd-4564-923e-04e0f548d58c-plugins-config-volume\") on node \"ip-10-0-128-111.ec2.internal\" DevicePath \"\""
Sep 16 02:37:12.926348 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:37:12.926292    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mdg6c\" (UniqueName: \"kubernetes.io/projected/02336b88-6308-4c8c-aa5f-e3e77e950b33-kube-api-access-mdg6c\") on node \"ip-10-0-128-111.ec2.internal\" DevicePath \"\""
Sep 16 02:37:12.926348 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:37:12.926302    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9929c\" (UniqueName: \"kubernetes.io/projected/a006042d-eddd-4564-923e-04e0f548d58c-kube-api-access-9929c\") on node \"ip-10-0-128-111.ec2.internal\" DevicePath \"\""
Sep 16 02:37:12.926348 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:37:12.926311    2583 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/02336b88-6308-4c8c-aa5f-e3e77e950b33-plugins-config-volume\") on node \"ip-10-0-128-111.ec2.internal\" DevicePath \"\""
Sep 16 02:37:13.416158 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:37:13.416125    2583 generic.go:356] "Generic (PLEG): container finished" podID="02336b88-6308-4c8c-aa5f-e3e77e950b33" containerID="33fa1376f0f074b966acee471bd0ec27560f281c011a142ff4e5d7d3413f4e35" exitCode=0
Sep 16 02:37:13.416646 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:37:13.416169    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-642a8155-756f57f47c-gwjv8"
Sep 16 02:37:13.416646 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:37:13.416215    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-642a8155-756f57f47c-gwjv8" event={"ID":"02336b88-6308-4c8c-aa5f-e3e77e950b33","Type":"ContainerDied","Data":"33fa1376f0f074b966acee471bd0ec27560f281c011a142ff4e5d7d3413f4e35"}
Sep 16 02:37:13.416646 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:37:13.416275    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-642a8155-756f57f47c-gwjv8" event={"ID":"02336b88-6308-4c8c-aa5f-e3e77e950b33","Type":"ContainerDied","Data":"8a7a60fc8e03cf74bb8736d0911f10058d2ecd60a15dba6d65031f2a05926422"}
Sep 16 02:37:13.416646 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:37:13.416303    2583 scope.go:117] "RemoveContainer" containerID="33fa1376f0f074b966acee471bd0ec27560f281c011a142ff4e5d7d3413f4e35"
Sep 16 02:37:13.419582 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:37:13.418830    2583 generic.go:356] "Generic (PLEG): container finished" podID="a006042d-eddd-4564-923e-04e0f548d58c" containerID="17478b9e67021c618bdb78652a9441454bafc11a91f8ef1c2cc0068b00eb1ce7" exitCode=0
Sep 16 02:37:13.419582 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:37:13.418944    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-642a8155-67d6745478-9lllv" event={"ID":"a006042d-eddd-4564-923e-04e0f548d58c","Type":"ContainerDied","Data":"17478b9e67021c618bdb78652a9441454bafc11a91f8ef1c2cc0068b00eb1ce7"}
Sep 16 02:37:13.419582 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:37:13.418990    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-642a8155-67d6745478-9lllv" event={"ID":"a006042d-eddd-4564-923e-04e0f548d58c","Type":"ContainerDied","Data":"d97d14fe15b3cea170724a21b4f497f0c7314a0f2fdec450282484ca92e2c690"}
Sep 16 02:37:13.419582 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:37:13.419053    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-642a8155-67d6745478-9lllv"
Sep 16 02:37:13.437894 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:37:13.437873    2583 scope.go:117] "RemoveContainer" containerID="33fa1376f0f074b966acee471bd0ec27560f281c011a142ff4e5d7d3413f4e35"
Sep 16 02:37:13.438219 ip-10-0-128-111 kubenswrapper[2583]: E0916 02:37:13.438195    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"33fa1376f0f074b966acee471bd0ec27560f281c011a142ff4e5d7d3413f4e35\": container with ID starting with 33fa1376f0f074b966acee471bd0ec27560f281c011a142ff4e5d7d3413f4e35 not found: ID does not exist" containerID="33fa1376f0f074b966acee471bd0ec27560f281c011a142ff4e5d7d3413f4e35"
Sep 16 02:37:13.438272 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:37:13.438229    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"33fa1376f0f074b966acee471bd0ec27560f281c011a142ff4e5d7d3413f4e35"} err="failed to get container status \"33fa1376f0f074b966acee471bd0ec27560f281c011a142ff4e5d7d3413f4e35\": rpc error: code = NotFound desc = could not find container \"33fa1376f0f074b966acee471bd0ec27560f281c011a142ff4e5d7d3413f4e35\": container with ID starting with 33fa1376f0f074b966acee471bd0ec27560f281c011a142ff4e5d7d3413f4e35 not found: ID does not exist"
Sep 16 02:37:13.438272 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:37:13.438245    2583 scope.go:117] "RemoveContainer" containerID="17478b9e67021c618bdb78652a9441454bafc11a91f8ef1c2cc0068b00eb1ce7"
Sep 16 02:37:13.446987 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:37:13.446923    2583 scope.go:117] "RemoveContainer" containerID="17478b9e67021c618bdb78652a9441454bafc11a91f8ef1c2cc0068b00eb1ce7"
Sep 16 02:37:13.447643 ip-10-0-128-111 kubenswrapper[2583]: E0916 02:37:13.447542    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"17478b9e67021c618bdb78652a9441454bafc11a91f8ef1c2cc0068b00eb1ce7\": container with ID starting with 17478b9e67021c618bdb78652a9441454bafc11a91f8ef1c2cc0068b00eb1ce7 not found: ID does not exist" containerID="17478b9e67021c618bdb78652a9441454bafc11a91f8ef1c2cc0068b00eb1ce7"
Sep 16 02:37:13.447739 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:37:13.447652    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"17478b9e67021c618bdb78652a9441454bafc11a91f8ef1c2cc0068b00eb1ce7"} err="failed to get container status \"17478b9e67021c618bdb78652a9441454bafc11a91f8ef1c2cc0068b00eb1ce7\": rpc error: code = NotFound desc = could not find container \"17478b9e67021c618bdb78652a9441454bafc11a91f8ef1c2cc0068b00eb1ce7\": container with ID starting with 17478b9e67021c618bdb78652a9441454bafc11a91f8ef1c2cc0068b00eb1ce7 not found: ID does not exist"
Sep 16 02:37:13.448192 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:37:13.448180    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-642a8155-67d6745478-9lllv"]
Sep 16 02:37:13.450133 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:37:13.450105    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-642a8155-67d6745478-9lllv"]
Sep 16 02:37:13.458220 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:37:13.458175    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-642a8155-756f57f47c-gwjv8"]
Sep 16 02:37:13.459964 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:37:13.459946    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-642a8155-756f57f47c-gwjv8"]
Sep 16 02:37:14.045273 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:37:14.045243    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="02336b88-6308-4c8c-aa5f-e3e77e950b33" path="/var/lib/kubelet/pods/02336b88-6308-4c8c-aa5f-e3e77e950b33/volumes"
Sep 16 02:37:14.045629 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:37:14.045615    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6e7e4add-97c8-4b9b-a793-d18322a78879" path="/var/lib/kubelet/pods/6e7e4add-97c8-4b9b-a793-d18322a78879/volumes"
Sep 16 02:37:14.045941 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:37:14.045930    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a006042d-eddd-4564-923e-04e0f548d58c" path="/var/lib/kubelet/pods/a006042d-eddd-4564-923e-04e0f548d58c/volumes"
Sep 16 02:37:21.355882 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:37:21.355846    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-a3d7eb-qdfz2"]
Sep 16 02:37:21.360304 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:37:21.360279    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-a3d7eb-qdfz2"]
Sep 16 02:37:22.045534 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:37:22.045487    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a5bebfdc-6ac7-4ce6-b12d-7bff79424333" path="/var/lib/kubelet/pods/a5bebfdc-6ac7-4ce6-b12d-7bff79424333/volumes"
Sep 16 02:37:23.971050 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:37:23.971009    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-models-e2e-worker-w3-a3d7eb/distinct-2-w3-a3d7eb-kserve-7b76c4fc77-pg8fv" podUID="02f506a2-c921-462d-afc7-5b47c3b30d54" containerName="main" containerID="cri-o://863217e97f1eccace04ca34e2287861a1a0fb5a39df7ed20866b1097ea95de7f" gracePeriod=45
Sep 16 02:37:23.975234 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:37:23.975203    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-models-e2e-worker-w3-a3d7eb/distinct-w3-a3d7eb-kserve-79d99dc9d6-mt2wx" podUID="8f08f681-a03c-49b8-a118-f7101aaeb69c" containerName="main" containerID="cri-o://eb619e491b6b4e343d5c2c90000de4c82e79eba46adce04bbb7a5c546158c33f" gracePeriod=45
Sep 16 02:37:23.994311 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:37:23.994284    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-models-e2e-worker-w3-a3d7eb/simulator-w3-a3d7eb-kserve-7bcf597646-zj6qt" podUID="cc9488c9-649c-4992-9c1e-fa311434f6d5" containerName="main" containerID="cri-o://00b3a10f8f77a6594565222cf05f2e6a5c46c5c6d787fa20b60f2968449e0690" gracePeriod=45
Sep 16 02:37:24.237402 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:37:24.237332    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-a3d7eb/distinct-2-w3-a3d7eb-kserve-7b76c4fc77-pg8fv"
Sep 16 02:37:24.281137 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:37:24.281090    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-a3d7eb/distinct-w3-a3d7eb-kserve-79d99dc9d6-mt2wx"
Sep 16 02:37:24.290830 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:37:24.290804    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-a3d7eb/simulator-w3-a3d7eb-kserve-7bcf597646-zj6qt"
Sep 16 02:37:24.342282 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:37:24.342247    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/02f506a2-c921-462d-afc7-5b47c3b30d54-model-cache\") pod \"02f506a2-c921-462d-afc7-5b47c3b30d54\" (UID: \"02f506a2-c921-462d-afc7-5b47c3b30d54\") "
Sep 16 02:37:24.342467 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:37:24.342330    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/02f506a2-c921-462d-afc7-5b47c3b30d54-tmp-dir\") pod \"02f506a2-c921-462d-afc7-5b47c3b30d54\" (UID: \"02f506a2-c921-462d-afc7-5b47c3b30d54\") "
Sep 16 02:37:24.342467 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:37:24.342355    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/02f506a2-c921-462d-afc7-5b47c3b30d54-home\") pod \"02f506a2-c921-462d-afc7-5b47c3b30d54\" (UID: \"02f506a2-c921-462d-afc7-5b47c3b30d54\") "
Sep 16 02:37:24.342467 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:37:24.342420    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/02f506a2-c921-462d-afc7-5b47c3b30d54-dshm\") pod \"02f506a2-c921-462d-afc7-5b47c3b30d54\" (UID: \"02f506a2-c921-462d-afc7-5b47c3b30d54\") "
Sep 16 02:37:24.342467 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:37:24.342449    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/02f506a2-c921-462d-afc7-5b47c3b30d54-tls-certs\") pod \"02f506a2-c921-462d-afc7-5b47c3b30d54\" (UID: \"02f506a2-c921-462d-afc7-5b47c3b30d54\") "
Sep 16 02:37:24.342698 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:37:24.342501    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/02f506a2-c921-462d-afc7-5b47c3b30d54-model-cache" (OuterVolumeSpecName: "model-cache") pod "02f506a2-c921-462d-afc7-5b47c3b30d54" (UID: "02f506a2-c921-462d-afc7-5b47c3b30d54"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 02:37:24.342698 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:37:24.342511    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-j7p2f\" (UniqueName: \"kubernetes.io/projected/02f506a2-c921-462d-afc7-5b47c3b30d54-kube-api-access-j7p2f\") pod \"02f506a2-c921-462d-afc7-5b47c3b30d54\" (UID: \"02f506a2-c921-462d-afc7-5b47c3b30d54\") "
Sep 16 02:37:24.342698 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:37:24.342651    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/02f506a2-c921-462d-afc7-5b47c3b30d54-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "02f506a2-c921-462d-afc7-5b47c3b30d54" (UID: "02f506a2-c921-462d-afc7-5b47c3b30d54"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 02:37:24.342849 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:37:24.342703    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/02f506a2-c921-462d-afc7-5b47c3b30d54-home" (OuterVolumeSpecName: "home") pod "02f506a2-c921-462d-afc7-5b47c3b30d54" (UID: "02f506a2-c921-462d-afc7-5b47c3b30d54"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 02:37:24.342963 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:37:24.342944    2583 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/02f506a2-c921-462d-afc7-5b47c3b30d54-model-cache\") on node \"ip-10-0-128-111.ec2.internal\" DevicePath \"\""
Sep 16 02:37:24.343045 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:37:24.342999    2583 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/02f506a2-c921-462d-afc7-5b47c3b30d54-tmp-dir\") on node \"ip-10-0-128-111.ec2.internal\" DevicePath \"\""
Sep 16 02:37:24.343045 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:37:24.343019    2583 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/02f506a2-c921-462d-afc7-5b47c3b30d54-home\") on node \"ip-10-0-128-111.ec2.internal\" DevicePath \"\""
Sep 16 02:37:24.345161 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:37:24.345129    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/02f506a2-c921-462d-afc7-5b47c3b30d54-dshm" (OuterVolumeSpecName: "dshm") pod "02f506a2-c921-462d-afc7-5b47c3b30d54" (UID: "02f506a2-c921-462d-afc7-5b47c3b30d54"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 02:37:24.345253 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:37:24.345161    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/02f506a2-c921-462d-afc7-5b47c3b30d54-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "02f506a2-c921-462d-afc7-5b47c3b30d54" (UID: "02f506a2-c921-462d-afc7-5b47c3b30d54"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 02:37:24.345253 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:37:24.345173    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/02f506a2-c921-462d-afc7-5b47c3b30d54-kube-api-access-j7p2f" (OuterVolumeSpecName: "kube-api-access-j7p2f") pod "02f506a2-c921-462d-afc7-5b47c3b30d54" (UID: "02f506a2-c921-462d-afc7-5b47c3b30d54"). InnerVolumeSpecName "kube-api-access-j7p2f". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 02:37:24.443858 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:37:24.443828    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8f08f681-a03c-49b8-a118-f7101aaeb69c-model-cache\") pod \"8f08f681-a03c-49b8-a118-f7101aaeb69c\" (UID: \"8f08f681-a03c-49b8-a118-f7101aaeb69c\") "
Sep 16 02:37:24.443858 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:37:24.443863    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8f08f681-a03c-49b8-a118-f7101aaeb69c-tmp-dir\") pod \"8f08f681-a03c-49b8-a118-f7101aaeb69c\" (UID: \"8f08f681-a03c-49b8-a118-f7101aaeb69c\") "
Sep 16 02:37:24.444100 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:37:24.443881    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cc9488c9-649c-4992-9c1e-fa311434f6d5-tmp-dir\") pod \"cc9488c9-649c-4992-9c1e-fa311434f6d5\" (UID: \"cc9488c9-649c-4992-9c1e-fa311434f6d5\") "
Sep 16 02:37:24.444100 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:37:24.443911    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cc9488c9-649c-4992-9c1e-fa311434f6d5-home\") pod \"cc9488c9-649c-4992-9c1e-fa311434f6d5\" (UID: \"cc9488c9-649c-4992-9c1e-fa311434f6d5\") "
Sep 16 02:37:24.444100 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:37:24.443943    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cc9488c9-649c-4992-9c1e-fa311434f6d5-tls-certs\") pod \"cc9488c9-649c-4992-9c1e-fa311434f6d5\" (UID: \"cc9488c9-649c-4992-9c1e-fa311434f6d5\") "
Sep 16 02:37:24.444100 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:37:24.443960    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cc9488c9-649c-4992-9c1e-fa311434f6d5-model-cache\") pod \"cc9488c9-649c-4992-9c1e-fa311434f6d5\" (UID: \"cc9488c9-649c-4992-9c1e-fa311434f6d5\") "
Sep 16 02:37:24.444100 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:37:24.443973    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8f08f681-a03c-49b8-a118-f7101aaeb69c-home\") pod \"8f08f681-a03c-49b8-a118-f7101aaeb69c\" (UID: \"8f08f681-a03c-49b8-a118-f7101aaeb69c\") "
Sep 16 02:37:24.444100 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:37:24.443996    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-d2zqt\" (UniqueName: \"kubernetes.io/projected/cc9488c9-649c-4992-9c1e-fa311434f6d5-kube-api-access-d2zqt\") pod \"cc9488c9-649c-4992-9c1e-fa311434f6d5\" (UID: \"cc9488c9-649c-4992-9c1e-fa311434f6d5\") "
Sep 16 02:37:24.444100 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:37:24.444018    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cc9488c9-649c-4992-9c1e-fa311434f6d5-dshm\") pod \"cc9488c9-649c-4992-9c1e-fa311434f6d5\" (UID: \"cc9488c9-649c-4992-9c1e-fa311434f6d5\") "
Sep 16 02:37:24.444100 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:37:24.444065    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8f08f681-a03c-49b8-a118-f7101aaeb69c-dshm\") pod \"8f08f681-a03c-49b8-a118-f7101aaeb69c\" (UID: \"8f08f681-a03c-49b8-a118-f7101aaeb69c\") "
Sep 16 02:37:24.444100 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:37:24.444095    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ncqf5\" (UniqueName: \"kubernetes.io/projected/8f08f681-a03c-49b8-a118-f7101aaeb69c-kube-api-access-ncqf5\") pod \"8f08f681-a03c-49b8-a118-f7101aaeb69c\" (UID: \"8f08f681-a03c-49b8-a118-f7101aaeb69c\") "
Sep 16 02:37:24.444543 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:37:24.444115    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8f08f681-a03c-49b8-a118-f7101aaeb69c-model-cache" (OuterVolumeSpecName: "model-cache") pod "8f08f681-a03c-49b8-a118-f7101aaeb69c" (UID: "8f08f681-a03c-49b8-a118-f7101aaeb69c"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 02:37:24.444543 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:37:24.444126    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cc9488c9-649c-4992-9c1e-fa311434f6d5-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "cc9488c9-649c-4992-9c1e-fa311434f6d5" (UID: "cc9488c9-649c-4992-9c1e-fa311434f6d5"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 02:37:24.444543 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:37:24.444134    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8f08f681-a03c-49b8-a118-f7101aaeb69c-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "8f08f681-a03c-49b8-a118-f7101aaeb69c" (UID: "8f08f681-a03c-49b8-a118-f7101aaeb69c"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 02:37:24.444543 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:37:24.444128    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8f08f681-a03c-49b8-a118-f7101aaeb69c-tls-certs\") pod \"8f08f681-a03c-49b8-a118-f7101aaeb69c\" (UID: \"8f08f681-a03c-49b8-a118-f7101aaeb69c\") "
Sep 16 02:37:24.444543 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:37:24.444252    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8f08f681-a03c-49b8-a118-f7101aaeb69c-home" (OuterVolumeSpecName: "home") pod "8f08f681-a03c-49b8-a118-f7101aaeb69c" (UID: "8f08f681-a03c-49b8-a118-f7101aaeb69c"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 02:37:24.444543 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:37:24.444341    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cc9488c9-649c-4992-9c1e-fa311434f6d5-home" (OuterVolumeSpecName: "home") pod "cc9488c9-649c-4992-9c1e-fa311434f6d5" (UID: "cc9488c9-649c-4992-9c1e-fa311434f6d5"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 02:37:24.444927 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:37:24.444599    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cc9488c9-649c-4992-9c1e-fa311434f6d5-model-cache" (OuterVolumeSpecName: "model-cache") pod "cc9488c9-649c-4992-9c1e-fa311434f6d5" (UID: "cc9488c9-649c-4992-9c1e-fa311434f6d5"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 02:37:24.444927 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:37:24.444613    2583 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8f08f681-a03c-49b8-a118-f7101aaeb69c-home\") on node \"ip-10-0-128-111.ec2.internal\" DevicePath \"\""
Sep 16 02:37:24.444927 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:37:24.444635    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j7p2f\" (UniqueName: \"kubernetes.io/projected/02f506a2-c921-462d-afc7-5b47c3b30d54-kube-api-access-j7p2f\") on node \"ip-10-0-128-111.ec2.internal\" DevicePath \"\""
Sep 16 02:37:24.444927 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:37:24.444651    2583 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8f08f681-a03c-49b8-a118-f7101aaeb69c-model-cache\") on node \"ip-10-0-128-111.ec2.internal\" DevicePath \"\""
Sep 16 02:37:24.444927 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:37:24.444665    2583 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8f08f681-a03c-49b8-a118-f7101aaeb69c-tmp-dir\") on node \"ip-10-0-128-111.ec2.internal\" DevicePath \"\""
Sep 16 02:37:24.444927 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:37:24.444678    2583 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cc9488c9-649c-4992-9c1e-fa311434f6d5-tmp-dir\") on node \"ip-10-0-128-111.ec2.internal\" DevicePath \"\""
Sep 16 02:37:24.444927 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:37:24.444690    2583 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/02f506a2-c921-462d-afc7-5b47c3b30d54-dshm\") on node \"ip-10-0-128-111.ec2.internal\" DevicePath \"\""
Sep 16 02:37:24.444927 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:37:24.444703    2583 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/02f506a2-c921-462d-afc7-5b47c3b30d54-tls-certs\") on node \"ip-10-0-128-111.ec2.internal\" DevicePath \"\""
Sep 16 02:37:24.444927 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:37:24.444715    2583 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cc9488c9-649c-4992-9c1e-fa311434f6d5-home\") on node \"ip-10-0-128-111.ec2.internal\" DevicePath \"\""
Sep 16 02:37:24.446782 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:37:24.446734    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cc9488c9-649c-4992-9c1e-fa311434f6d5-dshm" (OuterVolumeSpecName: "dshm") pod "cc9488c9-649c-4992-9c1e-fa311434f6d5" (UID: "cc9488c9-649c-4992-9c1e-fa311434f6d5"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 02:37:24.446782 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:37:24.446734    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cc9488c9-649c-4992-9c1e-fa311434f6d5-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "cc9488c9-649c-4992-9c1e-fa311434f6d5" (UID: "cc9488c9-649c-4992-9c1e-fa311434f6d5"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 02:37:24.446782 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:37:24.446742    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8f08f681-a03c-49b8-a118-f7101aaeb69c-dshm" (OuterVolumeSpecName: "dshm") pod "8f08f681-a03c-49b8-a118-f7101aaeb69c" (UID: "8f08f681-a03c-49b8-a118-f7101aaeb69c"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 02:37:24.446964 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:37:24.446790    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8f08f681-a03c-49b8-a118-f7101aaeb69c-kube-api-access-ncqf5" (OuterVolumeSpecName: "kube-api-access-ncqf5") pod "8f08f681-a03c-49b8-a118-f7101aaeb69c" (UID: "8f08f681-a03c-49b8-a118-f7101aaeb69c"). InnerVolumeSpecName "kube-api-access-ncqf5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 02:37:24.446964 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:37:24.446791    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8f08f681-a03c-49b8-a118-f7101aaeb69c-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "8f08f681-a03c-49b8-a118-f7101aaeb69c" (UID: "8f08f681-a03c-49b8-a118-f7101aaeb69c"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 02:37:24.446964 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:37:24.446826    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cc9488c9-649c-4992-9c1e-fa311434f6d5-kube-api-access-d2zqt" (OuterVolumeSpecName: "kube-api-access-d2zqt") pod "cc9488c9-649c-4992-9c1e-fa311434f6d5" (UID: "cc9488c9-649c-4992-9c1e-fa311434f6d5"). InnerVolumeSpecName "kube-api-access-d2zqt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 02:37:24.478656 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:37:24.478623    2583 generic.go:356] "Generic (PLEG): container finished" podID="cc9488c9-649c-4992-9c1e-fa311434f6d5" containerID="00b3a10f8f77a6594565222cf05f2e6a5c46c5c6d787fa20b60f2968449e0690" exitCode=0
Sep 16 02:37:24.478785 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:37:24.478671    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-a3d7eb/simulator-w3-a3d7eb-kserve-7bcf597646-zj6qt"
Sep 16 02:37:24.478785 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:37:24.478716    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-a3d7eb/simulator-w3-a3d7eb-kserve-7bcf597646-zj6qt" event={"ID":"cc9488c9-649c-4992-9c1e-fa311434f6d5","Type":"ContainerDied","Data":"00b3a10f8f77a6594565222cf05f2e6a5c46c5c6d787fa20b60f2968449e0690"}
Sep 16 02:37:24.478785 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:37:24.478760    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-a3d7eb/simulator-w3-a3d7eb-kserve-7bcf597646-zj6qt" event={"ID":"cc9488c9-649c-4992-9c1e-fa311434f6d5","Type":"ContainerDied","Data":"adab4e626ea014bb2d443fcdd54f7147954b8dc560d89c83aaf31dd2e51e7e32"}
Sep 16 02:37:24.478785 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:37:24.478777    2583 scope.go:117] "RemoveContainer" containerID="00b3a10f8f77a6594565222cf05f2e6a5c46c5c6d787fa20b60f2968449e0690"
Sep 16 02:37:24.480032 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:37:24.480011    2583 generic.go:356] "Generic (PLEG): container finished" podID="02f506a2-c921-462d-afc7-5b47c3b30d54" containerID="863217e97f1eccace04ca34e2287861a1a0fb5a39df7ed20866b1097ea95de7f" exitCode=0
Sep 16 02:37:24.480123 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:37:24.480059    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-a3d7eb/distinct-2-w3-a3d7eb-kserve-7b76c4fc77-pg8fv"
Sep 16 02:37:24.480123 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:37:24.480088    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-a3d7eb/distinct-2-w3-a3d7eb-kserve-7b76c4fc77-pg8fv" event={"ID":"02f506a2-c921-462d-afc7-5b47c3b30d54","Type":"ContainerDied","Data":"863217e97f1eccace04ca34e2287861a1a0fb5a39df7ed20866b1097ea95de7f"}
Sep 16 02:37:24.480257 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:37:24.480131    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-a3d7eb/distinct-2-w3-a3d7eb-kserve-7b76c4fc77-pg8fv" event={"ID":"02f506a2-c921-462d-afc7-5b47c3b30d54","Type":"ContainerDied","Data":"3ff118d28adfff41990e0f12e715e72e28b5364592e3b287c6bc524562ddbbfd"}
Sep 16 02:37:24.481466 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:37:24.481443    2583 generic.go:356] "Generic (PLEG): container finished" podID="8f08f681-a03c-49b8-a118-f7101aaeb69c" containerID="eb619e491b6b4e343d5c2c90000de4c82e79eba46adce04bbb7a5c546158c33f" exitCode=0
Sep 16 02:37:24.481589 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:37:24.481485    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-a3d7eb/distinct-w3-a3d7eb-kserve-79d99dc9d6-mt2wx" event={"ID":"8f08f681-a03c-49b8-a118-f7101aaeb69c","Type":"ContainerDied","Data":"eb619e491b6b4e343d5c2c90000de4c82e79eba46adce04bbb7a5c546158c33f"}
Sep 16 02:37:24.481589 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:37:24.481513    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-a3d7eb/distinct-w3-a3d7eb-kserve-79d99dc9d6-mt2wx" event={"ID":"8f08f681-a03c-49b8-a118-f7101aaeb69c","Type":"ContainerDied","Data":"13869204e7c859c103a9186a6c1c90678cfe0e946c7d48f72e60f53f4a2ecff3"}
Sep 16 02:37:24.481589 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:37:24.481531    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-a3d7eb/distinct-w3-a3d7eb-kserve-79d99dc9d6-mt2wx"
Sep 16 02:37:24.487924 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:37:24.487907    2583 scope.go:117] "RemoveContainer" containerID="00b3a10f8f77a6594565222cf05f2e6a5c46c5c6d787fa20b60f2968449e0690"
Sep 16 02:37:24.488205 ip-10-0-128-111 kubenswrapper[2583]: E0916 02:37:24.488188    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"00b3a10f8f77a6594565222cf05f2e6a5c46c5c6d787fa20b60f2968449e0690\": container with ID starting with 00b3a10f8f77a6594565222cf05f2e6a5c46c5c6d787fa20b60f2968449e0690 not found: ID does not exist" containerID="00b3a10f8f77a6594565222cf05f2e6a5c46c5c6d787fa20b60f2968449e0690"
Sep 16 02:37:24.488258 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:37:24.488213    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"00b3a10f8f77a6594565222cf05f2e6a5c46c5c6d787fa20b60f2968449e0690"} err="failed to get container status \"00b3a10f8f77a6594565222cf05f2e6a5c46c5c6d787fa20b60f2968449e0690\": rpc error: code = NotFound desc = could not find container \"00b3a10f8f77a6594565222cf05f2e6a5c46c5c6d787fa20b60f2968449e0690\": container with ID starting with 00b3a10f8f77a6594565222cf05f2e6a5c46c5c6d787fa20b60f2968449e0690 not found: ID does not exist"
Sep 16 02:37:24.488258 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:37:24.488229    2583 scope.go:117] "RemoveContainer" containerID="863217e97f1eccace04ca34e2287861a1a0fb5a39df7ed20866b1097ea95de7f"
Sep 16 02:37:24.496879 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:37:24.496862    2583 scope.go:117] "RemoveContainer" containerID="863217e97f1eccace04ca34e2287861a1a0fb5a39df7ed20866b1097ea95de7f"
Sep 16 02:37:24.497100 ip-10-0-128-111 kubenswrapper[2583]: E0916 02:37:24.497082    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"863217e97f1eccace04ca34e2287861a1a0fb5a39df7ed20866b1097ea95de7f\": container with ID starting with 863217e97f1eccace04ca34e2287861a1a0fb5a39df7ed20866b1097ea95de7f not found: ID does not exist" containerID="863217e97f1eccace04ca34e2287861a1a0fb5a39df7ed20866b1097ea95de7f"
Sep 16 02:37:24.497186 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:37:24.497106    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"863217e97f1eccace04ca34e2287861a1a0fb5a39df7ed20866b1097ea95de7f"} err="failed to get container status \"863217e97f1eccace04ca34e2287861a1a0fb5a39df7ed20866b1097ea95de7f\": rpc error: code = NotFound desc = could not find container \"863217e97f1eccace04ca34e2287861a1a0fb5a39df7ed20866b1097ea95de7f\": container with ID starting with 863217e97f1eccace04ca34e2287861a1a0fb5a39df7ed20866b1097ea95de7f not found: ID does not exist"
Sep 16 02:37:24.497186 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:37:24.497122    2583 scope.go:117] "RemoveContainer" containerID="eb619e491b6b4e343d5c2c90000de4c82e79eba46adce04bbb7a5c546158c33f"
Sep 16 02:37:24.504653 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:37:24.504633    2583 scope.go:117] "RemoveContainer" containerID="eb619e491b6b4e343d5c2c90000de4c82e79eba46adce04bbb7a5c546158c33f"
Sep 16 02:37:24.504938 ip-10-0-128-111 kubenswrapper[2583]: E0916 02:37:24.504919    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"eb619e491b6b4e343d5c2c90000de4c82e79eba46adce04bbb7a5c546158c33f\": container with ID starting with eb619e491b6b4e343d5c2c90000de4c82e79eba46adce04bbb7a5c546158c33f not found: ID does not exist" containerID="eb619e491b6b4e343d5c2c90000de4c82e79eba46adce04bbb7a5c546158c33f"
Sep 16 02:37:24.505002 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:37:24.504943    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"eb619e491b6b4e343d5c2c90000de4c82e79eba46adce04bbb7a5c546158c33f"} err="failed to get container status \"eb619e491b6b4e343d5c2c90000de4c82e79eba46adce04bbb7a5c546158c33f\": rpc error: code = NotFound desc = could not find container \"eb619e491b6b4e343d5c2c90000de4c82e79eba46adce04bbb7a5c546158c33f\": container with ID starting with eb619e491b6b4e343d5c2c90000de4c82e79eba46adce04bbb7a5c546158c33f not found: ID does not exist"
Sep 16 02:37:24.506402 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:37:24.506382    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-a3d7eb/distinct-2-w3-a3d7eb-kserve-7b76c4fc77-pg8fv"]
Sep 16 02:37:24.509281 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:37:24.509262    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w3-a3d7eb/distinct-2-w3-a3d7eb-kserve-7b76c4fc77-pg8fv"]
Sep 16 02:37:24.518116 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:37:24.518094    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-a3d7eb/simulator-w3-a3d7eb-kserve-7bcf597646-zj6qt"]
Sep 16 02:37:24.521950 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:37:24.521930    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w3-a3d7eb/simulator-w3-a3d7eb-kserve-7bcf597646-zj6qt"]
Sep 16 02:37:24.530926 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:37:24.530903    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-a3d7eb/distinct-w3-a3d7eb-kserve-79d99dc9d6-mt2wx"]
Sep 16 02:37:24.534136 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:37:24.534117    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w3-a3d7eb/distinct-w3-a3d7eb-kserve-79d99dc9d6-mt2wx"]
Sep 16 02:37:24.545152 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:37:24.545133    2583 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cc9488c9-649c-4992-9c1e-fa311434f6d5-tls-certs\") on node \"ip-10-0-128-111.ec2.internal\" DevicePath \"\""
Sep 16 02:37:24.545152 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:37:24.545150    2583 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cc9488c9-649c-4992-9c1e-fa311434f6d5-model-cache\") on node \"ip-10-0-128-111.ec2.internal\" DevicePath \"\""
Sep 16 02:37:24.545250 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:37:24.545158    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d2zqt\" (UniqueName: \"kubernetes.io/projected/cc9488c9-649c-4992-9c1e-fa311434f6d5-kube-api-access-d2zqt\") on node \"ip-10-0-128-111.ec2.internal\" DevicePath \"\""
Sep 16 02:37:24.545250 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:37:24.545169    2583 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cc9488c9-649c-4992-9c1e-fa311434f6d5-dshm\") on node \"ip-10-0-128-111.ec2.internal\" DevicePath \"\""
Sep 16 02:37:24.545250 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:37:24.545178    2583 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8f08f681-a03c-49b8-a118-f7101aaeb69c-dshm\") on node \"ip-10-0-128-111.ec2.internal\" DevicePath \"\""
Sep 16 02:37:24.545250 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:37:24.545185    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ncqf5\" (UniqueName: \"kubernetes.io/projected/8f08f681-a03c-49b8-a118-f7101aaeb69c-kube-api-access-ncqf5\") on node \"ip-10-0-128-111.ec2.internal\" DevicePath \"\""
Sep 16 02:37:24.545250 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:37:24.545194    2583 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8f08f681-a03c-49b8-a118-f7101aaeb69c-tls-certs\") on node \"ip-10-0-128-111.ec2.internal\" DevicePath \"\""
Sep 16 02:37:25.452530 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:37:25.452492    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-fzl8t"]
Sep 16 02:37:26.045711 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:37:26.045676    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="02f506a2-c921-462d-afc7-5b47c3b30d54" path="/var/lib/kubelet/pods/02f506a2-c921-462d-afc7-5b47c3b30d54/volumes"
Sep 16 02:37:26.046102 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:37:26.046089    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8f08f681-a03c-49b8-a118-f7101aaeb69c" path="/var/lib/kubelet/pods/8f08f681-a03c-49b8-a118-f7101aaeb69c/volumes"
Sep 16 02:37:26.046447 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:37:26.046434    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="cc9488c9-649c-4992-9c1e-fa311434f6d5" path="/var/lib/kubelet/pods/cc9488c9-649c-4992-9c1e-fa311434f6d5/volumes"
Sep 16 02:37:29.237931 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:37:29.237896    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-5b59aa/embedding-w0-5b59aa-kserve-6ff5887d57-zv75q"]
Sep 16 02:37:29.238445 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:37:29.238305    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8f08f681-a03c-49b8-a118-f7101aaeb69c" containerName="main"
Sep 16 02:37:29.238445 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:37:29.238317    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="8f08f681-a03c-49b8-a118-f7101aaeb69c" containerName="main"
Sep 16 02:37:29.238445 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:37:29.238326    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cc9488c9-649c-4992-9c1e-fa311434f6d5" containerName="main"
Sep 16 02:37:29.238445 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:37:29.238333    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="cc9488c9-649c-4992-9c1e-fa311434f6d5" containerName="main"
Sep 16 02:37:29.238445 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:37:29.238353    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="02336b88-6308-4c8c-aa5f-e3e77e950b33" containerName="payload-processing"
Sep 16 02:37:29.238445 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:37:29.238360    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="02336b88-6308-4c8c-aa5f-e3e77e950b33" containerName="payload-processing"
Sep 16 02:37:29.238445 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:37:29.238377    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a5bebfdc-6ac7-4ce6-b12d-7bff79424333" containerName="revoke-keys"
Sep 16 02:37:29.238445 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:37:29.238382    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="a5bebfdc-6ac7-4ce6-b12d-7bff79424333" containerName="revoke-keys"
Sep 16 02:37:29.238445 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:37:29.238391    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="02f506a2-c921-462d-afc7-5b47c3b30d54" containerName="main"
Sep 16 02:37:29.238445 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:37:29.238397    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="02f506a2-c921-462d-afc7-5b47c3b30d54" containerName="main"
Sep 16 02:37:29.238445 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:37:29.238411    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a006042d-eddd-4564-923e-04e0f548d58c" containerName="payload-pre-processing"
Sep 16 02:37:29.238445 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:37:29.238418    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="a006042d-eddd-4564-923e-04e0f548d58c" containerName="payload-pre-processing"
Sep 16 02:37:29.238971 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:37:29.238484    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="a006042d-eddd-4564-923e-04e0f548d58c" containerName="payload-pre-processing"
Sep 16 02:37:29.238971 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:37:29.238492    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="cc9488c9-649c-4992-9c1e-fa311434f6d5" containerName="main"
Sep 16 02:37:29.238971 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:37:29.238503    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="02f506a2-c921-462d-afc7-5b47c3b30d54" containerName="main"
Sep 16 02:37:29.238971 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:37:29.238512    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="a5bebfdc-6ac7-4ce6-b12d-7bff79424333" containerName="revoke-keys"
Sep 16 02:37:29.238971 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:37:29.238521    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="8f08f681-a03c-49b8-a118-f7101aaeb69c" containerName="main"
Sep 16 02:37:29.238971 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:37:29.238532    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="02336b88-6308-4c8c-aa5f-e3e77e950b33" containerName="payload-processing"
Sep 16 02:37:29.243083 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:37:29.243066    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-5b59aa/embedding-w0-5b59aa-kserve-6ff5887d57-zv75q"
Sep 16 02:37:29.245477 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:37:29.245452    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-5b59aa\"/\"kube-root-ca.crt\""
Sep 16 02:37:29.246599 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:37:29.246580    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-5b59aa\"/\"embedding-w0-5b59aa-kserve-self-signed-certs\""
Sep 16 02:37:29.246713 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:37:29.246633    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-5b59aa\"/\"default-dockercfg-w688b\""
Sep 16 02:37:29.246713 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:37:29.246664    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-5b59aa\"/\"openshift-service-ca.crt\""
Sep 16 02:37:29.251253 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:37:29.251233    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-5b59aa/embedding-w0-5b59aa-kserve-6ff5887d57-zv75q"]
Sep 16 02:37:29.281282 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:37:29.281249    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/163e3ae9-a733-4434-b17c-5b08a7d2af76-home\") pod \"embedding-w0-5b59aa-kserve-6ff5887d57-zv75q\" (UID: \"163e3ae9-a733-4434-b17c-5b08a7d2af76\") " pod="e2e-models-e2e-worker-w0-5b59aa/embedding-w0-5b59aa-kserve-6ff5887d57-zv75q"
Sep 16 02:37:29.281430 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:37:29.281293    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/163e3ae9-a733-4434-b17c-5b08a7d2af76-tmp-dir\") pod \"embedding-w0-5b59aa-kserve-6ff5887d57-zv75q\" (UID: \"163e3ae9-a733-4434-b17c-5b08a7d2af76\") " pod="e2e-models-e2e-worker-w0-5b59aa/embedding-w0-5b59aa-kserve-6ff5887d57-zv75q"
Sep 16 02:37:29.281430 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:37:29.281389    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/163e3ae9-a733-4434-b17c-5b08a7d2af76-dshm\") pod \"embedding-w0-5b59aa-kserve-6ff5887d57-zv75q\" (UID: \"163e3ae9-a733-4434-b17c-5b08a7d2af76\") " pod="e2e-models-e2e-worker-w0-5b59aa/embedding-w0-5b59aa-kserve-6ff5887d57-zv75q"
Sep 16 02:37:29.281430 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:37:29.281420    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wrxfs\" (UniqueName: \"kubernetes.io/projected/163e3ae9-a733-4434-b17c-5b08a7d2af76-kube-api-access-wrxfs\") pod \"embedding-w0-5b59aa-kserve-6ff5887d57-zv75q\" (UID: \"163e3ae9-a733-4434-b17c-5b08a7d2af76\") " pod="e2e-models-e2e-worker-w0-5b59aa/embedding-w0-5b59aa-kserve-6ff5887d57-zv75q"
Sep 16 02:37:29.281540 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:37:29.281441    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/163e3ae9-a733-4434-b17c-5b08a7d2af76-model-cache\") pod \"embedding-w0-5b59aa-kserve-6ff5887d57-zv75q\" (UID: \"163e3ae9-a733-4434-b17c-5b08a7d2af76\") " pod="e2e-models-e2e-worker-w0-5b59aa/embedding-w0-5b59aa-kserve-6ff5887d57-zv75q"
Sep 16 02:37:29.281540 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:37:29.281481    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/163e3ae9-a733-4434-b17c-5b08a7d2af76-tls-certs\") pod \"embedding-w0-5b59aa-kserve-6ff5887d57-zv75q\" (UID: \"163e3ae9-a733-4434-b17c-5b08a7d2af76\") " pod="e2e-models-e2e-worker-w0-5b59aa/embedding-w0-5b59aa-kserve-6ff5887d57-zv75q"
Sep 16 02:37:29.382753 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:37:29.382701    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/163e3ae9-a733-4434-b17c-5b08a7d2af76-home\") pod \"embedding-w0-5b59aa-kserve-6ff5887d57-zv75q\" (UID: \"163e3ae9-a733-4434-b17c-5b08a7d2af76\") " pod="e2e-models-e2e-worker-w0-5b59aa/embedding-w0-5b59aa-kserve-6ff5887d57-zv75q"
Sep 16 02:37:29.382753 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:37:29.382760    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/163e3ae9-a733-4434-b17c-5b08a7d2af76-tmp-dir\") pod \"embedding-w0-5b59aa-kserve-6ff5887d57-zv75q\" (UID: \"163e3ae9-a733-4434-b17c-5b08a7d2af76\") " pod="e2e-models-e2e-worker-w0-5b59aa/embedding-w0-5b59aa-kserve-6ff5887d57-zv75q"
Sep 16 02:37:29.383016 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:37:29.382819    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/163e3ae9-a733-4434-b17c-5b08a7d2af76-dshm\") pod \"embedding-w0-5b59aa-kserve-6ff5887d57-zv75q\" (UID: \"163e3ae9-a733-4434-b17c-5b08a7d2af76\") " pod="e2e-models-e2e-worker-w0-5b59aa/embedding-w0-5b59aa-kserve-6ff5887d57-zv75q"
Sep 16 02:37:29.383016 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:37:29.382849    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wrxfs\" (UniqueName: \"kubernetes.io/projected/163e3ae9-a733-4434-b17c-5b08a7d2af76-kube-api-access-wrxfs\") pod \"embedding-w0-5b59aa-kserve-6ff5887d57-zv75q\" (UID: \"163e3ae9-a733-4434-b17c-5b08a7d2af76\") " pod="e2e-models-e2e-worker-w0-5b59aa/embedding-w0-5b59aa-kserve-6ff5887d57-zv75q"
Sep 16 02:37:29.383016 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:37:29.382876    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/163e3ae9-a733-4434-b17c-5b08a7d2af76-model-cache\") pod \"embedding-w0-5b59aa-kserve-6ff5887d57-zv75q\" (UID: \"163e3ae9-a733-4434-b17c-5b08a7d2af76\") " pod="e2e-models-e2e-worker-w0-5b59aa/embedding-w0-5b59aa-kserve-6ff5887d57-zv75q"
Sep 16 02:37:29.383016 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:37:29.383002    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/163e3ae9-a733-4434-b17c-5b08a7d2af76-tls-certs\") pod \"embedding-w0-5b59aa-kserve-6ff5887d57-zv75q\" (UID: \"163e3ae9-a733-4434-b17c-5b08a7d2af76\") " pod="e2e-models-e2e-worker-w0-5b59aa/embedding-w0-5b59aa-kserve-6ff5887d57-zv75q"
Sep 16 02:37:29.383215 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:37:29.383126    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/163e3ae9-a733-4434-b17c-5b08a7d2af76-home\") pod \"embedding-w0-5b59aa-kserve-6ff5887d57-zv75q\" (UID: \"163e3ae9-a733-4434-b17c-5b08a7d2af76\") " pod="e2e-models-e2e-worker-w0-5b59aa/embedding-w0-5b59aa-kserve-6ff5887d57-zv75q"
Sep 16 02:37:29.383215 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:37:29.383171    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/163e3ae9-a733-4434-b17c-5b08a7d2af76-tmp-dir\") pod \"embedding-w0-5b59aa-kserve-6ff5887d57-zv75q\" (UID: \"163e3ae9-a733-4434-b17c-5b08a7d2af76\") " pod="e2e-models-e2e-worker-w0-5b59aa/embedding-w0-5b59aa-kserve-6ff5887d57-zv75q"
Sep 16 02:37:29.383287 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:37:29.383236    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/163e3ae9-a733-4434-b17c-5b08a7d2af76-model-cache\") pod \"embedding-w0-5b59aa-kserve-6ff5887d57-zv75q\" (UID: \"163e3ae9-a733-4434-b17c-5b08a7d2af76\") " pod="e2e-models-e2e-worker-w0-5b59aa/embedding-w0-5b59aa-kserve-6ff5887d57-zv75q"
Sep 16 02:37:29.385051 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:37:29.385028    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/163e3ae9-a733-4434-b17c-5b08a7d2af76-dshm\") pod \"embedding-w0-5b59aa-kserve-6ff5887d57-zv75q\" (UID: \"163e3ae9-a733-4434-b17c-5b08a7d2af76\") " pod="e2e-models-e2e-worker-w0-5b59aa/embedding-w0-5b59aa-kserve-6ff5887d57-zv75q"
Sep 16 02:37:29.385306 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:37:29.385286    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/163e3ae9-a733-4434-b17c-5b08a7d2af76-tls-certs\") pod \"embedding-w0-5b59aa-kserve-6ff5887d57-zv75q\" (UID: \"163e3ae9-a733-4434-b17c-5b08a7d2af76\") " pod="e2e-models-e2e-worker-w0-5b59aa/embedding-w0-5b59aa-kserve-6ff5887d57-zv75q"
Sep 16 02:37:29.390055 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:37:29.390021    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wrxfs\" (UniqueName: \"kubernetes.io/projected/163e3ae9-a733-4434-b17c-5b08a7d2af76-kube-api-access-wrxfs\") pod \"embedding-w0-5b59aa-kserve-6ff5887d57-zv75q\" (UID: \"163e3ae9-a733-4434-b17c-5b08a7d2af76\") " pod="e2e-models-e2e-worker-w0-5b59aa/embedding-w0-5b59aa-kserve-6ff5887d57-zv75q"
Sep 16 02:37:29.553431 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:37:29.553330    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-5b59aa/embedding-w0-5b59aa-kserve-6ff5887d57-zv75q"
Sep 16 02:37:29.823187 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:37:29.823080    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-5b59aa/embedding-w0-5b59aa-kserve-6ff5887d57-zv75q"]
Sep 16 02:37:29.825634 ip-10-0-128-111 kubenswrapper[2583]: W0916 02:37:29.825599    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod163e3ae9_a733_4434_b17c_5b08a7d2af76.slice/crio-f04d1151248316e18f2f147af64a131c58c03923837abeeb75fa052844345c5c WatchSource:0}: Error finding container f04d1151248316e18f2f147af64a131c58c03923837abeeb75fa052844345c5c: Status 404 returned error can't find the container with id f04d1151248316e18f2f147af64a131c58c03923837abeeb75fa052844345c5c
Sep 16 02:37:30.517222 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:37:30.517189    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-5b59aa/embedding-w0-5b59aa-kserve-6ff5887d57-zv75q" event={"ID":"163e3ae9-a733-4434-b17c-5b08a7d2af76","Type":"ContainerStarted","Data":"70fc9751edf748003d6e5fff6cfb209adbf41241fc9b3e053e4420b42eb346fc"}
Sep 16 02:37:30.517625 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:37:30.517234    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-5b59aa/embedding-w0-5b59aa-kserve-6ff5887d57-zv75q" event={"ID":"163e3ae9-a733-4434-b17c-5b08a7d2af76","Type":"ContainerStarted","Data":"f04d1151248316e18f2f147af64a131c58c03923837abeeb75fa052844345c5c"}
Sep 16 02:37:30.532989 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:37:30.532943    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-5b59aa/embedding-w0-5b59aa-kserve-6ff5887d57-zv75q" podStartSLOduration=1.532927517 podStartE2EDuration="1.532927517s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 02:37:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 02:37:30.531018115 +0000 UTC m=+1128.998806897" watchObservedRunningTime="2026-09-16 02:37:30.532927517 +0000 UTC m=+1129.000716338"
Sep 16 02:37:39.553786 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:37:39.553747    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-5b59aa/embedding-w0-5b59aa-kserve-6ff5887d57-zv75q"
Sep 16 02:37:39.553786 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:37:39.553794    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-5b59aa/embedding-w0-5b59aa-kserve-6ff5887d57-zv75q"
Sep 16 02:37:39.567362 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:37:39.567332    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-5b59aa/embedding-w0-5b59aa-kserve-6ff5887d57-zv75q"
Sep 16 02:37:39.578925 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:37:39.578903    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-5b59aa/embedding-w0-5b59aa-kserve-6ff5887d57-zv75q"
Sep 16 02:37:52.150484 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:37:52.150449    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-fzl8t"]
Sep 16 02:38:11.549133 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:38:11.549053    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-fzl8t"]
Sep 16 02:38:33.251861 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:38:33.251824    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-fzl8t"]
Sep 16 02:38:34.047979 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:38:34.047948    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-5b59aa/embedding-w0-5b59aa-kserve-6ff5887d57-zv75q"]
Sep 16 02:38:44.548700 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:38:44.548654    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-5b59aa-786d4c9486-fp9xw"]
Sep 16 02:38:44.549278 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:38:44.548904    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-5b59aa-786d4c9486-fp9xw" podUID="11497a78-de4b-493b-b889-420919a02a69" containerName="payload-pre-processing" containerID="cri-o://d0001a11fdf1b930520cc823d425ab6b8697f27d3af5bf5885602a3176bf4dae" gracePeriod=30
Sep 16 02:38:44.789939 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:38:44.789916    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-5b59aa-786d4c9486-fp9xw"
Sep 16 02:38:44.895196 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:38:44.895159    2583 generic.go:356] "Generic (PLEG): container finished" podID="11497a78-de4b-493b-b889-420919a02a69" containerID="d0001a11fdf1b930520cc823d425ab6b8697f27d3af5bf5885602a3176bf4dae" exitCode=0
Sep 16 02:38:44.895369 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:38:44.895201    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-5b59aa-786d4c9486-fp9xw" event={"ID":"11497a78-de4b-493b-b889-420919a02a69","Type":"ContainerDied","Data":"d0001a11fdf1b930520cc823d425ab6b8697f27d3af5bf5885602a3176bf4dae"}
Sep 16 02:38:44.895369 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:38:44.895204    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-5b59aa-786d4c9486-fp9xw"
Sep 16 02:38:44.895369 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:38:44.895242    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-5b59aa-786d4c9486-fp9xw" event={"ID":"11497a78-de4b-493b-b889-420919a02a69","Type":"ContainerDied","Data":"f1d9d8fe72c03d6667ccb1e7f799e2148b10fb10d627c177acc0582956cef25d"}
Sep 16 02:38:44.895369 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:38:44.895264    2583 scope.go:117] "RemoveContainer" containerID="d0001a11fdf1b930520cc823d425ab6b8697f27d3af5bf5885602a3176bf4dae"
Sep 16 02:38:44.903905 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:38:44.903888    2583 scope.go:117] "RemoveContainer" containerID="d0001a11fdf1b930520cc823d425ab6b8697f27d3af5bf5885602a3176bf4dae"
Sep 16 02:38:44.904167 ip-10-0-128-111 kubenswrapper[2583]: E0916 02:38:44.904145    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d0001a11fdf1b930520cc823d425ab6b8697f27d3af5bf5885602a3176bf4dae\": container with ID starting with d0001a11fdf1b930520cc823d425ab6b8697f27d3af5bf5885602a3176bf4dae not found: ID does not exist" containerID="d0001a11fdf1b930520cc823d425ab6b8697f27d3af5bf5885602a3176bf4dae"
Sep 16 02:38:44.904249 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:38:44.904178    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d0001a11fdf1b930520cc823d425ab6b8697f27d3af5bf5885602a3176bf4dae"} err="failed to get container status \"d0001a11fdf1b930520cc823d425ab6b8697f27d3af5bf5885602a3176bf4dae\": rpc error: code = NotFound desc = could not find container \"d0001a11fdf1b930520cc823d425ab6b8697f27d3af5bf5885602a3176bf4dae\": container with ID starting with d0001a11fdf1b930520cc823d425ab6b8697f27d3af5bf5885602a3176bf4dae not found: ID does not exist"
Sep 16 02:38:44.905441 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:38:44.905424    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qsk2z\" (UniqueName: \"kubernetes.io/projected/11497a78-de4b-493b-b889-420919a02a69-kube-api-access-qsk2z\") pod \"11497a78-de4b-493b-b889-420919a02a69\" (UID: \"11497a78-de4b-493b-b889-420919a02a69\") "
Sep 16 02:38:44.905492 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:38:44.905480    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/11497a78-de4b-493b-b889-420919a02a69-plugins-config-volume\") pod \"11497a78-de4b-493b-b889-420919a02a69\" (UID: \"11497a78-de4b-493b-b889-420919a02a69\") "
Sep 16 02:38:44.905861 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:38:44.905838    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/11497a78-de4b-493b-b889-420919a02a69-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "11497a78-de4b-493b-b889-420919a02a69" (UID: "11497a78-de4b-493b-b889-420919a02a69"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 02:38:44.907524 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:38:44.907506    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/11497a78-de4b-493b-b889-420919a02a69-kube-api-access-qsk2z" (OuterVolumeSpecName: "kube-api-access-qsk2z") pod "11497a78-de4b-493b-b889-420919a02a69" (UID: "11497a78-de4b-493b-b889-420919a02a69"). InnerVolumeSpecName "kube-api-access-qsk2z". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 02:38:45.006825 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:38:45.006798    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qsk2z\" (UniqueName: \"kubernetes.io/projected/11497a78-de4b-493b-b889-420919a02a69-kube-api-access-qsk2z\") on node \"ip-10-0-128-111.ec2.internal\" DevicePath \"\""
Sep 16 02:38:45.006825 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:38:45.006823    2583 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/11497a78-de4b-493b-b889-420919a02a69-plugins-config-volume\") on node \"ip-10-0-128-111.ec2.internal\" DevicePath \"\""
Sep 16 02:38:45.215299 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:38:45.215266    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-5b59aa-786d4c9486-fp9xw"]
Sep 16 02:38:45.217787 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:38:45.217762    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-5b59aa-786d4c9486-fp9xw"]
Sep 16 02:38:46.046266 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:38:46.046232    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="11497a78-de4b-493b-b889-420919a02a69" path="/var/lib/kubelet/pods/11497a78-de4b-493b-b889-420919a02a69/volumes"
Sep 16 02:38:49.068527 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:38:49.068482    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-5b59aa/embedding-w0-5b59aa-kserve-6ff5887d57-zv75q" podUID="163e3ae9-a733-4434-b17c-5b08a7d2af76" containerName="main" containerID="cri-o://70fc9751edf748003d6e5fff6cfb209adbf41241fc9b3e053e4420b42eb346fc" gracePeriod=45
Sep 16 02:38:49.311735 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:38:49.311710    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-5b59aa/embedding-w0-5b59aa-kserve-6ff5887d57-zv75q"
Sep 16 02:38:49.348406 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:38:49.348379    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/163e3ae9-a733-4434-b17c-5b08a7d2af76-tls-certs\") pod \"163e3ae9-a733-4434-b17c-5b08a7d2af76\" (UID: \"163e3ae9-a733-4434-b17c-5b08a7d2af76\") "
Sep 16 02:38:49.348616 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:38:49.348452    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/163e3ae9-a733-4434-b17c-5b08a7d2af76-tmp-dir\") pod \"163e3ae9-a733-4434-b17c-5b08a7d2af76\" (UID: \"163e3ae9-a733-4434-b17c-5b08a7d2af76\") "
Sep 16 02:38:49.348616 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:38:49.348471    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/163e3ae9-a733-4434-b17c-5b08a7d2af76-home\") pod \"163e3ae9-a733-4434-b17c-5b08a7d2af76\" (UID: \"163e3ae9-a733-4434-b17c-5b08a7d2af76\") "
Sep 16 02:38:49.348616 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:38:49.348489    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/163e3ae9-a733-4434-b17c-5b08a7d2af76-dshm\") pod \"163e3ae9-a733-4434-b17c-5b08a7d2af76\" (UID: \"163e3ae9-a733-4434-b17c-5b08a7d2af76\") "
Sep 16 02:38:49.348616 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:38:49.348561    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/163e3ae9-a733-4434-b17c-5b08a7d2af76-model-cache\") pod \"163e3ae9-a733-4434-b17c-5b08a7d2af76\" (UID: \"163e3ae9-a733-4434-b17c-5b08a7d2af76\") "
Sep 16 02:38:49.348861 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:38:49.348625    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wrxfs\" (UniqueName: \"kubernetes.io/projected/163e3ae9-a733-4434-b17c-5b08a7d2af76-kube-api-access-wrxfs\") pod \"163e3ae9-a733-4434-b17c-5b08a7d2af76\" (UID: \"163e3ae9-a733-4434-b17c-5b08a7d2af76\") "
Sep 16 02:38:49.348861 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:38:49.348781    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/163e3ae9-a733-4434-b17c-5b08a7d2af76-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "163e3ae9-a733-4434-b17c-5b08a7d2af76" (UID: "163e3ae9-a733-4434-b17c-5b08a7d2af76"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 02:38:49.348861 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:38:49.348794    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/163e3ae9-a733-4434-b17c-5b08a7d2af76-home" (OuterVolumeSpecName: "home") pod "163e3ae9-a733-4434-b17c-5b08a7d2af76" (UID: "163e3ae9-a733-4434-b17c-5b08a7d2af76"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 02:38:49.349032 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:38:49.348903    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/163e3ae9-a733-4434-b17c-5b08a7d2af76-model-cache" (OuterVolumeSpecName: "model-cache") pod "163e3ae9-a733-4434-b17c-5b08a7d2af76" (UID: "163e3ae9-a733-4434-b17c-5b08a7d2af76"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 02:38:49.349032 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:38:49.348997    2583 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/163e3ae9-a733-4434-b17c-5b08a7d2af76-tmp-dir\") on node \"ip-10-0-128-111.ec2.internal\" DevicePath \"\""
Sep 16 02:38:49.349032 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:38:49.349018    2583 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/163e3ae9-a733-4434-b17c-5b08a7d2af76-home\") on node \"ip-10-0-128-111.ec2.internal\" DevicePath \"\""
Sep 16 02:38:49.349190 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:38:49.349033    2583 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/163e3ae9-a733-4434-b17c-5b08a7d2af76-model-cache\") on node \"ip-10-0-128-111.ec2.internal\" DevicePath \"\""
Sep 16 02:38:49.350848 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:38:49.350822    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/163e3ae9-a733-4434-b17c-5b08a7d2af76-dshm" (OuterVolumeSpecName: "dshm") pod "163e3ae9-a733-4434-b17c-5b08a7d2af76" (UID: "163e3ae9-a733-4434-b17c-5b08a7d2af76"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 02:38:49.351026 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:38:49.350999    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/163e3ae9-a733-4434-b17c-5b08a7d2af76-kube-api-access-wrxfs" (OuterVolumeSpecName: "kube-api-access-wrxfs") pod "163e3ae9-a733-4434-b17c-5b08a7d2af76" (UID: "163e3ae9-a733-4434-b17c-5b08a7d2af76"). InnerVolumeSpecName "kube-api-access-wrxfs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 02:38:49.351121 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:38:49.351015    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/163e3ae9-a733-4434-b17c-5b08a7d2af76-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "163e3ae9-a733-4434-b17c-5b08a7d2af76" (UID: "163e3ae9-a733-4434-b17c-5b08a7d2af76"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 02:38:49.450526 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:38:49.450489    2583 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/163e3ae9-a733-4434-b17c-5b08a7d2af76-tls-certs\") on node \"ip-10-0-128-111.ec2.internal\" DevicePath \"\""
Sep 16 02:38:49.450526 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:38:49.450520    2583 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/163e3ae9-a733-4434-b17c-5b08a7d2af76-dshm\") on node \"ip-10-0-128-111.ec2.internal\" DevicePath \"\""
Sep 16 02:38:49.450526 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:38:49.450532    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wrxfs\" (UniqueName: \"kubernetes.io/projected/163e3ae9-a733-4434-b17c-5b08a7d2af76-kube-api-access-wrxfs\") on node \"ip-10-0-128-111.ec2.internal\" DevicePath \"\""
Sep 16 02:38:49.926072 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:38:49.926042    2583 generic.go:356] "Generic (PLEG): container finished" podID="163e3ae9-a733-4434-b17c-5b08a7d2af76" containerID="70fc9751edf748003d6e5fff6cfb209adbf41241fc9b3e053e4420b42eb346fc" exitCode=0
Sep 16 02:38:49.926262 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:38:49.926084    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-5b59aa/embedding-w0-5b59aa-kserve-6ff5887d57-zv75q"
Sep 16 02:38:49.926262 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:38:49.926092    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-5b59aa/embedding-w0-5b59aa-kserve-6ff5887d57-zv75q" event={"ID":"163e3ae9-a733-4434-b17c-5b08a7d2af76","Type":"ContainerDied","Data":"70fc9751edf748003d6e5fff6cfb209adbf41241fc9b3e053e4420b42eb346fc"}
Sep 16 02:38:49.926262 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:38:49.926187    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-5b59aa/embedding-w0-5b59aa-kserve-6ff5887d57-zv75q" event={"ID":"163e3ae9-a733-4434-b17c-5b08a7d2af76","Type":"ContainerDied","Data":"f04d1151248316e18f2f147af64a131c58c03923837abeeb75fa052844345c5c"}
Sep 16 02:38:49.926262 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:38:49.926204    2583 scope.go:117] "RemoveContainer" containerID="70fc9751edf748003d6e5fff6cfb209adbf41241fc9b3e053e4420b42eb346fc"
Sep 16 02:38:49.935504 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:38:49.935485    2583 scope.go:117] "RemoveContainer" containerID="70fc9751edf748003d6e5fff6cfb209adbf41241fc9b3e053e4420b42eb346fc"
Sep 16 02:38:49.935787 ip-10-0-128-111 kubenswrapper[2583]: E0916 02:38:49.935770    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"70fc9751edf748003d6e5fff6cfb209adbf41241fc9b3e053e4420b42eb346fc\": container with ID starting with 70fc9751edf748003d6e5fff6cfb209adbf41241fc9b3e053e4420b42eb346fc not found: ID does not exist" containerID="70fc9751edf748003d6e5fff6cfb209adbf41241fc9b3e053e4420b42eb346fc"
Sep 16 02:38:49.935848 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:38:49.935800    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"70fc9751edf748003d6e5fff6cfb209adbf41241fc9b3e053e4420b42eb346fc"} err="failed to get container status \"70fc9751edf748003d6e5fff6cfb209adbf41241fc9b3e053e4420b42eb346fc\": rpc error: code = NotFound desc = could not find container \"70fc9751edf748003d6e5fff6cfb209adbf41241fc9b3e053e4420b42eb346fc\": container with ID starting with 70fc9751edf748003d6e5fff6cfb209adbf41241fc9b3e053e4420b42eb346fc not found: ID does not exist"
Sep 16 02:38:49.951059 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:38:49.951025    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-5b59aa/embedding-w0-5b59aa-kserve-6ff5887d57-zv75q"]
Sep 16 02:38:49.953526 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:38:49.953503    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-5b59aa/embedding-w0-5b59aa-kserve-6ff5887d57-zv75q"]
Sep 16 02:38:50.045328 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:38:50.045299    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="163e3ae9-a733-4434-b17c-5b08a7d2af76" path="/var/lib/kubelet/pods/163e3ae9-a733-4434-b17c-5b08a7d2af76/volumes"
Sep 16 02:38:51.817048 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:38:51.817008    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-5b59aa-openshift-default-ff449c75d-6ckh5"]
Sep 16 02:38:51.817580 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:38:51.817255    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-5b59aa-openshift-default-ff449c75d-6ckh5" podUID="3a21aba3-2483-46a3-8b7f-acac0d646664" containerName="istio-proxy" containerID="cri-o://3c162bd7fdd33b01c85157b6d75034927fa5f72fc3e6bc6938cac244272ad9db" gracePeriod=30
Sep 16 02:38:51.913308 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:38:51.913272    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-efdec406-f765b5bf-rbj7z"]
Sep 16 02:38:51.913880 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:38:51.913858    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="163e3ae9-a733-4434-b17c-5b08a7d2af76" containerName="main"
Sep 16 02:38:51.913931 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:38:51.913885    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="163e3ae9-a733-4434-b17c-5b08a7d2af76" containerName="main"
Sep 16 02:38:51.913931 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:38:51.913913    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="11497a78-de4b-493b-b889-420919a02a69" containerName="payload-pre-processing"
Sep 16 02:38:51.913931 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:38:51.913923    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="11497a78-de4b-493b-b889-420919a02a69" containerName="payload-pre-processing"
Sep 16 02:38:51.914038 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:38:51.914019    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="163e3ae9-a733-4434-b17c-5b08a7d2af76" containerName="main"
Sep 16 02:38:51.914080 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:38:51.914045    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="11497a78-de4b-493b-b889-420919a02a69" containerName="payload-pre-processing"
Sep 16 02:38:51.918653 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:38:51.918635    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-efdec406-f765b5bf-rbj7z"
Sep 16 02:38:51.921201 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:38:51.921177    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-efdec406-dockercfg-vr25h\""
Sep 16 02:38:51.921201 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:38:51.921187    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-efdec406\""
Sep 16 02:38:51.926196 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:38:51.926176    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-efdec406-f765b5bf-rbj7z"]
Sep 16 02:38:51.974124 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:38:51.974092    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a8fae9c0-0ad9-4f73-a6ed-e6d092d33eda-plugins-config-volume\") pod \"payload-processing-e2e-mt-efdec406-f765b5bf-rbj7z\" (UID: \"a8fae9c0-0ad9-4f73-a6ed-e6d092d33eda\") " pod="openshift-ingress/payload-processing-e2e-mt-efdec406-f765b5bf-rbj7z"
Sep 16 02:38:51.974124 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:38:51.974128    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kzxfv\" (UniqueName: \"kubernetes.io/projected/a8fae9c0-0ad9-4f73-a6ed-e6d092d33eda-kube-api-access-kzxfv\") pod \"payload-processing-e2e-mt-efdec406-f765b5bf-rbj7z\" (UID: \"a8fae9c0-0ad9-4f73-a6ed-e6d092d33eda\") " pod="openshift-ingress/payload-processing-e2e-mt-efdec406-f765b5bf-rbj7z"
Sep 16 02:38:52.075702 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:38:52.075621    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a8fae9c0-0ad9-4f73-a6ed-e6d092d33eda-plugins-config-volume\") pod \"payload-processing-e2e-mt-efdec406-f765b5bf-rbj7z\" (UID: \"a8fae9c0-0ad9-4f73-a6ed-e6d092d33eda\") " pod="openshift-ingress/payload-processing-e2e-mt-efdec406-f765b5bf-rbj7z"
Sep 16 02:38:52.075702 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:38:52.075661    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kzxfv\" (UniqueName: \"kubernetes.io/projected/a8fae9c0-0ad9-4f73-a6ed-e6d092d33eda-kube-api-access-kzxfv\") pod \"payload-processing-e2e-mt-efdec406-f765b5bf-rbj7z\" (UID: \"a8fae9c0-0ad9-4f73-a6ed-e6d092d33eda\") " pod="openshift-ingress/payload-processing-e2e-mt-efdec406-f765b5bf-rbj7z"
Sep 16 02:38:52.076303 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:38:52.076283    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a8fae9c0-0ad9-4f73-a6ed-e6d092d33eda-plugins-config-volume\") pod \"payload-processing-e2e-mt-efdec406-f765b5bf-rbj7z\" (UID: \"a8fae9c0-0ad9-4f73-a6ed-e6d092d33eda\") " pod="openshift-ingress/payload-processing-e2e-mt-efdec406-f765b5bf-rbj7z"
Sep 16 02:38:52.083588 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:38:52.083541    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kzxfv\" (UniqueName: \"kubernetes.io/projected/a8fae9c0-0ad9-4f73-a6ed-e6d092d33eda-kube-api-access-kzxfv\") pod \"payload-processing-e2e-mt-efdec406-f765b5bf-rbj7z\" (UID: \"a8fae9c0-0ad9-4f73-a6ed-e6d092d33eda\") " pod="openshift-ingress/payload-processing-e2e-mt-efdec406-f765b5bf-rbj7z"
Sep 16 02:38:52.230133 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:38:52.230100    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-efdec406-f765b5bf-rbj7z"
Sep 16 02:38:52.501195 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:38:52.501172    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-efdec406-f765b5bf-rbj7z"]
Sep 16 02:38:52.503253 ip-10-0-128-111 kubenswrapper[2583]: W0916 02:38:52.503212    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda8fae9c0_0ad9_4f73_a6ed_e6d092d33eda.slice/crio-99897746379a9808b714e0471b494421f58ead3558cce6eaac6bfa3da788b186 WatchSource:0}: Error finding container 99897746379a9808b714e0471b494421f58ead3558cce6eaac6bfa3da788b186: Status 404 returned error can't find the container with id 99897746379a9808b714e0471b494421f58ead3558cce6eaac6bfa3da788b186
Sep 16 02:38:52.757380 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:38:52.757292    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-fzl8t"]
Sep 16 02:38:52.951355 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:38:52.951320    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-efdec406-f765b5bf-rbj7z" event={"ID":"a8fae9c0-0ad9-4f73-a6ed-e6d092d33eda","Type":"ContainerStarted","Data":"df747ac28169d824f8026edb98f6d6df6891dfb9d9dc0a7cd35a079fff3f45be"}
Sep 16 02:38:52.951749 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:38:52.951371    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-efdec406-f765b5bf-rbj7z" event={"ID":"a8fae9c0-0ad9-4f73-a6ed-e6d092d33eda","Type":"ContainerStarted","Data":"99897746379a9808b714e0471b494421f58ead3558cce6eaac6bfa3da788b186"}
Sep 16 02:38:52.951749 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:38:52.951459    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-mt-efdec406-f765b5bf-rbj7z"
Sep 16 02:38:52.967287 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:38:52.967243    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-mt-efdec406-f765b5bf-rbj7z" podStartSLOduration=1.967230062 podStartE2EDuration="1.967230062s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 02:38:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 02:38:52.965435686 +0000 UTC m=+1211.433224470" watchObservedRunningTime="2026-09-16 02:38:52.967230062 +0000 UTC m=+1211.435018844"
Sep 16 02:38:56.975626 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:38:56.975587    2583 generic.go:356] "Generic (PLEG): container finished" podID="3a21aba3-2483-46a3-8b7f-acac0d646664" containerID="3c162bd7fdd33b01c85157b6d75034927fa5f72fc3e6bc6938cac244272ad9db" exitCode=0
Sep 16 02:38:56.976023 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:38:56.975644    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-5b59aa-openshift-default-ff449c75d-6ckh5" event={"ID":"3a21aba3-2483-46a3-8b7f-acac0d646664","Type":"ContainerDied","Data":"3c162bd7fdd33b01c85157b6d75034927fa5f72fc3e6bc6938cac244272ad9db"}
Sep 16 02:38:56.976023 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:38:56.975705    2583 scope.go:117] "RemoveContainer" containerID="491b4b4e6dadc31073436a80f16e053c12b47ac8e6b4e1965b8e476f0a458623"
Sep 16 02:38:57.075505 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:38:57.075480    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-5b59aa-openshift-default-ff449c75d-6ckh5"
Sep 16 02:38:57.224112 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:38:57.224076    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3a21aba3-2483-46a3-8b7f-acac0d646664-workload-certs\") pod \"3a21aba3-2483-46a3-8b7f-acac0d646664\" (UID: \"3a21aba3-2483-46a3-8b7f-acac0d646664\") "
Sep 16 02:38:57.224112 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:38:57.224119    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3a21aba3-2483-46a3-8b7f-acac0d646664-istio-envoy\") pod \"3a21aba3-2483-46a3-8b7f-acac0d646664\" (UID: \"3a21aba3-2483-46a3-8b7f-acac0d646664\") "
Sep 16 02:38:57.224357 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:38:57.224152    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3a21aba3-2483-46a3-8b7f-acac0d646664-istio-data\") pod \"3a21aba3-2483-46a3-8b7f-acac0d646664\" (UID: \"3a21aba3-2483-46a3-8b7f-acac0d646664\") "
Sep 16 02:38:57.224357 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:38:57.224175    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3a21aba3-2483-46a3-8b7f-acac0d646664-istio-token\") pod \"3a21aba3-2483-46a3-8b7f-acac0d646664\" (UID: \"3a21aba3-2483-46a3-8b7f-acac0d646664\") "
Sep 16 02:38:57.224357 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:38:57.224202    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rhn8r\" (UniqueName: \"kubernetes.io/projected/3a21aba3-2483-46a3-8b7f-acac0d646664-kube-api-access-rhn8r\") pod \"3a21aba3-2483-46a3-8b7f-acac0d646664\" (UID: \"3a21aba3-2483-46a3-8b7f-acac0d646664\") "
Sep 16 02:38:57.224509 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:38:57.224354    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3a21aba3-2483-46a3-8b7f-acac0d646664-workload-socket\") pod \"3a21aba3-2483-46a3-8b7f-acac0d646664\" (UID: \"3a21aba3-2483-46a3-8b7f-acac0d646664\") "
Sep 16 02:38:57.224509 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:38:57.224387    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3a21aba3-2483-46a3-8b7f-acac0d646664-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "3a21aba3-2483-46a3-8b7f-acac0d646664" (UID: "3a21aba3-2483-46a3-8b7f-acac0d646664"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 02:38:57.224509 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:38:57.224408    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3a21aba3-2483-46a3-8b7f-acac0d646664-credential-socket\") pod \"3a21aba3-2483-46a3-8b7f-acac0d646664\" (UID: \"3a21aba3-2483-46a3-8b7f-acac0d646664\") "
Sep 16 02:38:57.224509 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:38:57.224418    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3a21aba3-2483-46a3-8b7f-acac0d646664-istio-data" (OuterVolumeSpecName: "istio-data") pod "3a21aba3-2483-46a3-8b7f-acac0d646664" (UID: "3a21aba3-2483-46a3-8b7f-acac0d646664"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 02:38:57.224509 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:38:57.224453    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3a21aba3-2483-46a3-8b7f-acac0d646664-istio-podinfo\") pod \"3a21aba3-2483-46a3-8b7f-acac0d646664\" (UID: \"3a21aba3-2483-46a3-8b7f-acac0d646664\") "
Sep 16 02:38:57.224509 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:38:57.224486    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3a21aba3-2483-46a3-8b7f-acac0d646664-istiod-ca-cert\") pod \"3a21aba3-2483-46a3-8b7f-acac0d646664\" (UID: \"3a21aba3-2483-46a3-8b7f-acac0d646664\") "
Sep 16 02:38:57.224862 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:38:57.224659    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3a21aba3-2483-46a3-8b7f-acac0d646664-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "3a21aba3-2483-46a3-8b7f-acac0d646664" (UID: "3a21aba3-2483-46a3-8b7f-acac0d646664"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 02:38:57.224862 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:38:57.224674    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3a21aba3-2483-46a3-8b7f-acac0d646664-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "3a21aba3-2483-46a3-8b7f-acac0d646664" (UID: "3a21aba3-2483-46a3-8b7f-acac0d646664"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 02:38:57.224980 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:38:57.224867    2583 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3a21aba3-2483-46a3-8b7f-acac0d646664-workload-certs\") on node \"ip-10-0-128-111.ec2.internal\" DevicePath \"\""
Sep 16 02:38:57.224980 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:38:57.224888    2583 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3a21aba3-2483-46a3-8b7f-acac0d646664-istio-data\") on node \"ip-10-0-128-111.ec2.internal\" DevicePath \"\""
Sep 16 02:38:57.224980 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:38:57.224900    2583 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3a21aba3-2483-46a3-8b7f-acac0d646664-workload-socket\") on node \"ip-10-0-128-111.ec2.internal\" DevicePath \"\""
Sep 16 02:38:57.224980 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:38:57.224914    2583 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3a21aba3-2483-46a3-8b7f-acac0d646664-credential-socket\") on node \"ip-10-0-128-111.ec2.internal\" DevicePath \"\""
Sep 16 02:38:57.224980 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:38:57.224864    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3a21aba3-2483-46a3-8b7f-acac0d646664-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "3a21aba3-2483-46a3-8b7f-acac0d646664" (UID: "3a21aba3-2483-46a3-8b7f-acac0d646664"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 02:38:57.226489 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:38:57.226461    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3a21aba3-2483-46a3-8b7f-acac0d646664-kube-api-access-rhn8r" (OuterVolumeSpecName: "kube-api-access-rhn8r") pod "3a21aba3-2483-46a3-8b7f-acac0d646664" (UID: "3a21aba3-2483-46a3-8b7f-acac0d646664"). InnerVolumeSpecName "kube-api-access-rhn8r". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 02:38:57.226627 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:38:57.226486    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3a21aba3-2483-46a3-8b7f-acac0d646664-istio-token" (OuterVolumeSpecName: "istio-token") pod "3a21aba3-2483-46a3-8b7f-acac0d646664" (UID: "3a21aba3-2483-46a3-8b7f-acac0d646664"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 02:38:57.226627 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:38:57.226580    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3a21aba3-2483-46a3-8b7f-acac0d646664-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "3a21aba3-2483-46a3-8b7f-acac0d646664" (UID: "3a21aba3-2483-46a3-8b7f-acac0d646664"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 02:38:57.226627 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:38:57.226620    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/3a21aba3-2483-46a3-8b7f-acac0d646664-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "3a21aba3-2483-46a3-8b7f-acac0d646664" (UID: "3a21aba3-2483-46a3-8b7f-acac0d646664"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 16 02:38:57.326291 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:38:57.326256    2583 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3a21aba3-2483-46a3-8b7f-acac0d646664-istio-podinfo\") on node \"ip-10-0-128-111.ec2.internal\" DevicePath \"\""
Sep 16 02:38:57.326291 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:38:57.326286    2583 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3a21aba3-2483-46a3-8b7f-acac0d646664-istiod-ca-cert\") on node \"ip-10-0-128-111.ec2.internal\" DevicePath \"\""
Sep 16 02:38:57.326291 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:38:57.326295    2583 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3a21aba3-2483-46a3-8b7f-acac0d646664-istio-envoy\") on node \"ip-10-0-128-111.ec2.internal\" DevicePath \"\""
Sep 16 02:38:57.326507 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:38:57.326303    2583 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3a21aba3-2483-46a3-8b7f-acac0d646664-istio-token\") on node \"ip-10-0-128-111.ec2.internal\" DevicePath \"\""
Sep 16 02:38:57.326507 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:38:57.326311    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rhn8r\" (UniqueName: \"kubernetes.io/projected/3a21aba3-2483-46a3-8b7f-acac0d646664-kube-api-access-rhn8r\") on node \"ip-10-0-128-111.ec2.internal\" DevicePath \"\""
Sep 16 02:38:57.982127 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:38:57.982091    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-5b59aa-openshift-default-ff449c75d-6ckh5"
Sep 16 02:38:57.982594 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:38:57.982129    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-5b59aa-openshift-default-ff449c75d-6ckh5" event={"ID":"3a21aba3-2483-46a3-8b7f-acac0d646664","Type":"ContainerDied","Data":"27c353be0c77316295e3fade9bdaf85dc2c08dbc5063b5190bea071bf2ecf421"}
Sep 16 02:38:57.982594 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:38:57.982188    2583 scope.go:117] "RemoveContainer" containerID="3c162bd7fdd33b01c85157b6d75034927fa5f72fc3e6bc6938cac244272ad9db"
Sep 16 02:38:58.006186 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:38:58.006148    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-5b59aa-openshift-default-ff449c75d-6ckh5"]
Sep 16 02:38:58.018185 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:38:58.018159    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-5b59aa-openshift-default-ff449c75d-6ckh5"]
Sep 16 02:38:58.047389 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:38:58.047359    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3a21aba3-2483-46a3-8b7f-acac0d646664" path="/var/lib/kubelet/pods/3a21aba3-2483-46a3-8b7f-acac0d646664/volumes"
Sep 16 02:39:03.959481 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:39:03.959448    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-mt-efdec406-f765b5bf-rbj7z"
Sep 16 02:39:11.849080 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:39:11.849041    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-fzl8t"]
Sep 16 02:39:27.547299 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:39:27.547268    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-fzl8t"]
Sep 16 02:39:32.317716 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:39:32.317684    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-efdec406-f765b5bf-rbj7z"]
Sep 16 02:39:32.318256 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:39:32.317937    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-mt-efdec406-f765b5bf-rbj7z" podUID="a8fae9c0-0ad9-4f73-a6ed-e6d092d33eda" containerName="payload-processing" containerID="cri-o://df747ac28169d824f8026edb98f6d6df6891dfb9d9dc0a7cd35a079fff3f45be" gracePeriod=30
Sep 16 02:39:32.577272 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:39:32.577210    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-efdec406-f765b5bf-rbj7z"
Sep 16 02:39:32.636461 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:39:32.636432    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kzxfv\" (UniqueName: \"kubernetes.io/projected/a8fae9c0-0ad9-4f73-a6ed-e6d092d33eda-kube-api-access-kzxfv\") pod \"a8fae9c0-0ad9-4f73-a6ed-e6d092d33eda\" (UID: \"a8fae9c0-0ad9-4f73-a6ed-e6d092d33eda\") "
Sep 16 02:39:32.636461 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:39:32.636466    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a8fae9c0-0ad9-4f73-a6ed-e6d092d33eda-plugins-config-volume\") pod \"a8fae9c0-0ad9-4f73-a6ed-e6d092d33eda\" (UID: \"a8fae9c0-0ad9-4f73-a6ed-e6d092d33eda\") "
Sep 16 02:39:32.636889 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:39:32.636867    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a8fae9c0-0ad9-4f73-a6ed-e6d092d33eda-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "a8fae9c0-0ad9-4f73-a6ed-e6d092d33eda" (UID: "a8fae9c0-0ad9-4f73-a6ed-e6d092d33eda"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 02:39:32.638445 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:39:32.638426    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a8fae9c0-0ad9-4f73-a6ed-e6d092d33eda-kube-api-access-kzxfv" (OuterVolumeSpecName: "kube-api-access-kzxfv") pod "a8fae9c0-0ad9-4f73-a6ed-e6d092d33eda" (UID: "a8fae9c0-0ad9-4f73-a6ed-e6d092d33eda"). InnerVolumeSpecName "kube-api-access-kzxfv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 02:39:32.737309 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:39:32.737280    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kzxfv\" (UniqueName: \"kubernetes.io/projected/a8fae9c0-0ad9-4f73-a6ed-e6d092d33eda-kube-api-access-kzxfv\") on node \"ip-10-0-128-111.ec2.internal\" DevicePath \"\""
Sep 16 02:39:32.737309 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:39:32.737306    2583 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a8fae9c0-0ad9-4f73-a6ed-e6d092d33eda-plugins-config-volume\") on node \"ip-10-0-128-111.ec2.internal\" DevicePath \"\""
Sep 16 02:39:33.149716 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:39:33.149681    2583 generic.go:356] "Generic (PLEG): container finished" podID="a8fae9c0-0ad9-4f73-a6ed-e6d092d33eda" containerID="df747ac28169d824f8026edb98f6d6df6891dfb9d9dc0a7cd35a079fff3f45be" exitCode=0
Sep 16 02:39:33.149885 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:39:33.149722    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-efdec406-f765b5bf-rbj7z"
Sep 16 02:39:33.149885 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:39:33.149766    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-efdec406-f765b5bf-rbj7z" event={"ID":"a8fae9c0-0ad9-4f73-a6ed-e6d092d33eda","Type":"ContainerDied","Data":"df747ac28169d824f8026edb98f6d6df6891dfb9d9dc0a7cd35a079fff3f45be"}
Sep 16 02:39:33.149885 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:39:33.149818    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-efdec406-f765b5bf-rbj7z" event={"ID":"a8fae9c0-0ad9-4f73-a6ed-e6d092d33eda","Type":"ContainerDied","Data":"99897746379a9808b714e0471b494421f58ead3558cce6eaac6bfa3da788b186"}
Sep 16 02:39:33.149885 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:39:33.149843    2583 scope.go:117] "RemoveContainer" containerID="df747ac28169d824f8026edb98f6d6df6891dfb9d9dc0a7cd35a079fff3f45be"
Sep 16 02:39:33.166544 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:39:33.166523    2583 scope.go:117] "RemoveContainer" containerID="df747ac28169d824f8026edb98f6d6df6891dfb9d9dc0a7cd35a079fff3f45be"
Sep 16 02:39:33.166860 ip-10-0-128-111 kubenswrapper[2583]: E0916 02:39:33.166835    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"df747ac28169d824f8026edb98f6d6df6891dfb9d9dc0a7cd35a079fff3f45be\": container with ID starting with df747ac28169d824f8026edb98f6d6df6891dfb9d9dc0a7cd35a079fff3f45be not found: ID does not exist" containerID="df747ac28169d824f8026edb98f6d6df6891dfb9d9dc0a7cd35a079fff3f45be"
Sep 16 02:39:33.166954 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:39:33.166864    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"df747ac28169d824f8026edb98f6d6df6891dfb9d9dc0a7cd35a079fff3f45be"} err="failed to get container status \"df747ac28169d824f8026edb98f6d6df6891dfb9d9dc0a7cd35a079fff3f45be\": rpc error: code = NotFound desc = could not find container \"df747ac28169d824f8026edb98f6d6df6891dfb9d9dc0a7cd35a079fff3f45be\": container with ID starting with df747ac28169d824f8026edb98f6d6df6891dfb9d9dc0a7cd35a079fff3f45be not found: ID does not exist"
Sep 16 02:39:33.174807 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:39:33.174782    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-efdec406-f765b5bf-rbj7z"]
Sep 16 02:39:33.178136 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:39:33.178114    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-efdec406-f765b5bf-rbj7z"]
Sep 16 02:39:34.045110 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:39:34.045027    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a8fae9c0-0ad9-4f73-a6ed-e6d092d33eda" path="/var/lib/kubelet/pods/a8fae9c0-0ad9-4f73-a6ed-e6d092d33eda/volumes"
Sep 16 02:39:42.450193 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:39:42.450098    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-fzl8t"]
Sep 16 02:39:53.932364 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:39:53.932332    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w2-db1ed7/auth-test-model-db1ed7-kserve-5474d84c75-q8j7z"]
Sep 16 02:39:55.457842 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:39:55.457809    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-fzl8t"]
Sep 16 02:39:58.483667 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:39:58.483635    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w1-fe9d17-6b5dbf5695-rxrld"]
Sep 16 02:39:58.484052 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:39:58.483974    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3a21aba3-2483-46a3-8b7f-acac0d646664" containerName="istio-proxy"
Sep 16 02:39:58.484052 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:39:58.483985    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="3a21aba3-2483-46a3-8b7f-acac0d646664" containerName="istio-proxy"
Sep 16 02:39:58.484052 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:39:58.484001    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3a21aba3-2483-46a3-8b7f-acac0d646664" containerName="istio-proxy"
Sep 16 02:39:58.484052 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:39:58.484006    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="3a21aba3-2483-46a3-8b7f-acac0d646664" containerName="istio-proxy"
Sep 16 02:39:58.484052 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:39:58.484020    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a8fae9c0-0ad9-4f73-a6ed-e6d092d33eda" containerName="payload-processing"
Sep 16 02:39:58.484052 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:39:58.484026    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="a8fae9c0-0ad9-4f73-a6ed-e6d092d33eda" containerName="payload-processing"
Sep 16 02:39:58.484279 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:39:58.484075    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="a8fae9c0-0ad9-4f73-a6ed-e6d092d33eda" containerName="payload-processing"
Sep 16 02:39:58.484279 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:39:58.484083    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="3a21aba3-2483-46a3-8b7f-acac0d646664" containerName="istio-proxy"
Sep 16 02:39:58.484279 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:39:58.484089    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="3a21aba3-2483-46a3-8b7f-acac0d646664" containerName="istio-proxy"
Sep 16 02:39:58.488363 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:39:58.488347    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-fe9d17-6b5dbf5695-rxrld"
Sep 16 02:39:58.490739 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:39:58.490711    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w1-fe9d17-dockercfg-xqstt\""
Sep 16 02:39:58.490883 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:39:58.490747    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w1-fe9d17\""
Sep 16 02:39:58.496094 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:39:58.496076    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w1-fe9d17-6b5dbf5695-rxrld"]
Sep 16 02:39:58.568020 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:39:58.567984    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1828a8ad-c4ec-4f54-b6e7-f34bc6735457-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w1-fe9d17-6b5dbf5695-rxrld\" (UID: \"1828a8ad-c4ec-4f54-b6e7-f34bc6735457\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w1-fe9d17-6b5dbf5695-rxrld"
Sep 16 02:39:58.568203 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:39:58.568040    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mszgq\" (UniqueName: \"kubernetes.io/projected/1828a8ad-c4ec-4f54-b6e7-f34bc6735457-kube-api-access-mszgq\") pod \"payload-processing-e2e-shared-a-w1-fe9d17-6b5dbf5695-rxrld\" (UID: \"1828a8ad-c4ec-4f54-b6e7-f34bc6735457\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w1-fe9d17-6b5dbf5695-rxrld"
Sep 16 02:39:58.669367 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:39:58.669328    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mszgq\" (UniqueName: \"kubernetes.io/projected/1828a8ad-c4ec-4f54-b6e7-f34bc6735457-kube-api-access-mszgq\") pod \"payload-processing-e2e-shared-a-w1-fe9d17-6b5dbf5695-rxrld\" (UID: \"1828a8ad-c4ec-4f54-b6e7-f34bc6735457\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w1-fe9d17-6b5dbf5695-rxrld"
Sep 16 02:39:58.669524 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:39:58.669416    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1828a8ad-c4ec-4f54-b6e7-f34bc6735457-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w1-fe9d17-6b5dbf5695-rxrld\" (UID: \"1828a8ad-c4ec-4f54-b6e7-f34bc6735457\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w1-fe9d17-6b5dbf5695-rxrld"
Sep 16 02:39:58.670081 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:39:58.670062    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1828a8ad-c4ec-4f54-b6e7-f34bc6735457-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w1-fe9d17-6b5dbf5695-rxrld\" (UID: \"1828a8ad-c4ec-4f54-b6e7-f34bc6735457\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w1-fe9d17-6b5dbf5695-rxrld"
Sep 16 02:39:58.676800 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:39:58.676773    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mszgq\" (UniqueName: \"kubernetes.io/projected/1828a8ad-c4ec-4f54-b6e7-f34bc6735457-kube-api-access-mszgq\") pod \"payload-processing-e2e-shared-a-w1-fe9d17-6b5dbf5695-rxrld\" (UID: \"1828a8ad-c4ec-4f54-b6e7-f34bc6735457\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w1-fe9d17-6b5dbf5695-rxrld"
Sep 16 02:39:58.799350 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:39:58.799262    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-fe9d17-6b5dbf5695-rxrld"
Sep 16 02:39:59.066783 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:39:59.066583    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w1-fe9d17-6b5dbf5695-rxrld"]
Sep 16 02:39:59.069654 ip-10-0-128-111 kubenswrapper[2583]: W0916 02:39:59.069623    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1828a8ad_c4ec_4f54_b6e7_f34bc6735457.slice/crio-5d29658a5d0ca8821ba3c474dcd2e9912885f64b3a8a87fca1a046c6976249ba WatchSource:0}: Error finding container 5d29658a5d0ca8821ba3c474dcd2e9912885f64b3a8a87fca1a046c6976249ba: Status 404 returned error can't find the container with id 5d29658a5d0ca8821ba3c474dcd2e9912885f64b3a8a87fca1a046c6976249ba
Sep 16 02:39:59.273626 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:39:59.273591    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-fe9d17-6b5dbf5695-rxrld" event={"ID":"1828a8ad-c4ec-4f54-b6e7-f34bc6735457","Type":"ContainerStarted","Data":"3586f74a0c5982b88161f5009a889a2eab9401871ccc1e37fe0b446a09418011"}
Sep 16 02:39:59.273973 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:39:59.273644    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-fe9d17-6b5dbf5695-rxrld" event={"ID":"1828a8ad-c4ec-4f54-b6e7-f34bc6735457","Type":"ContainerStarted","Data":"5d29658a5d0ca8821ba3c474dcd2e9912885f64b3a8a87fca1a046c6976249ba"}
Sep 16 02:39:59.273973 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:39:59.273711    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-fe9d17-6b5dbf5695-rxrld"
Sep 16 02:39:59.288158 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:39:59.288110    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-fe9d17-6b5dbf5695-rxrld" podStartSLOduration=1.288094185 podStartE2EDuration="1.288094185s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 02:39:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 02:39:59.286856673 +0000 UTC m=+1277.754645455" watchObservedRunningTime="2026-09-16 02:39:59.288094185 +0000 UTC m=+1277.755882966"
Sep 16 02:40:06.154598 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:40:06.154562    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-fzl8t"]
Sep 16 02:40:08.951932 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:40:08.951869    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w2-db1ed7/auth-test-model-db1ed7-kserve-5474d84c75-q8j7z" podUID="d073b71f-8e1d-4624-9192-08021795bf50" containerName="main" containerID="cri-o://768e589cfbdbab02561853d1f594974fdf7d08e0a071c351a81e250102a48fbd" gracePeriod=45
Sep 16 02:40:09.204058 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:40:09.203999    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w2-db1ed7/auth-test-model-db1ed7-kserve-5474d84c75-q8j7z"
Sep 16 02:40:09.326729 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:40:09.326690    2583 generic.go:356] "Generic (PLEG): container finished" podID="d073b71f-8e1d-4624-9192-08021795bf50" containerID="768e589cfbdbab02561853d1f594974fdf7d08e0a071c351a81e250102a48fbd" exitCode=0
Sep 16 02:40:09.326729 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:40:09.326734    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w2-db1ed7/auth-test-model-db1ed7-kserve-5474d84c75-q8j7z"
Sep 16 02:40:09.327000 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:40:09.326759    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w2-db1ed7/auth-test-model-db1ed7-kserve-5474d84c75-q8j7z" event={"ID":"d073b71f-8e1d-4624-9192-08021795bf50","Type":"ContainerDied","Data":"768e589cfbdbab02561853d1f594974fdf7d08e0a071c351a81e250102a48fbd"}
Sep 16 02:40:09.327000 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:40:09.326805    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w2-db1ed7/auth-test-model-db1ed7-kserve-5474d84c75-q8j7z" event={"ID":"d073b71f-8e1d-4624-9192-08021795bf50","Type":"ContainerDied","Data":"4cb8fbbe070f381984bc3d763ffac6d3037989de6595f50cab4d70af214b284e"}
Sep 16 02:40:09.327000 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:40:09.326821    2583 scope.go:117] "RemoveContainer" containerID="768e589cfbdbab02561853d1f594974fdf7d08e0a071c351a81e250102a48fbd"
Sep 16 02:40:09.334979 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:40:09.334934    2583 scope.go:117] "RemoveContainer" containerID="768e589cfbdbab02561853d1f594974fdf7d08e0a071c351a81e250102a48fbd"
Sep 16 02:40:09.335216 ip-10-0-128-111 kubenswrapper[2583]: E0916 02:40:09.335196    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"768e589cfbdbab02561853d1f594974fdf7d08e0a071c351a81e250102a48fbd\": container with ID starting with 768e589cfbdbab02561853d1f594974fdf7d08e0a071c351a81e250102a48fbd not found: ID does not exist" containerID="768e589cfbdbab02561853d1f594974fdf7d08e0a071c351a81e250102a48fbd"
Sep 16 02:40:09.335285 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:40:09.335230    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"768e589cfbdbab02561853d1f594974fdf7d08e0a071c351a81e250102a48fbd"} err="failed to get container status \"768e589cfbdbab02561853d1f594974fdf7d08e0a071c351a81e250102a48fbd\": rpc error: code = NotFound desc = could not find container \"768e589cfbdbab02561853d1f594974fdf7d08e0a071c351a81e250102a48fbd\": container with ID starting with 768e589cfbdbab02561853d1f594974fdf7d08e0a071c351a81e250102a48fbd not found: ID does not exist"
Sep 16 02:40:09.366967 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:40:09.366937    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d073b71f-8e1d-4624-9192-08021795bf50-dshm\") pod \"d073b71f-8e1d-4624-9192-08021795bf50\" (UID: \"d073b71f-8e1d-4624-9192-08021795bf50\") "
Sep 16 02:40:09.367094 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:40:09.366979    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d073b71f-8e1d-4624-9192-08021795bf50-model-cache\") pod \"d073b71f-8e1d-4624-9192-08021795bf50\" (UID: \"d073b71f-8e1d-4624-9192-08021795bf50\") "
Sep 16 02:40:09.367094 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:40:09.367019    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d073b71f-8e1d-4624-9192-08021795bf50-tmp-dir\") pod \"d073b71f-8e1d-4624-9192-08021795bf50\" (UID: \"d073b71f-8e1d-4624-9192-08021795bf50\") "
Sep 16 02:40:09.367094 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:40:09.367043    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d073b71f-8e1d-4624-9192-08021795bf50-tls-certs\") pod \"d073b71f-8e1d-4624-9192-08021795bf50\" (UID: \"d073b71f-8e1d-4624-9192-08021795bf50\") "
Sep 16 02:40:09.367231 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:40:09.367147    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mkx7d\" (UniqueName: \"kubernetes.io/projected/d073b71f-8e1d-4624-9192-08021795bf50-kube-api-access-mkx7d\") pod \"d073b71f-8e1d-4624-9192-08021795bf50\" (UID: \"d073b71f-8e1d-4624-9192-08021795bf50\") "
Sep 16 02:40:09.367231 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:40:09.367216    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d073b71f-8e1d-4624-9192-08021795bf50-home\") pod \"d073b71f-8e1d-4624-9192-08021795bf50\" (UID: \"d073b71f-8e1d-4624-9192-08021795bf50\") "
Sep 16 02:40:09.367338 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:40:09.367305    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d073b71f-8e1d-4624-9192-08021795bf50-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "d073b71f-8e1d-4624-9192-08021795bf50" (UID: "d073b71f-8e1d-4624-9192-08021795bf50"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 02:40:09.367338 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:40:09.367319    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d073b71f-8e1d-4624-9192-08021795bf50-model-cache" (OuterVolumeSpecName: "model-cache") pod "d073b71f-8e1d-4624-9192-08021795bf50" (UID: "d073b71f-8e1d-4624-9192-08021795bf50"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 02:40:09.367569 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:40:09.367528    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d073b71f-8e1d-4624-9192-08021795bf50-home" (OuterVolumeSpecName: "home") pod "d073b71f-8e1d-4624-9192-08021795bf50" (UID: "d073b71f-8e1d-4624-9192-08021795bf50"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 02:40:09.367660 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:40:09.367569    2583 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d073b71f-8e1d-4624-9192-08021795bf50-model-cache\") on node \"ip-10-0-128-111.ec2.internal\" DevicePath \"\""
Sep 16 02:40:09.367660 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:40:09.367589    2583 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d073b71f-8e1d-4624-9192-08021795bf50-tmp-dir\") on node \"ip-10-0-128-111.ec2.internal\" DevicePath \"\""
Sep 16 02:40:09.369194 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:40:09.369170    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d073b71f-8e1d-4624-9192-08021795bf50-dshm" (OuterVolumeSpecName: "dshm") pod "d073b71f-8e1d-4624-9192-08021795bf50" (UID: "d073b71f-8e1d-4624-9192-08021795bf50"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 02:40:09.369368 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:40:09.369345    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d073b71f-8e1d-4624-9192-08021795bf50-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "d073b71f-8e1d-4624-9192-08021795bf50" (UID: "d073b71f-8e1d-4624-9192-08021795bf50"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 02:40:09.369415 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:40:09.369352    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d073b71f-8e1d-4624-9192-08021795bf50-kube-api-access-mkx7d" (OuterVolumeSpecName: "kube-api-access-mkx7d") pod "d073b71f-8e1d-4624-9192-08021795bf50" (UID: "d073b71f-8e1d-4624-9192-08021795bf50"). InnerVolumeSpecName "kube-api-access-mkx7d". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 02:40:09.468636 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:40:09.468535    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mkx7d\" (UniqueName: \"kubernetes.io/projected/d073b71f-8e1d-4624-9192-08021795bf50-kube-api-access-mkx7d\") on node \"ip-10-0-128-111.ec2.internal\" DevicePath \"\""
Sep 16 02:40:09.468636 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:40:09.468582    2583 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d073b71f-8e1d-4624-9192-08021795bf50-home\") on node \"ip-10-0-128-111.ec2.internal\" DevicePath \"\""
Sep 16 02:40:09.468636 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:40:09.468595    2583 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d073b71f-8e1d-4624-9192-08021795bf50-dshm\") on node \"ip-10-0-128-111.ec2.internal\" DevicePath \"\""
Sep 16 02:40:09.468636 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:40:09.468603    2583 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d073b71f-8e1d-4624-9192-08021795bf50-tls-certs\") on node \"ip-10-0-128-111.ec2.internal\" DevicePath \"\""
Sep 16 02:40:09.648050 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:40:09.648019    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w2-db1ed7/auth-test-model-db1ed7-kserve-5474d84c75-q8j7z"]
Sep 16 02:40:09.651210 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:40:09.651185    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w2-db1ed7/auth-test-model-db1ed7-kserve-5474d84c75-q8j7z"]
Sep 16 02:40:10.045257 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:40:10.045217    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d073b71f-8e1d-4624-9192-08021795bf50" path="/var/lib/kubelet/pods/d073b71f-8e1d-4624-9192-08021795bf50/volumes"
Sep 16 02:40:10.280827 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:40:10.280798    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-fe9d17-6b5dbf5695-rxrld"
Sep 16 02:40:21.110655 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:40:21.110622    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w1-e86d9e-openshift-default-65bf4ddbdd-jnrbq"]
Sep 16 02:40:21.111172 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:40:21.111156    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d073b71f-8e1d-4624-9192-08021795bf50" containerName="main"
Sep 16 02:40:21.111219 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:40:21.111174    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="d073b71f-8e1d-4624-9192-08021795bf50" containerName="main"
Sep 16 02:40:21.111299 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:40:21.111288    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="d073b71f-8e1d-4624-9192-08021795bf50" containerName="main"
Sep 16 02:40:21.114783 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:40:21.114762    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-e86d9e-openshift-default-65bf4ddbdd-jnrbq"
Sep 16 02:40:21.117165 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:40:21.117145    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w1-e86d9e-openshift-default-dockercfg-9vhzv\""
Sep 16 02:40:21.123827 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:40:21.123793    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-e86d9e-openshift-default-65bf4ddbdd-jnrbq"]
Sep 16 02:40:21.173934 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:40:21.173897    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1490b32b-6d61-404a-ac86-f902766a0cfb-istio-data\") pod \"e2e-shared-b-w1-e86d9e-openshift-default-65bf4ddbdd-jnrbq\" (UID: \"1490b32b-6d61-404a-ac86-f902766a0cfb\") " pod="openshift-ingress/e2e-shared-b-w1-e86d9e-openshift-default-65bf4ddbdd-jnrbq"
Sep 16 02:40:21.173934 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:40:21.173941    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1490b32b-6d61-404a-ac86-f902766a0cfb-workload-certs\") pod \"e2e-shared-b-w1-e86d9e-openshift-default-65bf4ddbdd-jnrbq\" (UID: \"1490b32b-6d61-404a-ac86-f902766a0cfb\") " pod="openshift-ingress/e2e-shared-b-w1-e86d9e-openshift-default-65bf4ddbdd-jnrbq"
Sep 16 02:40:21.174140 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:40:21.173960    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1490b32b-6d61-404a-ac86-f902766a0cfb-istio-envoy\") pod \"e2e-shared-b-w1-e86d9e-openshift-default-65bf4ddbdd-jnrbq\" (UID: \"1490b32b-6d61-404a-ac86-f902766a0cfb\") " pod="openshift-ingress/e2e-shared-b-w1-e86d9e-openshift-default-65bf4ddbdd-jnrbq"
Sep 16 02:40:21.174140 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:40:21.174010    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1490b32b-6d61-404a-ac86-f902766a0cfb-workload-socket\") pod \"e2e-shared-b-w1-e86d9e-openshift-default-65bf4ddbdd-jnrbq\" (UID: \"1490b32b-6d61-404a-ac86-f902766a0cfb\") " pod="openshift-ingress/e2e-shared-b-w1-e86d9e-openshift-default-65bf4ddbdd-jnrbq"
Sep 16 02:40:21.174140 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:40:21.174052    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1490b32b-6d61-404a-ac86-f902766a0cfb-istio-token\") pod \"e2e-shared-b-w1-e86d9e-openshift-default-65bf4ddbdd-jnrbq\" (UID: \"1490b32b-6d61-404a-ac86-f902766a0cfb\") " pod="openshift-ingress/e2e-shared-b-w1-e86d9e-openshift-default-65bf4ddbdd-jnrbq"
Sep 16 02:40:21.174140 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:40:21.174090    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1490b32b-6d61-404a-ac86-f902766a0cfb-credential-socket\") pod \"e2e-shared-b-w1-e86d9e-openshift-default-65bf4ddbdd-jnrbq\" (UID: \"1490b32b-6d61-404a-ac86-f902766a0cfb\") " pod="openshift-ingress/e2e-shared-b-w1-e86d9e-openshift-default-65bf4ddbdd-jnrbq"
Sep 16 02:40:21.174140 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:40:21.174118    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1490b32b-6d61-404a-ac86-f902766a0cfb-istiod-ca-cert\") pod \"e2e-shared-b-w1-e86d9e-openshift-default-65bf4ddbdd-jnrbq\" (UID: \"1490b32b-6d61-404a-ac86-f902766a0cfb\") " pod="openshift-ingress/e2e-shared-b-w1-e86d9e-openshift-default-65bf4ddbdd-jnrbq"
Sep 16 02:40:21.174329 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:40:21.174146    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1490b32b-6d61-404a-ac86-f902766a0cfb-istio-podinfo\") pod \"e2e-shared-b-w1-e86d9e-openshift-default-65bf4ddbdd-jnrbq\" (UID: \"1490b32b-6d61-404a-ac86-f902766a0cfb\") " pod="openshift-ingress/e2e-shared-b-w1-e86d9e-openshift-default-65bf4ddbdd-jnrbq"
Sep 16 02:40:21.174329 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:40:21.174175    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p654j\" (UniqueName: \"kubernetes.io/projected/1490b32b-6d61-404a-ac86-f902766a0cfb-kube-api-access-p654j\") pod \"e2e-shared-b-w1-e86d9e-openshift-default-65bf4ddbdd-jnrbq\" (UID: \"1490b32b-6d61-404a-ac86-f902766a0cfb\") " pod="openshift-ingress/e2e-shared-b-w1-e86d9e-openshift-default-65bf4ddbdd-jnrbq"
Sep 16 02:40:21.275104 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:40:21.275068    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1490b32b-6d61-404a-ac86-f902766a0cfb-istio-data\") pod \"e2e-shared-b-w1-e86d9e-openshift-default-65bf4ddbdd-jnrbq\" (UID: \"1490b32b-6d61-404a-ac86-f902766a0cfb\") " pod="openshift-ingress/e2e-shared-b-w1-e86d9e-openshift-default-65bf4ddbdd-jnrbq"
Sep 16 02:40:21.275271 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:40:21.275110    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1490b32b-6d61-404a-ac86-f902766a0cfb-workload-certs\") pod \"e2e-shared-b-w1-e86d9e-openshift-default-65bf4ddbdd-jnrbq\" (UID: \"1490b32b-6d61-404a-ac86-f902766a0cfb\") " pod="openshift-ingress/e2e-shared-b-w1-e86d9e-openshift-default-65bf4ddbdd-jnrbq"
Sep 16 02:40:21.275271 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:40:21.275137    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1490b32b-6d61-404a-ac86-f902766a0cfb-istio-envoy\") pod \"e2e-shared-b-w1-e86d9e-openshift-default-65bf4ddbdd-jnrbq\" (UID: \"1490b32b-6d61-404a-ac86-f902766a0cfb\") " pod="openshift-ingress/e2e-shared-b-w1-e86d9e-openshift-default-65bf4ddbdd-jnrbq"
Sep 16 02:40:21.275271 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:40:21.275174    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1490b32b-6d61-404a-ac86-f902766a0cfb-workload-socket\") pod \"e2e-shared-b-w1-e86d9e-openshift-default-65bf4ddbdd-jnrbq\" (UID: \"1490b32b-6d61-404a-ac86-f902766a0cfb\") " pod="openshift-ingress/e2e-shared-b-w1-e86d9e-openshift-default-65bf4ddbdd-jnrbq"
Sep 16 02:40:21.275271 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:40:21.275213    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1490b32b-6d61-404a-ac86-f902766a0cfb-istio-token\") pod \"e2e-shared-b-w1-e86d9e-openshift-default-65bf4ddbdd-jnrbq\" (UID: \"1490b32b-6d61-404a-ac86-f902766a0cfb\") " pod="openshift-ingress/e2e-shared-b-w1-e86d9e-openshift-default-65bf4ddbdd-jnrbq"
Sep 16 02:40:21.275271 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:40:21.275262    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1490b32b-6d61-404a-ac86-f902766a0cfb-credential-socket\") pod \"e2e-shared-b-w1-e86d9e-openshift-default-65bf4ddbdd-jnrbq\" (UID: \"1490b32b-6d61-404a-ac86-f902766a0cfb\") " pod="openshift-ingress/e2e-shared-b-w1-e86d9e-openshift-default-65bf4ddbdd-jnrbq"
Sep 16 02:40:21.275532 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:40:21.275300    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1490b32b-6d61-404a-ac86-f902766a0cfb-istiod-ca-cert\") pod \"e2e-shared-b-w1-e86d9e-openshift-default-65bf4ddbdd-jnrbq\" (UID: \"1490b32b-6d61-404a-ac86-f902766a0cfb\") " pod="openshift-ingress/e2e-shared-b-w1-e86d9e-openshift-default-65bf4ddbdd-jnrbq"
Sep 16 02:40:21.275532 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:40:21.275335    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1490b32b-6d61-404a-ac86-f902766a0cfb-istio-podinfo\") pod \"e2e-shared-b-w1-e86d9e-openshift-default-65bf4ddbdd-jnrbq\" (UID: \"1490b32b-6d61-404a-ac86-f902766a0cfb\") " pod="openshift-ingress/e2e-shared-b-w1-e86d9e-openshift-default-65bf4ddbdd-jnrbq"
Sep 16 02:40:21.275532 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:40:21.275469    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1490b32b-6d61-404a-ac86-f902766a0cfb-workload-certs\") pod \"e2e-shared-b-w1-e86d9e-openshift-default-65bf4ddbdd-jnrbq\" (UID: \"1490b32b-6d61-404a-ac86-f902766a0cfb\") " pod="openshift-ingress/e2e-shared-b-w1-e86d9e-openshift-default-65bf4ddbdd-jnrbq"
Sep 16 02:40:21.275532 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:40:21.275520    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p654j\" (UniqueName: \"kubernetes.io/projected/1490b32b-6d61-404a-ac86-f902766a0cfb-kube-api-access-p654j\") pod \"e2e-shared-b-w1-e86d9e-openshift-default-65bf4ddbdd-jnrbq\" (UID: \"1490b32b-6d61-404a-ac86-f902766a0cfb\") " pod="openshift-ingress/e2e-shared-b-w1-e86d9e-openshift-default-65bf4ddbdd-jnrbq"
Sep 16 02:40:21.275787 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:40:21.275523    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1490b32b-6d61-404a-ac86-f902766a0cfb-istio-data\") pod \"e2e-shared-b-w1-e86d9e-openshift-default-65bf4ddbdd-jnrbq\" (UID: \"1490b32b-6d61-404a-ac86-f902766a0cfb\") " pod="openshift-ingress/e2e-shared-b-w1-e86d9e-openshift-default-65bf4ddbdd-jnrbq"
Sep 16 02:40:21.275787 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:40:21.275750    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1490b32b-6d61-404a-ac86-f902766a0cfb-workload-socket\") pod \"e2e-shared-b-w1-e86d9e-openshift-default-65bf4ddbdd-jnrbq\" (UID: \"1490b32b-6d61-404a-ac86-f902766a0cfb\") " pod="openshift-ingress/e2e-shared-b-w1-e86d9e-openshift-default-65bf4ddbdd-jnrbq"
Sep 16 02:40:21.275979 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:40:21.275959    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1490b32b-6d61-404a-ac86-f902766a0cfb-credential-socket\") pod \"e2e-shared-b-w1-e86d9e-openshift-default-65bf4ddbdd-jnrbq\" (UID: \"1490b32b-6d61-404a-ac86-f902766a0cfb\") " pod="openshift-ingress/e2e-shared-b-w1-e86d9e-openshift-default-65bf4ddbdd-jnrbq"
Sep 16 02:40:21.276104 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:40:21.276081    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1490b32b-6d61-404a-ac86-f902766a0cfb-istiod-ca-cert\") pod \"e2e-shared-b-w1-e86d9e-openshift-default-65bf4ddbdd-jnrbq\" (UID: \"1490b32b-6d61-404a-ac86-f902766a0cfb\") " pod="openshift-ingress/e2e-shared-b-w1-e86d9e-openshift-default-65bf4ddbdd-jnrbq"
Sep 16 02:40:21.277545 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:40:21.277525    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1490b32b-6d61-404a-ac86-f902766a0cfb-istio-envoy\") pod \"e2e-shared-b-w1-e86d9e-openshift-default-65bf4ddbdd-jnrbq\" (UID: \"1490b32b-6d61-404a-ac86-f902766a0cfb\") " pod="openshift-ingress/e2e-shared-b-w1-e86d9e-openshift-default-65bf4ddbdd-jnrbq"
Sep 16 02:40:21.277913 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:40:21.277893    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1490b32b-6d61-404a-ac86-f902766a0cfb-istio-podinfo\") pod \"e2e-shared-b-w1-e86d9e-openshift-default-65bf4ddbdd-jnrbq\" (UID: \"1490b32b-6d61-404a-ac86-f902766a0cfb\") " pod="openshift-ingress/e2e-shared-b-w1-e86d9e-openshift-default-65bf4ddbdd-jnrbq"
Sep 16 02:40:21.283890 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:40:21.283865    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-p654j\" (UniqueName: \"kubernetes.io/projected/1490b32b-6d61-404a-ac86-f902766a0cfb-kube-api-access-p654j\") pod \"e2e-shared-b-w1-e86d9e-openshift-default-65bf4ddbdd-jnrbq\" (UID: \"1490b32b-6d61-404a-ac86-f902766a0cfb\") " pod="openshift-ingress/e2e-shared-b-w1-e86d9e-openshift-default-65bf4ddbdd-jnrbq"
Sep 16 02:40:21.283977 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:40:21.283944    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1490b32b-6d61-404a-ac86-f902766a0cfb-istio-token\") pod \"e2e-shared-b-w1-e86d9e-openshift-default-65bf4ddbdd-jnrbq\" (UID: \"1490b32b-6d61-404a-ac86-f902766a0cfb\") " pod="openshift-ingress/e2e-shared-b-w1-e86d9e-openshift-default-65bf4ddbdd-jnrbq"
Sep 16 02:40:21.427425 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:40:21.427343    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-e86d9e-openshift-default-65bf4ddbdd-jnrbq"
Sep 16 02:40:21.700784 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:40:21.700732    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-e86d9e-openshift-default-65bf4ddbdd-jnrbq"]
Sep 16 02:40:21.703199 ip-10-0-128-111 kubenswrapper[2583]: W0916 02:40:21.703165    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1490b32b_6d61_404a_ac86_f902766a0cfb.slice/crio-ddae25ba735eb894babf6a228ac6a23b5ffdda3ffd318fd136a8b579663ee572 WatchSource:0}: Error finding container ddae25ba735eb894babf6a228ac6a23b5ffdda3ffd318fd136a8b579663ee572: Status 404 returned error can't find the container with id ddae25ba735eb894babf6a228ac6a23b5ffdda3ffd318fd136a8b579663ee572
Sep 16 02:40:21.705291 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:40:21.705262    2583 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 16 02:40:21.705361 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:40:21.705327    2583 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 16 02:40:21.705361 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:40:21.705356    2583 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 16 02:40:22.390354 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:40:22.390310    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-e86d9e-openshift-default-65bf4ddbdd-jnrbq" event={"ID":"1490b32b-6d61-404a-ac86-f902766a0cfb","Type":"ContainerStarted","Data":"caf2a731e53b6280980decc1fa9f3283e0a18c56dbb979f680229687a00edad3"}
Sep 16 02:40:22.390804 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:40:22.390368    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-e86d9e-openshift-default-65bf4ddbdd-jnrbq" event={"ID":"1490b32b-6d61-404a-ac86-f902766a0cfb","Type":"ContainerStarted","Data":"ddae25ba735eb894babf6a228ac6a23b5ffdda3ffd318fd136a8b579663ee572"}
Sep 16 02:40:22.410864 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:40:22.410811    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w1-e86d9e-openshift-default-65bf4ddbdd-jnrbq" podStartSLOduration=1.410796191 podStartE2EDuration="1.410796191s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 02:40:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 02:40:22.409164116 +0000 UTC m=+1300.876952929" watchObservedRunningTime="2026-09-16 02:40:22.410796191 +0000 UTC m=+1300.878584973"
Sep 16 02:40:22.428081 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:40:22.428053    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w1-e86d9e-openshift-default-65bf4ddbdd-jnrbq"
Sep 16 02:40:22.433330 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:40:22.433308    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w1-e86d9e-openshift-default-65bf4ddbdd-jnrbq"
Sep 16 02:40:23.396124 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:40:23.396089    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w1-e86d9e-openshift-default-65bf4ddbdd-jnrbq"
Sep 16 02:40:23.397096 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:40:23.397073    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w1-e86d9e-openshift-default-65bf4ddbdd-jnrbq"
Sep 16 02:40:31.744605 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:40:31.744568    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w2-db1ed7/sub-test-model-db1ed7-kserve-b99f74c57-55zfc"]
Sep 16 02:40:31.748475 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:40:31.748451    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w2-db1ed7/sub-test-model-db1ed7-kserve-b99f74c57-55zfc"
Sep 16 02:40:31.751942 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:40:31.751911    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w2-db1ed7\"/\"openshift-service-ca.crt\""
Sep 16 02:40:31.751942 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:40:31.751918    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w2-db1ed7\"/\"sub-test-model-db1ed7-kserve-self-signed-certs\""
Sep 16 02:40:31.752129 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:40:31.751920    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w2-db1ed7\"/\"kube-root-ca.crt\""
Sep 16 02:40:31.752129 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:40:31.751920    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w2-db1ed7\"/\"default-dockercfg-jjm52\""
Sep 16 02:40:31.758120 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:40:31.758095    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w2-db1ed7/sub-test-model-db1ed7-kserve-b99f74c57-55zfc"]
Sep 16 02:40:31.883869 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:40:31.883831    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/eb24e349-afc0-42e5-bf95-03cfcf5ff813-dshm\") pod \"sub-test-model-db1ed7-kserve-b99f74c57-55zfc\" (UID: \"eb24e349-afc0-42e5-bf95-03cfcf5ff813\") " pod="ai-tenant-e2e-shared-a-w2-db1ed7/sub-test-model-db1ed7-kserve-b99f74c57-55zfc"
Sep 16 02:40:31.884018 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:40:31.883940    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/eb24e349-afc0-42e5-bf95-03cfcf5ff813-tmp-dir\") pod \"sub-test-model-db1ed7-kserve-b99f74c57-55zfc\" (UID: \"eb24e349-afc0-42e5-bf95-03cfcf5ff813\") " pod="ai-tenant-e2e-shared-a-w2-db1ed7/sub-test-model-db1ed7-kserve-b99f74c57-55zfc"
Sep 16 02:40:31.884018 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:40:31.883989    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/eb24e349-afc0-42e5-bf95-03cfcf5ff813-home\") pod \"sub-test-model-db1ed7-kserve-b99f74c57-55zfc\" (UID: \"eb24e349-afc0-42e5-bf95-03cfcf5ff813\") " pod="ai-tenant-e2e-shared-a-w2-db1ed7/sub-test-model-db1ed7-kserve-b99f74c57-55zfc"
Sep 16 02:40:31.884018 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:40:31.884010    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dzxhk\" (UniqueName: \"kubernetes.io/projected/eb24e349-afc0-42e5-bf95-03cfcf5ff813-kube-api-access-dzxhk\") pod \"sub-test-model-db1ed7-kserve-b99f74c57-55zfc\" (UID: \"eb24e349-afc0-42e5-bf95-03cfcf5ff813\") " pod="ai-tenant-e2e-shared-a-w2-db1ed7/sub-test-model-db1ed7-kserve-b99f74c57-55zfc"
Sep 16 02:40:31.884147 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:40:31.884039    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/eb24e349-afc0-42e5-bf95-03cfcf5ff813-model-cache\") pod \"sub-test-model-db1ed7-kserve-b99f74c57-55zfc\" (UID: \"eb24e349-afc0-42e5-bf95-03cfcf5ff813\") " pod="ai-tenant-e2e-shared-a-w2-db1ed7/sub-test-model-db1ed7-kserve-b99f74c57-55zfc"
Sep 16 02:40:31.884147 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:40:31.884077    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/eb24e349-afc0-42e5-bf95-03cfcf5ff813-tls-certs\") pod \"sub-test-model-db1ed7-kserve-b99f74c57-55zfc\" (UID: \"eb24e349-afc0-42e5-bf95-03cfcf5ff813\") " pod="ai-tenant-e2e-shared-a-w2-db1ed7/sub-test-model-db1ed7-kserve-b99f74c57-55zfc"
Sep 16 02:40:31.951668 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:40:31.951634    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-fzl8t"]
Sep 16 02:40:31.984971 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:40:31.984929    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/eb24e349-afc0-42e5-bf95-03cfcf5ff813-tmp-dir\") pod \"sub-test-model-db1ed7-kserve-b99f74c57-55zfc\" (UID: \"eb24e349-afc0-42e5-bf95-03cfcf5ff813\") " pod="ai-tenant-e2e-shared-a-w2-db1ed7/sub-test-model-db1ed7-kserve-b99f74c57-55zfc"
Sep 16 02:40:31.985152 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:40:31.984981    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/eb24e349-afc0-42e5-bf95-03cfcf5ff813-home\") pod \"sub-test-model-db1ed7-kserve-b99f74c57-55zfc\" (UID: \"eb24e349-afc0-42e5-bf95-03cfcf5ff813\") " pod="ai-tenant-e2e-shared-a-w2-db1ed7/sub-test-model-db1ed7-kserve-b99f74c57-55zfc"
Sep 16 02:40:31.985152 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:40:31.985002    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dzxhk\" (UniqueName: \"kubernetes.io/projected/eb24e349-afc0-42e5-bf95-03cfcf5ff813-kube-api-access-dzxhk\") pod \"sub-test-model-db1ed7-kserve-b99f74c57-55zfc\" (UID: \"eb24e349-afc0-42e5-bf95-03cfcf5ff813\") " pod="ai-tenant-e2e-shared-a-w2-db1ed7/sub-test-model-db1ed7-kserve-b99f74c57-55zfc"
Sep 16 02:40:31.985152 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:40:31.985023    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/eb24e349-afc0-42e5-bf95-03cfcf5ff813-model-cache\") pod \"sub-test-model-db1ed7-kserve-b99f74c57-55zfc\" (UID: \"eb24e349-afc0-42e5-bf95-03cfcf5ff813\") " pod="ai-tenant-e2e-shared-a-w2-db1ed7/sub-test-model-db1ed7-kserve-b99f74c57-55zfc"
Sep 16 02:40:31.985152 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:40:31.985039    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/eb24e349-afc0-42e5-bf95-03cfcf5ff813-tls-certs\") pod \"sub-test-model-db1ed7-kserve-b99f74c57-55zfc\" (UID: \"eb24e349-afc0-42e5-bf95-03cfcf5ff813\") " pod="ai-tenant-e2e-shared-a-w2-db1ed7/sub-test-model-db1ed7-kserve-b99f74c57-55zfc"
Sep 16 02:40:31.985152 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:40:31.985101    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/eb24e349-afc0-42e5-bf95-03cfcf5ff813-dshm\") pod \"sub-test-model-db1ed7-kserve-b99f74c57-55zfc\" (UID: \"eb24e349-afc0-42e5-bf95-03cfcf5ff813\") " pod="ai-tenant-e2e-shared-a-w2-db1ed7/sub-test-model-db1ed7-kserve-b99f74c57-55zfc"
Sep 16 02:40:31.985436 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:40:31.985373    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/eb24e349-afc0-42e5-bf95-03cfcf5ff813-model-cache\") pod \"sub-test-model-db1ed7-kserve-b99f74c57-55zfc\" (UID: \"eb24e349-afc0-42e5-bf95-03cfcf5ff813\") " pod="ai-tenant-e2e-shared-a-w2-db1ed7/sub-test-model-db1ed7-kserve-b99f74c57-55zfc"
Sep 16 02:40:31.985436 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:40:31.985369    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/eb24e349-afc0-42e5-bf95-03cfcf5ff813-tmp-dir\") pod \"sub-test-model-db1ed7-kserve-b99f74c57-55zfc\" (UID: \"eb24e349-afc0-42e5-bf95-03cfcf5ff813\") " pod="ai-tenant-e2e-shared-a-w2-db1ed7/sub-test-model-db1ed7-kserve-b99f74c57-55zfc"
Sep 16 02:40:31.985436 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:40:31.985421    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/eb24e349-afc0-42e5-bf95-03cfcf5ff813-home\") pod \"sub-test-model-db1ed7-kserve-b99f74c57-55zfc\" (UID: \"eb24e349-afc0-42e5-bf95-03cfcf5ff813\") " pod="ai-tenant-e2e-shared-a-w2-db1ed7/sub-test-model-db1ed7-kserve-b99f74c57-55zfc"
Sep 16 02:40:31.987355 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:40:31.987330    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/eb24e349-afc0-42e5-bf95-03cfcf5ff813-dshm\") pod \"sub-test-model-db1ed7-kserve-b99f74c57-55zfc\" (UID: \"eb24e349-afc0-42e5-bf95-03cfcf5ff813\") " pod="ai-tenant-e2e-shared-a-w2-db1ed7/sub-test-model-db1ed7-kserve-b99f74c57-55zfc"
Sep 16 02:40:31.987534 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:40:31.987514    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/eb24e349-afc0-42e5-bf95-03cfcf5ff813-tls-certs\") pod \"sub-test-model-db1ed7-kserve-b99f74c57-55zfc\" (UID: \"eb24e349-afc0-42e5-bf95-03cfcf5ff813\") " pod="ai-tenant-e2e-shared-a-w2-db1ed7/sub-test-model-db1ed7-kserve-b99f74c57-55zfc"
Sep 16 02:40:31.991889 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:40:31.991868    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dzxhk\" (UniqueName: \"kubernetes.io/projected/eb24e349-afc0-42e5-bf95-03cfcf5ff813-kube-api-access-dzxhk\") pod \"sub-test-model-db1ed7-kserve-b99f74c57-55zfc\" (UID: \"eb24e349-afc0-42e5-bf95-03cfcf5ff813\") " pod="ai-tenant-e2e-shared-a-w2-db1ed7/sub-test-model-db1ed7-kserve-b99f74c57-55zfc"
Sep 16 02:40:32.059278 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:40:32.059194    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w2-db1ed7/sub-test-model-db1ed7-kserve-b99f74c57-55zfc"
Sep 16 02:40:32.332820 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:40:32.332735    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w2-db1ed7/sub-test-model-db1ed7-kserve-b99f74c57-55zfc"]
Sep 16 02:40:32.336705 ip-10-0-128-111 kubenswrapper[2583]: W0916 02:40:32.336679    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podeb24e349_afc0_42e5_bf95_03cfcf5ff813.slice/crio-5fdf69d22b7ffa3c2e3a055acc6370287427b21da7f83f3f786602b42d846687 WatchSource:0}: Error finding container 5fdf69d22b7ffa3c2e3a055acc6370287427b21da7f83f3f786602b42d846687: Status 404 returned error can't find the container with id 5fdf69d22b7ffa3c2e3a055acc6370287427b21da7f83f3f786602b42d846687
Sep 16 02:40:32.443264 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:40:32.443227    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w2-db1ed7/sub-test-model-db1ed7-kserve-b99f74c57-55zfc" event={"ID":"eb24e349-afc0-42e5-bf95-03cfcf5ff813","Type":"ContainerStarted","Data":"6dbeb0463c4fec9fd0ca1c354822c0f11ccb4c9ca073d7fae1bda69f8acad462"}
Sep 16 02:40:32.443433 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:40:32.443275    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w2-db1ed7/sub-test-model-db1ed7-kserve-b99f74c57-55zfc" event={"ID":"eb24e349-afc0-42e5-bf95-03cfcf5ff813","Type":"ContainerStarted","Data":"5fdf69d22b7ffa3c2e3a055acc6370287427b21da7f83f3f786602b42d846687"}
Sep 16 02:40:32.458899 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:40:32.458849    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w2-db1ed7/sub-test-model-db1ed7-kserve-b99f74c57-55zfc" podStartSLOduration=1.4588338699999999 podStartE2EDuration="1.45883387s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 02:40:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 02:40:32.457395527 +0000 UTC m=+1310.925184309" watchObservedRunningTime="2026-09-16 02:40:32.45883387 +0000 UTC m=+1310.926622652"
Sep 16 02:40:42.059801 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:40:42.059693    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w2-db1ed7/sub-test-model-db1ed7-kserve-b99f74c57-55zfc"
Sep 16 02:40:42.060319 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:40:42.059825    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w2-db1ed7/sub-test-model-db1ed7-kserve-b99f74c57-55zfc"
Sep 16 02:40:42.072454 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:40:42.072426    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w2-db1ed7/sub-test-model-db1ed7-kserve-b99f74c57-55zfc"
Sep 16 02:40:42.493327 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:40:42.493285    2583 generic.go:356] "Generic (PLEG): container finished" podID="1490b32b-6d61-404a-ac86-f902766a0cfb" containerID="caf2a731e53b6280980decc1fa9f3283e0a18c56dbb979f680229687a00edad3" exitCode=0
Sep 16 02:40:42.493507 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:40:42.493357    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-e86d9e-openshift-default-65bf4ddbdd-jnrbq" event={"ID":"1490b32b-6d61-404a-ac86-f902766a0cfb","Type":"ContainerDied","Data":"caf2a731e53b6280980decc1fa9f3283e0a18c56dbb979f680229687a00edad3"}
Sep 16 02:40:42.493854 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:40:42.493837    2583 scope.go:117] "RemoveContainer" containerID="caf2a731e53b6280980decc1fa9f3283e0a18c56dbb979f680229687a00edad3"
Sep 16 02:40:42.495092 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:40:42.495060    2583 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 16 02:40:42.495223 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:40:42.495140    2583 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 16 02:40:42.495223 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:40:42.495187    2583 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 16 02:40:42.505236 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:40:42.505213    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w2-db1ed7/sub-test-model-db1ed7-kserve-b99f74c57-55zfc"
Sep 16 02:40:43.501144 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:40:43.501112    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-e86d9e-openshift-default-65bf4ddbdd-jnrbq" event={"ID":"1490b32b-6d61-404a-ac86-f902766a0cfb","Type":"ContainerStarted","Data":"377984d084144614c81b3a83f3ef8f32c780f408f84ccb7490fc6338a658b88a"}
Sep 16 02:40:44.052251 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:40:44.052209    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-fzl8t"]
Sep 16 02:40:44.427933 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:40:44.427865    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w1-e86d9e-openshift-default-65bf4ddbdd-jnrbq"
Sep 16 02:40:44.428350 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:40:44.428323    2583 patch_prober.go:28] interesting pod/e2e-shared-b-w1-e86d9e-openshift-default-65bf4ddbdd-jnrbq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused" start-of-body=
Sep 16 02:40:44.428455 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:40:44.428390    2583 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-e86d9e-openshift-default-65bf4ddbdd-jnrbq" podUID="1490b32b-6d61-404a-ac86-f902766a0cfb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused"
Sep 16 02:40:45.428885 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:40:45.428843    2583 patch_prober.go:28] interesting pod/e2e-shared-b-w1-e86d9e-openshift-default-65bf4ddbdd-jnrbq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused" start-of-body=
Sep 16 02:40:45.429486 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:40:45.429453    2583 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-e86d9e-openshift-default-65bf4ddbdd-jnrbq" podUID="1490b32b-6d61-404a-ac86-f902766a0cfb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused"
Sep 16 02:40:46.428311 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:40:46.428271    2583 patch_prober.go:28] interesting pod/e2e-shared-b-w1-e86d9e-openshift-default-65bf4ddbdd-jnrbq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused" start-of-body=
Sep 16 02:40:46.428636 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:40:46.428604    2583 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-e86d9e-openshift-default-65bf4ddbdd-jnrbq" podUID="1490b32b-6d61-404a-ac86-f902766a0cfb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused"
Sep 16 02:40:47.428422 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:40:47.428385    2583 patch_prober.go:28] interesting pod/e2e-shared-b-w1-e86d9e-openshift-default-65bf4ddbdd-jnrbq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused" start-of-body=
Sep 16 02:40:47.429077 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:40:47.429042    2583 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-e86d9e-openshift-default-65bf4ddbdd-jnrbq" podUID="1490b32b-6d61-404a-ac86-f902766a0cfb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused"
Sep 16 02:40:48.428773 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:40:48.428733    2583 patch_prober.go:28] interesting pod/e2e-shared-b-w1-e86d9e-openshift-default-65bf4ddbdd-jnrbq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused" start-of-body=
Sep 16 02:40:48.429351 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:40:48.429319    2583 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-e86d9e-openshift-default-65bf4ddbdd-jnrbq" podUID="1490b32b-6d61-404a-ac86-f902766a0cfb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused"
Sep 16 02:40:49.428879 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:40:49.428837    2583 patch_prober.go:28] interesting pod/e2e-shared-b-w1-e86d9e-openshift-default-65bf4ddbdd-jnrbq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused" start-of-body=
Sep 16 02:40:49.429263 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:40:49.428946    2583 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-e86d9e-openshift-default-65bf4ddbdd-jnrbq" podUID="1490b32b-6d61-404a-ac86-f902766a0cfb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused"
Sep 16 02:40:50.431842 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:40:50.431807    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w1-e86d9e-openshift-default-65bf4ddbdd-jnrbq"
Sep 16 02:40:50.432183 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:40:50.432085    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w1-e86d9e-openshift-default-65bf4ddbdd-jnrbq"
Sep 16 02:40:50.432688 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:40:50.432668    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w1-e86d9e-openshift-default-65bf4ddbdd-jnrbq"
Sep 16 02:40:58.495673 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:40:58.495644    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-73bfc1e9-openshift-default-fc75695d8-8l6ww"]
Sep 16 02:40:58.499533 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:40:58.499509    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-73bfc1e9-openshift-default-fc75695d8-8l6ww"
Sep 16 02:40:58.501819 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:40:58.501794    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-73bfc1e9-openshift-default-dockercfg-h47t6\""
Sep 16 02:40:58.509764 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:40:58.509730    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-73bfc1e9-openshift-default-fc75695d8-8l6ww"]
Sep 16 02:40:58.545706 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:40:58.545675    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c9f199d5-ac58-4074-9ff0-b88bec554d35-istio-data\") pod \"e2e-mt-73bfc1e9-openshift-default-fc75695d8-8l6ww\" (UID: \"c9f199d5-ac58-4074-9ff0-b88bec554d35\") " pod="openshift-ingress/e2e-mt-73bfc1e9-openshift-default-fc75695d8-8l6ww"
Sep 16 02:40:58.545836 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:40:58.545721    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lrg7g\" (UniqueName: \"kubernetes.io/projected/c9f199d5-ac58-4074-9ff0-b88bec554d35-kube-api-access-lrg7g\") pod \"e2e-mt-73bfc1e9-openshift-default-fc75695d8-8l6ww\" (UID: \"c9f199d5-ac58-4074-9ff0-b88bec554d35\") " pod="openshift-ingress/e2e-mt-73bfc1e9-openshift-default-fc75695d8-8l6ww"
Sep 16 02:40:58.545836 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:40:58.545753    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c9f199d5-ac58-4074-9ff0-b88bec554d35-workload-socket\") pod \"e2e-mt-73bfc1e9-openshift-default-fc75695d8-8l6ww\" (UID: \"c9f199d5-ac58-4074-9ff0-b88bec554d35\") " pod="openshift-ingress/e2e-mt-73bfc1e9-openshift-default-fc75695d8-8l6ww"
Sep 16 02:40:58.545836 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:40:58.545781    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c9f199d5-ac58-4074-9ff0-b88bec554d35-credential-socket\") pod \"e2e-mt-73bfc1e9-openshift-default-fc75695d8-8l6ww\" (UID: \"c9f199d5-ac58-4074-9ff0-b88bec554d35\") " pod="openshift-ingress/e2e-mt-73bfc1e9-openshift-default-fc75695d8-8l6ww"
Sep 16 02:40:58.545836 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:40:58.545802    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c9f199d5-ac58-4074-9ff0-b88bec554d35-istio-envoy\") pod \"e2e-mt-73bfc1e9-openshift-default-fc75695d8-8l6ww\" (UID: \"c9f199d5-ac58-4074-9ff0-b88bec554d35\") " pod="openshift-ingress/e2e-mt-73bfc1e9-openshift-default-fc75695d8-8l6ww"
Sep 16 02:40:58.545976 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:40:58.545863    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c9f199d5-ac58-4074-9ff0-b88bec554d35-istio-token\") pod \"e2e-mt-73bfc1e9-openshift-default-fc75695d8-8l6ww\" (UID: \"c9f199d5-ac58-4074-9ff0-b88bec554d35\") " pod="openshift-ingress/e2e-mt-73bfc1e9-openshift-default-fc75695d8-8l6ww"
Sep 16 02:40:58.545976 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:40:58.545908    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c9f199d5-ac58-4074-9ff0-b88bec554d35-workload-certs\") pod \"e2e-mt-73bfc1e9-openshift-default-fc75695d8-8l6ww\" (UID: \"c9f199d5-ac58-4074-9ff0-b88bec554d35\") " pod="openshift-ingress/e2e-mt-73bfc1e9-openshift-default-fc75695d8-8l6ww"
Sep 16 02:40:58.545976 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:40:58.545926    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c9f199d5-ac58-4074-9ff0-b88bec554d35-istio-podinfo\") pod \"e2e-mt-73bfc1e9-openshift-default-fc75695d8-8l6ww\" (UID: \"c9f199d5-ac58-4074-9ff0-b88bec554d35\") " pod="openshift-ingress/e2e-mt-73bfc1e9-openshift-default-fc75695d8-8l6ww"
Sep 16 02:40:58.545976 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:40:58.545955    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c9f199d5-ac58-4074-9ff0-b88bec554d35-istiod-ca-cert\") pod \"e2e-mt-73bfc1e9-openshift-default-fc75695d8-8l6ww\" (UID: \"c9f199d5-ac58-4074-9ff0-b88bec554d35\") " pod="openshift-ingress/e2e-mt-73bfc1e9-openshift-default-fc75695d8-8l6ww"
Sep 16 02:40:58.646750 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:40:58.646714    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c9f199d5-ac58-4074-9ff0-b88bec554d35-workload-certs\") pod \"e2e-mt-73bfc1e9-openshift-default-fc75695d8-8l6ww\" (UID: \"c9f199d5-ac58-4074-9ff0-b88bec554d35\") " pod="openshift-ingress/e2e-mt-73bfc1e9-openshift-default-fc75695d8-8l6ww"
Sep 16 02:40:58.646750 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:40:58.646749    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c9f199d5-ac58-4074-9ff0-b88bec554d35-istio-podinfo\") pod \"e2e-mt-73bfc1e9-openshift-default-fc75695d8-8l6ww\" (UID: \"c9f199d5-ac58-4074-9ff0-b88bec554d35\") " pod="openshift-ingress/e2e-mt-73bfc1e9-openshift-default-fc75695d8-8l6ww"
Sep 16 02:40:58.646995 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:40:58.646783    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c9f199d5-ac58-4074-9ff0-b88bec554d35-istiod-ca-cert\") pod \"e2e-mt-73bfc1e9-openshift-default-fc75695d8-8l6ww\" (UID: \"c9f199d5-ac58-4074-9ff0-b88bec554d35\") " pod="openshift-ingress/e2e-mt-73bfc1e9-openshift-default-fc75695d8-8l6ww"
Sep 16 02:40:58.646995 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:40:58.646847    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c9f199d5-ac58-4074-9ff0-b88bec554d35-istio-data\") pod \"e2e-mt-73bfc1e9-openshift-default-fc75695d8-8l6ww\" (UID: \"c9f199d5-ac58-4074-9ff0-b88bec554d35\") " pod="openshift-ingress/e2e-mt-73bfc1e9-openshift-default-fc75695d8-8l6ww"
Sep 16 02:40:58.646995 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:40:58.646894    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lrg7g\" (UniqueName: \"kubernetes.io/projected/c9f199d5-ac58-4074-9ff0-b88bec554d35-kube-api-access-lrg7g\") pod \"e2e-mt-73bfc1e9-openshift-default-fc75695d8-8l6ww\" (UID: \"c9f199d5-ac58-4074-9ff0-b88bec554d35\") " pod="openshift-ingress/e2e-mt-73bfc1e9-openshift-default-fc75695d8-8l6ww"
Sep 16 02:40:58.646995 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:40:58.646928    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c9f199d5-ac58-4074-9ff0-b88bec554d35-workload-socket\") pod \"e2e-mt-73bfc1e9-openshift-default-fc75695d8-8l6ww\" (UID: \"c9f199d5-ac58-4074-9ff0-b88bec554d35\") " pod="openshift-ingress/e2e-mt-73bfc1e9-openshift-default-fc75695d8-8l6ww"
Sep 16 02:40:58.646995 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:40:58.646975    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c9f199d5-ac58-4074-9ff0-b88bec554d35-credential-socket\") pod \"e2e-mt-73bfc1e9-openshift-default-fc75695d8-8l6ww\" (UID: \"c9f199d5-ac58-4074-9ff0-b88bec554d35\") " pod="openshift-ingress/e2e-mt-73bfc1e9-openshift-default-fc75695d8-8l6ww"
Sep 16 02:40:58.647266 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:40:58.647009    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c9f199d5-ac58-4074-9ff0-b88bec554d35-istio-envoy\") pod \"e2e-mt-73bfc1e9-openshift-default-fc75695d8-8l6ww\" (UID: \"c9f199d5-ac58-4074-9ff0-b88bec554d35\") " pod="openshift-ingress/e2e-mt-73bfc1e9-openshift-default-fc75695d8-8l6ww"
Sep 16 02:40:58.647266 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:40:58.647046    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c9f199d5-ac58-4074-9ff0-b88bec554d35-istio-token\") pod \"e2e-mt-73bfc1e9-openshift-default-fc75695d8-8l6ww\" (UID: \"c9f199d5-ac58-4074-9ff0-b88bec554d35\") " pod="openshift-ingress/e2e-mt-73bfc1e9-openshift-default-fc75695d8-8l6ww"
Sep 16 02:40:58.647266 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:40:58.647182    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c9f199d5-ac58-4074-9ff0-b88bec554d35-workload-certs\") pod \"e2e-mt-73bfc1e9-openshift-default-fc75695d8-8l6ww\" (UID: \"c9f199d5-ac58-4074-9ff0-b88bec554d35\") " pod="openshift-ingress/e2e-mt-73bfc1e9-openshift-default-fc75695d8-8l6ww"
Sep 16 02:40:58.647428 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:40:58.647405    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c9f199d5-ac58-4074-9ff0-b88bec554d35-workload-socket\") pod \"e2e-mt-73bfc1e9-openshift-default-fc75695d8-8l6ww\" (UID: \"c9f199d5-ac58-4074-9ff0-b88bec554d35\") " pod="openshift-ingress/e2e-mt-73bfc1e9-openshift-default-fc75695d8-8l6ww"
Sep 16 02:40:58.647485 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:40:58.647439    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c9f199d5-ac58-4074-9ff0-b88bec554d35-istio-data\") pod \"e2e-mt-73bfc1e9-openshift-default-fc75695d8-8l6ww\" (UID: \"c9f199d5-ac58-4074-9ff0-b88bec554d35\") " pod="openshift-ingress/e2e-mt-73bfc1e9-openshift-default-fc75695d8-8l6ww"
Sep 16 02:40:58.647538 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:40:58.647506    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c9f199d5-ac58-4074-9ff0-b88bec554d35-credential-socket\") pod \"e2e-mt-73bfc1e9-openshift-default-fc75695d8-8l6ww\" (UID: \"c9f199d5-ac58-4074-9ff0-b88bec554d35\") " pod="openshift-ingress/e2e-mt-73bfc1e9-openshift-default-fc75695d8-8l6ww"
Sep 16 02:40:58.647599 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:40:58.647565    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c9f199d5-ac58-4074-9ff0-b88bec554d35-istiod-ca-cert\") pod \"e2e-mt-73bfc1e9-openshift-default-fc75695d8-8l6ww\" (UID: \"c9f199d5-ac58-4074-9ff0-b88bec554d35\") " pod="openshift-ingress/e2e-mt-73bfc1e9-openshift-default-fc75695d8-8l6ww"
Sep 16 02:40:58.649257 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:40:58.649234    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c9f199d5-ac58-4074-9ff0-b88bec554d35-istio-envoy\") pod \"e2e-mt-73bfc1e9-openshift-default-fc75695d8-8l6ww\" (UID: \"c9f199d5-ac58-4074-9ff0-b88bec554d35\") " pod="openshift-ingress/e2e-mt-73bfc1e9-openshift-default-fc75695d8-8l6ww"
Sep 16 02:40:58.649363 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:40:58.649347    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c9f199d5-ac58-4074-9ff0-b88bec554d35-istio-podinfo\") pod \"e2e-mt-73bfc1e9-openshift-default-fc75695d8-8l6ww\" (UID: \"c9f199d5-ac58-4074-9ff0-b88bec554d35\") " pod="openshift-ingress/e2e-mt-73bfc1e9-openshift-default-fc75695d8-8l6ww"
Sep 16 02:40:58.654307 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:40:58.654281    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c9f199d5-ac58-4074-9ff0-b88bec554d35-istio-token\") pod \"e2e-mt-73bfc1e9-openshift-default-fc75695d8-8l6ww\" (UID: \"c9f199d5-ac58-4074-9ff0-b88bec554d35\") " pod="openshift-ingress/e2e-mt-73bfc1e9-openshift-default-fc75695d8-8l6ww"
Sep 16 02:40:58.654668 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:40:58.654647    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lrg7g\" (UniqueName: \"kubernetes.io/projected/c9f199d5-ac58-4074-9ff0-b88bec554d35-kube-api-access-lrg7g\") pod \"e2e-mt-73bfc1e9-openshift-default-fc75695d8-8l6ww\" (UID: \"c9f199d5-ac58-4074-9ff0-b88bec554d35\") " pod="openshift-ingress/e2e-mt-73bfc1e9-openshift-default-fc75695d8-8l6ww"
Sep 16 02:40:58.810686 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:40:58.810606    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-73bfc1e9-openshift-default-fc75695d8-8l6ww"
Sep 16 02:40:59.082733 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:40:59.082709    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-73bfc1e9-openshift-default-fc75695d8-8l6ww"]
Sep 16 02:40:59.084945 ip-10-0-128-111 kubenswrapper[2583]: W0916 02:40:59.084903    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc9f199d5_ac58_4074_9ff0_b88bec554d35.slice/crio-579e641e4df31e1f368adcf44da585aa1f7a053acfb8fff46dacf93714a7f0e6 WatchSource:0}: Error finding container 579e641e4df31e1f368adcf44da585aa1f7a053acfb8fff46dacf93714a7f0e6: Status 404 returned error can't find the container with id 579e641e4df31e1f368adcf44da585aa1f7a053acfb8fff46dacf93714a7f0e6
Sep 16 02:40:59.087315 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:40:59.087284    2583 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 16 02:40:59.087405 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:40:59.087345    2583 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 16 02:40:59.087405 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:40:59.087375    2583 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 16 02:40:59.587533 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:40:59.587495    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-73bfc1e9-openshift-default-fc75695d8-8l6ww" event={"ID":"c9f199d5-ac58-4074-9ff0-b88bec554d35","Type":"ContainerStarted","Data":"f4bf3777bf2955e2c38c986a56ab72cf0832fcbdf2ab0e0564411358f7b26d2d"}
Sep 16 02:40:59.587930 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:40:59.587593    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-73bfc1e9-openshift-default-fc75695d8-8l6ww" event={"ID":"c9f199d5-ac58-4074-9ff0-b88bec554d35","Type":"ContainerStarted","Data":"579e641e4df31e1f368adcf44da585aa1f7a053acfb8fff46dacf93714a7f0e6"}
Sep 16 02:40:59.605678 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:40:59.605624    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-73bfc1e9-openshift-default-fc75695d8-8l6ww" podStartSLOduration=1.605607928 podStartE2EDuration="1.605607928s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 02:40:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 02:40:59.604049053 +0000 UTC m=+1338.071837836" watchObservedRunningTime="2026-09-16 02:40:59.605607928 +0000 UTC m=+1338.073396710"
Sep 16 02:40:59.811701 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:40:59.811668    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-73bfc1e9-openshift-default-fc75695d8-8l6ww"
Sep 16 02:41:00.816580 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:41:00.816516    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-73bfc1e9-openshift-default-fc75695d8-8l6ww"
Sep 16 02:41:01.598619 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:41:01.598584    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-73bfc1e9-openshift-default-fc75695d8-8l6ww"
Sep 16 02:41:01.599510 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:41:01.599495    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-73bfc1e9-openshift-default-fc75695d8-8l6ww"
Sep 16 02:41:04.501147 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:41:04.501063    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-73bfc1e9-76785bdc57-lh87l"]
Sep 16 02:41:04.504770 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:41:04.504748    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-73bfc1e9-76785bdc57-lh87l"
Sep 16 02:41:04.507110 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:41:04.507089    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-73bfc1e9-dockercfg-jzw6q\""
Sep 16 02:41:04.507230 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:41:04.507214    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-73bfc1e9\""
Sep 16 02:41:04.515091 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:41:04.515070    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-73bfc1e9-76785bdc57-lh87l"]
Sep 16 02:41:04.605293 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:41:04.605255    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z7pdv\" (UniqueName: \"kubernetes.io/projected/22af3c3e-31ca-4686-b2a0-09419bef9b93-kube-api-access-z7pdv\") pod \"payload-processing-e2e-mt-73bfc1e9-76785bdc57-lh87l\" (UID: \"22af3c3e-31ca-4686-b2a0-09419bef9b93\") " pod="openshift-ingress/payload-processing-e2e-mt-73bfc1e9-76785bdc57-lh87l"
Sep 16 02:41:04.605472 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:41:04.605375    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/22af3c3e-31ca-4686-b2a0-09419bef9b93-plugins-config-volume\") pod \"payload-processing-e2e-mt-73bfc1e9-76785bdc57-lh87l\" (UID: \"22af3c3e-31ca-4686-b2a0-09419bef9b93\") " pod="openshift-ingress/payload-processing-e2e-mt-73bfc1e9-76785bdc57-lh87l"
Sep 16 02:41:04.706731 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:41:04.706696    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/22af3c3e-31ca-4686-b2a0-09419bef9b93-plugins-config-volume\") pod \"payload-processing-e2e-mt-73bfc1e9-76785bdc57-lh87l\" (UID: \"22af3c3e-31ca-4686-b2a0-09419bef9b93\") " pod="openshift-ingress/payload-processing-e2e-mt-73bfc1e9-76785bdc57-lh87l"
Sep 16 02:41:04.706929 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:41:04.706792    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z7pdv\" (UniqueName: \"kubernetes.io/projected/22af3c3e-31ca-4686-b2a0-09419bef9b93-kube-api-access-z7pdv\") pod \"payload-processing-e2e-mt-73bfc1e9-76785bdc57-lh87l\" (UID: \"22af3c3e-31ca-4686-b2a0-09419bef9b93\") " pod="openshift-ingress/payload-processing-e2e-mt-73bfc1e9-76785bdc57-lh87l"
Sep 16 02:41:04.707446 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:41:04.707421    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/22af3c3e-31ca-4686-b2a0-09419bef9b93-plugins-config-volume\") pod \"payload-processing-e2e-mt-73bfc1e9-76785bdc57-lh87l\" (UID: \"22af3c3e-31ca-4686-b2a0-09419bef9b93\") " pod="openshift-ingress/payload-processing-e2e-mt-73bfc1e9-76785bdc57-lh87l"
Sep 16 02:41:04.714237 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:41:04.714215    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-z7pdv\" (UniqueName: \"kubernetes.io/projected/22af3c3e-31ca-4686-b2a0-09419bef9b93-kube-api-access-z7pdv\") pod \"payload-processing-e2e-mt-73bfc1e9-76785bdc57-lh87l\" (UID: \"22af3c3e-31ca-4686-b2a0-09419bef9b93\") " pod="openshift-ingress/payload-processing-e2e-mt-73bfc1e9-76785bdc57-lh87l"
Sep 16 02:41:04.815751 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:41:04.815670    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-73bfc1e9-76785bdc57-lh87l"
Sep 16 02:41:05.083980 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:41:05.083953    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-73bfc1e9-76785bdc57-lh87l"]
Sep 16 02:41:05.086159 ip-10-0-128-111 kubenswrapper[2583]: W0916 02:41:05.086130    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod22af3c3e_31ca_4686_b2a0_09419bef9b93.slice/crio-ccfa9b39dd88641eca63e2ceec3130a3a43b96a30112276e8027df4b2ccbcd3a WatchSource:0}: Error finding container ccfa9b39dd88641eca63e2ceec3130a3a43b96a30112276e8027df4b2ccbcd3a: Status 404 returned error can't find the container with id ccfa9b39dd88641eca63e2ceec3130a3a43b96a30112276e8027df4b2ccbcd3a
Sep 16 02:41:05.619784 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:41:05.619750    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-73bfc1e9-76785bdc57-lh87l" event={"ID":"22af3c3e-31ca-4686-b2a0-09419bef9b93","Type":"ContainerStarted","Data":"02bc18d002f519d2232d5f617428e90bd3157339805dabde1dea4a6429a12960"}
Sep 16 02:41:05.620182 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:41:05.619800    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-73bfc1e9-76785bdc57-lh87l" event={"ID":"22af3c3e-31ca-4686-b2a0-09419bef9b93","Type":"ContainerStarted","Data":"ccfa9b39dd88641eca63e2ceec3130a3a43b96a30112276e8027df4b2ccbcd3a"}
Sep 16 02:41:05.620182 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:41:05.619876    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-mt-73bfc1e9-76785bdc57-lh87l"
Sep 16 02:41:05.639056 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:41:05.639007    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-mt-73bfc1e9-76785bdc57-lh87l" podStartSLOduration=1.6389900769999999 podStartE2EDuration="1.638990077s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 02:41:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 02:41:05.637199874 +0000 UTC m=+1344.104988658" watchObservedRunningTime="2026-09-16 02:41:05.638990077 +0000 UTC m=+1344.106778859"
Sep 16 02:41:11.057004 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:41:11.056971    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-fzl8t"]
Sep 16 02:41:16.626741 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:41:16.626702    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-mt-73bfc1e9-76785bdc57-lh87l"
Sep 16 02:41:21.704040 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:41:21.704000    2583 generic.go:356] "Generic (PLEG): container finished" podID="c9f199d5-ac58-4074-9ff0-b88bec554d35" containerID="f4bf3777bf2955e2c38c986a56ab72cf0832fcbdf2ab0e0564411358f7b26d2d" exitCode=0
Sep 16 02:41:21.704603 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:41:21.704072    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-73bfc1e9-openshift-default-fc75695d8-8l6ww" event={"ID":"c9f199d5-ac58-4074-9ff0-b88bec554d35","Type":"ContainerDied","Data":"f4bf3777bf2955e2c38c986a56ab72cf0832fcbdf2ab0e0564411358f7b26d2d"}
Sep 16 02:41:21.704692 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:41:21.704632    2583 scope.go:117] "RemoveContainer" containerID="f4bf3777bf2955e2c38c986a56ab72cf0832fcbdf2ab0e0564411358f7b26d2d"
Sep 16 02:41:21.705797 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:41:21.705766    2583 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 16 02:41:21.705889 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:41:21.705833    2583 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 16 02:41:21.705889 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:41:21.705866    2583 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 16 02:41:22.710848 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:41:22.710810    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-73bfc1e9-openshift-default-fc75695d8-8l6ww" event={"ID":"c9f199d5-ac58-4074-9ff0-b88bec554d35","Type":"ContainerStarted","Data":"cb85373a9447cff483f0756c5c2c6efa8cd15baf50ce1fa72fe644752346f82b"}
Sep 16 02:41:22.811530 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:41:22.811495    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-73bfc1e9-openshift-default-fc75695d8-8l6ww"
Sep 16 02:41:22.811718 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:41:22.811696    2583 patch_prober.go:28] interesting pod/e2e-mt-73bfc1e9-openshift-default-fc75695d8-8l6ww container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused" start-of-body=
Sep 16 02:41:22.811772 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:41:22.811753    2583 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-73bfc1e9-openshift-default-fc75695d8-8l6ww" podUID="c9f199d5-ac58-4074-9ff0-b88bec554d35" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused"
Sep 16 02:41:23.250460 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:41:23.250420    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-fzl8t"]
Sep 16 02:41:23.811901 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:41:23.811860    2583 patch_prober.go:28] interesting pod/e2e-mt-73bfc1e9-openshift-default-fc75695d8-8l6ww container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused" start-of-body=
Sep 16 02:41:23.812390 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:41:23.811969    2583 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-73bfc1e9-openshift-default-fc75695d8-8l6ww" podUID="c9f199d5-ac58-4074-9ff0-b88bec554d35" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused"
Sep 16 02:41:24.811924 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:41:24.811888    2583 patch_prober.go:28] interesting pod/e2e-mt-73bfc1e9-openshift-default-fc75695d8-8l6ww container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused" start-of-body=
Sep 16 02:41:24.812301 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:41:24.811956    2583 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-73bfc1e9-openshift-default-fc75695d8-8l6ww" podUID="c9f199d5-ac58-4074-9ff0-b88bec554d35" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused"
Sep 16 02:41:25.811798 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:41:25.811758    2583 patch_prober.go:28] interesting pod/e2e-mt-73bfc1e9-openshift-default-fc75695d8-8l6ww container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused" start-of-body=
Sep 16 02:41:25.811978 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:41:25.811812    2583 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-73bfc1e9-openshift-default-fc75695d8-8l6ww" podUID="c9f199d5-ac58-4074-9ff0-b88bec554d35" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused"
Sep 16 02:41:26.811863 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:41:26.811828    2583 patch_prober.go:28] interesting pod/e2e-mt-73bfc1e9-openshift-default-fc75695d8-8l6ww container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused" start-of-body=
Sep 16 02:41:26.812043 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:41:26.811881    2583 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-73bfc1e9-openshift-default-fc75695d8-8l6ww" podUID="c9f199d5-ac58-4074-9ff0-b88bec554d35" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused"
Sep 16 02:41:27.811566 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:41:27.811526    2583 patch_prober.go:28] interesting pod/e2e-mt-73bfc1e9-openshift-default-fc75695d8-8l6ww container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused" start-of-body=
Sep 16 02:41:27.811741 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:41:27.811611    2583 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-73bfc1e9-openshift-default-fc75695d8-8l6ww" podUID="c9f199d5-ac58-4074-9ff0-b88bec554d35" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused"
Sep 16 02:41:28.815442 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:41:28.815411    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-73bfc1e9-openshift-default-fc75695d8-8l6ww"
Sep 16 02:41:28.815856 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:41:28.815762    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-73bfc1e9-openshift-default-fc75695d8-8l6ww"
Sep 16 02:41:28.816337 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:41:28.816319    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-73bfc1e9-openshift-default-fc75695d8-8l6ww"
Sep 16 02:41:41.053570 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:41:41.053516    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-fzl8t"]
Sep 16 02:41:42.156415 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:41:42.156385    2583 scope.go:117] "RemoveContainer" containerID="76c349fdc50fcd8b425c0ef05e86f2966e1c751f61869bbc86f9d15714363793"
Sep 16 02:41:53.435299 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:41:53.435263    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-9088-1c50b6"]
Sep 16 02:41:53.440275 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:41:53.440252    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-9088-1c50b6"
Sep 16 02:41:53.442633 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:41:53.442609    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-7nx29\""
Sep 16 02:41:53.445932 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:41:53.445909    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-9088-1c50b6"]
Sep 16 02:41:53.576498 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:41:53.576466    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8wntv\" (UniqueName: \"kubernetes.io/projected/39fb5e95-d8ed-4d8f-8ed4-be6664f0946a-kube-api-access-8wntv\") pod \"test-curl-9088-1c50b6\" (UID: \"39fb5e95-d8ed-4d8f-8ed4-be6664f0946a\") " pod="opendatahub/test-curl-9088-1c50b6"
Sep 16 02:41:53.677599 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:41:53.677537    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8wntv\" (UniqueName: \"kubernetes.io/projected/39fb5e95-d8ed-4d8f-8ed4-be6664f0946a-kube-api-access-8wntv\") pod \"test-curl-9088-1c50b6\" (UID: \"39fb5e95-d8ed-4d8f-8ed4-be6664f0946a\") " pod="opendatahub/test-curl-9088-1c50b6"
Sep 16 02:41:53.685062 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:41:53.685036    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8wntv\" (UniqueName: \"kubernetes.io/projected/39fb5e95-d8ed-4d8f-8ed4-be6664f0946a-kube-api-access-8wntv\") pod \"test-curl-9088-1c50b6\" (UID: \"39fb5e95-d8ed-4d8f-8ed4-be6664f0946a\") " pod="opendatahub/test-curl-9088-1c50b6"
Sep 16 02:41:53.751162 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:41:53.751091    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-9088-1c50b6"
Sep 16 02:41:54.016564 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:41:54.016518    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-9088-1c50b6"]
Sep 16 02:41:54.018440 ip-10-0-128-111 kubenswrapper[2583]: W0916 02:41:54.018412    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod39fb5e95_d8ed_4d8f_8ed4_be6664f0946a.slice/crio-6bc370dbd94d07938944e53a85f2d5e1de6a915719ccac6ccc771ede5398924f WatchSource:0}: Error finding container 6bc370dbd94d07938944e53a85f2d5e1de6a915719ccac6ccc771ede5398924f: Status 404 returned error can't find the container with id 6bc370dbd94d07938944e53a85f2d5e1de6a915719ccac6ccc771ede5398924f
Sep 16 02:41:54.020322 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:41:54.020308    2583 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 16 02:41:54.894102 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:41:54.894057    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-9088-1c50b6" event={"ID":"39fb5e95-d8ed-4d8f-8ed4-be6664f0946a","Type":"ContainerStarted","Data":"6bc370dbd94d07938944e53a85f2d5e1de6a915719ccac6ccc771ede5398924f"}
Sep 16 02:41:56.908629 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:41:56.908594    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-9088-1c50b6" event={"ID":"39fb5e95-d8ed-4d8f-8ed4-be6664f0946a","Type":"ContainerStarted","Data":"f3f81102ef40dd91ada98132ddf2b9dc9d44ce2f6ec3e3ec60b77229756aeb75"}
Sep 16 02:41:56.923389 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:41:56.923341    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-9088-1c50b6" podStartSLOduration=2.073688471 podStartE2EDuration="3.923323926s" totalImagesPullingTime="1.849635455s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 02:41:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 02:41:56.921446453 +0000 UTC m=+1395.389235236" watchObservedRunningTime="2026-09-16 02:41:56.923323926 +0000 UTC m=+1395.391112781"
Sep 16 02:41:57.488154 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:41:57.488090    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-9088-1c50b6"]
Sep 16 02:41:57.494942 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:41:57.494908    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-9088-1c50b6"]
Sep 16 02:41:57.600450 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:41:57.600420    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-9088-c59682"]
Sep 16 02:41:57.600874 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:41:57.600859    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="39fb5e95-d8ed-4d8f-8ed4-be6664f0946a" containerName="test-curl-9088-1c50b6"
Sep 16 02:41:57.600933 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:41:57.600876    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="39fb5e95-d8ed-4d8f-8ed4-be6664f0946a" containerName="test-curl-9088-1c50b6"
Sep 16 02:41:57.600966 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:41:57.600938    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="39fb5e95-d8ed-4d8f-8ed4-be6664f0946a" containerName="test-curl-9088-1c50b6"
Sep 16 02:41:57.603988 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:41:57.603974    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-9088-c59682"
Sep 16 02:41:57.609983 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:41:57.609958    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-9088-c59682"]
Sep 16 02:41:57.715541 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:41:57.715506    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zwvgb\" (UniqueName: \"kubernetes.io/projected/8159efc6-f4a1-48a5-bdd7-99aadf146823-kube-api-access-zwvgb\") pod \"test-curl-9088-c59682\" (UID: \"8159efc6-f4a1-48a5-bdd7-99aadf146823\") " pod="opendatahub/test-curl-9088-c59682"
Sep 16 02:41:57.816812 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:41:57.816780    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zwvgb\" (UniqueName: \"kubernetes.io/projected/8159efc6-f4a1-48a5-bdd7-99aadf146823-kube-api-access-zwvgb\") pod \"test-curl-9088-c59682\" (UID: \"8159efc6-f4a1-48a5-bdd7-99aadf146823\") " pod="opendatahub/test-curl-9088-c59682"
Sep 16 02:41:57.824059 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:41:57.824033    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zwvgb\" (UniqueName: \"kubernetes.io/projected/8159efc6-f4a1-48a5-bdd7-99aadf146823-kube-api-access-zwvgb\") pod \"test-curl-9088-c59682\" (UID: \"8159efc6-f4a1-48a5-bdd7-99aadf146823\") " pod="opendatahub/test-curl-9088-c59682"
Sep 16 02:41:57.913900 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:41:57.913825    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-9088-c59682"
Sep 16 02:41:57.914237 ip-10-0-128-111 kubenswrapper[2583]: E0916 02:41:57.913955    2583 kuberuntime_manager.go:649] "Missing actuated resource record" pod="opendatahub/test-curl-9088-1c50b6" container="test-curl-9088-1c50b6"
Sep 16 02:41:58.180812 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:41:58.180788    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-9088-c59682"]
Sep 16 02:41:58.182852 ip-10-0-128-111 kubenswrapper[2583]: W0916 02:41:58.182825    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod8159efc6_f4a1_48a5_bdd7_99aadf146823.slice/crio-c016e3d31859e831ed1d5b0c51e95810218cd4be371cc0fd0f987d932dd71cd8 WatchSource:0}: Error finding container c016e3d31859e831ed1d5b0c51e95810218cd4be371cc0fd0f987d932dd71cd8: Status 404 returned error can't find the container with id c016e3d31859e831ed1d5b0c51e95810218cd4be371cc0fd0f987d932dd71cd8
Sep 16 02:41:58.920475 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:41:58.920436    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-9088-c59682" event={"ID":"8159efc6-f4a1-48a5-bdd7-99aadf146823","Type":"ContainerStarted","Data":"a487bdf6d2d37df354748098dd899faaff4fac80e858de59775cd02545b07b88"}
Sep 16 02:41:58.920973 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:41:58.920473    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-9088-1c50b6" podUID="39fb5e95-d8ed-4d8f-8ed4-be6664f0946a" containerName="test-curl-9088-1c50b6" containerID="cri-o://f3f81102ef40dd91ada98132ddf2b9dc9d44ce2f6ec3e3ec60b77229756aeb75" gracePeriod=2
Sep 16 02:41:58.920973 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:41:58.920486    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-9088-c59682" event={"ID":"8159efc6-f4a1-48a5-bdd7-99aadf146823","Type":"ContainerStarted","Data":"c016e3d31859e831ed1d5b0c51e95810218cd4be371cc0fd0f987d932dd71cd8"}
Sep 16 02:41:58.933216 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:41:58.933162    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-9088-c59682" podStartSLOduration=1.9331320349999999 podStartE2EDuration="1.933132035s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 02:41:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 02:41:58.932571847 +0000 UTC m=+1397.400360624" watchObservedRunningTime="2026-09-16 02:41:58.933132035 +0000 UTC m=+1397.400920817"
Sep 16 02:41:59.485924 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:41:59.485892    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-9088-c59682"]
Sep 16 02:41:59.488041 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:41:59.488011    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-9088-c59682"]
Sep 16 02:41:59.853262 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:41:59.853231    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-fzl8t"]
Sep 16 02:41:59.926008 ip-10-0-128-111 kubenswrapper[2583]: E0916 02:41:59.925977    2583 kuberuntime_manager.go:649] "Missing actuated resource record" pod="opendatahub/test-curl-9088-c59682" container="test-curl-9088-c59682"
Sep 16 02:42:00.930586 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:42:00.930537    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-9088-c59682" podUID="8159efc6-f4a1-48a5-bdd7-99aadf146823" containerName="test-curl-9088-c59682" containerID="cri-o://a487bdf6d2d37df354748098dd899faaff4fac80e858de59775cd02545b07b88" gracePeriod=2
Sep 16 02:42:01.088314 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:42:01.088289    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-9088-1c50b6"
Sep 16 02:42:01.147573 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:42:01.147513    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8wntv\" (UniqueName: \"kubernetes.io/projected/39fb5e95-d8ed-4d8f-8ed4-be6664f0946a-kube-api-access-8wntv\") pod \"39fb5e95-d8ed-4d8f-8ed4-be6664f0946a\" (UID: \"39fb5e95-d8ed-4d8f-8ed4-be6664f0946a\") "
Sep 16 02:42:01.149659 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:42:01.149633    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/39fb5e95-d8ed-4d8f-8ed4-be6664f0946a-kube-api-access-8wntv" (OuterVolumeSpecName: "kube-api-access-8wntv") pod "39fb5e95-d8ed-4d8f-8ed4-be6664f0946a" (UID: "39fb5e95-d8ed-4d8f-8ed4-be6664f0946a"). InnerVolumeSpecName "kube-api-access-8wntv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 02:42:01.249095 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:42:01.249003    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8wntv\" (UniqueName: \"kubernetes.io/projected/39fb5e95-d8ed-4d8f-8ed4-be6664f0946a-kube-api-access-8wntv\") on node \"ip-10-0-128-111.ec2.internal\" DevicePath \"\""
Sep 16 02:42:01.431331 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:42:01.431299    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w2-db1ed7/sub-test-model-db1ed7-kserve-b99f74c57-55zfc"]
Sep 16 02:42:01.936150 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:42:01.936116    2583 generic.go:356] "Generic (PLEG): container finished" podID="39fb5e95-d8ed-4d8f-8ed4-be6664f0946a" containerID="f3f81102ef40dd91ada98132ddf2b9dc9d44ce2f6ec3e3ec60b77229756aeb75" exitCode=137
Sep 16 02:42:01.936540 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:42:01.936156    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-9088-1c50b6"
Sep 16 02:42:01.936540 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:42:01.936224    2583 scope.go:117] "RemoveContainer" containerID="f3f81102ef40dd91ada98132ddf2b9dc9d44ce2f6ec3e3ec60b77229756aeb75"
Sep 16 02:42:01.945033 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:42:01.945016    2583 scope.go:117] "RemoveContainer" containerID="f3f81102ef40dd91ada98132ddf2b9dc9d44ce2f6ec3e3ec60b77229756aeb75"
Sep 16 02:42:01.945286 ip-10-0-128-111 kubenswrapper[2583]: E0916 02:42:01.945270    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f3f81102ef40dd91ada98132ddf2b9dc9d44ce2f6ec3e3ec60b77229756aeb75\": container with ID starting with f3f81102ef40dd91ada98132ddf2b9dc9d44ce2f6ec3e3ec60b77229756aeb75 not found: ID does not exist" containerID="f3f81102ef40dd91ada98132ddf2b9dc9d44ce2f6ec3e3ec60b77229756aeb75"
Sep 16 02:42:01.945332 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:42:01.945292    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f3f81102ef40dd91ada98132ddf2b9dc9d44ce2f6ec3e3ec60b77229756aeb75"} err="failed to get container status \"f3f81102ef40dd91ada98132ddf2b9dc9d44ce2f6ec3e3ec60b77229756aeb75\": rpc error: code = NotFound desc = could not find container \"f3f81102ef40dd91ada98132ddf2b9dc9d44ce2f6ec3e3ec60b77229756aeb75\": container with ID starting with f3f81102ef40dd91ada98132ddf2b9dc9d44ce2f6ec3e3ec60b77229756aeb75 not found: ID does not exist"
Sep 16 02:42:01.946608 ip-10-0-128-111 kubenswrapper[2583]: E0916 02:42:01.946583    2583 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-9088-1c50b6\" is forbidden: User \"system:node:ip-10-0-128-111.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-128-111.ec2.internal' and this object" podUID="39fb5e95-d8ed-4d8f-8ed4-be6664f0946a" pod="opendatahub/test-curl-9088-1c50b6"
Sep 16 02:42:02.043451 ip-10-0-128-111 kubenswrapper[2583]: E0916 02:42:02.043418    2583 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-9088-1c50b6\" is forbidden: User \"system:node:ip-10-0-128-111.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-128-111.ec2.internal' and this object" podUID="39fb5e95-d8ed-4d8f-8ed4-be6664f0946a" pod="opendatahub/test-curl-9088-1c50b6"
Sep 16 02:42:02.045195 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:42:02.045172    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="39fb5e95-d8ed-4d8f-8ed4-be6664f0946a" path="/var/lib/kubelet/pods/39fb5e95-d8ed-4d8f-8ed4-be6664f0946a/volumes"
Sep 16 02:42:03.124172 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:42:03.124147    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-9088-c59682"
Sep 16 02:42:03.267606 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:42:03.267497    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zwvgb\" (UniqueName: \"kubernetes.io/projected/8159efc6-f4a1-48a5-bdd7-99aadf146823-kube-api-access-zwvgb\") pod \"8159efc6-f4a1-48a5-bdd7-99aadf146823\" (UID: \"8159efc6-f4a1-48a5-bdd7-99aadf146823\") "
Sep 16 02:42:03.269589 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:42:03.269524    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8159efc6-f4a1-48a5-bdd7-99aadf146823-kube-api-access-zwvgb" (OuterVolumeSpecName: "kube-api-access-zwvgb") pod "8159efc6-f4a1-48a5-bdd7-99aadf146823" (UID: "8159efc6-f4a1-48a5-bdd7-99aadf146823"). InnerVolumeSpecName "kube-api-access-zwvgb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 02:42:03.369117 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:42:03.369082    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zwvgb\" (UniqueName: \"kubernetes.io/projected/8159efc6-f4a1-48a5-bdd7-99aadf146823-kube-api-access-zwvgb\") on node \"ip-10-0-128-111.ec2.internal\" DevicePath \"\""
Sep 16 02:42:03.949532 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:42:03.949495    2583 generic.go:356] "Generic (PLEG): container finished" podID="8159efc6-f4a1-48a5-bdd7-99aadf146823" containerID="a487bdf6d2d37df354748098dd899faaff4fac80e858de59775cd02545b07b88" exitCode=137
Sep 16 02:42:03.949532 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:42:03.949529    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-9088-c59682"
Sep 16 02:42:03.949783 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:42:03.949598    2583 scope.go:117] "RemoveContainer" containerID="a487bdf6d2d37df354748098dd899faaff4fac80e858de59775cd02545b07b88"
Sep 16 02:42:03.958064 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:42:03.958047    2583 scope.go:117] "RemoveContainer" containerID="a487bdf6d2d37df354748098dd899faaff4fac80e858de59775cd02545b07b88"
Sep 16 02:42:03.958322 ip-10-0-128-111 kubenswrapper[2583]: E0916 02:42:03.958305    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a487bdf6d2d37df354748098dd899faaff4fac80e858de59775cd02545b07b88\": container with ID starting with a487bdf6d2d37df354748098dd899faaff4fac80e858de59775cd02545b07b88 not found: ID does not exist" containerID="a487bdf6d2d37df354748098dd899faaff4fac80e858de59775cd02545b07b88"
Sep 16 02:42:03.958387 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:42:03.958327    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a487bdf6d2d37df354748098dd899faaff4fac80e858de59775cd02545b07b88"} err="failed to get container status \"a487bdf6d2d37df354748098dd899faaff4fac80e858de59775cd02545b07b88\": rpc error: code = NotFound desc = could not find container \"a487bdf6d2d37df354748098dd899faaff4fac80e858de59775cd02545b07b88\": container with ID starting with a487bdf6d2d37df354748098dd899faaff4fac80e858de59775cd02545b07b88 not found: ID does not exist"
Sep 16 02:42:03.959618 ip-10-0-128-111 kubenswrapper[2583]: E0916 02:42:03.959598    2583 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-9088-c59682\" is forbidden: User \"system:node:ip-10-0-128-111.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-128-111.ec2.internal' and this object" podUID="8159efc6-f4a1-48a5-bdd7-99aadf146823" pod="opendatahub/test-curl-9088-c59682"
Sep 16 02:42:04.045705 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:42:04.045673    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8159efc6-f4a1-48a5-bdd7-99aadf146823" path="/var/lib/kubelet/pods/8159efc6-f4a1-48a5-bdd7-99aadf146823/volumes"
Sep 16 02:42:12.254222 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:42:12.254187    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w2-db1ed7/rate-test-model-db1ed7-kserve-7f9997cf75-csxff"]
Sep 16 02:42:12.254874 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:42:12.254845    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8159efc6-f4a1-48a5-bdd7-99aadf146823" containerName="test-curl-9088-c59682"
Sep 16 02:42:12.254874 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:42:12.254874    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="8159efc6-f4a1-48a5-bdd7-99aadf146823" containerName="test-curl-9088-c59682"
Sep 16 02:42:12.255085 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:42:12.255017    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="8159efc6-f4a1-48a5-bdd7-99aadf146823" containerName="test-curl-9088-c59682"
Sep 16 02:42:12.257886 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:42:12.257868    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w2-db1ed7/rate-test-model-db1ed7-kserve-7f9997cf75-csxff"
Sep 16 02:42:12.260223 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:42:12.260201    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w2-db1ed7\"/\"rate-test-model-db1ed7-kserve-self-signed-certs\""
Sep 16 02:42:12.268702 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:42:12.268681    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w2-db1ed7/rate-test-model-db1ed7-kserve-7f9997cf75-csxff"]
Sep 16 02:42:12.353074 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:42:12.353036    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4694ad1d-3bf3-4f02-afc3-31b4282778cd-dshm\") pod \"rate-test-model-db1ed7-kserve-7f9997cf75-csxff\" (UID: \"4694ad1d-3bf3-4f02-afc3-31b4282778cd\") " pod="ai-tenant-e2e-shared-a-w2-db1ed7/rate-test-model-db1ed7-kserve-7f9997cf75-csxff"
Sep 16 02:42:12.353228 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:42:12.353088    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4694ad1d-3bf3-4f02-afc3-31b4282778cd-home\") pod \"rate-test-model-db1ed7-kserve-7f9997cf75-csxff\" (UID: \"4694ad1d-3bf3-4f02-afc3-31b4282778cd\") " pod="ai-tenant-e2e-shared-a-w2-db1ed7/rate-test-model-db1ed7-kserve-7f9997cf75-csxff"
Sep 16 02:42:12.353228 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:42:12.353146    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4694ad1d-3bf3-4f02-afc3-31b4282778cd-model-cache\") pod \"rate-test-model-db1ed7-kserve-7f9997cf75-csxff\" (UID: \"4694ad1d-3bf3-4f02-afc3-31b4282778cd\") " pod="ai-tenant-e2e-shared-a-w2-db1ed7/rate-test-model-db1ed7-kserve-7f9997cf75-csxff"
Sep 16 02:42:12.353228 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:42:12.353167    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-khhqj\" (UniqueName: \"kubernetes.io/projected/4694ad1d-3bf3-4f02-afc3-31b4282778cd-kube-api-access-khhqj\") pod \"rate-test-model-db1ed7-kserve-7f9997cf75-csxff\" (UID: \"4694ad1d-3bf3-4f02-afc3-31b4282778cd\") " pod="ai-tenant-e2e-shared-a-w2-db1ed7/rate-test-model-db1ed7-kserve-7f9997cf75-csxff"
Sep 16 02:42:12.353228 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:42:12.353207    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4694ad1d-3bf3-4f02-afc3-31b4282778cd-tls-certs\") pod \"rate-test-model-db1ed7-kserve-7f9997cf75-csxff\" (UID: \"4694ad1d-3bf3-4f02-afc3-31b4282778cd\") " pod="ai-tenant-e2e-shared-a-w2-db1ed7/rate-test-model-db1ed7-kserve-7f9997cf75-csxff"
Sep 16 02:42:12.353465 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:42:12.353237    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4694ad1d-3bf3-4f02-afc3-31b4282778cd-tmp-dir\") pod \"rate-test-model-db1ed7-kserve-7f9997cf75-csxff\" (UID: \"4694ad1d-3bf3-4f02-afc3-31b4282778cd\") " pod="ai-tenant-e2e-shared-a-w2-db1ed7/rate-test-model-db1ed7-kserve-7f9997cf75-csxff"
Sep 16 02:42:12.454417 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:42:12.454381    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4694ad1d-3bf3-4f02-afc3-31b4282778cd-model-cache\") pod \"rate-test-model-db1ed7-kserve-7f9997cf75-csxff\" (UID: \"4694ad1d-3bf3-4f02-afc3-31b4282778cd\") " pod="ai-tenant-e2e-shared-a-w2-db1ed7/rate-test-model-db1ed7-kserve-7f9997cf75-csxff"
Sep 16 02:42:12.454616 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:42:12.454434    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-khhqj\" (UniqueName: \"kubernetes.io/projected/4694ad1d-3bf3-4f02-afc3-31b4282778cd-kube-api-access-khhqj\") pod \"rate-test-model-db1ed7-kserve-7f9997cf75-csxff\" (UID: \"4694ad1d-3bf3-4f02-afc3-31b4282778cd\") " pod="ai-tenant-e2e-shared-a-w2-db1ed7/rate-test-model-db1ed7-kserve-7f9997cf75-csxff"
Sep 16 02:42:12.454616 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:42:12.454487    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4694ad1d-3bf3-4f02-afc3-31b4282778cd-tls-certs\") pod \"rate-test-model-db1ed7-kserve-7f9997cf75-csxff\" (UID: \"4694ad1d-3bf3-4f02-afc3-31b4282778cd\") " pod="ai-tenant-e2e-shared-a-w2-db1ed7/rate-test-model-db1ed7-kserve-7f9997cf75-csxff"
Sep 16 02:42:12.454616 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:42:12.454525    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4694ad1d-3bf3-4f02-afc3-31b4282778cd-tmp-dir\") pod \"rate-test-model-db1ed7-kserve-7f9997cf75-csxff\" (UID: \"4694ad1d-3bf3-4f02-afc3-31b4282778cd\") " pod="ai-tenant-e2e-shared-a-w2-db1ed7/rate-test-model-db1ed7-kserve-7f9997cf75-csxff"
Sep 16 02:42:12.454821 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:42:12.454641    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4694ad1d-3bf3-4f02-afc3-31b4282778cd-dshm\") pod \"rate-test-model-db1ed7-kserve-7f9997cf75-csxff\" (UID: \"4694ad1d-3bf3-4f02-afc3-31b4282778cd\") " pod="ai-tenant-e2e-shared-a-w2-db1ed7/rate-test-model-db1ed7-kserve-7f9997cf75-csxff"
Sep 16 02:42:12.454821 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:42:12.454703    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4694ad1d-3bf3-4f02-afc3-31b4282778cd-home\") pod \"rate-test-model-db1ed7-kserve-7f9997cf75-csxff\" (UID: \"4694ad1d-3bf3-4f02-afc3-31b4282778cd\") " pod="ai-tenant-e2e-shared-a-w2-db1ed7/rate-test-model-db1ed7-kserve-7f9997cf75-csxff"
Sep 16 02:42:12.454930 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:42:12.454884    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4694ad1d-3bf3-4f02-afc3-31b4282778cd-model-cache\") pod \"rate-test-model-db1ed7-kserve-7f9997cf75-csxff\" (UID: \"4694ad1d-3bf3-4f02-afc3-31b4282778cd\") " pod="ai-tenant-e2e-shared-a-w2-db1ed7/rate-test-model-db1ed7-kserve-7f9997cf75-csxff"
Sep 16 02:42:12.454983 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:42:12.454926    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4694ad1d-3bf3-4f02-afc3-31b4282778cd-tmp-dir\") pod \"rate-test-model-db1ed7-kserve-7f9997cf75-csxff\" (UID: \"4694ad1d-3bf3-4f02-afc3-31b4282778cd\") " pod="ai-tenant-e2e-shared-a-w2-db1ed7/rate-test-model-db1ed7-kserve-7f9997cf75-csxff"
Sep 16 02:42:12.455116 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:42:12.455086    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4694ad1d-3bf3-4f02-afc3-31b4282778cd-home\") pod \"rate-test-model-db1ed7-kserve-7f9997cf75-csxff\" (UID: \"4694ad1d-3bf3-4f02-afc3-31b4282778cd\") " pod="ai-tenant-e2e-shared-a-w2-db1ed7/rate-test-model-db1ed7-kserve-7f9997cf75-csxff"
Sep 16 02:42:12.457834 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:42:12.457805    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4694ad1d-3bf3-4f02-afc3-31b4282778cd-dshm\") pod \"rate-test-model-db1ed7-kserve-7f9997cf75-csxff\" (UID: \"4694ad1d-3bf3-4f02-afc3-31b4282778cd\") " pod="ai-tenant-e2e-shared-a-w2-db1ed7/rate-test-model-db1ed7-kserve-7f9997cf75-csxff"
Sep 16 02:42:12.458818 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:42:12.458796    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4694ad1d-3bf3-4f02-afc3-31b4282778cd-tls-certs\") pod \"rate-test-model-db1ed7-kserve-7f9997cf75-csxff\" (UID: \"4694ad1d-3bf3-4f02-afc3-31b4282778cd\") " pod="ai-tenant-e2e-shared-a-w2-db1ed7/rate-test-model-db1ed7-kserve-7f9997cf75-csxff"
Sep 16 02:42:12.461509 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:42:12.461481    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-khhqj\" (UniqueName: \"kubernetes.io/projected/4694ad1d-3bf3-4f02-afc3-31b4282778cd-kube-api-access-khhqj\") pod \"rate-test-model-db1ed7-kserve-7f9997cf75-csxff\" (UID: \"4694ad1d-3bf3-4f02-afc3-31b4282778cd\") " pod="ai-tenant-e2e-shared-a-w2-db1ed7/rate-test-model-db1ed7-kserve-7f9997cf75-csxff"
Sep 16 02:42:12.568739 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:42:12.568646    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w2-db1ed7/rate-test-model-db1ed7-kserve-7f9997cf75-csxff"
Sep 16 02:42:12.839689 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:42:12.839655    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w2-db1ed7/rate-test-model-db1ed7-kserve-7f9997cf75-csxff"]
Sep 16 02:42:12.842657 ip-10-0-128-111 kubenswrapper[2583]: W0916 02:42:12.842627    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod4694ad1d_3bf3_4f02_afc3_31b4282778cd.slice/crio-50213d832d24057375aca436a749f2e6db06f7bcfebd58b427ee65be4a4709ca WatchSource:0}: Error finding container 50213d832d24057375aca436a749f2e6db06f7bcfebd58b427ee65be4a4709ca: Status 404 returned error can't find the container with id 50213d832d24057375aca436a749f2e6db06f7bcfebd58b427ee65be4a4709ca
Sep 16 02:42:12.997957 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:42:12.997918    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w2-db1ed7/rate-test-model-db1ed7-kserve-7f9997cf75-csxff" event={"ID":"4694ad1d-3bf3-4f02-afc3-31b4282778cd","Type":"ContainerStarted","Data":"9e038aa2e7b4611dd467b6562551cb66e9c02df00d4158c8ba9f29aecc35e20e"}
Sep 16 02:42:12.998157 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:42:12.997965    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w2-db1ed7/rate-test-model-db1ed7-kserve-7f9997cf75-csxff" event={"ID":"4694ad1d-3bf3-4f02-afc3-31b4282778cd","Type":"ContainerStarted","Data":"50213d832d24057375aca436a749f2e6db06f7bcfebd58b427ee65be4a4709ca"}
Sep 16 02:42:13.012877 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:42:13.012827    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w2-db1ed7/rate-test-model-db1ed7-kserve-7f9997cf75-csxff" podStartSLOduration=1.012811675 podStartE2EDuration="1.012811675s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 02:42:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 02:42:13.011655215 +0000 UTC m=+1411.479443988" watchObservedRunningTime="2026-09-16 02:42:13.012811675 +0000 UTC m=+1411.480600453"
Sep 16 02:42:16.452113 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:42:16.452066    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w2-db1ed7/sub-test-model-db1ed7-kserve-b99f74c57-55zfc" podUID="eb24e349-afc0-42e5-bf95-03cfcf5ff813" containerName="main" containerID="cri-o://6dbeb0463c4fec9fd0ca1c354822c0f11ccb4c9ca073d7fae1bda69f8acad462" gracePeriod=45
Sep 16 02:42:16.705843 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:42:16.705776    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w2-db1ed7/sub-test-model-db1ed7-kserve-b99f74c57-55zfc"
Sep 16 02:42:16.795478 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:42:16.795449    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dzxhk\" (UniqueName: \"kubernetes.io/projected/eb24e349-afc0-42e5-bf95-03cfcf5ff813-kube-api-access-dzxhk\") pod \"eb24e349-afc0-42e5-bf95-03cfcf5ff813\" (UID: \"eb24e349-afc0-42e5-bf95-03cfcf5ff813\") "
Sep 16 02:42:16.795686 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:42:16.795497    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/eb24e349-afc0-42e5-bf95-03cfcf5ff813-tmp-dir\") pod \"eb24e349-afc0-42e5-bf95-03cfcf5ff813\" (UID: \"eb24e349-afc0-42e5-bf95-03cfcf5ff813\") "
Sep 16 02:42:16.795686 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:42:16.795576    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/eb24e349-afc0-42e5-bf95-03cfcf5ff813-home\") pod \"eb24e349-afc0-42e5-bf95-03cfcf5ff813\" (UID: \"eb24e349-afc0-42e5-bf95-03cfcf5ff813\") "
Sep 16 02:42:16.795686 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:42:16.795601    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/eb24e349-afc0-42e5-bf95-03cfcf5ff813-tls-certs\") pod \"eb24e349-afc0-42e5-bf95-03cfcf5ff813\" (UID: \"eb24e349-afc0-42e5-bf95-03cfcf5ff813\") "
Sep 16 02:42:16.795686 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:42:16.795622    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/eb24e349-afc0-42e5-bf95-03cfcf5ff813-dshm\") pod \"eb24e349-afc0-42e5-bf95-03cfcf5ff813\" (UID: \"eb24e349-afc0-42e5-bf95-03cfcf5ff813\") "
Sep 16 02:42:16.795686 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:42:16.795655    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/eb24e349-afc0-42e5-bf95-03cfcf5ff813-model-cache\") pod \"eb24e349-afc0-42e5-bf95-03cfcf5ff813\" (UID: \"eb24e349-afc0-42e5-bf95-03cfcf5ff813\") "
Sep 16 02:42:16.795949 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:42:16.795828    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eb24e349-afc0-42e5-bf95-03cfcf5ff813-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "eb24e349-afc0-42e5-bf95-03cfcf5ff813" (UID: "eb24e349-afc0-42e5-bf95-03cfcf5ff813"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 02:42:16.795949 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:42:16.795886    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eb24e349-afc0-42e5-bf95-03cfcf5ff813-home" (OuterVolumeSpecName: "home") pod "eb24e349-afc0-42e5-bf95-03cfcf5ff813" (UID: "eb24e349-afc0-42e5-bf95-03cfcf5ff813"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 02:42:16.796034 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:42:16.795973    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eb24e349-afc0-42e5-bf95-03cfcf5ff813-model-cache" (OuterVolumeSpecName: "model-cache") pod "eb24e349-afc0-42e5-bf95-03cfcf5ff813" (UID: "eb24e349-afc0-42e5-bf95-03cfcf5ff813"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 02:42:16.796165 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:42:16.796134    2583 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/eb24e349-afc0-42e5-bf95-03cfcf5ff813-tmp-dir\") on node \"ip-10-0-128-111.ec2.internal\" DevicePath \"\""
Sep 16 02:42:16.796165 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:42:16.796155    2583 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/eb24e349-afc0-42e5-bf95-03cfcf5ff813-home\") on node \"ip-10-0-128-111.ec2.internal\" DevicePath \"\""
Sep 16 02:42:16.796165 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:42:16.796165    2583 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/eb24e349-afc0-42e5-bf95-03cfcf5ff813-model-cache\") on node \"ip-10-0-128-111.ec2.internal\" DevicePath \"\""
Sep 16 02:42:16.797724 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:42:16.797701    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/eb24e349-afc0-42e5-bf95-03cfcf5ff813-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "eb24e349-afc0-42e5-bf95-03cfcf5ff813" (UID: "eb24e349-afc0-42e5-bf95-03cfcf5ff813"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 02:42:16.797724 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:42:16.797716    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eb24e349-afc0-42e5-bf95-03cfcf5ff813-dshm" (OuterVolumeSpecName: "dshm") pod "eb24e349-afc0-42e5-bf95-03cfcf5ff813" (UID: "eb24e349-afc0-42e5-bf95-03cfcf5ff813"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 02:42:16.797841 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:42:16.797722    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eb24e349-afc0-42e5-bf95-03cfcf5ff813-kube-api-access-dzxhk" (OuterVolumeSpecName: "kube-api-access-dzxhk") pod "eb24e349-afc0-42e5-bf95-03cfcf5ff813" (UID: "eb24e349-afc0-42e5-bf95-03cfcf5ff813"). InnerVolumeSpecName "kube-api-access-dzxhk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 02:42:16.896933 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:42:16.896896    2583 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/eb24e349-afc0-42e5-bf95-03cfcf5ff813-tls-certs\") on node \"ip-10-0-128-111.ec2.internal\" DevicePath \"\""
Sep 16 02:42:16.896933 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:42:16.896926    2583 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/eb24e349-afc0-42e5-bf95-03cfcf5ff813-dshm\") on node \"ip-10-0-128-111.ec2.internal\" DevicePath \"\""
Sep 16 02:42:16.896933 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:42:16.896937    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dzxhk\" (UniqueName: \"kubernetes.io/projected/eb24e349-afc0-42e5-bf95-03cfcf5ff813-kube-api-access-dzxhk\") on node \"ip-10-0-128-111.ec2.internal\" DevicePath \"\""
Sep 16 02:42:17.018947 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:42:17.018854    2583 generic.go:356] "Generic (PLEG): container finished" podID="eb24e349-afc0-42e5-bf95-03cfcf5ff813" containerID="6dbeb0463c4fec9fd0ca1c354822c0f11ccb4c9ca073d7fae1bda69f8acad462" exitCode=0
Sep 16 02:42:17.018947 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:42:17.018898    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w2-db1ed7/sub-test-model-db1ed7-kserve-b99f74c57-55zfc"
Sep 16 02:42:17.019230 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:42:17.018938    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w2-db1ed7/sub-test-model-db1ed7-kserve-b99f74c57-55zfc" event={"ID":"eb24e349-afc0-42e5-bf95-03cfcf5ff813","Type":"ContainerDied","Data":"6dbeb0463c4fec9fd0ca1c354822c0f11ccb4c9ca073d7fae1bda69f8acad462"}
Sep 16 02:42:17.019230 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:42:17.018995    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w2-db1ed7/sub-test-model-db1ed7-kserve-b99f74c57-55zfc" event={"ID":"eb24e349-afc0-42e5-bf95-03cfcf5ff813","Type":"ContainerDied","Data":"5fdf69d22b7ffa3c2e3a055acc6370287427b21da7f83f3f786602b42d846687"}
Sep 16 02:42:17.019230 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:42:17.019019    2583 scope.go:117] "RemoveContainer" containerID="6dbeb0463c4fec9fd0ca1c354822c0f11ccb4c9ca073d7fae1bda69f8acad462"
Sep 16 02:42:17.028881 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:42:17.028857    2583 scope.go:117] "RemoveContainer" containerID="6dbeb0463c4fec9fd0ca1c354822c0f11ccb4c9ca073d7fae1bda69f8acad462"
Sep 16 02:42:17.029159 ip-10-0-128-111 kubenswrapper[2583]: E0916 02:42:17.029138    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6dbeb0463c4fec9fd0ca1c354822c0f11ccb4c9ca073d7fae1bda69f8acad462\": container with ID starting with 6dbeb0463c4fec9fd0ca1c354822c0f11ccb4c9ca073d7fae1bda69f8acad462 not found: ID does not exist" containerID="6dbeb0463c4fec9fd0ca1c354822c0f11ccb4c9ca073d7fae1bda69f8acad462"
Sep 16 02:42:17.029221 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:42:17.029170    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6dbeb0463c4fec9fd0ca1c354822c0f11ccb4c9ca073d7fae1bda69f8acad462"} err="failed to get container status \"6dbeb0463c4fec9fd0ca1c354822c0f11ccb4c9ca073d7fae1bda69f8acad462\": rpc error: code = NotFound desc = could not find container \"6dbeb0463c4fec9fd0ca1c354822c0f11ccb4c9ca073d7fae1bda69f8acad462\": container with ID starting with 6dbeb0463c4fec9fd0ca1c354822c0f11ccb4c9ca073d7fae1bda69f8acad462 not found: ID does not exist"
Sep 16 02:42:17.043714 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:42:17.043679    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w2-db1ed7/sub-test-model-db1ed7-kserve-b99f74c57-55zfc"]
Sep 16 02:42:17.048609 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:42:17.048586    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w2-db1ed7/sub-test-model-db1ed7-kserve-b99f74c57-55zfc"]
Sep 16 02:42:17.056456 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:42:17.056435    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-fzl8t"]
Sep 16 02:42:18.046472 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:42:18.046429    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="eb24e349-afc0-42e5-bf95-03cfcf5ff813" path="/var/lib/kubelet/pods/eb24e349-afc0-42e5-bf95-03cfcf5ff813/volumes"
Sep 16 02:42:22.568889 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:42:22.568849    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w2-db1ed7/rate-test-model-db1ed7-kserve-7f9997cf75-csxff"
Sep 16 02:42:22.568889 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:42:22.568897    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w2-db1ed7/rate-test-model-db1ed7-kserve-7f9997cf75-csxff"
Sep 16 02:42:22.581259 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:42:22.581225    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w2-db1ed7/rate-test-model-db1ed7-kserve-7f9997cf75-csxff"
Sep 16 02:42:23.063193 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:42:23.063165    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w2-db1ed7/rate-test-model-db1ed7-kserve-7f9997cf75-csxff"
Sep 16 02:42:26.514984 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:42:26.514947    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-73bfc1e9-76785bdc57-lh87l"]
Sep 16 02:42:26.515520 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:42:26.515187    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-mt-73bfc1e9-76785bdc57-lh87l" podUID="22af3c3e-31ca-4686-b2a0-09419bef9b93" containerName="payload-processing" containerID="cri-o://02bc18d002f519d2232d5f617428e90bd3157339805dabde1dea4a6429a12960" gracePeriod=30
Sep 16 02:42:26.625545 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:42:26.625503    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-mt-73bfc1e9-76785bdc57-lh87l" podUID="22af3c3e-31ca-4686-b2a0-09419bef9b93" containerName="payload-processing" probeResult="failure" output="dial tcp 10.134.0.62:9004: connect: connection refused"
Sep 16 02:42:26.779400 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:42:26.779340    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-73bfc1e9-76785bdc57-lh87l"
Sep 16 02:42:26.809336 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:42:26.809300    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-z7pdv\" (UniqueName: \"kubernetes.io/projected/22af3c3e-31ca-4686-b2a0-09419bef9b93-kube-api-access-z7pdv\") pod \"22af3c3e-31ca-4686-b2a0-09419bef9b93\" (UID: \"22af3c3e-31ca-4686-b2a0-09419bef9b93\") "
Sep 16 02:42:26.809500 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:42:26.809352    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/22af3c3e-31ca-4686-b2a0-09419bef9b93-plugins-config-volume\") pod \"22af3c3e-31ca-4686-b2a0-09419bef9b93\" (UID: \"22af3c3e-31ca-4686-b2a0-09419bef9b93\") "
Sep 16 02:42:26.809745 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:42:26.809723    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/22af3c3e-31ca-4686-b2a0-09419bef9b93-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "22af3c3e-31ca-4686-b2a0-09419bef9b93" (UID: "22af3c3e-31ca-4686-b2a0-09419bef9b93"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 02:42:26.811411 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:42:26.811388    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/22af3c3e-31ca-4686-b2a0-09419bef9b93-kube-api-access-z7pdv" (OuterVolumeSpecName: "kube-api-access-z7pdv") pod "22af3c3e-31ca-4686-b2a0-09419bef9b93" (UID: "22af3c3e-31ca-4686-b2a0-09419bef9b93"). InnerVolumeSpecName "kube-api-access-z7pdv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 02:42:26.910348 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:42:26.910297    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z7pdv\" (UniqueName: \"kubernetes.io/projected/22af3c3e-31ca-4686-b2a0-09419bef9b93-kube-api-access-z7pdv\") on node \"ip-10-0-128-111.ec2.internal\" DevicePath \"\""
Sep 16 02:42:26.910348 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:42:26.910340    2583 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/22af3c3e-31ca-4686-b2a0-09419bef9b93-plugins-config-volume\") on node \"ip-10-0-128-111.ec2.internal\" DevicePath \"\""
Sep 16 02:42:27.075278 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:42:27.075194    2583 generic.go:356] "Generic (PLEG): container finished" podID="22af3c3e-31ca-4686-b2a0-09419bef9b93" containerID="02bc18d002f519d2232d5f617428e90bd3157339805dabde1dea4a6429a12960" exitCode=0
Sep 16 02:42:27.075278 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:42:27.075237    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-73bfc1e9-76785bdc57-lh87l"
Sep 16 02:42:27.075278 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:42:27.075260    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-73bfc1e9-76785bdc57-lh87l" event={"ID":"22af3c3e-31ca-4686-b2a0-09419bef9b93","Type":"ContainerDied","Data":"02bc18d002f519d2232d5f617428e90bd3157339805dabde1dea4a6429a12960"}
Sep 16 02:42:27.075505 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:42:27.075301    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-73bfc1e9-76785bdc57-lh87l" event={"ID":"22af3c3e-31ca-4686-b2a0-09419bef9b93","Type":"ContainerDied","Data":"ccfa9b39dd88641eca63e2ceec3130a3a43b96a30112276e8027df4b2ccbcd3a"}
Sep 16 02:42:27.075505 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:42:27.075323    2583 scope.go:117] "RemoveContainer" containerID="02bc18d002f519d2232d5f617428e90bd3157339805dabde1dea4a6429a12960"
Sep 16 02:42:27.084117 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:42:27.084101    2583 scope.go:117] "RemoveContainer" containerID="02bc18d002f519d2232d5f617428e90bd3157339805dabde1dea4a6429a12960"
Sep 16 02:42:27.084384 ip-10-0-128-111 kubenswrapper[2583]: E0916 02:42:27.084363    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"02bc18d002f519d2232d5f617428e90bd3157339805dabde1dea4a6429a12960\": container with ID starting with 02bc18d002f519d2232d5f617428e90bd3157339805dabde1dea4a6429a12960 not found: ID does not exist" containerID="02bc18d002f519d2232d5f617428e90bd3157339805dabde1dea4a6429a12960"
Sep 16 02:42:27.084443 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:42:27.084391    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"02bc18d002f519d2232d5f617428e90bd3157339805dabde1dea4a6429a12960"} err="failed to get container status \"02bc18d002f519d2232d5f617428e90bd3157339805dabde1dea4a6429a12960\": rpc error: code = NotFound desc = could not find container \"02bc18d002f519d2232d5f617428e90bd3157339805dabde1dea4a6429a12960\": container with ID starting with 02bc18d002f519d2232d5f617428e90bd3157339805dabde1dea4a6429a12960 not found: ID does not exist"
Sep 16 02:42:27.109688 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:42:27.109656    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-73bfc1e9-76785bdc57-lh87l"]
Sep 16 02:42:27.126291 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:42:27.126269    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-73bfc1e9-76785bdc57-lh87l"]
Sep 16 02:42:28.045482 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:42:28.045446    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="22af3c3e-31ca-4686-b2a0-09419bef9b93" path="/var/lib/kubelet/pods/22af3c3e-31ca-4686-b2a0-09419bef9b93/volumes"
Sep 16 02:42:32.052462 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:42:32.052428    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w2-eee708/rate-test-model-eee708-kserve-77bb8d6bcf-lpfhm"]
Sep 16 02:42:32.053023 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:42:32.052977    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="22af3c3e-31ca-4686-b2a0-09419bef9b93" containerName="payload-processing"
Sep 16 02:42:32.053023 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:42:32.052994    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="22af3c3e-31ca-4686-b2a0-09419bef9b93" containerName="payload-processing"
Sep 16 02:42:32.053023 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:42:32.053005    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="eb24e349-afc0-42e5-bf95-03cfcf5ff813" containerName="main"
Sep 16 02:42:32.053023 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:42:32.053010    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="eb24e349-afc0-42e5-bf95-03cfcf5ff813" containerName="main"
Sep 16 02:42:32.053184 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:42:32.053084    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="eb24e349-afc0-42e5-bf95-03cfcf5ff813" containerName="main"
Sep 16 02:42:32.053184 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:42:32.053093    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="22af3c3e-31ca-4686-b2a0-09419bef9b93" containerName="payload-processing"
Sep 16 02:42:32.061285 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:42:32.061254    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w2-eee708/rate-test-model-eee708-kserve-77bb8d6bcf-lpfhm"
Sep 16 02:42:32.063664 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:42:32.063640    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w2-eee708/rate-test-model-eee708-kserve-77bb8d6bcf-lpfhm"]
Sep 16 02:42:32.064668 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:42:32.064642    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w2-eee708\"/\"openshift-service-ca.crt\""
Sep 16 02:42:32.064800 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:42:32.064665    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w2-eee708\"/\"rate-test-model-eee708-kserve-self-signed-certs\""
Sep 16 02:42:32.064800 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:42:32.064683    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w2-eee708\"/\"kube-root-ca.crt\""
Sep 16 02:42:32.064800 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:42:32.064665    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w2-eee708\"/\"default-dockercfg-2w275\""
Sep 16 02:42:32.162895 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:42:32.162846    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ceb8aa19-128b-4aa2-a6f8-91013dfdb0c1-home\") pod \"rate-test-model-eee708-kserve-77bb8d6bcf-lpfhm\" (UID: \"ceb8aa19-128b-4aa2-a6f8-91013dfdb0c1\") " pod="ai-tenant-e2e-shared-b-w2-eee708/rate-test-model-eee708-kserve-77bb8d6bcf-lpfhm"
Sep 16 02:42:32.162895 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:42:32.162901    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ceb8aa19-128b-4aa2-a6f8-91013dfdb0c1-dshm\") pod \"rate-test-model-eee708-kserve-77bb8d6bcf-lpfhm\" (UID: \"ceb8aa19-128b-4aa2-a6f8-91013dfdb0c1\") " pod="ai-tenant-e2e-shared-b-w2-eee708/rate-test-model-eee708-kserve-77bb8d6bcf-lpfhm"
Sep 16 02:42:32.163119 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:42:32.162930    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hv5gw\" (UniqueName: \"kubernetes.io/projected/ceb8aa19-128b-4aa2-a6f8-91013dfdb0c1-kube-api-access-hv5gw\") pod \"rate-test-model-eee708-kserve-77bb8d6bcf-lpfhm\" (UID: \"ceb8aa19-128b-4aa2-a6f8-91013dfdb0c1\") " pod="ai-tenant-e2e-shared-b-w2-eee708/rate-test-model-eee708-kserve-77bb8d6bcf-lpfhm"
Sep 16 02:42:32.163119 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:42:32.162951    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ceb8aa19-128b-4aa2-a6f8-91013dfdb0c1-model-cache\") pod \"rate-test-model-eee708-kserve-77bb8d6bcf-lpfhm\" (UID: \"ceb8aa19-128b-4aa2-a6f8-91013dfdb0c1\") " pod="ai-tenant-e2e-shared-b-w2-eee708/rate-test-model-eee708-kserve-77bb8d6bcf-lpfhm"
Sep 16 02:42:32.163119 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:42:32.162988    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ceb8aa19-128b-4aa2-a6f8-91013dfdb0c1-tmp-dir\") pod \"rate-test-model-eee708-kserve-77bb8d6bcf-lpfhm\" (UID: \"ceb8aa19-128b-4aa2-a6f8-91013dfdb0c1\") " pod="ai-tenant-e2e-shared-b-w2-eee708/rate-test-model-eee708-kserve-77bb8d6bcf-lpfhm"
Sep 16 02:42:32.163119 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:42:32.163018    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ceb8aa19-128b-4aa2-a6f8-91013dfdb0c1-tls-certs\") pod \"rate-test-model-eee708-kserve-77bb8d6bcf-lpfhm\" (UID: \"ceb8aa19-128b-4aa2-a6f8-91013dfdb0c1\") " pod="ai-tenant-e2e-shared-b-w2-eee708/rate-test-model-eee708-kserve-77bb8d6bcf-lpfhm"
Sep 16 02:42:32.264075 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:42:32.264042    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ceb8aa19-128b-4aa2-a6f8-91013dfdb0c1-home\") pod \"rate-test-model-eee708-kserve-77bb8d6bcf-lpfhm\" (UID: \"ceb8aa19-128b-4aa2-a6f8-91013dfdb0c1\") " pod="ai-tenant-e2e-shared-b-w2-eee708/rate-test-model-eee708-kserve-77bb8d6bcf-lpfhm"
Sep 16 02:42:32.264273 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:42:32.264082    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ceb8aa19-128b-4aa2-a6f8-91013dfdb0c1-dshm\") pod \"rate-test-model-eee708-kserve-77bb8d6bcf-lpfhm\" (UID: \"ceb8aa19-128b-4aa2-a6f8-91013dfdb0c1\") " pod="ai-tenant-e2e-shared-b-w2-eee708/rate-test-model-eee708-kserve-77bb8d6bcf-lpfhm"
Sep 16 02:42:32.264273 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:42:32.264212    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hv5gw\" (UniqueName: \"kubernetes.io/projected/ceb8aa19-128b-4aa2-a6f8-91013dfdb0c1-kube-api-access-hv5gw\") pod \"rate-test-model-eee708-kserve-77bb8d6bcf-lpfhm\" (UID: \"ceb8aa19-128b-4aa2-a6f8-91013dfdb0c1\") " pod="ai-tenant-e2e-shared-b-w2-eee708/rate-test-model-eee708-kserve-77bb8d6bcf-lpfhm"
Sep 16 02:42:32.264273 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:42:32.264257    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ceb8aa19-128b-4aa2-a6f8-91013dfdb0c1-model-cache\") pod \"rate-test-model-eee708-kserve-77bb8d6bcf-lpfhm\" (UID: \"ceb8aa19-128b-4aa2-a6f8-91013dfdb0c1\") " pod="ai-tenant-e2e-shared-b-w2-eee708/rate-test-model-eee708-kserve-77bb8d6bcf-lpfhm"
Sep 16 02:42:32.264450 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:42:32.264329    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ceb8aa19-128b-4aa2-a6f8-91013dfdb0c1-tmp-dir\") pod \"rate-test-model-eee708-kserve-77bb8d6bcf-lpfhm\" (UID: \"ceb8aa19-128b-4aa2-a6f8-91013dfdb0c1\") " pod="ai-tenant-e2e-shared-b-w2-eee708/rate-test-model-eee708-kserve-77bb8d6bcf-lpfhm"
Sep 16 02:42:32.264450 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:42:32.264353    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ceb8aa19-128b-4aa2-a6f8-91013dfdb0c1-tls-certs\") pod \"rate-test-model-eee708-kserve-77bb8d6bcf-lpfhm\" (UID: \"ceb8aa19-128b-4aa2-a6f8-91013dfdb0c1\") " pod="ai-tenant-e2e-shared-b-w2-eee708/rate-test-model-eee708-kserve-77bb8d6bcf-lpfhm"
Sep 16 02:42:32.264450 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:42:32.264425    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ceb8aa19-128b-4aa2-a6f8-91013dfdb0c1-home\") pod \"rate-test-model-eee708-kserve-77bb8d6bcf-lpfhm\" (UID: \"ceb8aa19-128b-4aa2-a6f8-91013dfdb0c1\") " pod="ai-tenant-e2e-shared-b-w2-eee708/rate-test-model-eee708-kserve-77bb8d6bcf-lpfhm"
Sep 16 02:42:32.264625 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:42:32.264592    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ceb8aa19-128b-4aa2-a6f8-91013dfdb0c1-model-cache\") pod \"rate-test-model-eee708-kserve-77bb8d6bcf-lpfhm\" (UID: \"ceb8aa19-128b-4aa2-a6f8-91013dfdb0c1\") " pod="ai-tenant-e2e-shared-b-w2-eee708/rate-test-model-eee708-kserve-77bb8d6bcf-lpfhm"
Sep 16 02:42:32.264690 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:42:32.264623    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ceb8aa19-128b-4aa2-a6f8-91013dfdb0c1-tmp-dir\") pod \"rate-test-model-eee708-kserve-77bb8d6bcf-lpfhm\" (UID: \"ceb8aa19-128b-4aa2-a6f8-91013dfdb0c1\") " pod="ai-tenant-e2e-shared-b-w2-eee708/rate-test-model-eee708-kserve-77bb8d6bcf-lpfhm"
Sep 16 02:42:32.266395 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:42:32.266370    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ceb8aa19-128b-4aa2-a6f8-91013dfdb0c1-dshm\") pod \"rate-test-model-eee708-kserve-77bb8d6bcf-lpfhm\" (UID: \"ceb8aa19-128b-4aa2-a6f8-91013dfdb0c1\") " pod="ai-tenant-e2e-shared-b-w2-eee708/rate-test-model-eee708-kserve-77bb8d6bcf-lpfhm"
Sep 16 02:42:32.266794 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:42:32.266778    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ceb8aa19-128b-4aa2-a6f8-91013dfdb0c1-tls-certs\") pod \"rate-test-model-eee708-kserve-77bb8d6bcf-lpfhm\" (UID: \"ceb8aa19-128b-4aa2-a6f8-91013dfdb0c1\") " pod="ai-tenant-e2e-shared-b-w2-eee708/rate-test-model-eee708-kserve-77bb8d6bcf-lpfhm"
Sep 16 02:42:32.271508 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:42:32.271484    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hv5gw\" (UniqueName: \"kubernetes.io/projected/ceb8aa19-128b-4aa2-a6f8-91013dfdb0c1-kube-api-access-hv5gw\") pod \"rate-test-model-eee708-kserve-77bb8d6bcf-lpfhm\" (UID: \"ceb8aa19-128b-4aa2-a6f8-91013dfdb0c1\") " pod="ai-tenant-e2e-shared-b-w2-eee708/rate-test-model-eee708-kserve-77bb8d6bcf-lpfhm"
Sep 16 02:42:32.372388 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:42:32.372356    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w2-eee708/rate-test-model-eee708-kserve-77bb8d6bcf-lpfhm"
Sep 16 02:42:32.642410 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:42:32.642373    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w2-eee708/rate-test-model-eee708-kserve-77bb8d6bcf-lpfhm"]
Sep 16 02:42:32.643711 ip-10-0-128-111 kubenswrapper[2583]: W0916 02:42:32.643684    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podceb8aa19_128b_4aa2_a6f8_91013dfdb0c1.slice/crio-2e4d5dac8a2d4c61ab0b9c6242f1f9323b3f20f5b95cb326172756ceb6a82e04 WatchSource:0}: Error finding container 2e4d5dac8a2d4c61ab0b9c6242f1f9323b3f20f5b95cb326172756ceb6a82e04: Status 404 returned error can't find the container with id 2e4d5dac8a2d4c61ab0b9c6242f1f9323b3f20f5b95cb326172756ceb6a82e04
Sep 16 02:42:33.107689 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:42:33.107653    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w2-eee708/rate-test-model-eee708-kserve-77bb8d6bcf-lpfhm" event={"ID":"ceb8aa19-128b-4aa2-a6f8-91013dfdb0c1","Type":"ContainerStarted","Data":"f11cc08066303dff4bb6cc6c3135d006122f166065f4f3a69c3e001d88b58d82"}
Sep 16 02:42:33.107689 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:42:33.107699    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w2-eee708/rate-test-model-eee708-kserve-77bb8d6bcf-lpfhm" event={"ID":"ceb8aa19-128b-4aa2-a6f8-91013dfdb0c1","Type":"ContainerStarted","Data":"2e4d5dac8a2d4c61ab0b9c6242f1f9323b3f20f5b95cb326172756ceb6a82e04"}
Sep 16 02:42:33.124244 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:42:33.124196    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w2-eee708/rate-test-model-eee708-kserve-77bb8d6bcf-lpfhm" podStartSLOduration=1.124180487 podStartE2EDuration="1.124180487s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 02:42:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 02:42:33.122583475 +0000 UTC m=+1431.590372256" watchObservedRunningTime="2026-09-16 02:42:33.124180487 +0000 UTC m=+1431.591969266"
Sep 16 02:42:33.548328 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:42:33.548235    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-fzl8t"]
Sep 16 02:42:42.372726 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:42:42.372695    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w2-eee708/rate-test-model-eee708-kserve-77bb8d6bcf-lpfhm"
Sep 16 02:42:42.373151 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:42:42.372754    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w2-eee708/rate-test-model-eee708-kserve-77bb8d6bcf-lpfhm"
Sep 16 02:42:42.385411 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:42:42.385385    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w2-eee708/rate-test-model-eee708-kserve-77bb8d6bcf-lpfhm"
Sep 16 02:42:43.171009 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:42:43.170975    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w2-eee708/rate-test-model-eee708-kserve-77bb8d6bcf-lpfhm"
Sep 16 02:42:44.545973 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:42:44.545938    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-fzl8t"]
Sep 16 02:42:56.446625 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:42:56.446580    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-fzl8t"]
Sep 16 02:43:10.846412 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:43:10.846378    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-fzl8t"]
Sep 16 02:43:22.968876 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:43:22.968832    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w2-db1ed7/rate-test-model-db1ed7-kserve-7f9997cf75-csxff"]
Sep 16 02:43:24.279021 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:43:24.278989    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w2-eee708/rate-test-model-eee708-kserve-77bb8d6bcf-lpfhm"]
Sep 16 02:43:27.551367 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:43:27.551331    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-fzl8t"]
Sep 16 02:43:32.081952 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:43:32.081917    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-73bfc1e9-openshift-default-fc75695d8-8l6ww"]
Sep 16 02:43:32.082439 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:43:32.082171    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-73bfc1e9-openshift-default-fc75695d8-8l6ww" podUID="c9f199d5-ac58-4074-9ff0-b88bec554d35" containerName="istio-proxy" containerID="cri-o://cb85373a9447cff483f0756c5c2c6efa8cd15baf50ce1fa72fe644752346f82b" gracePeriod=30
Sep 16 02:43:37.332093 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:43:37.332064    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-73bfc1e9-openshift-default-fc75695d8-8l6ww"
Sep 16 02:43:37.388854 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:43:37.388760    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c9f199d5-ac58-4074-9ff0-b88bec554d35-istio-data\") pod \"c9f199d5-ac58-4074-9ff0-b88bec554d35\" (UID: \"c9f199d5-ac58-4074-9ff0-b88bec554d35\") "
Sep 16 02:43:37.388854 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:43:37.388794    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c9f199d5-ac58-4074-9ff0-b88bec554d35-workload-socket\") pod \"c9f199d5-ac58-4074-9ff0-b88bec554d35\" (UID: \"c9f199d5-ac58-4074-9ff0-b88bec554d35\") "
Sep 16 02:43:37.388854 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:43:37.388823    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c9f199d5-ac58-4074-9ff0-b88bec554d35-istio-envoy\") pod \"c9f199d5-ac58-4074-9ff0-b88bec554d35\" (UID: \"c9f199d5-ac58-4074-9ff0-b88bec554d35\") "
Sep 16 02:43:37.389157 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:43:37.388894    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c9f199d5-ac58-4074-9ff0-b88bec554d35-istiod-ca-cert\") pod \"c9f199d5-ac58-4074-9ff0-b88bec554d35\" (UID: \"c9f199d5-ac58-4074-9ff0-b88bec554d35\") "
Sep 16 02:43:37.389157 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:43:37.388933    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c9f199d5-ac58-4074-9ff0-b88bec554d35-istio-podinfo\") pod \"c9f199d5-ac58-4074-9ff0-b88bec554d35\" (UID: \"c9f199d5-ac58-4074-9ff0-b88bec554d35\") "
Sep 16 02:43:37.389157 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:43:37.388963    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c9f199d5-ac58-4074-9ff0-b88bec554d35-workload-certs\") pod \"c9f199d5-ac58-4074-9ff0-b88bec554d35\" (UID: \"c9f199d5-ac58-4074-9ff0-b88bec554d35\") "
Sep 16 02:43:37.389157 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:43:37.389005    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c9f199d5-ac58-4074-9ff0-b88bec554d35-istio-token\") pod \"c9f199d5-ac58-4074-9ff0-b88bec554d35\" (UID: \"c9f199d5-ac58-4074-9ff0-b88bec554d35\") "
Sep 16 02:43:37.389157 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:43:37.389032    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c9f199d5-ac58-4074-9ff0-b88bec554d35-credential-socket\") pod \"c9f199d5-ac58-4074-9ff0-b88bec554d35\" (UID: \"c9f199d5-ac58-4074-9ff0-b88bec554d35\") "
Sep 16 02:43:37.389157 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:43:37.389067    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c9f199d5-ac58-4074-9ff0-b88bec554d35-istio-data" (OuterVolumeSpecName: "istio-data") pod "c9f199d5-ac58-4074-9ff0-b88bec554d35" (UID: "c9f199d5-ac58-4074-9ff0-b88bec554d35"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 02:43:37.389157 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:43:37.389074    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lrg7g\" (UniqueName: \"kubernetes.io/projected/c9f199d5-ac58-4074-9ff0-b88bec554d35-kube-api-access-lrg7g\") pod \"c9f199d5-ac58-4074-9ff0-b88bec554d35\" (UID: \"c9f199d5-ac58-4074-9ff0-b88bec554d35\") "
Sep 16 02:43:37.389157 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:43:37.389081    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c9f199d5-ac58-4074-9ff0-b88bec554d35-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "c9f199d5-ac58-4074-9ff0-b88bec554d35" (UID: "c9f199d5-ac58-4074-9ff0-b88bec554d35"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 02:43:37.389634 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:43:37.389350    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c9f199d5-ac58-4074-9ff0-b88bec554d35-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "c9f199d5-ac58-4074-9ff0-b88bec554d35" (UID: "c9f199d5-ac58-4074-9ff0-b88bec554d35"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 02:43:37.389634 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:43:37.389578    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c9f199d5-ac58-4074-9ff0-b88bec554d35-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "c9f199d5-ac58-4074-9ff0-b88bec554d35" (UID: "c9f199d5-ac58-4074-9ff0-b88bec554d35"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 02:43:37.389634 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:43:37.389592    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c9f199d5-ac58-4074-9ff0-b88bec554d35-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "c9f199d5-ac58-4074-9ff0-b88bec554d35" (UID: "c9f199d5-ac58-4074-9ff0-b88bec554d35"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 02:43:37.389878 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:43:37.389833    2583 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c9f199d5-ac58-4074-9ff0-b88bec554d35-istiod-ca-cert\") on node \"ip-10-0-128-111.ec2.internal\" DevicePath \"\""
Sep 16 02:43:37.389878 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:43:37.389855    2583 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c9f199d5-ac58-4074-9ff0-b88bec554d35-workload-certs\") on node \"ip-10-0-128-111.ec2.internal\" DevicePath \"\""
Sep 16 02:43:37.389878 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:43:37.389870    2583 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c9f199d5-ac58-4074-9ff0-b88bec554d35-credential-socket\") on node \"ip-10-0-128-111.ec2.internal\" DevicePath \"\""
Sep 16 02:43:37.390069 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:43:37.389894    2583 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c9f199d5-ac58-4074-9ff0-b88bec554d35-istio-data\") on node \"ip-10-0-128-111.ec2.internal\" DevicePath \"\""
Sep 16 02:43:37.390069 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:43:37.389907    2583 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c9f199d5-ac58-4074-9ff0-b88bec554d35-workload-socket\") on node \"ip-10-0-128-111.ec2.internal\" DevicePath \"\""
Sep 16 02:43:37.391321 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:43:37.391293    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c9f199d5-ac58-4074-9ff0-b88bec554d35-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "c9f199d5-ac58-4074-9ff0-b88bec554d35" (UID: "c9f199d5-ac58-4074-9ff0-b88bec554d35"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 16 02:43:37.391432 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:43:37.391301    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c9f199d5-ac58-4074-9ff0-b88bec554d35-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "c9f199d5-ac58-4074-9ff0-b88bec554d35" (UID: "c9f199d5-ac58-4074-9ff0-b88bec554d35"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 02:43:37.391602 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:43:37.391584    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c9f199d5-ac58-4074-9ff0-b88bec554d35-kube-api-access-lrg7g" (OuterVolumeSpecName: "kube-api-access-lrg7g") pod "c9f199d5-ac58-4074-9ff0-b88bec554d35" (UID: "c9f199d5-ac58-4074-9ff0-b88bec554d35"). InnerVolumeSpecName "kube-api-access-lrg7g". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 02:43:37.391670 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:43:37.391644    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c9f199d5-ac58-4074-9ff0-b88bec554d35-istio-token" (OuterVolumeSpecName: "istio-token") pod "c9f199d5-ac58-4074-9ff0-b88bec554d35" (UID: "c9f199d5-ac58-4074-9ff0-b88bec554d35"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 02:43:37.431662 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:43:37.431627    2583 generic.go:356] "Generic (PLEG): container finished" podID="c9f199d5-ac58-4074-9ff0-b88bec554d35" containerID="cb85373a9447cff483f0756c5c2c6efa8cd15baf50ce1fa72fe644752346f82b" exitCode=0
Sep 16 02:43:37.431833 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:43:37.431680    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-73bfc1e9-openshift-default-fc75695d8-8l6ww"
Sep 16 02:43:37.431833 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:43:37.431706    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-73bfc1e9-openshift-default-fc75695d8-8l6ww" event={"ID":"c9f199d5-ac58-4074-9ff0-b88bec554d35","Type":"ContainerDied","Data":"cb85373a9447cff483f0756c5c2c6efa8cd15baf50ce1fa72fe644752346f82b"}
Sep 16 02:43:37.431833 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:43:37.431770    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-73bfc1e9-openshift-default-fc75695d8-8l6ww" event={"ID":"c9f199d5-ac58-4074-9ff0-b88bec554d35","Type":"ContainerDied","Data":"579e641e4df31e1f368adcf44da585aa1f7a053acfb8fff46dacf93714a7f0e6"}
Sep 16 02:43:37.431833 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:43:37.431800    2583 scope.go:117] "RemoveContainer" containerID="cb85373a9447cff483f0756c5c2c6efa8cd15baf50ce1fa72fe644752346f82b"
Sep 16 02:43:37.441187 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:43:37.441171    2583 scope.go:117] "RemoveContainer" containerID="f4bf3777bf2955e2c38c986a56ab72cf0832fcbdf2ab0e0564411358f7b26d2d"
Sep 16 02:43:37.449201 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:43:37.449183    2583 scope.go:117] "RemoveContainer" containerID="cb85373a9447cff483f0756c5c2c6efa8cd15baf50ce1fa72fe644752346f82b"
Sep 16 02:43:37.449519 ip-10-0-128-111 kubenswrapper[2583]: E0916 02:43:37.449495    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cb85373a9447cff483f0756c5c2c6efa8cd15baf50ce1fa72fe644752346f82b\": container with ID starting with cb85373a9447cff483f0756c5c2c6efa8cd15baf50ce1fa72fe644752346f82b not found: ID does not exist" containerID="cb85373a9447cff483f0756c5c2c6efa8cd15baf50ce1fa72fe644752346f82b"
Sep 16 02:43:37.449746 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:43:37.449522    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cb85373a9447cff483f0756c5c2c6efa8cd15baf50ce1fa72fe644752346f82b"} err="failed to get container status \"cb85373a9447cff483f0756c5c2c6efa8cd15baf50ce1fa72fe644752346f82b\": rpc error: code = NotFound desc = could not find container \"cb85373a9447cff483f0756c5c2c6efa8cd15baf50ce1fa72fe644752346f82b\": container with ID starting with cb85373a9447cff483f0756c5c2c6efa8cd15baf50ce1fa72fe644752346f82b not found: ID does not exist"
Sep 16 02:43:37.449746 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:43:37.449539    2583 scope.go:117] "RemoveContainer" containerID="f4bf3777bf2955e2c38c986a56ab72cf0832fcbdf2ab0e0564411358f7b26d2d"
Sep 16 02:43:37.449887 ip-10-0-128-111 kubenswrapper[2583]: E0916 02:43:37.449795    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f4bf3777bf2955e2c38c986a56ab72cf0832fcbdf2ab0e0564411358f7b26d2d\": container with ID starting with f4bf3777bf2955e2c38c986a56ab72cf0832fcbdf2ab0e0564411358f7b26d2d not found: ID does not exist" containerID="f4bf3777bf2955e2c38c986a56ab72cf0832fcbdf2ab0e0564411358f7b26d2d"
Sep 16 02:43:37.449887 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:43:37.449818    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f4bf3777bf2955e2c38c986a56ab72cf0832fcbdf2ab0e0564411358f7b26d2d"} err="failed to get container status \"f4bf3777bf2955e2c38c986a56ab72cf0832fcbdf2ab0e0564411358f7b26d2d\": rpc error: code = NotFound desc = could not find container \"f4bf3777bf2955e2c38c986a56ab72cf0832fcbdf2ab0e0564411358f7b26d2d\": container with ID starting with f4bf3777bf2955e2c38c986a56ab72cf0832fcbdf2ab0e0564411358f7b26d2d not found: ID does not exist"
Sep 16 02:43:37.452982 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:43:37.452957    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-73bfc1e9-openshift-default-fc75695d8-8l6ww"]
Sep 16 02:43:37.456619 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:43:37.456597    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-73bfc1e9-openshift-default-fc75695d8-8l6ww"]
Sep 16 02:43:37.490564 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:43:37.490524    2583 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c9f199d5-ac58-4074-9ff0-b88bec554d35-istio-envoy\") on node \"ip-10-0-128-111.ec2.internal\" DevicePath \"\""
Sep 16 02:43:37.490564 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:43:37.490545    2583 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c9f199d5-ac58-4074-9ff0-b88bec554d35-istio-podinfo\") on node \"ip-10-0-128-111.ec2.internal\" DevicePath \"\""
Sep 16 02:43:37.490564 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:43:37.490564    2583 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c9f199d5-ac58-4074-9ff0-b88bec554d35-istio-token\") on node \"ip-10-0-128-111.ec2.internal\" DevicePath \"\""
Sep 16 02:43:37.490828 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:43:37.490574    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lrg7g\" (UniqueName: \"kubernetes.io/projected/c9f199d5-ac58-4074-9ff0-b88bec554d35-kube-api-access-lrg7g\") on node \"ip-10-0-128-111.ec2.internal\" DevicePath \"\""
Sep 16 02:43:37.989362 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:43:37.989318    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w2-db1ed7/rate-test-model-db1ed7-kserve-7f9997cf75-csxff" podUID="4694ad1d-3bf3-4f02-afc3-31b4282778cd" containerName="main" containerID="cri-o://9e038aa2e7b4611dd467b6562551cb66e9c02df00d4158c8ba9f29aecc35e20e" gracePeriod=45
Sep 16 02:43:38.046155 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:43:38.046115    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c9f199d5-ac58-4074-9ff0-b88bec554d35" path="/var/lib/kubelet/pods/c9f199d5-ac58-4074-9ff0-b88bec554d35/volumes"
Sep 16 02:43:38.234537 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:43:38.234512    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w2-db1ed7/rate-test-model-db1ed7-kserve-7f9997cf75-csxff"
Sep 16 02:43:38.297956 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:43:38.297867    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4694ad1d-3bf3-4f02-afc3-31b4282778cd-tls-certs\") pod \"4694ad1d-3bf3-4f02-afc3-31b4282778cd\" (UID: \"4694ad1d-3bf3-4f02-afc3-31b4282778cd\") "
Sep 16 02:43:38.297956 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:43:38.297934    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4694ad1d-3bf3-4f02-afc3-31b4282778cd-home\") pod \"4694ad1d-3bf3-4f02-afc3-31b4282778cd\" (UID: \"4694ad1d-3bf3-4f02-afc3-31b4282778cd\") "
Sep 16 02:43:38.298189 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:43:38.297963    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4694ad1d-3bf3-4f02-afc3-31b4282778cd-dshm\") pod \"4694ad1d-3bf3-4f02-afc3-31b4282778cd\" (UID: \"4694ad1d-3bf3-4f02-afc3-31b4282778cd\") "
Sep 16 02:43:38.298189 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:43:38.298033    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4694ad1d-3bf3-4f02-afc3-31b4282778cd-tmp-dir\") pod \"4694ad1d-3bf3-4f02-afc3-31b4282778cd\" (UID: \"4694ad1d-3bf3-4f02-afc3-31b4282778cd\") "
Sep 16 02:43:38.298189 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:43:38.298065    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4694ad1d-3bf3-4f02-afc3-31b4282778cd-model-cache\") pod \"4694ad1d-3bf3-4f02-afc3-31b4282778cd\" (UID: \"4694ad1d-3bf3-4f02-afc3-31b4282778cd\") "
Sep 16 02:43:38.298189 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:43:38.298094    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-khhqj\" (UniqueName: \"kubernetes.io/projected/4694ad1d-3bf3-4f02-afc3-31b4282778cd-kube-api-access-khhqj\") pod \"4694ad1d-3bf3-4f02-afc3-31b4282778cd\" (UID: \"4694ad1d-3bf3-4f02-afc3-31b4282778cd\") "
Sep 16 02:43:38.298406 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:43:38.298235    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4694ad1d-3bf3-4f02-afc3-31b4282778cd-home" (OuterVolumeSpecName: "home") pod "4694ad1d-3bf3-4f02-afc3-31b4282778cd" (UID: "4694ad1d-3bf3-4f02-afc3-31b4282778cd"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 02:43:38.298406 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:43:38.298295    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4694ad1d-3bf3-4f02-afc3-31b4282778cd-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "4694ad1d-3bf3-4f02-afc3-31b4282778cd" (UID: "4694ad1d-3bf3-4f02-afc3-31b4282778cd"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 02:43:38.298406 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:43:38.298319    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4694ad1d-3bf3-4f02-afc3-31b4282778cd-model-cache" (OuterVolumeSpecName: "model-cache") pod "4694ad1d-3bf3-4f02-afc3-31b4282778cd" (UID: "4694ad1d-3bf3-4f02-afc3-31b4282778cd"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 02:43:38.298527 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:43:38.298470    2583 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4694ad1d-3bf3-4f02-afc3-31b4282778cd-home\") on node \"ip-10-0-128-111.ec2.internal\" DevicePath \"\""
Sep 16 02:43:38.298527 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:43:38.298504    2583 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4694ad1d-3bf3-4f02-afc3-31b4282778cd-tmp-dir\") on node \"ip-10-0-128-111.ec2.internal\" DevicePath \"\""
Sep 16 02:43:38.298527 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:43:38.298515    2583 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4694ad1d-3bf3-4f02-afc3-31b4282778cd-model-cache\") on node \"ip-10-0-128-111.ec2.internal\" DevicePath \"\""
Sep 16 02:43:38.300169 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:43:38.300146    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4694ad1d-3bf3-4f02-afc3-31b4282778cd-dshm" (OuterVolumeSpecName: "dshm") pod "4694ad1d-3bf3-4f02-afc3-31b4282778cd" (UID: "4694ad1d-3bf3-4f02-afc3-31b4282778cd"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 02:43:38.300273 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:43:38.300187    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4694ad1d-3bf3-4f02-afc3-31b4282778cd-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "4694ad1d-3bf3-4f02-afc3-31b4282778cd" (UID: "4694ad1d-3bf3-4f02-afc3-31b4282778cd"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 02:43:38.300273 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:43:38.300264    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4694ad1d-3bf3-4f02-afc3-31b4282778cd-kube-api-access-khhqj" (OuterVolumeSpecName: "kube-api-access-khhqj") pod "4694ad1d-3bf3-4f02-afc3-31b4282778cd" (UID: "4694ad1d-3bf3-4f02-afc3-31b4282778cd"). InnerVolumeSpecName "kube-api-access-khhqj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 02:43:38.399365 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:43:38.399331    2583 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4694ad1d-3bf3-4f02-afc3-31b4282778cd-tls-certs\") on node \"ip-10-0-128-111.ec2.internal\" DevicePath \"\""
Sep 16 02:43:38.399365 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:43:38.399359    2583 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4694ad1d-3bf3-4f02-afc3-31b4282778cd-dshm\") on node \"ip-10-0-128-111.ec2.internal\" DevicePath \"\""
Sep 16 02:43:38.399365 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:43:38.399372    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-khhqj\" (UniqueName: \"kubernetes.io/projected/4694ad1d-3bf3-4f02-afc3-31b4282778cd-kube-api-access-khhqj\") on node \"ip-10-0-128-111.ec2.internal\" DevicePath \"\""
Sep 16 02:43:38.438400 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:43:38.438365    2583 generic.go:356] "Generic (PLEG): container finished" podID="4694ad1d-3bf3-4f02-afc3-31b4282778cd" containerID="9e038aa2e7b4611dd467b6562551cb66e9c02df00d4158c8ba9f29aecc35e20e" exitCode=0
Sep 16 02:43:38.438545 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:43:38.438411    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w2-db1ed7/rate-test-model-db1ed7-kserve-7f9997cf75-csxff"
Sep 16 02:43:38.438545 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:43:38.438449    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w2-db1ed7/rate-test-model-db1ed7-kserve-7f9997cf75-csxff" event={"ID":"4694ad1d-3bf3-4f02-afc3-31b4282778cd","Type":"ContainerDied","Data":"9e038aa2e7b4611dd467b6562551cb66e9c02df00d4158c8ba9f29aecc35e20e"}
Sep 16 02:43:38.438545 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:43:38.438504    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w2-db1ed7/rate-test-model-db1ed7-kserve-7f9997cf75-csxff" event={"ID":"4694ad1d-3bf3-4f02-afc3-31b4282778cd","Type":"ContainerDied","Data":"50213d832d24057375aca436a749f2e6db06f7bcfebd58b427ee65be4a4709ca"}
Sep 16 02:43:38.438545 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:43:38.438526    2583 scope.go:117] "RemoveContainer" containerID="9e038aa2e7b4611dd467b6562551cb66e9c02df00d4158c8ba9f29aecc35e20e"
Sep 16 02:43:38.447743 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:43:38.447726    2583 scope.go:117] "RemoveContainer" containerID="9e038aa2e7b4611dd467b6562551cb66e9c02df00d4158c8ba9f29aecc35e20e"
Sep 16 02:43:38.448053 ip-10-0-128-111 kubenswrapper[2583]: E0916 02:43:38.448035    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9e038aa2e7b4611dd467b6562551cb66e9c02df00d4158c8ba9f29aecc35e20e\": container with ID starting with 9e038aa2e7b4611dd467b6562551cb66e9c02df00d4158c8ba9f29aecc35e20e not found: ID does not exist" containerID="9e038aa2e7b4611dd467b6562551cb66e9c02df00d4158c8ba9f29aecc35e20e"
Sep 16 02:43:38.448101 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:43:38.448060    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9e038aa2e7b4611dd467b6562551cb66e9c02df00d4158c8ba9f29aecc35e20e"} err="failed to get container status \"9e038aa2e7b4611dd467b6562551cb66e9c02df00d4158c8ba9f29aecc35e20e\": rpc error: code = NotFound desc = could not find container \"9e038aa2e7b4611dd467b6562551cb66e9c02df00d4158c8ba9f29aecc35e20e\": container with ID starting with 9e038aa2e7b4611dd467b6562551cb66e9c02df00d4158c8ba9f29aecc35e20e not found: ID does not exist"
Sep 16 02:43:38.458517 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:43:38.458495    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w2-db1ed7/rate-test-model-db1ed7-kserve-7f9997cf75-csxff"]
Sep 16 02:43:38.462381 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:43:38.462358    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w2-db1ed7/rate-test-model-db1ed7-kserve-7f9997cf75-csxff"]
Sep 16 02:43:39.298374 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:43:39.298330    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w2-eee708/rate-test-model-eee708-kserve-77bb8d6bcf-lpfhm" podUID="ceb8aa19-128b-4aa2-a6f8-91013dfdb0c1" containerName="main" containerID="cri-o://f11cc08066303dff4bb6cc6c3135d006122f166065f4f3a69c3e001d88b58d82" gracePeriod=45
Sep 16 02:43:39.450437 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:43:39.450403    2583 generic.go:356] "Generic (PLEG): container finished" podID="ceb8aa19-128b-4aa2-a6f8-91013dfdb0c1" containerID="f11cc08066303dff4bb6cc6c3135d006122f166065f4f3a69c3e001d88b58d82" exitCode=0
Sep 16 02:43:39.450437 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:43:39.450441    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w2-eee708/rate-test-model-eee708-kserve-77bb8d6bcf-lpfhm" event={"ID":"ceb8aa19-128b-4aa2-a6f8-91013dfdb0c1","Type":"ContainerDied","Data":"f11cc08066303dff4bb6cc6c3135d006122f166065f4f3a69c3e001d88b58d82"}
Sep 16 02:43:39.556217 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:43:39.556150    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w2-eee708/rate-test-model-eee708-kserve-77bb8d6bcf-lpfhm"
Sep 16 02:43:39.610376 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:43:39.610341    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ceb8aa19-128b-4aa2-a6f8-91013dfdb0c1-dshm\") pod \"ceb8aa19-128b-4aa2-a6f8-91013dfdb0c1\" (UID: \"ceb8aa19-128b-4aa2-a6f8-91013dfdb0c1\") "
Sep 16 02:43:39.610543 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:43:39.610399    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ceb8aa19-128b-4aa2-a6f8-91013dfdb0c1-home\") pod \"ceb8aa19-128b-4aa2-a6f8-91013dfdb0c1\" (UID: \"ceb8aa19-128b-4aa2-a6f8-91013dfdb0c1\") "
Sep 16 02:43:39.610543 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:43:39.610419    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ceb8aa19-128b-4aa2-a6f8-91013dfdb0c1-tmp-dir\") pod \"ceb8aa19-128b-4aa2-a6f8-91013dfdb0c1\" (UID: \"ceb8aa19-128b-4aa2-a6f8-91013dfdb0c1\") "
Sep 16 02:43:39.610543 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:43:39.610452    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ceb8aa19-128b-4aa2-a6f8-91013dfdb0c1-model-cache\") pod \"ceb8aa19-128b-4aa2-a6f8-91013dfdb0c1\" (UID: \"ceb8aa19-128b-4aa2-a6f8-91013dfdb0c1\") "
Sep 16 02:43:39.610543 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:43:39.610508    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ceb8aa19-128b-4aa2-a6f8-91013dfdb0c1-tls-certs\") pod \"ceb8aa19-128b-4aa2-a6f8-91013dfdb0c1\" (UID: \"ceb8aa19-128b-4aa2-a6f8-91013dfdb0c1\") "
Sep 16 02:43:39.610817 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:43:39.610584    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hv5gw\" (UniqueName: \"kubernetes.io/projected/ceb8aa19-128b-4aa2-a6f8-91013dfdb0c1-kube-api-access-hv5gw\") pod \"ceb8aa19-128b-4aa2-a6f8-91013dfdb0c1\" (UID: \"ceb8aa19-128b-4aa2-a6f8-91013dfdb0c1\") "
Sep 16 02:43:39.610817 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:43:39.610745    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ceb8aa19-128b-4aa2-a6f8-91013dfdb0c1-home" (OuterVolumeSpecName: "home") pod "ceb8aa19-128b-4aa2-a6f8-91013dfdb0c1" (UID: "ceb8aa19-128b-4aa2-a6f8-91013dfdb0c1"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 02:43:39.610817 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:43:39.610764    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ceb8aa19-128b-4aa2-a6f8-91013dfdb0c1-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "ceb8aa19-128b-4aa2-a6f8-91013dfdb0c1" (UID: "ceb8aa19-128b-4aa2-a6f8-91013dfdb0c1"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 02:43:39.610817 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:43:39.610794    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ceb8aa19-128b-4aa2-a6f8-91013dfdb0c1-model-cache" (OuterVolumeSpecName: "model-cache") pod "ceb8aa19-128b-4aa2-a6f8-91013dfdb0c1" (UID: "ceb8aa19-128b-4aa2-a6f8-91013dfdb0c1"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 02:43:39.610973 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:43:39.610895    2583 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ceb8aa19-128b-4aa2-a6f8-91013dfdb0c1-model-cache\") on node \"ip-10-0-128-111.ec2.internal\" DevicePath \"\""
Sep 16 02:43:39.610973 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:43:39.610912    2583 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ceb8aa19-128b-4aa2-a6f8-91013dfdb0c1-home\") on node \"ip-10-0-128-111.ec2.internal\" DevicePath \"\""
Sep 16 02:43:39.610973 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:43:39.610924    2583 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ceb8aa19-128b-4aa2-a6f8-91013dfdb0c1-tmp-dir\") on node \"ip-10-0-128-111.ec2.internal\" DevicePath \"\""
Sep 16 02:43:39.612773 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:43:39.612752    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ceb8aa19-128b-4aa2-a6f8-91013dfdb0c1-dshm" (OuterVolumeSpecName: "dshm") pod "ceb8aa19-128b-4aa2-a6f8-91013dfdb0c1" (UID: "ceb8aa19-128b-4aa2-a6f8-91013dfdb0c1"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 02:43:39.612773 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:43:39.612759    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ceb8aa19-128b-4aa2-a6f8-91013dfdb0c1-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "ceb8aa19-128b-4aa2-a6f8-91013dfdb0c1" (UID: "ceb8aa19-128b-4aa2-a6f8-91013dfdb0c1"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 02:43:39.612892 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:43:39.612841    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ceb8aa19-128b-4aa2-a6f8-91013dfdb0c1-kube-api-access-hv5gw" (OuterVolumeSpecName: "kube-api-access-hv5gw") pod "ceb8aa19-128b-4aa2-a6f8-91013dfdb0c1" (UID: "ceb8aa19-128b-4aa2-a6f8-91013dfdb0c1"). InnerVolumeSpecName "kube-api-access-hv5gw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 02:43:39.711672 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:43:39.711626    2583 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ceb8aa19-128b-4aa2-a6f8-91013dfdb0c1-tls-certs\") on node \"ip-10-0-128-111.ec2.internal\" DevicePath \"\""
Sep 16 02:43:39.711672 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:43:39.711664    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hv5gw\" (UniqueName: \"kubernetes.io/projected/ceb8aa19-128b-4aa2-a6f8-91013dfdb0c1-kube-api-access-hv5gw\") on node \"ip-10-0-128-111.ec2.internal\" DevicePath \"\""
Sep 16 02:43:39.711672 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:43:39.711679    2583 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ceb8aa19-128b-4aa2-a6f8-91013dfdb0c1-dshm\") on node \"ip-10-0-128-111.ec2.internal\" DevicePath \"\""
Sep 16 02:43:40.046108 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:43:40.046075    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4694ad1d-3bf3-4f02-afc3-31b4282778cd" path="/var/lib/kubelet/pods/4694ad1d-3bf3-4f02-afc3-31b4282778cd/volumes"
Sep 16 02:43:40.456684 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:43:40.456648    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w2-eee708/rate-test-model-eee708-kserve-77bb8d6bcf-lpfhm"
Sep 16 02:43:40.457077 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:43:40.456697    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w2-eee708/rate-test-model-eee708-kserve-77bb8d6bcf-lpfhm" event={"ID":"ceb8aa19-128b-4aa2-a6f8-91013dfdb0c1","Type":"ContainerDied","Data":"2e4d5dac8a2d4c61ab0b9c6242f1f9323b3f20f5b95cb326172756ceb6a82e04"}
Sep 16 02:43:40.457077 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:43:40.456752    2583 scope.go:117] "RemoveContainer" containerID="f11cc08066303dff4bb6cc6c3135d006122f166065f4f3a69c3e001d88b58d82"
Sep 16 02:43:40.471716 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:43:40.471689    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w2-eee708/rate-test-model-eee708-kserve-77bb8d6bcf-lpfhm"]
Sep 16 02:43:40.475196 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:43:40.475174    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w2-eee708/rate-test-model-eee708-kserve-77bb8d6bcf-lpfhm"]
Sep 16 02:43:42.047418 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:43:42.047389    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ceb8aa19-128b-4aa2-a6f8-91013dfdb0c1" path="/var/lib/kubelet/pods/ceb8aa19-128b-4aa2-a6f8-91013dfdb0c1/volumes"
Sep 16 02:43:42.204681 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:43:42.204645    2583 scope.go:117] "RemoveContainer" containerID="371fbbe45fe526b8488b386f6b98ceb5c8f63618aae0469cad60eb2fc634a5d4"
Sep 16 02:43:42.212616 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:43:42.212596    2583 scope.go:117] "RemoveContainer" containerID="5d21b9b4fe65eb21b660ddb39982857e6d1fb02b63984a3609980fafd4b3ea97"
Sep 16 02:43:46.045845 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:43:46.045816    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-fzl8t"]
Sep 16 02:43:53.606489 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:43:53.606456    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-b-4fe974-openshift-default-59b7dbcd46-cpkvz"]
Sep 16 02:43:53.607098 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:43:53.606840    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4694ad1d-3bf3-4f02-afc3-31b4282778cd" containerName="main"
Sep 16 02:43:53.607098 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:43:53.606852    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="4694ad1d-3bf3-4f02-afc3-31b4282778cd" containerName="main"
Sep 16 02:43:53.607098 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:43:53.606864    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ceb8aa19-128b-4aa2-a6f8-91013dfdb0c1" containerName="main"
Sep 16 02:43:53.607098 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:43:53.606869    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="ceb8aa19-128b-4aa2-a6f8-91013dfdb0c1" containerName="main"
Sep 16 02:43:53.607098 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:43:53.606882    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c9f199d5-ac58-4074-9ff0-b88bec554d35" containerName="istio-proxy"
Sep 16 02:43:53.607098 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:43:53.606888    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="c9f199d5-ac58-4074-9ff0-b88bec554d35" containerName="istio-proxy"
Sep 16 02:43:53.607098 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:43:53.606901    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c9f199d5-ac58-4074-9ff0-b88bec554d35" containerName="istio-proxy"
Sep 16 02:43:53.607098 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:43:53.606905    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="c9f199d5-ac58-4074-9ff0-b88bec554d35" containerName="istio-proxy"
Sep 16 02:43:53.607098 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:43:53.606964    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="c9f199d5-ac58-4074-9ff0-b88bec554d35" containerName="istio-proxy"
Sep 16 02:43:53.607098 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:43:53.606971    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="ceb8aa19-128b-4aa2-a6f8-91013dfdb0c1" containerName="main"
Sep 16 02:43:53.607098 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:43:53.606977    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="c9f199d5-ac58-4074-9ff0-b88bec554d35" containerName="istio-proxy"
Sep 16 02:43:53.607098 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:43:53.606983    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="4694ad1d-3bf3-4f02-afc3-31b4282778cd" containerName="main"
Sep 16 02:43:53.611908 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:43:53.611885    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-4fe974-openshift-default-59b7dbcd46-cpkvz"
Sep 16 02:43:53.614834 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:43:53.614814    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-b-4fe974-openshift-default-dockercfg-d2cjb\""
Sep 16 02:43:53.622801 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:43:53.622777    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-4fe974-openshift-default-59b7dbcd46-cpkvz"]
Sep 16 02:43:53.741741 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:43:53.741708    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4b2c7d51-9fcb-4bae-a99c-66ad0a8e62eb-istio-data\") pod \"e2e-ipp-b-4fe974-openshift-default-59b7dbcd46-cpkvz\" (UID: \"4b2c7d51-9fcb-4bae-a99c-66ad0a8e62eb\") " pod="openshift-ingress/e2e-ipp-b-4fe974-openshift-default-59b7dbcd46-cpkvz"
Sep 16 02:43:53.741934 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:43:53.741749    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b97v2\" (UniqueName: \"kubernetes.io/projected/4b2c7d51-9fcb-4bae-a99c-66ad0a8e62eb-kube-api-access-b97v2\") pod \"e2e-ipp-b-4fe974-openshift-default-59b7dbcd46-cpkvz\" (UID: \"4b2c7d51-9fcb-4bae-a99c-66ad0a8e62eb\") " pod="openshift-ingress/e2e-ipp-b-4fe974-openshift-default-59b7dbcd46-cpkvz"
Sep 16 02:43:53.741934 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:43:53.741793    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4b2c7d51-9fcb-4bae-a99c-66ad0a8e62eb-workload-socket\") pod \"e2e-ipp-b-4fe974-openshift-default-59b7dbcd46-cpkvz\" (UID: \"4b2c7d51-9fcb-4bae-a99c-66ad0a8e62eb\") " pod="openshift-ingress/e2e-ipp-b-4fe974-openshift-default-59b7dbcd46-cpkvz"
Sep 16 02:43:53.741934 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:43:53.741879    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4b2c7d51-9fcb-4bae-a99c-66ad0a8e62eb-istio-token\") pod \"e2e-ipp-b-4fe974-openshift-default-59b7dbcd46-cpkvz\" (UID: \"4b2c7d51-9fcb-4bae-a99c-66ad0a8e62eb\") " pod="openshift-ingress/e2e-ipp-b-4fe974-openshift-default-59b7dbcd46-cpkvz"
Sep 16 02:43:53.741934 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:43:53.741911    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4b2c7d51-9fcb-4bae-a99c-66ad0a8e62eb-istiod-ca-cert\") pod \"e2e-ipp-b-4fe974-openshift-default-59b7dbcd46-cpkvz\" (UID: \"4b2c7d51-9fcb-4bae-a99c-66ad0a8e62eb\") " pod="openshift-ingress/e2e-ipp-b-4fe974-openshift-default-59b7dbcd46-cpkvz"
Sep 16 02:43:53.742174 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:43:53.741971    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4b2c7d51-9fcb-4bae-a99c-66ad0a8e62eb-credential-socket\") pod \"e2e-ipp-b-4fe974-openshift-default-59b7dbcd46-cpkvz\" (UID: \"4b2c7d51-9fcb-4bae-a99c-66ad0a8e62eb\") " pod="openshift-ingress/e2e-ipp-b-4fe974-openshift-default-59b7dbcd46-cpkvz"
Sep 16 02:43:53.742174 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:43:53.742021    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4b2c7d51-9fcb-4bae-a99c-66ad0a8e62eb-istio-podinfo\") pod \"e2e-ipp-b-4fe974-openshift-default-59b7dbcd46-cpkvz\" (UID: \"4b2c7d51-9fcb-4bae-a99c-66ad0a8e62eb\") " pod="openshift-ingress/e2e-ipp-b-4fe974-openshift-default-59b7dbcd46-cpkvz"
Sep 16 02:43:53.742174 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:43:53.742056    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4b2c7d51-9fcb-4bae-a99c-66ad0a8e62eb-istio-envoy\") pod \"e2e-ipp-b-4fe974-openshift-default-59b7dbcd46-cpkvz\" (UID: \"4b2c7d51-9fcb-4bae-a99c-66ad0a8e62eb\") " pod="openshift-ingress/e2e-ipp-b-4fe974-openshift-default-59b7dbcd46-cpkvz"
Sep 16 02:43:53.742174 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:43:53.742094    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4b2c7d51-9fcb-4bae-a99c-66ad0a8e62eb-workload-certs\") pod \"e2e-ipp-b-4fe974-openshift-default-59b7dbcd46-cpkvz\" (UID: \"4b2c7d51-9fcb-4bae-a99c-66ad0a8e62eb\") " pod="openshift-ingress/e2e-ipp-b-4fe974-openshift-default-59b7dbcd46-cpkvz"
Sep 16 02:43:53.842769 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:43:53.842739    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4b2c7d51-9fcb-4bae-a99c-66ad0a8e62eb-istio-podinfo\") pod \"e2e-ipp-b-4fe974-openshift-default-59b7dbcd46-cpkvz\" (UID: \"4b2c7d51-9fcb-4bae-a99c-66ad0a8e62eb\") " pod="openshift-ingress/e2e-ipp-b-4fe974-openshift-default-59b7dbcd46-cpkvz"
Sep 16 02:43:53.842769 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:43:53.842771    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4b2c7d51-9fcb-4bae-a99c-66ad0a8e62eb-istio-envoy\") pod \"e2e-ipp-b-4fe974-openshift-default-59b7dbcd46-cpkvz\" (UID: \"4b2c7d51-9fcb-4bae-a99c-66ad0a8e62eb\") " pod="openshift-ingress/e2e-ipp-b-4fe974-openshift-default-59b7dbcd46-cpkvz"
Sep 16 02:43:53.843020 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:43:53.842794    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4b2c7d51-9fcb-4bae-a99c-66ad0a8e62eb-workload-certs\") pod \"e2e-ipp-b-4fe974-openshift-default-59b7dbcd46-cpkvz\" (UID: \"4b2c7d51-9fcb-4bae-a99c-66ad0a8e62eb\") " pod="openshift-ingress/e2e-ipp-b-4fe974-openshift-default-59b7dbcd46-cpkvz"
Sep 16 02:43:53.843020 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:43:53.842917    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4b2c7d51-9fcb-4bae-a99c-66ad0a8e62eb-istio-data\") pod \"e2e-ipp-b-4fe974-openshift-default-59b7dbcd46-cpkvz\" (UID: \"4b2c7d51-9fcb-4bae-a99c-66ad0a8e62eb\") " pod="openshift-ingress/e2e-ipp-b-4fe974-openshift-default-59b7dbcd46-cpkvz"
Sep 16 02:43:53.843020 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:43:53.842956    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b97v2\" (UniqueName: \"kubernetes.io/projected/4b2c7d51-9fcb-4bae-a99c-66ad0a8e62eb-kube-api-access-b97v2\") pod \"e2e-ipp-b-4fe974-openshift-default-59b7dbcd46-cpkvz\" (UID: \"4b2c7d51-9fcb-4bae-a99c-66ad0a8e62eb\") " pod="openshift-ingress/e2e-ipp-b-4fe974-openshift-default-59b7dbcd46-cpkvz"
Sep 16 02:43:53.843020 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:43:53.842983    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4b2c7d51-9fcb-4bae-a99c-66ad0a8e62eb-workload-socket\") pod \"e2e-ipp-b-4fe974-openshift-default-59b7dbcd46-cpkvz\" (UID: \"4b2c7d51-9fcb-4bae-a99c-66ad0a8e62eb\") " pod="openshift-ingress/e2e-ipp-b-4fe974-openshift-default-59b7dbcd46-cpkvz"
Sep 16 02:43:53.843247 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:43:53.843062    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4b2c7d51-9fcb-4bae-a99c-66ad0a8e62eb-istio-token\") pod \"e2e-ipp-b-4fe974-openshift-default-59b7dbcd46-cpkvz\" (UID: \"4b2c7d51-9fcb-4bae-a99c-66ad0a8e62eb\") " pod="openshift-ingress/e2e-ipp-b-4fe974-openshift-default-59b7dbcd46-cpkvz"
Sep 16 02:43:53.843247 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:43:53.843097    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4b2c7d51-9fcb-4bae-a99c-66ad0a8e62eb-istiod-ca-cert\") pod \"e2e-ipp-b-4fe974-openshift-default-59b7dbcd46-cpkvz\" (UID: \"4b2c7d51-9fcb-4bae-a99c-66ad0a8e62eb\") " pod="openshift-ingress/e2e-ipp-b-4fe974-openshift-default-59b7dbcd46-cpkvz"
Sep 16 02:43:53.843247 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:43:53.843147    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4b2c7d51-9fcb-4bae-a99c-66ad0a8e62eb-credential-socket\") pod \"e2e-ipp-b-4fe974-openshift-default-59b7dbcd46-cpkvz\" (UID: \"4b2c7d51-9fcb-4bae-a99c-66ad0a8e62eb\") " pod="openshift-ingress/e2e-ipp-b-4fe974-openshift-default-59b7dbcd46-cpkvz"
Sep 16 02:43:53.843247 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:43:53.843169    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4b2c7d51-9fcb-4bae-a99c-66ad0a8e62eb-workload-certs\") pod \"e2e-ipp-b-4fe974-openshift-default-59b7dbcd46-cpkvz\" (UID: \"4b2c7d51-9fcb-4bae-a99c-66ad0a8e62eb\") " pod="openshift-ingress/e2e-ipp-b-4fe974-openshift-default-59b7dbcd46-cpkvz"
Sep 16 02:43:53.843247 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:43:53.843224    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4b2c7d51-9fcb-4bae-a99c-66ad0a8e62eb-istio-data\") pod \"e2e-ipp-b-4fe974-openshift-default-59b7dbcd46-cpkvz\" (UID: \"4b2c7d51-9fcb-4bae-a99c-66ad0a8e62eb\") " pod="openshift-ingress/e2e-ipp-b-4fe974-openshift-default-59b7dbcd46-cpkvz"
Sep 16 02:43:53.843509 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:43:53.843449    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4b2c7d51-9fcb-4bae-a99c-66ad0a8e62eb-workload-socket\") pod \"e2e-ipp-b-4fe974-openshift-default-59b7dbcd46-cpkvz\" (UID: \"4b2c7d51-9fcb-4bae-a99c-66ad0a8e62eb\") " pod="openshift-ingress/e2e-ipp-b-4fe974-openshift-default-59b7dbcd46-cpkvz"
Sep 16 02:43:53.843509 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:43:53.843469    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4b2c7d51-9fcb-4bae-a99c-66ad0a8e62eb-credential-socket\") pod \"e2e-ipp-b-4fe974-openshift-default-59b7dbcd46-cpkvz\" (UID: \"4b2c7d51-9fcb-4bae-a99c-66ad0a8e62eb\") " pod="openshift-ingress/e2e-ipp-b-4fe974-openshift-default-59b7dbcd46-cpkvz"
Sep 16 02:43:53.843818 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:43:53.843799    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4b2c7d51-9fcb-4bae-a99c-66ad0a8e62eb-istiod-ca-cert\") pod \"e2e-ipp-b-4fe974-openshift-default-59b7dbcd46-cpkvz\" (UID: \"4b2c7d51-9fcb-4bae-a99c-66ad0a8e62eb\") " pod="openshift-ingress/e2e-ipp-b-4fe974-openshift-default-59b7dbcd46-cpkvz"
Sep 16 02:43:53.845213 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:43:53.845189    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4b2c7d51-9fcb-4bae-a99c-66ad0a8e62eb-istio-envoy\") pod \"e2e-ipp-b-4fe974-openshift-default-59b7dbcd46-cpkvz\" (UID: \"4b2c7d51-9fcb-4bae-a99c-66ad0a8e62eb\") " pod="openshift-ingress/e2e-ipp-b-4fe974-openshift-default-59b7dbcd46-cpkvz"
Sep 16 02:43:53.845364 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:43:53.845349    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4b2c7d51-9fcb-4bae-a99c-66ad0a8e62eb-istio-podinfo\") pod \"e2e-ipp-b-4fe974-openshift-default-59b7dbcd46-cpkvz\" (UID: \"4b2c7d51-9fcb-4bae-a99c-66ad0a8e62eb\") " pod="openshift-ingress/e2e-ipp-b-4fe974-openshift-default-59b7dbcd46-cpkvz"
Sep 16 02:43:53.850233 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:43:53.850201    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4b2c7d51-9fcb-4bae-a99c-66ad0a8e62eb-istio-token\") pod \"e2e-ipp-b-4fe974-openshift-default-59b7dbcd46-cpkvz\" (UID: \"4b2c7d51-9fcb-4bae-a99c-66ad0a8e62eb\") " pod="openshift-ingress/e2e-ipp-b-4fe974-openshift-default-59b7dbcd46-cpkvz"
Sep 16 02:43:53.850343 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:43:53.850281    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-b97v2\" (UniqueName: \"kubernetes.io/projected/4b2c7d51-9fcb-4bae-a99c-66ad0a8e62eb-kube-api-access-b97v2\") pod \"e2e-ipp-b-4fe974-openshift-default-59b7dbcd46-cpkvz\" (UID: \"4b2c7d51-9fcb-4bae-a99c-66ad0a8e62eb\") " pod="openshift-ingress/e2e-ipp-b-4fe974-openshift-default-59b7dbcd46-cpkvz"
Sep 16 02:43:53.923530 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:43:53.923448    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-4fe974-openshift-default-59b7dbcd46-cpkvz"
Sep 16 02:43:54.195806 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:43:54.195779    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-4fe974-openshift-default-59b7dbcd46-cpkvz"]
Sep 16 02:43:54.197683 ip-10-0-128-111 kubenswrapper[2583]: W0916 02:43:54.197654    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4b2c7d51_9fcb_4bae_a99c_66ad0a8e62eb.slice/crio-8f1cc1b52728c0440cabb5c2632e0a0e4d059a2a27a583ea1f7e799632a2fd6f WatchSource:0}: Error finding container 8f1cc1b52728c0440cabb5c2632e0a0e4d059a2a27a583ea1f7e799632a2fd6f: Status 404 returned error can't find the container with id 8f1cc1b52728c0440cabb5c2632e0a0e4d059a2a27a583ea1f7e799632a2fd6f
Sep 16 02:43:54.200226 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:43:54.200189    2583 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 16 02:43:54.200292 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:43:54.200277    2583 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 16 02:43:54.200337 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:43:54.200323    2583 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 16 02:43:54.524674 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:43:54.524584    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-4fe974-openshift-default-59b7dbcd46-cpkvz" event={"ID":"4b2c7d51-9fcb-4bae-a99c-66ad0a8e62eb","Type":"ContainerStarted","Data":"3c8c75e419f914054215a63cf3ddba4923d913959f3d76a9c8aa9af0d19e8adb"}
Sep 16 02:43:54.524674 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:43:54.524651    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-4fe974-openshift-default-59b7dbcd46-cpkvz" event={"ID":"4b2c7d51-9fcb-4bae-a99c-66ad0a8e62eb","Type":"ContainerStarted","Data":"8f1cc1b52728c0440cabb5c2632e0a0e4d059a2a27a583ea1f7e799632a2fd6f"}
Sep 16 02:43:54.542776 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:43:54.542729    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-b-4fe974-openshift-default-59b7dbcd46-cpkvz" podStartSLOduration=1.54271292 podStartE2EDuration="1.54271292s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 02:43:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 02:43:54.54173612 +0000 UTC m=+1513.009524926" watchObservedRunningTime="2026-09-16 02:43:54.54271292 +0000 UTC m=+1513.010501701"
Sep 16 02:43:54.923857 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:43:54.923828    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-4fe974-openshift-default-59b7dbcd46-cpkvz"
Sep 16 02:43:55.928914 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:43:55.928883    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-4fe974-openshift-default-59b7dbcd46-cpkvz"
Sep 16 02:43:56.534071 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:43:56.534040    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-4fe974-openshift-default-59b7dbcd46-cpkvz"
Sep 16 02:43:56.535087 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:43:56.535070    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-4fe974-openshift-default-59b7dbcd46-cpkvz"
Sep 16 02:44:00.349289 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:44:00.349249    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-4fe974-659977f8c4-926tc"]
Sep 16 02:44:00.353038 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:44:00.353016    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-4fe974-659977f8c4-926tc"
Sep 16 02:44:00.355273 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:44:00.355249    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-4fe974\""
Sep 16 02:44:00.355377 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:44:00.355320    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-4fe974-dockercfg-px7l7\""
Sep 16 02:44:00.361655 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:44:00.361620    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-4fe974-659977f8c4-926tc"]
Sep 16 02:44:00.370461 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:44:00.370438    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-4fe974-5c5d79b89c-jfqrs"]
Sep 16 02:44:00.374080 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:44:00.374041    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-4fe974-5c5d79b89c-jfqrs"
Sep 16 02:44:00.384774 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:44:00.384755    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-4fe974-5c5d79b89c-jfqrs"]
Sep 16 02:44:00.508089 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:44:00.508053    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qdxq5\" (UniqueName: \"kubernetes.io/projected/a18ebddf-5578-4752-852a-3dcfe8eb881b-kube-api-access-qdxq5\") pod \"payload-processing-e2e-ipp-b-4fe974-5c5d79b89c-jfqrs\" (UID: \"a18ebddf-5578-4752-852a-3dcfe8eb881b\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-4fe974-5c5d79b89c-jfqrs"
Sep 16 02:44:00.508089 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:44:00.508090    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a18ebddf-5578-4752-852a-3dcfe8eb881b-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-4fe974-5c5d79b89c-jfqrs\" (UID: \"a18ebddf-5578-4752-852a-3dcfe8eb881b\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-4fe974-5c5d79b89c-jfqrs"
Sep 16 02:44:00.508324 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:44:00.508217    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7j8pp\" (UniqueName: \"kubernetes.io/projected/7d492f3f-cf1b-4902-b111-4c0f248d6fe1-kube-api-access-7j8pp\") pod \"payload-pre-processing-e2e-ipp-b-4fe974-659977f8c4-926tc\" (UID: \"7d492f3f-cf1b-4902-b111-4c0f248d6fe1\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-4fe974-659977f8c4-926tc"
Sep 16 02:44:00.508324 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:44:00.508271    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7d492f3f-cf1b-4902-b111-4c0f248d6fe1-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-4fe974-659977f8c4-926tc\" (UID: \"7d492f3f-cf1b-4902-b111-4c0f248d6fe1\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-4fe974-659977f8c4-926tc"
Sep 16 02:44:00.608938 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:44:00.608852    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7d492f3f-cf1b-4902-b111-4c0f248d6fe1-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-4fe974-659977f8c4-926tc\" (UID: \"7d492f3f-cf1b-4902-b111-4c0f248d6fe1\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-4fe974-659977f8c4-926tc"
Sep 16 02:44:00.608938 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:44:00.608907    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qdxq5\" (UniqueName: \"kubernetes.io/projected/a18ebddf-5578-4752-852a-3dcfe8eb881b-kube-api-access-qdxq5\") pod \"payload-processing-e2e-ipp-b-4fe974-5c5d79b89c-jfqrs\" (UID: \"a18ebddf-5578-4752-852a-3dcfe8eb881b\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-4fe974-5c5d79b89c-jfqrs"
Sep 16 02:44:00.609129 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:44:00.609022    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a18ebddf-5578-4752-852a-3dcfe8eb881b-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-4fe974-5c5d79b89c-jfqrs\" (UID: \"a18ebddf-5578-4752-852a-3dcfe8eb881b\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-4fe974-5c5d79b89c-jfqrs"
Sep 16 02:44:00.609177 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:44:00.609150    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7j8pp\" (UniqueName: \"kubernetes.io/projected/7d492f3f-cf1b-4902-b111-4c0f248d6fe1-kube-api-access-7j8pp\") pod \"payload-pre-processing-e2e-ipp-b-4fe974-659977f8c4-926tc\" (UID: \"7d492f3f-cf1b-4902-b111-4c0f248d6fe1\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-4fe974-659977f8c4-926tc"
Sep 16 02:44:00.609654 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:44:00.609637    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7d492f3f-cf1b-4902-b111-4c0f248d6fe1-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-4fe974-659977f8c4-926tc\" (UID: \"7d492f3f-cf1b-4902-b111-4c0f248d6fe1\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-4fe974-659977f8c4-926tc"
Sep 16 02:44:00.609728 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:44:00.609710    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a18ebddf-5578-4752-852a-3dcfe8eb881b-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-4fe974-5c5d79b89c-jfqrs\" (UID: \"a18ebddf-5578-4752-852a-3dcfe8eb881b\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-4fe974-5c5d79b89c-jfqrs"
Sep 16 02:44:00.616181 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:44:00.616149    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qdxq5\" (UniqueName: \"kubernetes.io/projected/a18ebddf-5578-4752-852a-3dcfe8eb881b-kube-api-access-qdxq5\") pod \"payload-processing-e2e-ipp-b-4fe974-5c5d79b89c-jfqrs\" (UID: \"a18ebddf-5578-4752-852a-3dcfe8eb881b\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-4fe974-5c5d79b89c-jfqrs"
Sep 16 02:44:00.616181 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:44:00.616173    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7j8pp\" (UniqueName: \"kubernetes.io/projected/7d492f3f-cf1b-4902-b111-4c0f248d6fe1-kube-api-access-7j8pp\") pod \"payload-pre-processing-e2e-ipp-b-4fe974-659977f8c4-926tc\" (UID: \"7d492f3f-cf1b-4902-b111-4c0f248d6fe1\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-4fe974-659977f8c4-926tc"
Sep 16 02:44:00.665994 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:44:00.665962    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-4fe974-659977f8c4-926tc"
Sep 16 02:44:00.685196 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:44:00.685168    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-4fe974-5c5d79b89c-jfqrs"
Sep 16 02:44:00.939766 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:44:00.939692    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-4fe974-659977f8c4-926tc"]
Sep 16 02:44:00.941968 ip-10-0-128-111 kubenswrapper[2583]: W0916 02:44:00.941945    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7d492f3f_cf1b_4902_b111_4c0f248d6fe1.slice/crio-21bd20969d1770036b6376964db22c19c8b60069900c4e299d99a4f2310d7fd2 WatchSource:0}: Error finding container 21bd20969d1770036b6376964db22c19c8b60069900c4e299d99a4f2310d7fd2: Status 404 returned error can't find the container with id 21bd20969d1770036b6376964db22c19c8b60069900c4e299d99a4f2310d7fd2
Sep 16 02:44:00.965913 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:44:00.965885    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-4fe974-5c5d79b89c-jfqrs"]
Sep 16 02:44:00.979850 ip-10-0-128-111 kubenswrapper[2583]: W0916 02:44:00.979827    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda18ebddf_5578_4752_852a_3dcfe8eb881b.slice/crio-55528f2447a525d7cae1c313034c488a91fcf171b20f3fb63a0aab4f7bf93fee WatchSource:0}: Error finding container 55528f2447a525d7cae1c313034c488a91fcf171b20f3fb63a0aab4f7bf93fee: Status 404 returned error can't find the container with id 55528f2447a525d7cae1c313034c488a91fcf171b20f3fb63a0aab4f7bf93fee
Sep 16 02:44:01.557587 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:44:01.557534    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-4fe974-5c5d79b89c-jfqrs" event={"ID":"a18ebddf-5578-4752-852a-3dcfe8eb881b","Type":"ContainerStarted","Data":"01e59d1d2026dc539ddbd2a39a32f8f42b51a44bfb204d4faaa55022110c692a"}
Sep 16 02:44:01.558067 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:44:01.557612    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-4fe974-5c5d79b89c-jfqrs" event={"ID":"a18ebddf-5578-4752-852a-3dcfe8eb881b","Type":"ContainerStarted","Data":"55528f2447a525d7cae1c313034c488a91fcf171b20f3fb63a0aab4f7bf93fee"}
Sep 16 02:44:01.558067 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:44:01.557720    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-4fe974-5c5d79b89c-jfqrs"
Sep 16 02:44:01.559278 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:44:01.559253    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-4fe974-659977f8c4-926tc" event={"ID":"7d492f3f-cf1b-4902-b111-4c0f248d6fe1","Type":"ContainerStarted","Data":"54d8eb8b89d8f12cae91787ed97d35b6e1ffb1c0cabbacc9e3c061b57f92e2d0"}
Sep 16 02:44:01.559403 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:44:01.559301    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-4fe974-659977f8c4-926tc" event={"ID":"7d492f3f-cf1b-4902-b111-4c0f248d6fe1","Type":"ContainerStarted","Data":"21bd20969d1770036b6376964db22c19c8b60069900c4e299d99a4f2310d7fd2"}
Sep 16 02:44:01.559403 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:44:01.559334    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-4fe974-659977f8c4-926tc"
Sep 16 02:44:01.574951 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:44:01.574898    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-4fe974-5c5d79b89c-jfqrs" podStartSLOduration=1.574881291 podStartE2EDuration="1.574881291s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 02:44:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 02:44:01.573008954 +0000 UTC m=+1520.040797738" watchObservedRunningTime="2026-09-16 02:44:01.574881291 +0000 UTC m=+1520.042670074"
Sep 16 02:44:01.586889 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:44:01.586828    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-4fe974-659977f8c4-926tc" podStartSLOduration=1.586806674 podStartE2EDuration="1.586806674s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 02:44:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 02:44:01.586437561 +0000 UTC m=+1520.054226345" watchObservedRunningTime="2026-09-16 02:44:01.586806674 +0000 UTC m=+1520.054595459"
Sep 16 02:44:06.764166 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:44:06.764075    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-fzl8t"]
Sep 16 02:44:12.565658 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:44:12.565623    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-4fe974-659977f8c4-926tc"
Sep 16 02:44:12.566063 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:44:12.565698    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-4fe974-5c5d79b89c-jfqrs"
Sep 16 02:44:17.647427 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:44:17.647391    2583 generic.go:356] "Generic (PLEG): container finished" podID="4b2c7d51-9fcb-4bae-a99c-66ad0a8e62eb" containerID="3c8c75e419f914054215a63cf3ddba4923d913959f3d76a9c8aa9af0d19e8adb" exitCode=0
Sep 16 02:44:17.647896 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:44:17.647444    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-4fe974-openshift-default-59b7dbcd46-cpkvz" event={"ID":"4b2c7d51-9fcb-4bae-a99c-66ad0a8e62eb","Type":"ContainerDied","Data":"3c8c75e419f914054215a63cf3ddba4923d913959f3d76a9c8aa9af0d19e8adb"}
Sep 16 02:44:17.647896 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:44:17.647850    2583 scope.go:117] "RemoveContainer" containerID="3c8c75e419f914054215a63cf3ddba4923d913959f3d76a9c8aa9af0d19e8adb"
Sep 16 02:44:17.648989 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:44:17.648959    2583 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 16 02:44:17.649051 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:44:17.649024    2583 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 16 02:44:17.649096 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:44:17.649056    2583 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 16 02:44:18.654049 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:44:18.654008    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-4fe974-openshift-default-59b7dbcd46-cpkvz" event={"ID":"4b2c7d51-9fcb-4bae-a99c-66ad0a8e62eb","Type":"ContainerStarted","Data":"8002db96b2b0dc6ea48f3e73037be4a28fa3d3fcf1ece34705154668d7f21e72"}
Sep 16 02:44:18.923804 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:44:18.923697    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-4fe974-openshift-default-59b7dbcd46-cpkvz"
Sep 16 02:44:18.924095 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:44:18.923992    2583 patch_prober.go:28] interesting pod/e2e-ipp-b-4fe974-openshift-default-59b7dbcd46-cpkvz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" start-of-body=
Sep 16 02:44:18.924263 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:44:18.924239    2583 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-4fe974-openshift-default-59b7dbcd46-cpkvz" podUID="4b2c7d51-9fcb-4bae-a99c-66ad0a8e62eb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused"
Sep 16 02:44:19.931577 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:44:19.923987    2583 patch_prober.go:28] interesting pod/e2e-ipp-b-4fe974-openshift-default-59b7dbcd46-cpkvz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" start-of-body=
Sep 16 02:44:19.931577 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:44:19.924074    2583 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-4fe974-openshift-default-59b7dbcd46-cpkvz" podUID="4b2c7d51-9fcb-4bae-a99c-66ad0a8e62eb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused"
Sep 16 02:44:20.924474 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:44:20.924015    2583 patch_prober.go:28] interesting pod/e2e-ipp-b-4fe974-openshift-default-59b7dbcd46-cpkvz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" start-of-body=
Sep 16 02:44:20.924474 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:44:20.924092    2583 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-4fe974-openshift-default-59b7dbcd46-cpkvz" podUID="4b2c7d51-9fcb-4bae-a99c-66ad0a8e62eb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused"
Sep 16 02:44:21.928992 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:44:21.928947    2583 patch_prober.go:28] interesting pod/e2e-ipp-b-4fe974-openshift-default-59b7dbcd46-cpkvz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" start-of-body=
Sep 16 02:44:21.929502 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:44:21.929013    2583 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-4fe974-openshift-default-59b7dbcd46-cpkvz" podUID="4b2c7d51-9fcb-4bae-a99c-66ad0a8e62eb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused"
Sep 16 02:44:22.924858 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:44:22.924810    2583 patch_prober.go:28] interesting pod/e2e-ipp-b-4fe974-openshift-default-59b7dbcd46-cpkvz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" start-of-body=
Sep 16 02:44:22.925081 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:44:22.924896    2583 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-4fe974-openshift-default-59b7dbcd46-cpkvz" podUID="4b2c7d51-9fcb-4bae-a99c-66ad0a8e62eb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused"
Sep 16 02:44:23.941922 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:44:23.941889    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-4fe974-openshift-default-59b7dbcd46-cpkvz"
Sep 16 02:44:23.942373 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:44:23.942188    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-4fe974-openshift-default-59b7dbcd46-cpkvz"
Sep 16 02:44:23.943886 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:44:23.943861    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-4fe974-openshift-default-59b7dbcd46-cpkvz"
Sep 16 02:44:26.631023 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:44:26.630994    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-b-4fe974-5c5d79b89c-jfqrs_a18ebddf-5578-4752-852a-3dcfe8eb881b/payload-processing/0.log"
Sep 16 02:44:30.838771 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:44:30.838740    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-4178a6-openshift-default-596589786-tm9nc"]
Sep 16 02:44:30.841863 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:44:30.841842    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-4178a6-openshift-default-596589786-tm9nc"
Sep 16 02:44:30.844145 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:44:30.844117    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-4178a6-openshift-default-dockercfg-9rdhz\""
Sep 16 02:44:30.854503 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:44:30.854480    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-4178a6-openshift-default-596589786-tm9nc"]
Sep 16 02:44:30.921150 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:44:30.921118    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a7e3b05a-b2e6-4a33-b102-c7c98550b985-istio-envoy\") pod \"e2e-api-a-4178a6-openshift-default-596589786-tm9nc\" (UID: \"a7e3b05a-b2e6-4a33-b102-c7c98550b985\") " pod="openshift-ingress/e2e-api-a-4178a6-openshift-default-596589786-tm9nc"
Sep 16 02:44:30.921297 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:44:30.921156    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2hl6k\" (UniqueName: \"kubernetes.io/projected/a7e3b05a-b2e6-4a33-b102-c7c98550b985-kube-api-access-2hl6k\") pod \"e2e-api-a-4178a6-openshift-default-596589786-tm9nc\" (UID: \"a7e3b05a-b2e6-4a33-b102-c7c98550b985\") " pod="openshift-ingress/e2e-api-a-4178a6-openshift-default-596589786-tm9nc"
Sep 16 02:44:30.921297 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:44:30.921187    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a7e3b05a-b2e6-4a33-b102-c7c98550b985-credential-socket\") pod \"e2e-api-a-4178a6-openshift-default-596589786-tm9nc\" (UID: \"a7e3b05a-b2e6-4a33-b102-c7c98550b985\") " pod="openshift-ingress/e2e-api-a-4178a6-openshift-default-596589786-tm9nc"
Sep 16 02:44:30.921297 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:44:30.921248    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a7e3b05a-b2e6-4a33-b102-c7c98550b985-workload-certs\") pod \"e2e-api-a-4178a6-openshift-default-596589786-tm9nc\" (UID: \"a7e3b05a-b2e6-4a33-b102-c7c98550b985\") " pod="openshift-ingress/e2e-api-a-4178a6-openshift-default-596589786-tm9nc"
Sep 16 02:44:30.921297 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:44:30.921289    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a7e3b05a-b2e6-4a33-b102-c7c98550b985-istio-data\") pod \"e2e-api-a-4178a6-openshift-default-596589786-tm9nc\" (UID: \"a7e3b05a-b2e6-4a33-b102-c7c98550b985\") " pod="openshift-ingress/e2e-api-a-4178a6-openshift-default-596589786-tm9nc"
Sep 16 02:44:30.921438 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:44:30.921320    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a7e3b05a-b2e6-4a33-b102-c7c98550b985-istio-token\") pod \"e2e-api-a-4178a6-openshift-default-596589786-tm9nc\" (UID: \"a7e3b05a-b2e6-4a33-b102-c7c98550b985\") " pod="openshift-ingress/e2e-api-a-4178a6-openshift-default-596589786-tm9nc"
Sep 16 02:44:30.921438 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:44:30.921367    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a7e3b05a-b2e6-4a33-b102-c7c98550b985-workload-socket\") pod \"e2e-api-a-4178a6-openshift-default-596589786-tm9nc\" (UID: \"a7e3b05a-b2e6-4a33-b102-c7c98550b985\") " pod="openshift-ingress/e2e-api-a-4178a6-openshift-default-596589786-tm9nc"
Sep 16 02:44:30.921438 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:44:30.921407    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a7e3b05a-b2e6-4a33-b102-c7c98550b985-istio-podinfo\") pod \"e2e-api-a-4178a6-openshift-default-596589786-tm9nc\" (UID: \"a7e3b05a-b2e6-4a33-b102-c7c98550b985\") " pod="openshift-ingress/e2e-api-a-4178a6-openshift-default-596589786-tm9nc"
Sep 16 02:44:30.921438 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:44:30.921430    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a7e3b05a-b2e6-4a33-b102-c7c98550b985-istiod-ca-cert\") pod \"e2e-api-a-4178a6-openshift-default-596589786-tm9nc\" (UID: \"a7e3b05a-b2e6-4a33-b102-c7c98550b985\") " pod="openshift-ingress/e2e-api-a-4178a6-openshift-default-596589786-tm9nc"
Sep 16 02:44:31.022024 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:44:31.021987    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a7e3b05a-b2e6-4a33-b102-c7c98550b985-istio-token\") pod \"e2e-api-a-4178a6-openshift-default-596589786-tm9nc\" (UID: \"a7e3b05a-b2e6-4a33-b102-c7c98550b985\") " pod="openshift-ingress/e2e-api-a-4178a6-openshift-default-596589786-tm9nc"
Sep 16 02:44:31.022199 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:44:31.022042    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a7e3b05a-b2e6-4a33-b102-c7c98550b985-workload-socket\") pod \"e2e-api-a-4178a6-openshift-default-596589786-tm9nc\" (UID: \"a7e3b05a-b2e6-4a33-b102-c7c98550b985\") " pod="openshift-ingress/e2e-api-a-4178a6-openshift-default-596589786-tm9nc"
Sep 16 02:44:31.022199 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:44:31.022070    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a7e3b05a-b2e6-4a33-b102-c7c98550b985-istio-podinfo\") pod \"e2e-api-a-4178a6-openshift-default-596589786-tm9nc\" (UID: \"a7e3b05a-b2e6-4a33-b102-c7c98550b985\") " pod="openshift-ingress/e2e-api-a-4178a6-openshift-default-596589786-tm9nc"
Sep 16 02:44:31.022199 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:44:31.022086    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a7e3b05a-b2e6-4a33-b102-c7c98550b985-istiod-ca-cert\") pod \"e2e-api-a-4178a6-openshift-default-596589786-tm9nc\" (UID: \"a7e3b05a-b2e6-4a33-b102-c7c98550b985\") " pod="openshift-ingress/e2e-api-a-4178a6-openshift-default-596589786-tm9nc"
Sep 16 02:44:31.022199 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:44:31.022136    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a7e3b05a-b2e6-4a33-b102-c7c98550b985-istio-envoy\") pod \"e2e-api-a-4178a6-openshift-default-596589786-tm9nc\" (UID: \"a7e3b05a-b2e6-4a33-b102-c7c98550b985\") " pod="openshift-ingress/e2e-api-a-4178a6-openshift-default-596589786-tm9nc"
Sep 16 02:44:31.022442 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:44:31.022416    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2hl6k\" (UniqueName: \"kubernetes.io/projected/a7e3b05a-b2e6-4a33-b102-c7c98550b985-kube-api-access-2hl6k\") pod \"e2e-api-a-4178a6-openshift-default-596589786-tm9nc\" (UID: \"a7e3b05a-b2e6-4a33-b102-c7c98550b985\") " pod="openshift-ingress/e2e-api-a-4178a6-openshift-default-596589786-tm9nc"
Sep 16 02:44:31.022479 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:44:31.022465    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a7e3b05a-b2e6-4a33-b102-c7c98550b985-credential-socket\") pod \"e2e-api-a-4178a6-openshift-default-596589786-tm9nc\" (UID: \"a7e3b05a-b2e6-4a33-b102-c7c98550b985\") " pod="openshift-ingress/e2e-api-a-4178a6-openshift-default-596589786-tm9nc"
Sep 16 02:44:31.022530 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:44:31.022493    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a7e3b05a-b2e6-4a33-b102-c7c98550b985-workload-socket\") pod \"e2e-api-a-4178a6-openshift-default-596589786-tm9nc\" (UID: \"a7e3b05a-b2e6-4a33-b102-c7c98550b985\") " pod="openshift-ingress/e2e-api-a-4178a6-openshift-default-596589786-tm9nc"
Sep 16 02:44:31.022530 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:44:31.022506    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a7e3b05a-b2e6-4a33-b102-c7c98550b985-workload-certs\") pod \"e2e-api-a-4178a6-openshift-default-596589786-tm9nc\" (UID: \"a7e3b05a-b2e6-4a33-b102-c7c98550b985\") " pod="openshift-ingress/e2e-api-a-4178a6-openshift-default-596589786-tm9nc"
Sep 16 02:44:31.022666 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:44:31.022588    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a7e3b05a-b2e6-4a33-b102-c7c98550b985-istio-data\") pod \"e2e-api-a-4178a6-openshift-default-596589786-tm9nc\" (UID: \"a7e3b05a-b2e6-4a33-b102-c7c98550b985\") " pod="openshift-ingress/e2e-api-a-4178a6-openshift-default-596589786-tm9nc"
Sep 16 02:44:31.022823 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:44:31.022763    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a7e3b05a-b2e6-4a33-b102-c7c98550b985-workload-certs\") pod \"e2e-api-a-4178a6-openshift-default-596589786-tm9nc\" (UID: \"a7e3b05a-b2e6-4a33-b102-c7c98550b985\") " pod="openshift-ingress/e2e-api-a-4178a6-openshift-default-596589786-tm9nc"
Sep 16 02:44:31.022948 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:44:31.022905    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a7e3b05a-b2e6-4a33-b102-c7c98550b985-istiod-ca-cert\") pod \"e2e-api-a-4178a6-openshift-default-596589786-tm9nc\" (UID: \"a7e3b05a-b2e6-4a33-b102-c7c98550b985\") " pod="openshift-ingress/e2e-api-a-4178a6-openshift-default-596589786-tm9nc"
Sep 16 02:44:31.022948 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:44:31.022933    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a7e3b05a-b2e6-4a33-b102-c7c98550b985-istio-data\") pod \"e2e-api-a-4178a6-openshift-default-596589786-tm9nc\" (UID: \"a7e3b05a-b2e6-4a33-b102-c7c98550b985\") " pod="openshift-ingress/e2e-api-a-4178a6-openshift-default-596589786-tm9nc"
Sep 16 02:44:31.023028 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:44:31.022949    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a7e3b05a-b2e6-4a33-b102-c7c98550b985-credential-socket\") pod \"e2e-api-a-4178a6-openshift-default-596589786-tm9nc\" (UID: \"a7e3b05a-b2e6-4a33-b102-c7c98550b985\") " pod="openshift-ingress/e2e-api-a-4178a6-openshift-default-596589786-tm9nc"
Sep 16 02:44:31.025052 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:44:31.025028    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a7e3b05a-b2e6-4a33-b102-c7c98550b985-istio-envoy\") pod \"e2e-api-a-4178a6-openshift-default-596589786-tm9nc\" (UID: \"a7e3b05a-b2e6-4a33-b102-c7c98550b985\") " pod="openshift-ingress/e2e-api-a-4178a6-openshift-default-596589786-tm9nc"
Sep 16 02:44:31.025147 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:44:31.025130    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a7e3b05a-b2e6-4a33-b102-c7c98550b985-istio-podinfo\") pod \"e2e-api-a-4178a6-openshift-default-596589786-tm9nc\" (UID: \"a7e3b05a-b2e6-4a33-b102-c7c98550b985\") " pod="openshift-ingress/e2e-api-a-4178a6-openshift-default-596589786-tm9nc"
Sep 16 02:44:31.029404 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:44:31.029379    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a7e3b05a-b2e6-4a33-b102-c7c98550b985-istio-token\") pod \"e2e-api-a-4178a6-openshift-default-596589786-tm9nc\" (UID: \"a7e3b05a-b2e6-4a33-b102-c7c98550b985\") " pod="openshift-ingress/e2e-api-a-4178a6-openshift-default-596589786-tm9nc"
Sep 16 02:44:31.030236 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:44:31.030217    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2hl6k\" (UniqueName: \"kubernetes.io/projected/a7e3b05a-b2e6-4a33-b102-c7c98550b985-kube-api-access-2hl6k\") pod \"e2e-api-a-4178a6-openshift-default-596589786-tm9nc\" (UID: \"a7e3b05a-b2e6-4a33-b102-c7c98550b985\") " pod="openshift-ingress/e2e-api-a-4178a6-openshift-default-596589786-tm9nc"
Sep 16 02:44:31.153918 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:44:31.153829    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-4178a6-openshift-default-596589786-tm9nc"
Sep 16 02:44:31.430783 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:44:31.430753    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-4178a6-openshift-default-596589786-tm9nc"]
Sep 16 02:44:31.432331 ip-10-0-128-111 kubenswrapper[2583]: W0916 02:44:31.432304    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda7e3b05a_b2e6_4a33_b102_c7c98550b985.slice/crio-c99c3bb91d03055d234a7206e1423cf7aa8056609d163c5097de05d16f4428ef WatchSource:0}: Error finding container c99c3bb91d03055d234a7206e1423cf7aa8056609d163c5097de05d16f4428ef: Status 404 returned error can't find the container with id c99c3bb91d03055d234a7206e1423cf7aa8056609d163c5097de05d16f4428ef
Sep 16 02:44:31.434900 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:44:31.434850    2583 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 16 02:44:31.434978 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:44:31.434964    2583 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 16 02:44:31.435016 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:44:31.435006    2583 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 16 02:44:31.721122 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:44:31.721027    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-4178a6-openshift-default-596589786-tm9nc" event={"ID":"a7e3b05a-b2e6-4a33-b102-c7c98550b985","Type":"ContainerStarted","Data":"09c436387ff7aed3dcff8780c92064c7307ad2034cc3f53e62b1c352b6f49f87"}
Sep 16 02:44:31.721122 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:44:31.721088    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-4178a6-openshift-default-596589786-tm9nc" event={"ID":"a7e3b05a-b2e6-4a33-b102-c7c98550b985","Type":"ContainerStarted","Data":"c99c3bb91d03055d234a7206e1423cf7aa8056609d163c5097de05d16f4428ef"}
Sep 16 02:44:31.740852 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:44:31.740794    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-4178a6-openshift-default-596589786-tm9nc" podStartSLOduration=1.7407771269999999 podStartE2EDuration="1.740777127s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 02:44:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 02:44:31.738793456 +0000 UTC m=+1550.206582243" watchObservedRunningTime="2026-09-16 02:44:31.740777127 +0000 UTC m=+1550.208565908"
Sep 16 02:44:32.154273 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:44:32.154237    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-4178a6-openshift-default-596589786-tm9nc"
Sep 16 02:44:32.159306 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:44:32.159283    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-4178a6-openshift-default-596589786-tm9nc"
Sep 16 02:44:32.726141 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:44:32.726111    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-4178a6-openshift-default-596589786-tm9nc"
Sep 16 02:44:32.727017 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:44:32.726994    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-4178a6-openshift-default-596589786-tm9nc"
Sep 16 02:44:38.386381 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:44:38.386341    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-4178a6-66b5858d78-fdlkw"]
Sep 16 02:44:38.391111 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:44:38.391094    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-4178a6-66b5858d78-fdlkw"
Sep 16 02:44:38.393347 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:44:38.393314    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-a-4178a6-dockercfg-jlfkk\""
Sep 16 02:44:38.393501 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:44:38.393406    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-a-4178a6\""
Sep 16 02:44:38.399439 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:44:38.399415    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-4178a6-66b5858d78-fdlkw"]
Sep 16 02:44:38.420626 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:44:38.420600    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-4178a6-67c58859d5-mc9tp"]
Sep 16 02:44:38.424091 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:44:38.424074    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-4178a6-67c58859d5-mc9tp"
Sep 16 02:44:38.433891 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:44:38.433867    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-4178a6-67c58859d5-mc9tp"]
Sep 16 02:44:38.497269 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:44:38.497231    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/934c93b5-25af-450b-8be2-1cffaae7fb9a-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-4178a6-66b5858d78-fdlkw\" (UID: \"934c93b5-25af-450b-8be2-1cffaae7fb9a\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-4178a6-66b5858d78-fdlkw"
Sep 16 02:44:38.497479 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:44:38.497298    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4bb17a92-fec6-4725-aca1-dd044f679ed0-plugins-config-volume\") pod \"payload-processing-e2e-api-a-4178a6-67c58859d5-mc9tp\" (UID: \"4bb17a92-fec6-4725-aca1-dd044f679ed0\") " pod="openshift-ingress/payload-processing-e2e-api-a-4178a6-67c58859d5-mc9tp"
Sep 16 02:44:38.497479 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:44:38.497364    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7w7cs\" (UniqueName: \"kubernetes.io/projected/4bb17a92-fec6-4725-aca1-dd044f679ed0-kube-api-access-7w7cs\") pod \"payload-processing-e2e-api-a-4178a6-67c58859d5-mc9tp\" (UID: \"4bb17a92-fec6-4725-aca1-dd044f679ed0\") " pod="openshift-ingress/payload-processing-e2e-api-a-4178a6-67c58859d5-mc9tp"
Sep 16 02:44:38.497479 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:44:38.497393    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8ljxp\" (UniqueName: \"kubernetes.io/projected/934c93b5-25af-450b-8be2-1cffaae7fb9a-kube-api-access-8ljxp\") pod \"payload-pre-processing-e2e-api-a-4178a6-66b5858d78-fdlkw\" (UID: \"934c93b5-25af-450b-8be2-1cffaae7fb9a\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-4178a6-66b5858d78-fdlkw"
Sep 16 02:44:38.598718 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:44:38.598681    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/934c93b5-25af-450b-8be2-1cffaae7fb9a-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-4178a6-66b5858d78-fdlkw\" (UID: \"934c93b5-25af-450b-8be2-1cffaae7fb9a\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-4178a6-66b5858d78-fdlkw"
Sep 16 02:44:38.598901 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:44:38.598727    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4bb17a92-fec6-4725-aca1-dd044f679ed0-plugins-config-volume\") pod \"payload-processing-e2e-api-a-4178a6-67c58859d5-mc9tp\" (UID: \"4bb17a92-fec6-4725-aca1-dd044f679ed0\") " pod="openshift-ingress/payload-processing-e2e-api-a-4178a6-67c58859d5-mc9tp"
Sep 16 02:44:38.598901 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:44:38.598774    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7w7cs\" (UniqueName: \"kubernetes.io/projected/4bb17a92-fec6-4725-aca1-dd044f679ed0-kube-api-access-7w7cs\") pod \"payload-processing-e2e-api-a-4178a6-67c58859d5-mc9tp\" (UID: \"4bb17a92-fec6-4725-aca1-dd044f679ed0\") " pod="openshift-ingress/payload-processing-e2e-api-a-4178a6-67c58859d5-mc9tp"
Sep 16 02:44:38.598901 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:44:38.598797    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8ljxp\" (UniqueName: \"kubernetes.io/projected/934c93b5-25af-450b-8be2-1cffaae7fb9a-kube-api-access-8ljxp\") pod \"payload-pre-processing-e2e-api-a-4178a6-66b5858d78-fdlkw\" (UID: \"934c93b5-25af-450b-8be2-1cffaae7fb9a\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-4178a6-66b5858d78-fdlkw"
Sep 16 02:44:38.599454 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:44:38.599435    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4bb17a92-fec6-4725-aca1-dd044f679ed0-plugins-config-volume\") pod \"payload-processing-e2e-api-a-4178a6-67c58859d5-mc9tp\" (UID: \"4bb17a92-fec6-4725-aca1-dd044f679ed0\") " pod="openshift-ingress/payload-processing-e2e-api-a-4178a6-67c58859d5-mc9tp"
Sep 16 02:44:38.599497 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:44:38.599460    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/934c93b5-25af-450b-8be2-1cffaae7fb9a-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-4178a6-66b5858d78-fdlkw\" (UID: \"934c93b5-25af-450b-8be2-1cffaae7fb9a\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-4178a6-66b5858d78-fdlkw"
Sep 16 02:44:38.606244 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:44:38.606219    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8ljxp\" (UniqueName: \"kubernetes.io/projected/934c93b5-25af-450b-8be2-1cffaae7fb9a-kube-api-access-8ljxp\") pod \"payload-pre-processing-e2e-api-a-4178a6-66b5858d78-fdlkw\" (UID: \"934c93b5-25af-450b-8be2-1cffaae7fb9a\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-4178a6-66b5858d78-fdlkw"
Sep 16 02:44:38.606381 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:44:38.606361    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7w7cs\" (UniqueName: \"kubernetes.io/projected/4bb17a92-fec6-4725-aca1-dd044f679ed0-kube-api-access-7w7cs\") pod \"payload-processing-e2e-api-a-4178a6-67c58859d5-mc9tp\" (UID: \"4bb17a92-fec6-4725-aca1-dd044f679ed0\") " pod="openshift-ingress/payload-processing-e2e-api-a-4178a6-67c58859d5-mc9tp"
Sep 16 02:44:38.703659 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:44:38.703568    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-4178a6-66b5858d78-fdlkw"
Sep 16 02:44:38.735770 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:44:38.735741    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-4178a6-67c58859d5-mc9tp"
Sep 16 02:44:38.978479 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:44:38.978454    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-4178a6-66b5858d78-fdlkw"]
Sep 16 02:44:38.980262 ip-10-0-128-111 kubenswrapper[2583]: W0916 02:44:38.980234    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod934c93b5_25af_450b_8be2_1cffaae7fb9a.slice/crio-c4551338c79eb3bf47a86d0f497ed7773f61331a9b58ffca5f3ff0be72b442e1 WatchSource:0}: Error finding container c4551338c79eb3bf47a86d0f497ed7773f61331a9b58ffca5f3ff0be72b442e1: Status 404 returned error can't find the container with id c4551338c79eb3bf47a86d0f497ed7773f61331a9b58ffca5f3ff0be72b442e1
Sep 16 02:44:39.021502 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:44:39.021471    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-4178a6-67c58859d5-mc9tp"]
Sep 16 02:44:39.022008 ip-10-0-128-111 kubenswrapper[2583]: W0916 02:44:39.021985    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4bb17a92_fec6_4725_aca1_dd044f679ed0.slice/crio-7edda015ddebbbc605ea6d6d378575fb07ffcd93881527ad3884a6817a9d56b9 WatchSource:0}: Error finding container 7edda015ddebbbc605ea6d6d378575fb07ffcd93881527ad3884a6817a9d56b9: Status 404 returned error can't find the container with id 7edda015ddebbbc605ea6d6d378575fb07ffcd93881527ad3884a6817a9d56b9
Sep 16 02:44:39.764968 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:44:39.764927    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-4178a6-67c58859d5-mc9tp" event={"ID":"4bb17a92-fec6-4725-aca1-dd044f679ed0","Type":"ContainerStarted","Data":"d54243f590268e4f89ef06b0ec994fde28aedddc1b73761359b132b9908c5104"}
Sep 16 02:44:39.765457 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:44:39.765000    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-4178a6-67c58859d5-mc9tp" event={"ID":"4bb17a92-fec6-4725-aca1-dd044f679ed0","Type":"ContainerStarted","Data":"7edda015ddebbbc605ea6d6d378575fb07ffcd93881527ad3884a6817a9d56b9"}
Sep 16 02:44:39.765457 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:44:39.765167    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-a-4178a6-67c58859d5-mc9tp"
Sep 16 02:44:39.766642 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:44:39.766615    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-4178a6-66b5858d78-fdlkw" event={"ID":"934c93b5-25af-450b-8be2-1cffaae7fb9a","Type":"ContainerStarted","Data":"1016d63b6a8dd721a0f79b8037be6b4905b45da03b16a29e53f84161f9c084fd"}
Sep 16 02:44:39.766777 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:44:39.766659    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-4178a6-66b5858d78-fdlkw" event={"ID":"934c93b5-25af-450b-8be2-1cffaae7fb9a","Type":"ContainerStarted","Data":"c4551338c79eb3bf47a86d0f497ed7773f61331a9b58ffca5f3ff0be72b442e1"}
Sep 16 02:44:39.766777 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:44:39.766681    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-4178a6-66b5858d78-fdlkw"
Sep 16 02:44:39.779516 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:44:39.779468    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-a-4178a6-67c58859d5-mc9tp" podStartSLOduration=1.779451335 podStartE2EDuration="1.779451335s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 02:44:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 02:44:39.778796879 +0000 UTC m=+1558.246585661" watchObservedRunningTime="2026-09-16 02:44:39.779451335 +0000 UTC m=+1558.247240117"
Sep 16 02:44:39.793306 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:44:39.793260    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-a-4178a6-66b5858d78-fdlkw" podStartSLOduration=1.793246415 podStartE2EDuration="1.793246415s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 02:44:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 02:44:39.792314151 +0000 UTC m=+1558.260102935" watchObservedRunningTime="2026-09-16 02:44:39.793246415 +0000 UTC m=+1558.261035197"
Sep 16 02:44:44.150177 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:44:44.150139    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-fzl8t"]
Sep 16 02:44:50.773661 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:44:50.773628    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-a-4178a6-67c58859d5-mc9tp"
Sep 16 02:44:50.774022 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:44:50.773703    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-4178a6-66b5858d78-fdlkw"
Sep 16 02:44:53.838701 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:44:53.838671    2583 generic.go:356] "Generic (PLEG): container finished" podID="a7e3b05a-b2e6-4a33-b102-c7c98550b985" containerID="09c436387ff7aed3dcff8780c92064c7307ad2034cc3f53e62b1c352b6f49f87" exitCode=0
Sep 16 02:44:53.839101 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:44:53.838752    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-4178a6-openshift-default-596589786-tm9nc" event={"ID":"a7e3b05a-b2e6-4a33-b102-c7c98550b985","Type":"ContainerDied","Data":"09c436387ff7aed3dcff8780c92064c7307ad2034cc3f53e62b1c352b6f49f87"}
Sep 16 02:44:53.839199 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:44:53.839183    2583 scope.go:117] "RemoveContainer" containerID="09c436387ff7aed3dcff8780c92064c7307ad2034cc3f53e62b1c352b6f49f87"
Sep 16 02:44:53.840321 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:44:53.840292    2583 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 16 02:44:53.840410 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:44:53.840359    2583 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 16 02:44:53.840410 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:44:53.840390    2583 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 16 02:44:54.846879 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:44:54.846844    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-4178a6-openshift-default-596589786-tm9nc" event={"ID":"a7e3b05a-b2e6-4a33-b102-c7c98550b985","Type":"ContainerStarted","Data":"7e453e0c33b890e8afe37bdf6255c778f886d0c29b8abaea032652fb02e9b2d1"}
Sep 16 02:44:55.154624 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:44:55.154504    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-4178a6-openshift-default-596589786-tm9nc"
Sep 16 02:44:55.154774 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:44:55.154756    2583 patch_prober.go:28] interesting pod/e2e-api-a-4178a6-openshift-default-596589786-tm9nc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" start-of-body=
Sep 16 02:44:55.154818 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:44:55.154800    2583 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-4178a6-openshift-default-596589786-tm9nc" podUID="a7e3b05a-b2e6-4a33-b102-c7c98550b985" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused"
Sep 16 02:44:56.154774 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:44:56.154741    2583 patch_prober.go:28] interesting pod/e2e-api-a-4178a6-openshift-default-596589786-tm9nc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" start-of-body=
Sep 16 02:44:56.155171 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:44:56.154794    2583 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-4178a6-openshift-default-596589786-tm9nc" podUID="a7e3b05a-b2e6-4a33-b102-c7c98550b985" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused"
Sep 16 02:44:57.155055 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:44:57.155010    2583 patch_prober.go:28] interesting pod/e2e-api-a-4178a6-openshift-default-596589786-tm9nc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" start-of-body=
Sep 16 02:44:57.155462 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:44:57.155077    2583 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-4178a6-openshift-default-596589786-tm9nc" podUID="a7e3b05a-b2e6-4a33-b102-c7c98550b985" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused"
Sep 16 02:44:58.155156 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:44:58.155115    2583 patch_prober.go:28] interesting pod/e2e-api-a-4178a6-openshift-default-596589786-tm9nc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" start-of-body=
Sep 16 02:44:58.155573 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:44:58.155175    2583 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-4178a6-openshift-default-596589786-tm9nc" podUID="a7e3b05a-b2e6-4a33-b102-c7c98550b985" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused"
Sep 16 02:44:59.154496 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:44:59.154452    2583 patch_prober.go:28] interesting pod/e2e-api-a-4178a6-openshift-default-596589786-tm9nc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" start-of-body=
Sep 16 02:44:59.154724 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:44:59.154533    2583 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-4178a6-openshift-default-596589786-tm9nc" podUID="a7e3b05a-b2e6-4a33-b102-c7c98550b985" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused"
Sep 16 02:45:00.136344 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:45:00.136306    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-4178a6-29825445-lwq6f"]
Sep 16 02:45:00.140116 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:45:00.140094    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-db1ed7-29825445-tnkp6"]
Sep 16 02:45:00.140250 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:45:00.140210    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-4178a6-29825445-lwq6f"
Sep 16 02:45:00.143187 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:45:00.143157    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-w8hnp\""
Sep 16 02:45:00.143882 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:45:00.143854    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-db1ed7-29825445-tnkp6"
Sep 16 02:45:00.147191 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:45:00.147161    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-4178a6-29825445-lwq6f"]
Sep 16 02:45:00.152594 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:45:00.151692    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-db1ed7-29825445-tnkp6"]
Sep 16 02:45:00.157582 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:45:00.157543    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-10a6b9-29825445-5n5s4"]
Sep 16 02:45:00.159180 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:45:00.159149    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-4178a6-29825445-lwq6f"
Sep 16 02:45:00.164118 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:45:00.164092    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-fe9d17-29825445-qjxl4"]
Sep 16 02:45:00.165647 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:45:00.165623    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-10a6b9-29825445-5n5s4"
Sep 16 02:45:00.176978 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:45:00.176933    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-db1ed7-29825445-tnkp6"
Sep 16 02:45:00.180976 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:45:00.180953    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-10a6b9-29825445-5n5s4"]
Sep 16 02:45:00.181159 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:45:00.181145    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-fe9d17-29825445-qjxl4"]
Sep 16 02:45:00.181344 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:45:00.181329    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-fe9d17-29825445-qjxl4"
Sep 16 02:45:00.224909 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:45:00.224709    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-10a6b9-29825445-5n5s4"
Sep 16 02:45:00.234449 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:45:00.234249    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-fe9d17-29825445-qjxl4"
Sep 16 02:45:00.268541 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:45:00.268381    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-4178a6-openshift-default-596589786-tm9nc"
Sep 16 02:45:00.269823 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:45:00.269663    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-4178a6-openshift-default-596589786-tm9nc"
Sep 16 02:45:00.270799 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:45:00.270755    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-4178a6-openshift-default-596589786-tm9nc"
Sep 16 02:45:00.476253 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:45:00.476231    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-4178a6-29825445-lwq6f"]
Sep 16 02:45:00.478427 ip-10-0-128-111 kubenswrapper[2583]: W0916 02:45:00.478401    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1890f865_a55e_435c_adfb_fee2d3754825.slice/crio-46d2b3351b149f6ded10297b950730b76b39519d4a882f0baff9b881466d190a WatchSource:0}: Error finding container 46d2b3351b149f6ded10297b950730b76b39519d4a882f0baff9b881466d190a: Status 404 returned error can't find the container with id 46d2b3351b149f6ded10297b950730b76b39519d4a882f0baff9b881466d190a
Sep 16 02:45:00.489593 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:45:00.488347    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-db1ed7-29825445-tnkp6"]
Sep 16 02:45:00.546191 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:45:00.546166    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-10a6b9-29825445-5n5s4"]
Sep 16 02:45:00.548266 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:45:00.548241    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-fe9d17-29825445-qjxl4"]
Sep 16 02:45:00.548399 ip-10-0-128-111 kubenswrapper[2583]: W0916 02:45:00.548381    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb8b8b76a_5f01_4a45_9750_4b5814c0eedc.slice/crio-4bdbe081dfde3a25049352cd7db5bb7303e6da203893295bf0381478412358c1 WatchSource:0}: Error finding container 4bdbe081dfde3a25049352cd7db5bb7303e6da203893295bf0381478412358c1: Status 404 returned error can't find the container with id 4bdbe081dfde3a25049352cd7db5bb7303e6da203893295bf0381478412358c1
Sep 16 02:45:00.549362 ip-10-0-128-111 kubenswrapper[2583]: W0916 02:45:00.549336    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5f700448_b33c_43ec_ba1d_b850e3e7618a.slice/crio-d8eaeb56e74228f99176983a25b4af6075c7d63dcb01b86b29e06c3846b6ad00 WatchSource:0}: Error finding container d8eaeb56e74228f99176983a25b4af6075c7d63dcb01b86b29e06c3846b6ad00: Status 404 returned error can't find the container with id d8eaeb56e74228f99176983a25b4af6075c7d63dcb01b86b29e06c3846b6ad00
Sep 16 02:45:00.876826 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:45:00.876793    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-fe9d17-29825445-qjxl4" event={"ID":"5f700448-b33c-43ec-ba1d-b850e3e7618a","Type":"ContainerStarted","Data":"ee71db41c3a3cc305eca954c8cff5227a4f531e66c279f742c95f65bf0890aff"}
Sep 16 02:45:00.876972 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:45:00.876850    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-fe9d17-29825445-qjxl4" event={"ID":"5f700448-b33c-43ec-ba1d-b850e3e7618a","Type":"ContainerStarted","Data":"d8eaeb56e74228f99176983a25b4af6075c7d63dcb01b86b29e06c3846b6ad00"}
Sep 16 02:45:00.878372 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:45:00.878340    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-10a6b9-29825445-5n5s4" event={"ID":"b8b8b76a-5f01-4a45-9750-4b5814c0eedc","Type":"ContainerStarted","Data":"33c0e7601aab0514fd6551cc5a457193b28cff2971bb7daad7d7bc6624eac362"}
Sep 16 02:45:00.878501 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:45:00.878388    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-10a6b9-29825445-5n5s4" event={"ID":"b8b8b76a-5f01-4a45-9750-4b5814c0eedc","Type":"ContainerStarted","Data":"4bdbe081dfde3a25049352cd7db5bb7303e6da203893295bf0381478412358c1"}
Sep 16 02:45:00.879955 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:45:00.879931    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-db1ed7-29825445-tnkp6" event={"ID":"12bfa699-c47a-495e-8c11-5c0ff7927bf3","Type":"ContainerStarted","Data":"9cf60f4f256bbb7679ad9a591101fbfe00beb63f6d794d05254bc39b7ad516b2"}
Sep 16 02:45:00.880064 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:45:00.879979    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-db1ed7-29825445-tnkp6" event={"ID":"12bfa699-c47a-495e-8c11-5c0ff7927bf3","Type":"ContainerStarted","Data":"d259b4e4ffcdd428c2bf412ac58b53be05bb4c45bea21bb6fbab76fb9a11a214"}
Sep 16 02:45:00.881485 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:45:00.881460    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-4178a6-29825445-lwq6f" event={"ID":"1890f865-a55e-435c-adfb-fee2d3754825","Type":"ContainerStarted","Data":"7a4714cc303cbbd7a3384553ddbc126ee3a6feffb10755d0a621c602abe1c72a"}
Sep 16 02:45:00.881604 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:45:00.881508    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-4178a6-29825445-lwq6f" event={"ID":"1890f865-a55e-435c-adfb-fee2d3754825","Type":"ContainerStarted","Data":"46d2b3351b149f6ded10297b950730b76b39519d4a882f0baff9b881466d190a"}
Sep 16 02:45:00.896032 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:45:00.895982    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-fe9d17-29825445-qjxl4" podStartSLOduration=0.895963572 podStartE2EDuration="895.963572ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 02:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 02:45:00.894681332 +0000 UTC m=+1579.362470124" watchObservedRunningTime="2026-09-16 02:45:00.895963572 +0000 UTC m=+1579.363752355"
Sep 16 02:45:00.908870 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:45:00.908830    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-4178a6-29825445-lwq6f" podStartSLOduration=0.908817597 podStartE2EDuration="908.817597ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 02:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 02:45:00.907227426 +0000 UTC m=+1579.375016209" watchObservedRunningTime="2026-09-16 02:45:00.908817597 +0000 UTC m=+1579.376606378"
Sep 16 02:45:00.920909 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:45:00.920864    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-db1ed7-29825445-tnkp6" podStartSLOduration=0.920847689 podStartE2EDuration="920.847689ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 02:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 02:45:00.918733433 +0000 UTC m=+1579.386522218" watchObservedRunningTime="2026-09-16 02:45:00.920847689 +0000 UTC m=+1579.388636471"
Sep 16 02:45:00.931746 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:45:00.931714    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-10a6b9-29825445-5n5s4" podStartSLOduration=0.931703137 podStartE2EDuration="931.703137ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 02:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 02:45:00.930760148 +0000 UTC m=+1579.398548933" watchObservedRunningTime="2026-09-16 02:45:00.931703137 +0000 UTC m=+1579.399491918"
Sep 16 02:45:01.888648 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:45:01.888613    2583 generic.go:356] "Generic (PLEG): container finished" podID="5f700448-b33c-43ec-ba1d-b850e3e7618a" containerID="ee71db41c3a3cc305eca954c8cff5227a4f531e66c279f742c95f65bf0890aff" exitCode=0
Sep 16 02:45:01.889119 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:45:01.888706    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-fe9d17-29825445-qjxl4" event={"ID":"5f700448-b33c-43ec-ba1d-b850e3e7618a","Type":"ContainerDied","Data":"ee71db41c3a3cc305eca954c8cff5227a4f531e66c279f742c95f65bf0890aff"}
Sep 16 02:45:01.892505 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:45:01.892475    2583 generic.go:356] "Generic (PLEG): container finished" podID="b8b8b76a-5f01-4a45-9750-4b5814c0eedc" containerID="33c0e7601aab0514fd6551cc5a457193b28cff2971bb7daad7d7bc6624eac362" exitCode=0
Sep 16 02:45:01.892653 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:45:01.892592    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-10a6b9-29825445-5n5s4" event={"ID":"b8b8b76a-5f01-4a45-9750-4b5814c0eedc","Type":"ContainerDied","Data":"33c0e7601aab0514fd6551cc5a457193b28cff2971bb7daad7d7bc6624eac362"}
Sep 16 02:45:01.895002 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:45:01.894974    2583 generic.go:356] "Generic (PLEG): container finished" podID="12bfa699-c47a-495e-8c11-5c0ff7927bf3" containerID="9cf60f4f256bbb7679ad9a591101fbfe00beb63f6d794d05254bc39b7ad516b2" exitCode=0
Sep 16 02:45:01.895125 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:45:01.895058    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-db1ed7-29825445-tnkp6" event={"ID":"12bfa699-c47a-495e-8c11-5c0ff7927bf3","Type":"ContainerDied","Data":"9cf60f4f256bbb7679ad9a591101fbfe00beb63f6d794d05254bc39b7ad516b2"}
Sep 16 02:45:01.896858 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:45:01.896834    2583 generic.go:356] "Generic (PLEG): container finished" podID="1890f865-a55e-435c-adfb-fee2d3754825" containerID="7a4714cc303cbbd7a3384553ddbc126ee3a6feffb10755d0a621c602abe1c72a" exitCode=0
Sep 16 02:45:01.896963 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:45:01.896907    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-4178a6-29825445-lwq6f" event={"ID":"1890f865-a55e-435c-adfb-fee2d3754825","Type":"ContainerDied","Data":"7a4714cc303cbbd7a3384553ddbc126ee3a6feffb10755d0a621c602abe1c72a"}
Sep 16 02:45:03.043039 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:45:03.043010    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-4178a6-29825445-lwq6f"
Sep 16 02:45:03.095565 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:45:03.095520    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-fe9d17-29825445-qjxl4"
Sep 16 02:45:03.128059 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:45:03.128028    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-db1ed7-29825445-tnkp6"
Sep 16 02:45:03.139722 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:45:03.139694    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-10a6b9-29825445-5n5s4"
Sep 16 02:45:03.917005 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:45:03.916976    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-fe9d17-29825445-qjxl4"
Sep 16 02:45:03.917208 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:45:03.917015    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-fe9d17-29825445-qjxl4" event={"ID":"5f700448-b33c-43ec-ba1d-b850e3e7618a","Type":"ContainerDied","Data":"d8eaeb56e74228f99176983a25b4af6075c7d63dcb01b86b29e06c3846b6ad00"}
Sep 16 02:45:03.917208 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:45:03.917089    2583 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d8eaeb56e74228f99176983a25b4af6075c7d63dcb01b86b29e06c3846b6ad00"
Sep 16 02:45:03.918452 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:45:03.918425    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-10a6b9-29825445-5n5s4"
Sep 16 02:45:03.918593 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:45:03.918415    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-10a6b9-29825445-5n5s4" event={"ID":"b8b8b76a-5f01-4a45-9750-4b5814c0eedc","Type":"ContainerDied","Data":"4bdbe081dfde3a25049352cd7db5bb7303e6da203893295bf0381478412358c1"}
Sep 16 02:45:03.918593 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:45:03.918567    2583 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4bdbe081dfde3a25049352cd7db5bb7303e6da203893295bf0381478412358c1"
Sep 16 02:45:03.919727 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:45:03.919706    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-db1ed7-29825445-tnkp6"
Sep 16 02:45:03.919855 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:45:03.919707    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-db1ed7-29825445-tnkp6" event={"ID":"12bfa699-c47a-495e-8c11-5c0ff7927bf3","Type":"ContainerDied","Data":"d259b4e4ffcdd428c2bf412ac58b53be05bb4c45bea21bb6fbab76fb9a11a214"}
Sep 16 02:45:03.919855 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:45:03.919817    2583 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d259b4e4ffcdd428c2bf412ac58b53be05bb4c45bea21bb6fbab76fb9a11a214"
Sep 16 02:45:03.920968 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:45:03.920950    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-4178a6-29825445-lwq6f"
Sep 16 02:45:03.921062 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:45:03.920949    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-4178a6-29825445-lwq6f" event={"ID":"1890f865-a55e-435c-adfb-fee2d3754825","Type":"ContainerDied","Data":"46d2b3351b149f6ded10297b950730b76b39519d4a882f0baff9b881466d190a"}
Sep 16 02:45:03.921062 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:45:03.921059    2583 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="46d2b3351b149f6ded10297b950730b76b39519d4a882f0baff9b881466d190a"
Sep 16 02:45:10.561641 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:45:10.561602    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-fzl8t"]
Sep 16 02:45:29.448198 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:45:29.448162    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-fzl8t"]
Sep 16 02:45:40.259790 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:45:40.259715    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-4178a6-7shm4"]
Sep 16 02:45:40.260138 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:45:40.260126    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5f700448-b33c-43ec-ba1d-b850e3e7618a" containerName="cleanup"
Sep 16 02:45:40.260138 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:45:40.260139    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="5f700448-b33c-43ec-ba1d-b850e3e7618a" containerName="cleanup"
Sep 16 02:45:40.260219 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:45:40.260149    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="12bfa699-c47a-495e-8c11-5c0ff7927bf3" containerName="cleanup"
Sep 16 02:45:40.260219 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:45:40.260155    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="12bfa699-c47a-495e-8c11-5c0ff7927bf3" containerName="cleanup"
Sep 16 02:45:40.260219 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:45:40.260163    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b8b8b76a-5f01-4a45-9750-4b5814c0eedc" containerName="cleanup"
Sep 16 02:45:40.260219 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:45:40.260168    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="b8b8b76a-5f01-4a45-9750-4b5814c0eedc" containerName="cleanup"
Sep 16 02:45:40.260219 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:45:40.260175    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1890f865-a55e-435c-adfb-fee2d3754825" containerName="cleanup"
Sep 16 02:45:40.260219 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:45:40.260180    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="1890f865-a55e-435c-adfb-fee2d3754825" containerName="cleanup"
Sep 16 02:45:40.260404 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:45:40.260238    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="5f700448-b33c-43ec-ba1d-b850e3e7618a" containerName="cleanup"
Sep 16 02:45:40.260404 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:45:40.260246    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="12bfa699-c47a-495e-8c11-5c0ff7927bf3" containerName="cleanup"
Sep 16 02:45:40.260404 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:45:40.260252    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="b8b8b76a-5f01-4a45-9750-4b5814c0eedc" containerName="cleanup"
Sep 16 02:45:40.260404 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:45:40.260259    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="1890f865-a55e-435c-adfb-fee2d3754825" containerName="cleanup"
Sep 16 02:45:40.264745 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:45:40.264728    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-4178a6-7shm4"
Sep 16 02:45:40.267362 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:45:40.267334    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-w8hnp\""
Sep 16 02:45:40.267620 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:45:40.267602    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 16 02:45:40.268958 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:45:40.268933    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-4178a6-7shm4"]
Sep 16 02:45:40.320561 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:45:40.320518    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/385d1b69-c49c-4613-89e0-faa8025b3dd7-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-a-4178a6-7shm4\" (UID: \"385d1b69-c49c-4613-89e0-faa8025b3dd7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-4178a6-7shm4"
Sep 16 02:45:40.421325 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:45:40.421282    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/385d1b69-c49c-4613-89e0-faa8025b3dd7-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-a-4178a6-7shm4\" (UID: \"385d1b69-c49c-4613-89e0-faa8025b3dd7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-4178a6-7shm4"
Sep 16 02:45:40.421953 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:45:40.421933    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/385d1b69-c49c-4613-89e0-faa8025b3dd7-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-a-4178a6-7shm4\" (UID: \"385d1b69-c49c-4613-89e0-faa8025b3dd7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-4178a6-7shm4"
Sep 16 02:45:40.576959 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:45:40.576868    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-4178a6-7shm4"
Sep 16 02:45:40.848263 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:45:40.848242    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-4178a6-7shm4"]
Sep 16 02:45:40.850377 ip-10-0-128-111 kubenswrapper[2583]: W0916 02:45:40.850349    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod385d1b69_c49c_4613_89e0_faa8025b3dd7.slice/crio-c7d5a16b7575fdc27affdbc25ed8a56740447d559736280f620591622a85000b WatchSource:0}: Error finding container c7d5a16b7575fdc27affdbc25ed8a56740447d559736280f620591622a85000b: Status 404 returned error can't find the container with id c7d5a16b7575fdc27affdbc25ed8a56740447d559736280f620591622a85000b
Sep 16 02:45:41.113063 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:45:41.112979    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-4178a6-7shm4" event={"ID":"385d1b69-c49c-4613-89e0-faa8025b3dd7","Type":"ContainerStarted","Data":"c7d5a16b7575fdc27affdbc25ed8a56740447d559736280f620591622a85000b"}
Sep 16 02:45:42.120129 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:45:42.120092    2583 generic.go:356] "Generic (PLEG): container finished" podID="385d1b69-c49c-4613-89e0-faa8025b3dd7" containerID="844ad7487aa5c0d5843e3bb2bbaf827b591e71344095b9518012ce5d7c2ce8d0" exitCode=0
Sep 16 02:45:42.120493 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:45:42.120142    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-4178a6-7shm4" event={"ID":"385d1b69-c49c-4613-89e0-faa8025b3dd7","Type":"ContainerDied","Data":"844ad7487aa5c0d5843e3bb2bbaf827b591e71344095b9518012ce5d7c2ce8d0"}
Sep 16 02:45:42.858199 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:45:42.858171    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-b-4fe974-5c5d79b89c-jfqrs_a18ebddf-5578-4752-852a-3dcfe8eb881b/payload-processing/0.log"
Sep 16 02:45:42.965891 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:45:42.965857    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-b-4fe974-5c5d79b89c-jfqrs_a18ebddf-5578-4752-852a-3dcfe8eb881b/payload-processing/0.log"
Sep 16 02:45:43.254288 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:45:43.254256    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-4178a6-7shm4"
Sep 16 02:45:43.349950 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:45:43.349919    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/385d1b69-c49c-4613-89e0-faa8025b3dd7-maas-api-service-ca\") pod \"385d1b69-c49c-4613-89e0-faa8025b3dd7\" (UID: \"385d1b69-c49c-4613-89e0-faa8025b3dd7\") "
Sep 16 02:45:43.350287 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:45:43.350263    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/385d1b69-c49c-4613-89e0-faa8025b3dd7-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "385d1b69-c49c-4613-89e0-faa8025b3dd7" (UID: "385d1b69-c49c-4613-89e0-faa8025b3dd7"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 02:45:43.450924 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:45:43.450838    2583 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/385d1b69-c49c-4613-89e0-faa8025b3dd7-maas-api-service-ca\") on node \"ip-10-0-128-111.ec2.internal\" DevicePath \"\""
Sep 16 02:45:43.480010 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:45:43.479973    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-342933-openshift-default-66f6f7b544-c5hzx"]
Sep 16 02:45:43.480471 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:45:43.480452    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="385d1b69-c49c-4613-89e0-faa8025b3dd7" containerName="revoke-keys"
Sep 16 02:45:43.480592 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:45:43.480474    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="385d1b69-c49c-4613-89e0-faa8025b3dd7" containerName="revoke-keys"
Sep 16 02:45:43.480662 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:45:43.480594    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="385d1b69-c49c-4613-89e0-faa8025b3dd7" containerName="revoke-keys"
Sep 16 02:45:43.483751 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:45:43.483729    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-cleanup-342933-openshift-default-66f6f7b544-c5hzx"
Sep 16 02:45:43.486241 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:45:43.486220    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-cleanup-342933-openshift-default-dockercfg-f6rz8\""
Sep 16 02:45:43.496620 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:45:43.496590    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-342933-openshift-default-66f6f7b544-c5hzx"]
Sep 16 02:45:43.551750 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:45:43.551716    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e2ae0cd7-7d55-4dc1-8ea2-1eb9020290dd-istio-data\") pod \"e2e-ipp-cleanup-342933-openshift-default-66f6f7b544-c5hzx\" (UID: \"e2ae0cd7-7d55-4dc1-8ea2-1eb9020290dd\") " pod="openshift-ingress/e2e-ipp-cleanup-342933-openshift-default-66f6f7b544-c5hzx"
Sep 16 02:45:43.551944 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:45:43.551761    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9mqdg\" (UniqueName: \"kubernetes.io/projected/e2ae0cd7-7d55-4dc1-8ea2-1eb9020290dd-kube-api-access-9mqdg\") pod \"e2e-ipp-cleanup-342933-openshift-default-66f6f7b544-c5hzx\" (UID: \"e2ae0cd7-7d55-4dc1-8ea2-1eb9020290dd\") " pod="openshift-ingress/e2e-ipp-cleanup-342933-openshift-default-66f6f7b544-c5hzx"
Sep 16 02:45:43.551944 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:45:43.551840    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e2ae0cd7-7d55-4dc1-8ea2-1eb9020290dd-workload-socket\") pod \"e2e-ipp-cleanup-342933-openshift-default-66f6f7b544-c5hzx\" (UID: \"e2ae0cd7-7d55-4dc1-8ea2-1eb9020290dd\") " pod="openshift-ingress/e2e-ipp-cleanup-342933-openshift-default-66f6f7b544-c5hzx"
Sep 16 02:45:43.551944 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:45:43.551883    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e2ae0cd7-7d55-4dc1-8ea2-1eb9020290dd-istio-token\") pod \"e2e-ipp-cleanup-342933-openshift-default-66f6f7b544-c5hzx\" (UID: \"e2ae0cd7-7d55-4dc1-8ea2-1eb9020290dd\") " pod="openshift-ingress/e2e-ipp-cleanup-342933-openshift-default-66f6f7b544-c5hzx"
Sep 16 02:45:43.551944 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:45:43.551934    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e2ae0cd7-7d55-4dc1-8ea2-1eb9020290dd-credential-socket\") pod \"e2e-ipp-cleanup-342933-openshift-default-66f6f7b544-c5hzx\" (UID: \"e2ae0cd7-7d55-4dc1-8ea2-1eb9020290dd\") " pod="openshift-ingress/e2e-ipp-cleanup-342933-openshift-default-66f6f7b544-c5hzx"
Sep 16 02:45:43.552117 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:45:43.551992    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e2ae0cd7-7d55-4dc1-8ea2-1eb9020290dd-workload-certs\") pod \"e2e-ipp-cleanup-342933-openshift-default-66f6f7b544-c5hzx\" (UID: \"e2ae0cd7-7d55-4dc1-8ea2-1eb9020290dd\") " pod="openshift-ingress/e2e-ipp-cleanup-342933-openshift-default-66f6f7b544-c5hzx"
Sep 16 02:45:43.552117 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:45:43.552040    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e2ae0cd7-7d55-4dc1-8ea2-1eb9020290dd-istiod-ca-cert\") pod \"e2e-ipp-cleanup-342933-openshift-default-66f6f7b544-c5hzx\" (UID: \"e2ae0cd7-7d55-4dc1-8ea2-1eb9020290dd\") " pod="openshift-ingress/e2e-ipp-cleanup-342933-openshift-default-66f6f7b544-c5hzx"
Sep 16 02:45:43.552117 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:45:43.552085    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e2ae0cd7-7d55-4dc1-8ea2-1eb9020290dd-istio-podinfo\") pod \"e2e-ipp-cleanup-342933-openshift-default-66f6f7b544-c5hzx\" (UID: \"e2ae0cd7-7d55-4dc1-8ea2-1eb9020290dd\") " pod="openshift-ingress/e2e-ipp-cleanup-342933-openshift-default-66f6f7b544-c5hzx"
Sep 16 02:45:43.552220 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:45:43.552143    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e2ae0cd7-7d55-4dc1-8ea2-1eb9020290dd-istio-envoy\") pod \"e2e-ipp-cleanup-342933-openshift-default-66f6f7b544-c5hzx\" (UID: \"e2ae0cd7-7d55-4dc1-8ea2-1eb9020290dd\") " pod="openshift-ingress/e2e-ipp-cleanup-342933-openshift-default-66f6f7b544-c5hzx"
Sep 16 02:45:43.653032 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:45:43.652991    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e2ae0cd7-7d55-4dc1-8ea2-1eb9020290dd-istio-data\") pod \"e2e-ipp-cleanup-342933-openshift-default-66f6f7b544-c5hzx\" (UID: \"e2ae0cd7-7d55-4dc1-8ea2-1eb9020290dd\") " pod="openshift-ingress/e2e-ipp-cleanup-342933-openshift-default-66f6f7b544-c5hzx"
Sep 16 02:45:43.653238 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:45:43.653044    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9mqdg\" (UniqueName: \"kubernetes.io/projected/e2ae0cd7-7d55-4dc1-8ea2-1eb9020290dd-kube-api-access-9mqdg\") pod \"e2e-ipp-cleanup-342933-openshift-default-66f6f7b544-c5hzx\" (UID: \"e2ae0cd7-7d55-4dc1-8ea2-1eb9020290dd\") " pod="openshift-ingress/e2e-ipp-cleanup-342933-openshift-default-66f6f7b544-c5hzx"
Sep 16 02:45:43.653238 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:45:43.653075    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e2ae0cd7-7d55-4dc1-8ea2-1eb9020290dd-workload-socket\") pod \"e2e-ipp-cleanup-342933-openshift-default-66f6f7b544-c5hzx\" (UID: \"e2ae0cd7-7d55-4dc1-8ea2-1eb9020290dd\") " pod="openshift-ingress/e2e-ipp-cleanup-342933-openshift-default-66f6f7b544-c5hzx"
Sep 16 02:45:43.653238 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:45:43.653099    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e2ae0cd7-7d55-4dc1-8ea2-1eb9020290dd-istio-token\") pod \"e2e-ipp-cleanup-342933-openshift-default-66f6f7b544-c5hzx\" (UID: \"e2ae0cd7-7d55-4dc1-8ea2-1eb9020290dd\") " pod="openshift-ingress/e2e-ipp-cleanup-342933-openshift-default-66f6f7b544-c5hzx"
Sep 16 02:45:43.653238 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:45:43.653129    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e2ae0cd7-7d55-4dc1-8ea2-1eb9020290dd-credential-socket\") pod \"e2e-ipp-cleanup-342933-openshift-default-66f6f7b544-c5hzx\" (UID: \"e2ae0cd7-7d55-4dc1-8ea2-1eb9020290dd\") " pod="openshift-ingress/e2e-ipp-cleanup-342933-openshift-default-66f6f7b544-c5hzx"
Sep 16 02:45:43.653238 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:45:43.653171    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e2ae0cd7-7d55-4dc1-8ea2-1eb9020290dd-workload-certs\") pod \"e2e-ipp-cleanup-342933-openshift-default-66f6f7b544-c5hzx\" (UID: \"e2ae0cd7-7d55-4dc1-8ea2-1eb9020290dd\") " pod="openshift-ingress/e2e-ipp-cleanup-342933-openshift-default-66f6f7b544-c5hzx"
Sep 16 02:45:43.653238 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:45:43.653197    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e2ae0cd7-7d55-4dc1-8ea2-1eb9020290dd-istiod-ca-cert\") pod \"e2e-ipp-cleanup-342933-openshift-default-66f6f7b544-c5hzx\" (UID: \"e2ae0cd7-7d55-4dc1-8ea2-1eb9020290dd\") " pod="openshift-ingress/e2e-ipp-cleanup-342933-openshift-default-66f6f7b544-c5hzx"
Sep 16 02:45:43.653629 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:45:43.653248    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e2ae0cd7-7d55-4dc1-8ea2-1eb9020290dd-istio-podinfo\") pod \"e2e-ipp-cleanup-342933-openshift-default-66f6f7b544-c5hzx\" (UID: \"e2ae0cd7-7d55-4dc1-8ea2-1eb9020290dd\") " pod="openshift-ingress/e2e-ipp-cleanup-342933-openshift-default-66f6f7b544-c5hzx"
Sep 16 02:45:43.653629 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:45:43.653308    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e2ae0cd7-7d55-4dc1-8ea2-1eb9020290dd-istio-envoy\") pod \"e2e-ipp-cleanup-342933-openshift-default-66f6f7b544-c5hzx\" (UID: \"e2ae0cd7-7d55-4dc1-8ea2-1eb9020290dd\") " pod="openshift-ingress/e2e-ipp-cleanup-342933-openshift-default-66f6f7b544-c5hzx"
Sep 16 02:45:43.653629 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:45:43.653458    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e2ae0cd7-7d55-4dc1-8ea2-1eb9020290dd-istio-data\") pod \"e2e-ipp-cleanup-342933-openshift-default-66f6f7b544-c5hzx\" (UID: \"e2ae0cd7-7d55-4dc1-8ea2-1eb9020290dd\") " pod="openshift-ingress/e2e-ipp-cleanup-342933-openshift-default-66f6f7b544-c5hzx"
Sep 16 02:45:43.653629 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:45:43.653517    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e2ae0cd7-7d55-4dc1-8ea2-1eb9020290dd-workload-socket\") pod \"e2e-ipp-cleanup-342933-openshift-default-66f6f7b544-c5hzx\" (UID: \"e2ae0cd7-7d55-4dc1-8ea2-1eb9020290dd\") " pod="openshift-ingress/e2e-ipp-cleanup-342933-openshift-default-66f6f7b544-c5hzx"
Sep 16 02:45:43.653849 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:45:43.653726    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e2ae0cd7-7d55-4dc1-8ea2-1eb9020290dd-credential-socket\") pod \"e2e-ipp-cleanup-342933-openshift-default-66f6f7b544-c5hzx\" (UID: \"e2ae0cd7-7d55-4dc1-8ea2-1eb9020290dd\") " pod="openshift-ingress/e2e-ipp-cleanup-342933-openshift-default-66f6f7b544-c5hzx"
Sep 16 02:45:43.653849 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:45:43.653808    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e2ae0cd7-7d55-4dc1-8ea2-1eb9020290dd-workload-certs\") pod \"e2e-ipp-cleanup-342933-openshift-default-66f6f7b544-c5hzx\" (UID: \"e2ae0cd7-7d55-4dc1-8ea2-1eb9020290dd\") " pod="openshift-ingress/e2e-ipp-cleanup-342933-openshift-default-66f6f7b544-c5hzx"
Sep 16 02:45:43.654303 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:45:43.654278    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e2ae0cd7-7d55-4dc1-8ea2-1eb9020290dd-istiod-ca-cert\") pod \"e2e-ipp-cleanup-342933-openshift-default-66f6f7b544-c5hzx\" (UID: \"e2ae0cd7-7d55-4dc1-8ea2-1eb9020290dd\") " pod="openshift-ingress/e2e-ipp-cleanup-342933-openshift-default-66f6f7b544-c5hzx"
Sep 16 02:45:43.656117 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:45:43.656093    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e2ae0cd7-7d55-4dc1-8ea2-1eb9020290dd-istio-envoy\") pod \"e2e-ipp-cleanup-342933-openshift-default-66f6f7b544-c5hzx\" (UID: \"e2ae0cd7-7d55-4dc1-8ea2-1eb9020290dd\") " pod="openshift-ingress/e2e-ipp-cleanup-342933-openshift-default-66f6f7b544-c5hzx"
Sep 16 02:45:43.656330 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:45:43.656310    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e2ae0cd7-7d55-4dc1-8ea2-1eb9020290dd-istio-podinfo\") pod \"e2e-ipp-cleanup-342933-openshift-default-66f6f7b544-c5hzx\" (UID: \"e2ae0cd7-7d55-4dc1-8ea2-1eb9020290dd\") " pod="openshift-ingress/e2e-ipp-cleanup-342933-openshift-default-66f6f7b544-c5hzx"
Sep 16 02:45:43.661210 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:45:43.661188    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e2ae0cd7-7d55-4dc1-8ea2-1eb9020290dd-istio-token\") pod \"e2e-ipp-cleanup-342933-openshift-default-66f6f7b544-c5hzx\" (UID: \"e2ae0cd7-7d55-4dc1-8ea2-1eb9020290dd\") " pod="openshift-ingress/e2e-ipp-cleanup-342933-openshift-default-66f6f7b544-c5hzx"
Sep 16 02:45:43.661799 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:45:43.661778    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9mqdg\" (UniqueName: \"kubernetes.io/projected/e2ae0cd7-7d55-4dc1-8ea2-1eb9020290dd-kube-api-access-9mqdg\") pod \"e2e-ipp-cleanup-342933-openshift-default-66f6f7b544-c5hzx\" (UID: \"e2ae0cd7-7d55-4dc1-8ea2-1eb9020290dd\") " pod="openshift-ingress/e2e-ipp-cleanup-342933-openshift-default-66f6f7b544-c5hzx"
Sep 16 02:45:43.803692 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:45:43.803611    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-cleanup-342933-openshift-default-66f6f7b544-c5hzx"
Sep 16 02:45:44.079344 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:45:44.079319    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-342933-openshift-default-66f6f7b544-c5hzx"]
Sep 16 02:45:44.080610 ip-10-0-128-111 kubenswrapper[2583]: W0916 02:45:44.080580    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode2ae0cd7_7d55_4dc1_8ea2_1eb9020290dd.slice/crio-16faa3a3f7417e728c53f5d3cc1ce1a55aa1f26f56a61390fd6c7c7f6362da41 WatchSource:0}: Error finding container 16faa3a3f7417e728c53f5d3cc1ce1a55aa1f26f56a61390fd6c7c7f6362da41: Status 404 returned error can't find the container with id 16faa3a3f7417e728c53f5d3cc1ce1a55aa1f26f56a61390fd6c7c7f6362da41
Sep 16 02:45:44.082994 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:45:44.082963    2583 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 16 02:45:44.083083 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:45:44.083025    2583 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 16 02:45:44.083083 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:45:44.083054    2583 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 16 02:45:44.131717 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:45:44.131695    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-4178a6-7shm4"
Sep 16 02:45:44.131833 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:45:44.131725    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-4178a6-7shm4" event={"ID":"385d1b69-c49c-4613-89e0-faa8025b3dd7","Type":"ContainerDied","Data":"c7d5a16b7575fdc27affdbc25ed8a56740447d559736280f620591622a85000b"}
Sep 16 02:45:44.131833 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:45:44.131782    2583 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c7d5a16b7575fdc27affdbc25ed8a56740447d559736280f620591622a85000b"
Sep 16 02:45:44.133041 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:45:44.133019    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-342933-openshift-default-66f6f7b544-c5hzx" event={"ID":"e2ae0cd7-7d55-4dc1-8ea2-1eb9020290dd","Type":"ContainerStarted","Data":"16faa3a3f7417e728c53f5d3cc1ce1a55aa1f26f56a61390fd6c7c7f6362da41"}
Sep 16 02:45:44.346087 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:45:44.346051    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-fzl8t"]
Sep 16 02:45:45.139545 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:45:45.139507    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-342933-openshift-default-66f6f7b544-c5hzx" event={"ID":"e2ae0cd7-7d55-4dc1-8ea2-1eb9020290dd","Type":"ContainerStarted","Data":"14de09c0f6ff121463cb92a82976c4fb156a33dc8d66672fd3627e71f0ad77d8"}
Sep 16 02:45:45.159447 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:45:45.159395    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-cleanup-342933-openshift-default-66f6f7b544-c5hzx" podStartSLOduration=2.159380198 podStartE2EDuration="2.159380198s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 02:45:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 02:45:45.156674487 +0000 UTC m=+1623.624463270" watchObservedRunningTime="2026-09-16 02:45:45.159380198 +0000 UTC m=+1623.627169036"
Sep 16 02:45:45.804178 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:45:45.804145    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-cleanup-342933-openshift-default-66f6f7b544-c5hzx"
Sep 16 02:45:45.809488 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:45:45.809465    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-cleanup-342933-openshift-default-66f6f7b544-c5hzx"
Sep 16 02:45:46.145036 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:45:46.145008    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-cleanup-342933-openshift-default-66f6f7b544-c5hzx"
Sep 16 02:45:46.145966 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:45:46.145941    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-cleanup-342933-openshift-default-66f6f7b544-c5hzx"
Sep 16 02:45:52.690434 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:45:52.690402    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-342933-5884dfdb95-npwr9"]
Sep 16 02:45:52.694089 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:45:52.694068    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-342933-5884dfdb95-npwr9"
Sep 16 02:45:52.696641 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:45:52.696612    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-cleanup-342933\""
Sep 16 02:45:52.696961 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:45:52.696942    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-cleanup-342933-dockercfg-lmrdb\""
Sep 16 02:45:52.701366 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:45:52.701344    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-342933-5884dfdb95-npwr9"]
Sep 16 02:45:52.843794 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:45:52.843755    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a5df25a6-8c74-484a-b32c-974d9b321fc6-plugins-config-volume\") pod \"payload-processing-e2e-ipp-cleanup-342933-5884dfdb95-npwr9\" (UID: \"a5df25a6-8c74-484a-b32c-974d9b321fc6\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-342933-5884dfdb95-npwr9"
Sep 16 02:45:52.843998 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:45:52.843974    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-569j8\" (UniqueName: \"kubernetes.io/projected/a5df25a6-8c74-484a-b32c-974d9b321fc6-kube-api-access-569j8\") pod \"payload-processing-e2e-ipp-cleanup-342933-5884dfdb95-npwr9\" (UID: \"a5df25a6-8c74-484a-b32c-974d9b321fc6\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-342933-5884dfdb95-npwr9"
Sep 16 02:45:52.945154 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:45:52.945044    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-569j8\" (UniqueName: \"kubernetes.io/projected/a5df25a6-8c74-484a-b32c-974d9b321fc6-kube-api-access-569j8\") pod \"payload-processing-e2e-ipp-cleanup-342933-5884dfdb95-npwr9\" (UID: \"a5df25a6-8c74-484a-b32c-974d9b321fc6\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-342933-5884dfdb95-npwr9"
Sep 16 02:45:52.945154 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:45:52.945123    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a5df25a6-8c74-484a-b32c-974d9b321fc6-plugins-config-volume\") pod \"payload-processing-e2e-ipp-cleanup-342933-5884dfdb95-npwr9\" (UID: \"a5df25a6-8c74-484a-b32c-974d9b321fc6\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-342933-5884dfdb95-npwr9"
Sep 16 02:45:52.945888 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:45:52.945866    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a5df25a6-8c74-484a-b32c-974d9b321fc6-plugins-config-volume\") pod \"payload-processing-e2e-ipp-cleanup-342933-5884dfdb95-npwr9\" (UID: \"a5df25a6-8c74-484a-b32c-974d9b321fc6\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-342933-5884dfdb95-npwr9"
Sep 16 02:45:52.952658 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:45:52.952633    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-569j8\" (UniqueName: \"kubernetes.io/projected/a5df25a6-8c74-484a-b32c-974d9b321fc6-kube-api-access-569j8\") pod \"payload-processing-e2e-ipp-cleanup-342933-5884dfdb95-npwr9\" (UID: \"a5df25a6-8c74-484a-b32c-974d9b321fc6\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-342933-5884dfdb95-npwr9"
Sep 16 02:45:53.006216 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:45:53.006184    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-342933-5884dfdb95-npwr9"
Sep 16 02:45:53.276096 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:45:53.276072    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-342933-5884dfdb95-npwr9"]
Sep 16 02:45:53.277988 ip-10-0-128-111 kubenswrapper[2583]: W0916 02:45:53.277958    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda5df25a6_8c74_484a_b32c_974d9b321fc6.slice/crio-0cc82a564556d89a8e4f8952865ba4ab5882cd3919dd26bc715b6be63633b874 WatchSource:0}: Error finding container 0cc82a564556d89a8e4f8952865ba4ab5882cd3919dd26bc715b6be63633b874: Status 404 returned error can't find the container with id 0cc82a564556d89a8e4f8952865ba4ab5882cd3919dd26bc715b6be63633b874
Sep 16 02:45:54.186367 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:45:54.186334    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-342933-5884dfdb95-npwr9" event={"ID":"a5df25a6-8c74-484a-b32c-974d9b321fc6","Type":"ContainerStarted","Data":"a1eee1d8e5bafa5e197c920dfcb3b92b2e227d5336c95613d364ccb411ffb110"}
Sep 16 02:45:54.186757 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:45:54.186385    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-342933-5884dfdb95-npwr9" event={"ID":"a5df25a6-8c74-484a-b32c-974d9b321fc6","Type":"ContainerStarted","Data":"0cc82a564556d89a8e4f8952865ba4ab5882cd3919dd26bc715b6be63633b874"}
Sep 16 02:45:54.186757 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:45:54.186533    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-342933-5884dfdb95-npwr9"
Sep 16 02:45:54.205411 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:45:54.205358    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-342933-5884dfdb95-npwr9" podStartSLOduration=2.205336703 podStartE2EDuration="2.205336703s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 02:45:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 02:45:54.202150538 +0000 UTC m=+1632.669939320" watchObservedRunningTime="2026-09-16 02:45:54.205336703 +0000 UTC m=+1632.673125489"
Sep 16 02:45:58.136394 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:45:58.136356    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-4178a6-29825445-lwq6f"]
Sep 16 02:45:58.138794 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:45:58.138765    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-4178a6-29825445-lwq6f"]
Sep 16 02:45:58.164043 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:45:58.164006    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-4178a6-67c58859d5-mc9tp"]
Sep 16 02:45:58.164274 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:45:58.164246    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-a-4178a6-67c58859d5-mc9tp" podUID="4bb17a92-fec6-4725-aca1-dd044f679ed0" containerName="payload-processing" containerID="cri-o://d54243f590268e4f89ef06b0ec994fde28aedddc1b73761359b132b9908c5104" gracePeriod=30
Sep 16 02:45:58.169819 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:45:58.169792    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-4178a6-66b5858d78-fdlkw"]
Sep 16 02:45:58.170481 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:45:58.169980    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-a-4178a6-66b5858d78-fdlkw" podUID="934c93b5-25af-450b-8be2-1cffaae7fb9a" containerName="payload-pre-processing" containerID="cri-o://1016d63b6a8dd721a0f79b8037be6b4905b45da03b16a29e53f84161f9c084fd" gracePeriod=30
Sep 16 02:45:58.304688 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:45:58.304648    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-4178a6-7shm4"]
Sep 16 02:45:58.311514 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:45:58.311480    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-4178a6-7shm4"]
Sep 16 02:45:58.450001 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:45:58.449972    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-4178a6-67c58859d5-mc9tp"
Sep 16 02:45:58.471936 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:45:58.471914    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-4178a6-66b5858d78-fdlkw"
Sep 16 02:45:58.607241 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:45:58.607208    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7w7cs\" (UniqueName: \"kubernetes.io/projected/4bb17a92-fec6-4725-aca1-dd044f679ed0-kube-api-access-7w7cs\") pod \"4bb17a92-fec6-4725-aca1-dd044f679ed0\" (UID: \"4bb17a92-fec6-4725-aca1-dd044f679ed0\") "
Sep 16 02:45:58.607405 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:45:58.607331    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8ljxp\" (UniqueName: \"kubernetes.io/projected/934c93b5-25af-450b-8be2-1cffaae7fb9a-kube-api-access-8ljxp\") pod \"934c93b5-25af-450b-8be2-1cffaae7fb9a\" (UID: \"934c93b5-25af-450b-8be2-1cffaae7fb9a\") "
Sep 16 02:45:58.607405 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:45:58.607364    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/934c93b5-25af-450b-8be2-1cffaae7fb9a-plugins-config-volume\") pod \"934c93b5-25af-450b-8be2-1cffaae7fb9a\" (UID: \"934c93b5-25af-450b-8be2-1cffaae7fb9a\") "
Sep 16 02:45:58.607405 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:45:58.607390    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4bb17a92-fec6-4725-aca1-dd044f679ed0-plugins-config-volume\") pod \"4bb17a92-fec6-4725-aca1-dd044f679ed0\" (UID: \"4bb17a92-fec6-4725-aca1-dd044f679ed0\") "
Sep 16 02:45:58.607808 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:45:58.607788    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/934c93b5-25af-450b-8be2-1cffaae7fb9a-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "934c93b5-25af-450b-8be2-1cffaae7fb9a" (UID: "934c93b5-25af-450b-8be2-1cffaae7fb9a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 02:45:58.607896 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:45:58.607827    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4bb17a92-fec6-4725-aca1-dd044f679ed0-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "4bb17a92-fec6-4725-aca1-dd044f679ed0" (UID: "4bb17a92-fec6-4725-aca1-dd044f679ed0"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 02:45:58.609281 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:45:58.609260    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4bb17a92-fec6-4725-aca1-dd044f679ed0-kube-api-access-7w7cs" (OuterVolumeSpecName: "kube-api-access-7w7cs") pod "4bb17a92-fec6-4725-aca1-dd044f679ed0" (UID: "4bb17a92-fec6-4725-aca1-dd044f679ed0"). InnerVolumeSpecName "kube-api-access-7w7cs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 02:45:58.609445 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:45:58.609426    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/934c93b5-25af-450b-8be2-1cffaae7fb9a-kube-api-access-8ljxp" (OuterVolumeSpecName: "kube-api-access-8ljxp") pod "934c93b5-25af-450b-8be2-1cffaae7fb9a" (UID: "934c93b5-25af-450b-8be2-1cffaae7fb9a"). InnerVolumeSpecName "kube-api-access-8ljxp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 02:45:58.708868 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:45:58.708791    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8ljxp\" (UniqueName: \"kubernetes.io/projected/934c93b5-25af-450b-8be2-1cffaae7fb9a-kube-api-access-8ljxp\") on node \"ip-10-0-128-111.ec2.internal\" DevicePath \"\""
Sep 16 02:45:58.708868 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:45:58.708817    2583 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/934c93b5-25af-450b-8be2-1cffaae7fb9a-plugins-config-volume\") on node \"ip-10-0-128-111.ec2.internal\" DevicePath \"\""
Sep 16 02:45:58.708868 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:45:58.708827    2583 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4bb17a92-fec6-4725-aca1-dd044f679ed0-plugins-config-volume\") on node \"ip-10-0-128-111.ec2.internal\" DevicePath \"\""
Sep 16 02:45:58.708868 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:45:58.708836    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7w7cs\" (UniqueName: \"kubernetes.io/projected/4bb17a92-fec6-4725-aca1-dd044f679ed0-kube-api-access-7w7cs\") on node \"ip-10-0-128-111.ec2.internal\" DevicePath \"\""
Sep 16 02:45:59.221885 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:45:59.221853    2583 generic.go:356] "Generic (PLEG): container finished" podID="4bb17a92-fec6-4725-aca1-dd044f679ed0" containerID="d54243f590268e4f89ef06b0ec994fde28aedddc1b73761359b132b9908c5104" exitCode=0
Sep 16 02:45:59.222335 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:45:59.221895    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-4178a6-67c58859d5-mc9tp"
Sep 16 02:45:59.222335 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:45:59.221936    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-4178a6-67c58859d5-mc9tp" event={"ID":"4bb17a92-fec6-4725-aca1-dd044f679ed0","Type":"ContainerDied","Data":"d54243f590268e4f89ef06b0ec994fde28aedddc1b73761359b132b9908c5104"}
Sep 16 02:45:59.222335 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:45:59.221988    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-4178a6-67c58859d5-mc9tp" event={"ID":"4bb17a92-fec6-4725-aca1-dd044f679ed0","Type":"ContainerDied","Data":"7edda015ddebbbc605ea6d6d378575fb07ffcd93881527ad3884a6817a9d56b9"}
Sep 16 02:45:59.222335 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:45:59.222010    2583 scope.go:117] "RemoveContainer" containerID="d54243f590268e4f89ef06b0ec994fde28aedddc1b73761359b132b9908c5104"
Sep 16 02:45:59.223388 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:45:59.223281    2583 generic.go:356] "Generic (PLEG): container finished" podID="934c93b5-25af-450b-8be2-1cffaae7fb9a" containerID="1016d63b6a8dd721a0f79b8037be6b4905b45da03b16a29e53f84161f9c084fd" exitCode=0
Sep 16 02:45:59.223388 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:45:59.223332    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-4178a6-66b5858d78-fdlkw"
Sep 16 02:45:59.223388 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:45:59.223369    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-4178a6-66b5858d78-fdlkw" event={"ID":"934c93b5-25af-450b-8be2-1cffaae7fb9a","Type":"ContainerDied","Data":"1016d63b6a8dd721a0f79b8037be6b4905b45da03b16a29e53f84161f9c084fd"}
Sep 16 02:45:59.223522 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:45:59.223421    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-4178a6-66b5858d78-fdlkw" event={"ID":"934c93b5-25af-450b-8be2-1cffaae7fb9a","Type":"ContainerDied","Data":"c4551338c79eb3bf47a86d0f497ed7773f61331a9b58ffca5f3ff0be72b442e1"}
Sep 16 02:45:59.233503 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:45:59.233487    2583 scope.go:117] "RemoveContainer" containerID="d54243f590268e4f89ef06b0ec994fde28aedddc1b73761359b132b9908c5104"
Sep 16 02:45:59.233855 ip-10-0-128-111 kubenswrapper[2583]: E0916 02:45:59.233829    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d54243f590268e4f89ef06b0ec994fde28aedddc1b73761359b132b9908c5104\": container with ID starting with d54243f590268e4f89ef06b0ec994fde28aedddc1b73761359b132b9908c5104 not found: ID does not exist" containerID="d54243f590268e4f89ef06b0ec994fde28aedddc1b73761359b132b9908c5104"
Sep 16 02:45:59.233964 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:45:59.233867    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d54243f590268e4f89ef06b0ec994fde28aedddc1b73761359b132b9908c5104"} err="failed to get container status \"d54243f590268e4f89ef06b0ec994fde28aedddc1b73761359b132b9908c5104\": rpc error: code = NotFound desc = could not find container \"d54243f590268e4f89ef06b0ec994fde28aedddc1b73761359b132b9908c5104\": container with ID starting with d54243f590268e4f89ef06b0ec994fde28aedddc1b73761359b132b9908c5104 not found: ID does not exist"
Sep 16 02:45:59.233964 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:45:59.233891    2583 scope.go:117] "RemoveContainer" containerID="1016d63b6a8dd721a0f79b8037be6b4905b45da03b16a29e53f84161f9c084fd"
Sep 16 02:45:59.242278 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:45:59.242262    2583 scope.go:117] "RemoveContainer" containerID="1016d63b6a8dd721a0f79b8037be6b4905b45da03b16a29e53f84161f9c084fd"
Sep 16 02:45:59.242514 ip-10-0-128-111 kubenswrapper[2583]: E0916 02:45:59.242496    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1016d63b6a8dd721a0f79b8037be6b4905b45da03b16a29e53f84161f9c084fd\": container with ID starting with 1016d63b6a8dd721a0f79b8037be6b4905b45da03b16a29e53f84161f9c084fd not found: ID does not exist" containerID="1016d63b6a8dd721a0f79b8037be6b4905b45da03b16a29e53f84161f9c084fd"
Sep 16 02:45:59.242630 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:45:59.242520    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1016d63b6a8dd721a0f79b8037be6b4905b45da03b16a29e53f84161f9c084fd"} err="failed to get container status \"1016d63b6a8dd721a0f79b8037be6b4905b45da03b16a29e53f84161f9c084fd\": rpc error: code = NotFound desc = could not find container \"1016d63b6a8dd721a0f79b8037be6b4905b45da03b16a29e53f84161f9c084fd\": container with ID starting with 1016d63b6a8dd721a0f79b8037be6b4905b45da03b16a29e53f84161f9c084fd not found: ID does not exist"
Sep 16 02:45:59.244298 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:45:59.244276    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-4178a6-67c58859d5-mc9tp"]
Sep 16 02:45:59.247309 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:45:59.247292    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-4178a6-67c58859d5-mc9tp"]
Sep 16 02:45:59.257012 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:45:59.256992    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-4178a6-66b5858d78-fdlkw"]
Sep 16 02:45:59.260453 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:45:59.260434    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-4178a6-66b5858d78-fdlkw"]
Sep 16 02:45:59.851059 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:45:59.851022    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-fzl8t"]
Sep 16 02:46:00.045637 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:46:00.045606    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1890f865-a55e-435c-adfb-fee2d3754825" path="/var/lib/kubelet/pods/1890f865-a55e-435c-adfb-fee2d3754825/volumes"
Sep 16 02:46:00.045945 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:46:00.045932    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="385d1b69-c49c-4613-89e0-faa8025b3dd7" path="/var/lib/kubelet/pods/385d1b69-c49c-4613-89e0-faa8025b3dd7/volumes"
Sep 16 02:46:00.046283 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:46:00.046270    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4bb17a92-fec6-4725-aca1-dd044f679ed0" path="/var/lib/kubelet/pods/4bb17a92-fec6-4725-aca1-dd044f679ed0/volumes"
Sep 16 02:46:00.046630 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:46:00.046604    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="934c93b5-25af-450b-8be2-1cffaae7fb9a" path="/var/lib/kubelet/pods/934c93b5-25af-450b-8be2-1cffaae7fb9a/volumes"
Sep 16 02:46:04.653569 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:46:04.653518    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-4178a6-openshift-default-596589786-tm9nc"]
Sep 16 02:46:04.654072 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:46:04.653764    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-4178a6-openshift-default-596589786-tm9nc" podUID="a7e3b05a-b2e6-4a33-b102-c7c98550b985" containerName="istio-proxy" containerID="cri-o://7e453e0c33b890e8afe37bdf6255c778f886d0c29b8abaea032652fb02e9b2d1" gracePeriod=30
Sep 16 02:46:04.979347 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:46:04.979270    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-1a50f4-stg4v"]
Sep 16 02:46:04.979733 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:46:04.979718    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="934c93b5-25af-450b-8be2-1cffaae7fb9a" containerName="payload-pre-processing"
Sep 16 02:46:04.979733 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:46:04.979734    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="934c93b5-25af-450b-8be2-1cffaae7fb9a" containerName="payload-pre-processing"
Sep 16 02:46:04.979822 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:46:04.979743    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4bb17a92-fec6-4725-aca1-dd044f679ed0" containerName="payload-processing"
Sep 16 02:46:04.979822 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:46:04.979750    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="4bb17a92-fec6-4725-aca1-dd044f679ed0" containerName="payload-processing"
Sep 16 02:46:04.979822 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:46:04.979808    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="934c93b5-25af-450b-8be2-1cffaae7fb9a" containerName="payload-pre-processing"
Sep 16 02:46:04.979822 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:46:04.979816    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="4bb17a92-fec6-4725-aca1-dd044f679ed0" containerName="payload-processing"
Sep 16 02:46:04.984383 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:46:04.984361    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-1a50f4-stg4v"
Sep 16 02:46:04.986894 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:46:04.986868    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 16 02:46:04.987051 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:46:04.986880    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-w8hnp\""
Sep 16 02:46:04.989004 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:46:04.988974    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-1a50f4-stg4v"]
Sep 16 02:46:05.071944 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:46:05.071909    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/81b1434e-0098-4ff6-b603-7fc0d04dc0b0-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-b-1a50f4-stg4v\" (UID: \"81b1434e-0098-4ff6-b603-7fc0d04dc0b0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-1a50f4-stg4v"
Sep 16 02:46:05.172766 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:46:05.172720    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/81b1434e-0098-4ff6-b603-7fc0d04dc0b0-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-b-1a50f4-stg4v\" (UID: \"81b1434e-0098-4ff6-b603-7fc0d04dc0b0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-1a50f4-stg4v"
Sep 16 02:46:05.173394 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:46:05.173371    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/81b1434e-0098-4ff6-b603-7fc0d04dc0b0-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-b-1a50f4-stg4v\" (UID: \"81b1434e-0098-4ff6-b603-7fc0d04dc0b0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-1a50f4-stg4v"
Sep 16 02:46:05.194601 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:46:05.194575    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-342933-5884dfdb95-npwr9"
Sep 16 02:46:05.297796 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:46:05.297716    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-1a50f4-stg4v"
Sep 16 02:46:05.592329 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:46:05.592301    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-1a50f4-stg4v"]
Sep 16 02:46:05.594726 ip-10-0-128-111 kubenswrapper[2583]: W0916 02:46:05.594699    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod81b1434e_0098_4ff6_b603_7fc0d04dc0b0.slice/crio-8ddbc0dadceb86f8263d1da9fab28869ff264d4d4860260e432825afb8811a5f WatchSource:0}: Error finding container 8ddbc0dadceb86f8263d1da9fab28869ff264d4d4860260e432825afb8811a5f: Status 404 returned error can't find the container with id 8ddbc0dadceb86f8263d1da9fab28869ff264d4d4860260e432825afb8811a5f
Sep 16 02:46:06.267324 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:46:06.267286    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-1a50f4-stg4v" event={"ID":"81b1434e-0098-4ff6-b603-7fc0d04dc0b0","Type":"ContainerStarted","Data":"3798671a60d65e0e531261233bd251c6c3f4b8ffd722f391239cec7e98f39570"}
Sep 16 02:46:06.267724 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:46:06.267339    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-1a50f4-stg4v" event={"ID":"81b1434e-0098-4ff6-b603-7fc0d04dc0b0","Type":"ContainerStarted","Data":"8ddbc0dadceb86f8263d1da9fab28869ff264d4d4860260e432825afb8811a5f"}
Sep 16 02:46:06.283650 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:46:06.283526    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-1a50f4-stg4v" podStartSLOduration=2.283512492 podStartE2EDuration="2.283512492s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 02:46:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 02:46:06.28108505 +0000 UTC m=+1644.748873843" watchObservedRunningTime="2026-09-16 02:46:06.283512492 +0000 UTC m=+1644.751301265"
Sep 16 02:46:07.274413 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:46:07.274374    2583 generic.go:356] "Generic (PLEG): container finished" podID="81b1434e-0098-4ff6-b603-7fc0d04dc0b0" containerID="3798671a60d65e0e531261233bd251c6c3f4b8ffd722f391239cec7e98f39570" exitCode=0
Sep 16 02:46:07.274807 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:46:07.274458    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-1a50f4-stg4v" event={"ID":"81b1434e-0098-4ff6-b603-7fc0d04dc0b0","Type":"ContainerDied","Data":"3798671a60d65e0e531261233bd251c6c3f4b8ffd722f391239cec7e98f39570"}
Sep 16 02:46:08.429359 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:46:08.429325    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-1a50f4-stg4v"
Sep 16 02:46:08.609530 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:46:08.609503    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/81b1434e-0098-4ff6-b603-7fc0d04dc0b0-maas-api-service-ca\") pod \"81b1434e-0098-4ff6-b603-7fc0d04dc0b0\" (UID: \"81b1434e-0098-4ff6-b603-7fc0d04dc0b0\") "
Sep 16 02:46:08.609889 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:46:08.609862    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/81b1434e-0098-4ff6-b603-7fc0d04dc0b0-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "81b1434e-0098-4ff6-b603-7fc0d04dc0b0" (UID: "81b1434e-0098-4ff6-b603-7fc0d04dc0b0"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 02:46:08.711210 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:46:08.711173    2583 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/81b1434e-0098-4ff6-b603-7fc0d04dc0b0-maas-api-service-ca\") on node \"ip-10-0-128-111.ec2.internal\" DevicePath \"\""
Sep 16 02:46:09.287726 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:46:09.287686    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-1a50f4-stg4v" event={"ID":"81b1434e-0098-4ff6-b603-7fc0d04dc0b0","Type":"ContainerDied","Data":"8ddbc0dadceb86f8263d1da9fab28869ff264d4d4860260e432825afb8811a5f"}
Sep 16 02:46:09.287904 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:46:09.287744    2583 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8ddbc0dadceb86f8263d1da9fab28869ff264d4d4860260e432825afb8811a5f"
Sep 16 02:46:09.287904 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:46:09.287694    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-1a50f4-stg4v"
Sep 16 02:46:09.897701 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:46:09.897675    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-4178a6-openshift-default-596589786-tm9nc"
Sep 16 02:46:09.920317 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:46:09.920234    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a7e3b05a-b2e6-4a33-b102-c7c98550b985-istio-token\") pod \"a7e3b05a-b2e6-4a33-b102-c7c98550b985\" (UID: \"a7e3b05a-b2e6-4a33-b102-c7c98550b985\") "
Sep 16 02:46:09.920317 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:46:09.920283    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a7e3b05a-b2e6-4a33-b102-c7c98550b985-istio-podinfo\") pod \"a7e3b05a-b2e6-4a33-b102-c7c98550b985\" (UID: \"a7e3b05a-b2e6-4a33-b102-c7c98550b985\") "
Sep 16 02:46:09.920734 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:46:09.920336    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a7e3b05a-b2e6-4a33-b102-c7c98550b985-istio-data\") pod \"a7e3b05a-b2e6-4a33-b102-c7c98550b985\" (UID: \"a7e3b05a-b2e6-4a33-b102-c7c98550b985\") "
Sep 16 02:46:09.920734 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:46:09.920371    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a7e3b05a-b2e6-4a33-b102-c7c98550b985-workload-certs\") pod \"a7e3b05a-b2e6-4a33-b102-c7c98550b985\" (UID: \"a7e3b05a-b2e6-4a33-b102-c7c98550b985\") "
Sep 16 02:46:09.920734 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:46:09.920394    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a7e3b05a-b2e6-4a33-b102-c7c98550b985-istiod-ca-cert\") pod \"a7e3b05a-b2e6-4a33-b102-c7c98550b985\" (UID: \"a7e3b05a-b2e6-4a33-b102-c7c98550b985\") "
Sep 16 02:46:09.920734 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:46:09.920454    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a7e3b05a-b2e6-4a33-b102-c7c98550b985-istio-envoy\") pod \"a7e3b05a-b2e6-4a33-b102-c7c98550b985\" (UID: \"a7e3b05a-b2e6-4a33-b102-c7c98550b985\") "
Sep 16 02:46:09.920734 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:46:09.920497    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a7e3b05a-b2e6-4a33-b102-c7c98550b985-credential-socket\") pod \"a7e3b05a-b2e6-4a33-b102-c7c98550b985\" (UID: \"a7e3b05a-b2e6-4a33-b102-c7c98550b985\") "
Sep 16 02:46:09.920734 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:46:09.920584    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2hl6k\" (UniqueName: \"kubernetes.io/projected/a7e3b05a-b2e6-4a33-b102-c7c98550b985-kube-api-access-2hl6k\") pod \"a7e3b05a-b2e6-4a33-b102-c7c98550b985\" (UID: \"a7e3b05a-b2e6-4a33-b102-c7c98550b985\") "
Sep 16 02:46:09.920734 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:46:09.920645    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a7e3b05a-b2e6-4a33-b102-c7c98550b985-workload-socket\") pod \"a7e3b05a-b2e6-4a33-b102-c7c98550b985\" (UID: \"a7e3b05a-b2e6-4a33-b102-c7c98550b985\") "
Sep 16 02:46:09.921105 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:46:09.920834    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a7e3b05a-b2e6-4a33-b102-c7c98550b985-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "a7e3b05a-b2e6-4a33-b102-c7c98550b985" (UID: "a7e3b05a-b2e6-4a33-b102-c7c98550b985"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 02:46:09.921105 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:46:09.921011    2583 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a7e3b05a-b2e6-4a33-b102-c7c98550b985-workload-certs\") on node \"ip-10-0-128-111.ec2.internal\" DevicePath \"\""
Sep 16 02:46:09.921354 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:46:09.921331    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a7e3b05a-b2e6-4a33-b102-c7c98550b985-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "a7e3b05a-b2e6-4a33-b102-c7c98550b985" (UID: "a7e3b05a-b2e6-4a33-b102-c7c98550b985"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 02:46:09.921603 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:46:09.921402    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a7e3b05a-b2e6-4a33-b102-c7c98550b985-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "a7e3b05a-b2e6-4a33-b102-c7c98550b985" (UID: "a7e3b05a-b2e6-4a33-b102-c7c98550b985"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 02:46:09.921926 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:46:09.921903    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a7e3b05a-b2e6-4a33-b102-c7c98550b985-istio-data" (OuterVolumeSpecName: "istio-data") pod "a7e3b05a-b2e6-4a33-b102-c7c98550b985" (UID: "a7e3b05a-b2e6-4a33-b102-c7c98550b985"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 02:46:09.922006 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:46:09.921956    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a7e3b05a-b2e6-4a33-b102-c7c98550b985-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "a7e3b05a-b2e6-4a33-b102-c7c98550b985" (UID: "a7e3b05a-b2e6-4a33-b102-c7c98550b985"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 02:46:09.926256 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:46:09.926214    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a7e3b05a-b2e6-4a33-b102-c7c98550b985-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "a7e3b05a-b2e6-4a33-b102-c7c98550b985" (UID: "a7e3b05a-b2e6-4a33-b102-c7c98550b985"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 16 02:46:09.926256 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:46:09.926228    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a7e3b05a-b2e6-4a33-b102-c7c98550b985-kube-api-access-2hl6k" (OuterVolumeSpecName: "kube-api-access-2hl6k") pod "a7e3b05a-b2e6-4a33-b102-c7c98550b985" (UID: "a7e3b05a-b2e6-4a33-b102-c7c98550b985"). InnerVolumeSpecName "kube-api-access-2hl6k". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 02:46:09.926648 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:46:09.926623    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a7e3b05a-b2e6-4a33-b102-c7c98550b985-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "a7e3b05a-b2e6-4a33-b102-c7c98550b985" (UID: "a7e3b05a-b2e6-4a33-b102-c7c98550b985"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 02:46:09.926787 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:46:09.926768    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a7e3b05a-b2e6-4a33-b102-c7c98550b985-istio-token" (OuterVolumeSpecName: "istio-token") pod "a7e3b05a-b2e6-4a33-b102-c7c98550b985" (UID: "a7e3b05a-b2e6-4a33-b102-c7c98550b985"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 02:46:10.021473 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:46:10.021442    2583 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a7e3b05a-b2e6-4a33-b102-c7c98550b985-istio-envoy\") on node \"ip-10-0-128-111.ec2.internal\" DevicePath \"\""
Sep 16 02:46:10.021473 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:46:10.021469    2583 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a7e3b05a-b2e6-4a33-b102-c7c98550b985-credential-socket\") on node \"ip-10-0-128-111.ec2.internal\" DevicePath \"\""
Sep 16 02:46:10.021473 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:46:10.021479    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2hl6k\" (UniqueName: \"kubernetes.io/projected/a7e3b05a-b2e6-4a33-b102-c7c98550b985-kube-api-access-2hl6k\") on node \"ip-10-0-128-111.ec2.internal\" DevicePath \"\""
Sep 16 02:46:10.021716 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:46:10.021488    2583 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a7e3b05a-b2e6-4a33-b102-c7c98550b985-workload-socket\") on node \"ip-10-0-128-111.ec2.internal\" DevicePath \"\""
Sep 16 02:46:10.021716 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:46:10.021497    2583 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a7e3b05a-b2e6-4a33-b102-c7c98550b985-istio-token\") on node \"ip-10-0-128-111.ec2.internal\" DevicePath \"\""
Sep 16 02:46:10.021716 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:46:10.021506    2583 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a7e3b05a-b2e6-4a33-b102-c7c98550b985-istio-podinfo\") on node \"ip-10-0-128-111.ec2.internal\" DevicePath \"\""
Sep 16 02:46:10.021716 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:46:10.021515    2583 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a7e3b05a-b2e6-4a33-b102-c7c98550b985-istio-data\") on node \"ip-10-0-128-111.ec2.internal\" DevicePath \"\""
Sep 16 02:46:10.021716 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:46:10.021523    2583 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a7e3b05a-b2e6-4a33-b102-c7c98550b985-istiod-ca-cert\") on node \"ip-10-0-128-111.ec2.internal\" DevicePath \"\""
Sep 16 02:46:10.295357 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:46:10.295273    2583 generic.go:356] "Generic (PLEG): container finished" podID="a7e3b05a-b2e6-4a33-b102-c7c98550b985" containerID="7e453e0c33b890e8afe37bdf6255c778f886d0c29b8abaea032652fb02e9b2d1" exitCode=0
Sep 16 02:46:10.295357 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:46:10.295332    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-4178a6-openshift-default-596589786-tm9nc" event={"ID":"a7e3b05a-b2e6-4a33-b102-c7c98550b985","Type":"ContainerDied","Data":"7e453e0c33b890e8afe37bdf6255c778f886d0c29b8abaea032652fb02e9b2d1"}
Sep 16 02:46:10.295694 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:46:10.295375    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-4178a6-openshift-default-596589786-tm9nc"
Sep 16 02:46:10.295694 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:46:10.295388    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-4178a6-openshift-default-596589786-tm9nc" event={"ID":"a7e3b05a-b2e6-4a33-b102-c7c98550b985","Type":"ContainerDied","Data":"c99c3bb91d03055d234a7206e1423cf7aa8056609d163c5097de05d16f4428ef"}
Sep 16 02:46:10.295694 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:46:10.295415    2583 scope.go:117] "RemoveContainer" containerID="7e453e0c33b890e8afe37bdf6255c778f886d0c29b8abaea032652fb02e9b2d1"
Sep 16 02:46:10.307795 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:46:10.307775    2583 scope.go:117] "RemoveContainer" containerID="09c436387ff7aed3dcff8780c92064c7307ad2034cc3f53e62b1c352b6f49f87"
Sep 16 02:46:10.314730 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:46:10.314694    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-4178a6-openshift-default-596589786-tm9nc"]
Sep 16 02:46:10.318136 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:46:10.317918    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-4178a6-openshift-default-596589786-tm9nc"]
Sep 16 02:46:10.318136 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:46:10.318005    2583 scope.go:117] "RemoveContainer" containerID="7e453e0c33b890e8afe37bdf6255c778f886d0c29b8abaea032652fb02e9b2d1"
Sep 16 02:46:10.318671 ip-10-0-128-111 kubenswrapper[2583]: E0916 02:46:10.318580    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7e453e0c33b890e8afe37bdf6255c778f886d0c29b8abaea032652fb02e9b2d1\": container with ID starting with 7e453e0c33b890e8afe37bdf6255c778f886d0c29b8abaea032652fb02e9b2d1 not found: ID does not exist" containerID="7e453e0c33b890e8afe37bdf6255c778f886d0c29b8abaea032652fb02e9b2d1"
Sep 16 02:46:10.318671 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:46:10.318617    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7e453e0c33b890e8afe37bdf6255c778f886d0c29b8abaea032652fb02e9b2d1"} err="failed to get container status \"7e453e0c33b890e8afe37bdf6255c778f886d0c29b8abaea032652fb02e9b2d1\": rpc error: code = NotFound desc = could not find container \"7e453e0c33b890e8afe37bdf6255c778f886d0c29b8abaea032652fb02e9b2d1\": container with ID starting with 7e453e0c33b890e8afe37bdf6255c778f886d0c29b8abaea032652fb02e9b2d1 not found: ID does not exist"
Sep 16 02:46:10.318671 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:46:10.318642    2583 scope.go:117] "RemoveContainer" containerID="09c436387ff7aed3dcff8780c92064c7307ad2034cc3f53e62b1c352b6f49f87"
Sep 16 02:46:10.319012 ip-10-0-128-111 kubenswrapper[2583]: E0916 02:46:10.318982    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"09c436387ff7aed3dcff8780c92064c7307ad2034cc3f53e62b1c352b6f49f87\": container with ID starting with 09c436387ff7aed3dcff8780c92064c7307ad2034cc3f53e62b1c352b6f49f87 not found: ID does not exist" containerID="09c436387ff7aed3dcff8780c92064c7307ad2034cc3f53e62b1c352b6f49f87"
Sep 16 02:46:10.319083 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:46:10.319009    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"09c436387ff7aed3dcff8780c92064c7307ad2034cc3f53e62b1c352b6f49f87"} err="failed to get container status \"09c436387ff7aed3dcff8780c92064c7307ad2034cc3f53e62b1c352b6f49f87\": rpc error: code = NotFound desc = could not find container \"09c436387ff7aed3dcff8780c92064c7307ad2034cc3f53e62b1c352b6f49f87\": container with ID starting with 09c436387ff7aed3dcff8780c92064c7307ad2034cc3f53e62b1c352b6f49f87 not found: ID does not exist"
Sep 16 02:46:11.303459 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:46:11.303428    2583 generic.go:356] "Generic (PLEG): container finished" podID="e2ae0cd7-7d55-4dc1-8ea2-1eb9020290dd" containerID="14de09c0f6ff121463cb92a82976c4fb156a33dc8d66672fd3627e71f0ad77d8" exitCode=0
Sep 16 02:46:11.303974 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:46:11.303512    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-342933-openshift-default-66f6f7b544-c5hzx" event={"ID":"e2ae0cd7-7d55-4dc1-8ea2-1eb9020290dd","Type":"ContainerDied","Data":"14de09c0f6ff121463cb92a82976c4fb156a33dc8d66672fd3627e71f0ad77d8"}
Sep 16 02:46:11.303974 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:46:11.303958    2583 scope.go:117] "RemoveContainer" containerID="14de09c0f6ff121463cb92a82976c4fb156a33dc8d66672fd3627e71f0ad77d8"
Sep 16 02:46:11.305204 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:46:11.305176    2583 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 16 02:46:11.305288 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:46:11.305254    2583 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 16 02:46:11.305327 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:46:11.305297    2583 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 16 02:46:12.046207 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:46:12.046173    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a7e3b05a-b2e6-4a33-b102-c7c98550b985" path="/var/lib/kubelet/pods/a7e3b05a-b2e6-4a33-b102-c7c98550b985/volumes"
Sep 16 02:46:12.311022 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:46:12.310937    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-342933-openshift-default-66f6f7b544-c5hzx" event={"ID":"e2ae0cd7-7d55-4dc1-8ea2-1eb9020290dd","Type":"ContainerStarted","Data":"bd27330f5e2003d3a5f2e9c6d8fef57a47955b553312d958746e27f4688be485"}
Sep 16 02:46:12.804291 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:46:12.804255    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-cleanup-342933-openshift-default-66f6f7b544-c5hzx"
Sep 16 02:46:12.805594 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:46:12.805545    2583 patch_prober.go:28] interesting pod/e2e-ipp-cleanup-342933-openshift-default-66f6f7b544-c5hzx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" start-of-body=
Sep 16 02:46:12.805716 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:46:12.805619    2583 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-cleanup-342933-openshift-default-66f6f7b544-c5hzx" podUID="e2ae0cd7-7d55-4dc1-8ea2-1eb9020290dd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused"
Sep 16 02:46:13.804064 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:46:13.804021    2583 patch_prober.go:28] interesting pod/e2e-ipp-cleanup-342933-openshift-default-66f6f7b544-c5hzx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" start-of-body=
Sep 16 02:46:13.804531 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:46:13.804084    2583 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-cleanup-342933-openshift-default-66f6f7b544-c5hzx" podUID="e2ae0cd7-7d55-4dc1-8ea2-1eb9020290dd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused"
Sep 16 02:46:14.804759 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:46:14.804721    2583 patch_prober.go:28] interesting pod/e2e-ipp-cleanup-342933-openshift-default-66f6f7b544-c5hzx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" start-of-body=
Sep 16 02:46:14.805139 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:46:14.804782    2583 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-cleanup-342933-openshift-default-66f6f7b544-c5hzx" podUID="e2ae0cd7-7d55-4dc1-8ea2-1eb9020290dd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused"
Sep 16 02:46:15.804804 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:46:15.804767    2583 patch_prober.go:28] interesting pod/e2e-ipp-cleanup-342933-openshift-default-66f6f7b544-c5hzx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" start-of-body=
Sep 16 02:46:15.805202 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:46:15.804828    2583 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-cleanup-342933-openshift-default-66f6f7b544-c5hzx" podUID="e2ae0cd7-7d55-4dc1-8ea2-1eb9020290dd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused"
Sep 16 02:46:16.145223 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:46:16.145167    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-cleanup-342933-openshift-default-66f6f7b544-c5hzx"
Sep 16 02:46:16.803956 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:46:16.803923    2583 patch_prober.go:28] interesting pod/e2e-ipp-cleanup-342933-openshift-default-66f6f7b544-c5hzx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" start-of-body=
Sep 16 02:46:16.804209 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:46:16.803975    2583 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-cleanup-342933-openshift-default-66f6f7b544-c5hzx" podUID="e2ae0cd7-7d55-4dc1-8ea2-1eb9020290dd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused"
Sep 16 02:46:17.804318 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:46:17.804279    2583 patch_prober.go:28] interesting pod/e2e-ipp-cleanup-342933-openshift-default-66f6f7b544-c5hzx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" start-of-body=
Sep 16 02:46:17.804723 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:46:17.804334    2583 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-cleanup-342933-openshift-default-66f6f7b544-c5hzx" podUID="e2ae0cd7-7d55-4dc1-8ea2-1eb9020290dd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused"
Sep 16 02:46:18.307150 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:46:18.307116    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-342933-5884dfdb95-npwr9"]
Sep 16 02:46:18.307383 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:46:18.307357    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-342933-5884dfdb95-npwr9" podUID="a5df25a6-8c74-484a-b32c-974d9b321fc6" containerName="payload-processing" containerID="cri-o://a1eee1d8e5bafa5e197c920dfcb3b92b2e227d5336c95613d364ccb411ffb110" gracePeriod=30
Sep 16 02:46:18.577695 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:46:18.577670    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-342933-5884dfdb95-npwr9"
Sep 16 02:46:18.601929 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:46:18.601891    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a5df25a6-8c74-484a-b32c-974d9b321fc6-plugins-config-volume\") pod \"a5df25a6-8c74-484a-b32c-974d9b321fc6\" (UID: \"a5df25a6-8c74-484a-b32c-974d9b321fc6\") "
Sep 16 02:46:18.602124 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:46:18.601982    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-569j8\" (UniqueName: \"kubernetes.io/projected/a5df25a6-8c74-484a-b32c-974d9b321fc6-kube-api-access-569j8\") pod \"a5df25a6-8c74-484a-b32c-974d9b321fc6\" (UID: \"a5df25a6-8c74-484a-b32c-974d9b321fc6\") "
Sep 16 02:46:18.602367 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:46:18.602341    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a5df25a6-8c74-484a-b32c-974d9b321fc6-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "a5df25a6-8c74-484a-b32c-974d9b321fc6" (UID: "a5df25a6-8c74-484a-b32c-974d9b321fc6"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 02:46:18.604335 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:46:18.604303    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a5df25a6-8c74-484a-b32c-974d9b321fc6-kube-api-access-569j8" (OuterVolumeSpecName: "kube-api-access-569j8") pod "a5df25a6-8c74-484a-b32c-974d9b321fc6" (UID: "a5df25a6-8c74-484a-b32c-974d9b321fc6"). InnerVolumeSpecName "kube-api-access-569j8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 02:46:18.702993 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:46:18.702959    2583 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a5df25a6-8c74-484a-b32c-974d9b321fc6-plugins-config-volume\") on node \"ip-10-0-128-111.ec2.internal\" DevicePath \"\""
Sep 16 02:46:18.702993 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:46:18.702989    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-569j8\" (UniqueName: \"kubernetes.io/projected/a5df25a6-8c74-484a-b32c-974d9b321fc6-kube-api-access-569j8\") on node \"ip-10-0-128-111.ec2.internal\" DevicePath \"\""
Sep 16 02:46:18.808066 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:46:18.808029    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-cleanup-342933-openshift-default-66f6f7b544-c5hzx"
Sep 16 02:46:18.809128 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:46:18.809105    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-cleanup-342933-openshift-default-66f6f7b544-c5hzx"
Sep 16 02:46:19.363016 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:46:19.362980    2583 generic.go:356] "Generic (PLEG): container finished" podID="a5df25a6-8c74-484a-b32c-974d9b321fc6" containerID="a1eee1d8e5bafa5e197c920dfcb3b92b2e227d5336c95613d364ccb411ffb110" exitCode=0
Sep 16 02:46:19.363193 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:46:19.363022    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-342933-5884dfdb95-npwr9"
Sep 16 02:46:19.363193 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:46:19.363059    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-342933-5884dfdb95-npwr9" event={"ID":"a5df25a6-8c74-484a-b32c-974d9b321fc6","Type":"ContainerDied","Data":"a1eee1d8e5bafa5e197c920dfcb3b92b2e227d5336c95613d364ccb411ffb110"}
Sep 16 02:46:19.363193 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:46:19.363108    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-342933-5884dfdb95-npwr9" event={"ID":"a5df25a6-8c74-484a-b32c-974d9b321fc6","Type":"ContainerDied","Data":"0cc82a564556d89a8e4f8952865ba4ab5882cd3919dd26bc715b6be63633b874"}
Sep 16 02:46:19.363193 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:46:19.363133    2583 scope.go:117] "RemoveContainer" containerID="a1eee1d8e5bafa5e197c920dfcb3b92b2e227d5336c95613d364ccb411ffb110"
Sep 16 02:46:19.372522 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:46:19.372502    2583 scope.go:117] "RemoveContainer" containerID="a1eee1d8e5bafa5e197c920dfcb3b92b2e227d5336c95613d364ccb411ffb110"
Sep 16 02:46:19.372936 ip-10-0-128-111 kubenswrapper[2583]: E0916 02:46:19.372917    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a1eee1d8e5bafa5e197c920dfcb3b92b2e227d5336c95613d364ccb411ffb110\": container with ID starting with a1eee1d8e5bafa5e197c920dfcb3b92b2e227d5336c95613d364ccb411ffb110 not found: ID does not exist" containerID="a1eee1d8e5bafa5e197c920dfcb3b92b2e227d5336c95613d364ccb411ffb110"
Sep 16 02:46:19.373006 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:46:19.372947    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a1eee1d8e5bafa5e197c920dfcb3b92b2e227d5336c95613d364ccb411ffb110"} err="failed to get container status \"a1eee1d8e5bafa5e197c920dfcb3b92b2e227d5336c95613d364ccb411ffb110\": rpc error: code = NotFound desc = could not find container \"a1eee1d8e5bafa5e197c920dfcb3b92b2e227d5336c95613d364ccb411ffb110\": container with ID starting with a1eee1d8e5bafa5e197c920dfcb3b92b2e227d5336c95613d364ccb411ffb110 not found: ID does not exist"
Sep 16 02:46:19.386338 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:46:19.386318    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-342933-5884dfdb95-npwr9"]
Sep 16 02:46:19.392354 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:46:19.392334    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-342933-5884dfdb95-npwr9"]
Sep 16 02:46:20.046202 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:46:20.046167    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a5df25a6-8c74-484a-b32c-974d9b321fc6" path="/var/lib/kubelet/pods/a5df25a6-8c74-484a-b32c-974d9b321fc6/volumes"
Sep 16 02:46:22.451230 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:46:22.451192    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-fzl8t"]
Sep 16 02:46:23.274735 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:46:23.274701    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-1a50f4-stg4v"]
Sep 16 02:46:23.276903 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:46:23.276878    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-1a50f4-stg4v"]
Sep 16 02:46:24.046198 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:46:24.046166    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="81b1434e-0098-4ff6-b603-7fc0d04dc0b0" path="/var/lib/kubelet/pods/81b1434e-0098-4ff6-b603-7fc0d04dc0b0/volumes"
Sep 16 02:46:25.691132 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:46:25.691095    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-342933-openshift-default-66f6f7b544-c5hzx"]
Sep 16 02:46:25.691589 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:46:25.691367    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-cleanup-342933-openshift-default-66f6f7b544-c5hzx" podUID="e2ae0cd7-7d55-4dc1-8ea2-1eb9020290dd" containerName="istio-proxy" containerID="cri-o://bd27330f5e2003d3a5f2e9c6d8fef57a47955b553312d958746e27f4688be485" gracePeriod=30
Sep 16 02:46:29.093736 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:46:29.093706    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-10a6b9-hc9fg"]
Sep 16 02:46:29.094128 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:46:29.094095    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a7e3b05a-b2e6-4a33-b102-c7c98550b985" containerName="istio-proxy"
Sep 16 02:46:29.094128 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:46:29.094107    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="a7e3b05a-b2e6-4a33-b102-c7c98550b985" containerName="istio-proxy"
Sep 16 02:46:29.094128 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:46:29.094117    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a7e3b05a-b2e6-4a33-b102-c7c98550b985" containerName="istio-proxy"
Sep 16 02:46:29.094128 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:46:29.094123    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="a7e3b05a-b2e6-4a33-b102-c7c98550b985" containerName="istio-proxy"
Sep 16 02:46:29.094128 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:46:29.094131    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="81b1434e-0098-4ff6-b603-7fc0d04dc0b0" containerName="revoke-keys"
Sep 16 02:46:29.094298 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:46:29.094137    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="81b1434e-0098-4ff6-b603-7fc0d04dc0b0" containerName="revoke-keys"
Sep 16 02:46:29.094298 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:46:29.094148    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a5df25a6-8c74-484a-b32c-974d9b321fc6" containerName="payload-processing"
Sep 16 02:46:29.094298 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:46:29.094154    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="a5df25a6-8c74-484a-b32c-974d9b321fc6" containerName="payload-processing"
Sep 16 02:46:29.094298 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:46:29.094232    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="a7e3b05a-b2e6-4a33-b102-c7c98550b985" containerName="istio-proxy"
Sep 16 02:46:29.094298 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:46:29.094240    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="81b1434e-0098-4ff6-b603-7fc0d04dc0b0" containerName="revoke-keys"
Sep 16 02:46:29.094298 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:46:29.094248    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="a5df25a6-8c74-484a-b32c-974d9b321fc6" containerName="payload-processing"
Sep 16 02:46:29.098612 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:46:29.098595    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-10a6b9-hc9fg"
Sep 16 02:46:29.101160 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:46:29.101134    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 16 02:46:29.101160 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:46:29.101158    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-w8hnp\""
Sep 16 02:46:29.103533 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:46:29.103507    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-10a6b9-hc9fg"]
Sep 16 02:46:29.206355 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:46:29.206321    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/97866e0b-aab7-4115-9e3f-e6895c50a644-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-10a6b9-hc9fg\" (UID: \"97866e0b-aab7-4115-9e3f-e6895c50a644\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-10a6b9-hc9fg"
Sep 16 02:46:29.307437 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:46:29.307391    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/97866e0b-aab7-4115-9e3f-e6895c50a644-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-10a6b9-hc9fg\" (UID: \"97866e0b-aab7-4115-9e3f-e6895c50a644\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-10a6b9-hc9fg"
Sep 16 02:46:29.307998 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:46:29.307980    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/97866e0b-aab7-4115-9e3f-e6895c50a644-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-10a6b9-hc9fg\" (UID: \"97866e0b-aab7-4115-9e3f-e6895c50a644\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-10a6b9-hc9fg"
Sep 16 02:46:29.412313 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:46:29.412218    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-10a6b9-hc9fg"
Sep 16 02:46:29.681880 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:46:29.681804    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-10a6b9-hc9fg"]
Sep 16 02:46:29.685082 ip-10-0-128-111 kubenswrapper[2583]: W0916 02:46:29.685057    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod97866e0b_aab7_4115_9e3f_e6895c50a644.slice/crio-1eb0f986c8544552d0352da422e1c278c8258cf5e0373c51026a2182411e5663 WatchSource:0}: Error finding container 1eb0f986c8544552d0352da422e1c278c8258cf5e0373c51026a2182411e5663: Status 404 returned error can't find the container with id 1eb0f986c8544552d0352da422e1c278c8258cf5e0373c51026a2182411e5663
Sep 16 02:46:30.425987 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:46:30.425949    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-10a6b9-hc9fg" event={"ID":"97866e0b-aab7-4115-9e3f-e6895c50a644","Type":"ContainerStarted","Data":"e7e22381d4f095d7b12d928dfaafb799e70c87f4dd5009e6c367cc2c900b9254"}
Sep 16 02:46:30.426371 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:46:30.426008    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-10a6b9-hc9fg" event={"ID":"97866e0b-aab7-4115-9e3f-e6895c50a644","Type":"ContainerStarted","Data":"1eb0f986c8544552d0352da422e1c278c8258cf5e0373c51026a2182411e5663"}
Sep 16 02:46:30.441823 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:46:30.441778    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-10a6b9-hc9fg" podStartSLOduration=1.441757963 podStartE2EDuration="1.441757963s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 02:46:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 02:46:30.439128103 +0000 UTC m=+1668.906916885" watchObservedRunningTime="2026-09-16 02:46:30.441757963 +0000 UTC m=+1668.909546746"
Sep 16 02:46:30.951627 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:46:30.951598    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-cleanup-342933-openshift-default-66f6f7b544-c5hzx"
Sep 16 02:46:31.024015 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:46:31.023930    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e2ae0cd7-7d55-4dc1-8ea2-1eb9020290dd-istio-podinfo\") pod \"e2ae0cd7-7d55-4dc1-8ea2-1eb9020290dd\" (UID: \"e2ae0cd7-7d55-4dc1-8ea2-1eb9020290dd\") "
Sep 16 02:46:31.024015 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:46:31.023968    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e2ae0cd7-7d55-4dc1-8ea2-1eb9020290dd-workload-certs\") pod \"e2ae0cd7-7d55-4dc1-8ea2-1eb9020290dd\" (UID: \"e2ae0cd7-7d55-4dc1-8ea2-1eb9020290dd\") "
Sep 16 02:46:31.024015 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:46:31.023990    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e2ae0cd7-7d55-4dc1-8ea2-1eb9020290dd-istiod-ca-cert\") pod \"e2ae0cd7-7d55-4dc1-8ea2-1eb9020290dd\" (UID: \"e2ae0cd7-7d55-4dc1-8ea2-1eb9020290dd\") "
Sep 16 02:46:31.024015 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:46:31.024013    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e2ae0cd7-7d55-4dc1-8ea2-1eb9020290dd-istio-envoy\") pod \"e2ae0cd7-7d55-4dc1-8ea2-1eb9020290dd\" (UID: \"e2ae0cd7-7d55-4dc1-8ea2-1eb9020290dd\") "
Sep 16 02:46:31.024362 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:46:31.024058    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e2ae0cd7-7d55-4dc1-8ea2-1eb9020290dd-istio-token\") pod \"e2ae0cd7-7d55-4dc1-8ea2-1eb9020290dd\" (UID: \"e2ae0cd7-7d55-4dc1-8ea2-1eb9020290dd\") "
Sep 16 02:46:31.024362 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:46:31.024085    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e2ae0cd7-7d55-4dc1-8ea2-1eb9020290dd-workload-socket\") pod \"e2ae0cd7-7d55-4dc1-8ea2-1eb9020290dd\" (UID: \"e2ae0cd7-7d55-4dc1-8ea2-1eb9020290dd\") "
Sep 16 02:46:31.024362 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:46:31.024128    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e2ae0cd7-7d55-4dc1-8ea2-1eb9020290dd-istio-data\") pod \"e2ae0cd7-7d55-4dc1-8ea2-1eb9020290dd\" (UID: \"e2ae0cd7-7d55-4dc1-8ea2-1eb9020290dd\") "
Sep 16 02:46:31.024362 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:46:31.024163    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9mqdg\" (UniqueName: \"kubernetes.io/projected/e2ae0cd7-7d55-4dc1-8ea2-1eb9020290dd-kube-api-access-9mqdg\") pod \"e2ae0cd7-7d55-4dc1-8ea2-1eb9020290dd\" (UID: \"e2ae0cd7-7d55-4dc1-8ea2-1eb9020290dd\") "
Sep 16 02:46:31.024362 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:46:31.024233    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e2ae0cd7-7d55-4dc1-8ea2-1eb9020290dd-credential-socket\") pod \"e2ae0cd7-7d55-4dc1-8ea2-1eb9020290dd\" (UID: \"e2ae0cd7-7d55-4dc1-8ea2-1eb9020290dd\") "
Sep 16 02:46:31.024362 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:46:31.024304    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e2ae0cd7-7d55-4dc1-8ea2-1eb9020290dd-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "e2ae0cd7-7d55-4dc1-8ea2-1eb9020290dd" (UID: "e2ae0cd7-7d55-4dc1-8ea2-1eb9020290dd"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 02:46:31.024664 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:46:31.024387    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e2ae0cd7-7d55-4dc1-8ea2-1eb9020290dd-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "e2ae0cd7-7d55-4dc1-8ea2-1eb9020290dd" (UID: "e2ae0cd7-7d55-4dc1-8ea2-1eb9020290dd"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 02:46:31.024664 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:46:31.024528    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e2ae0cd7-7d55-4dc1-8ea2-1eb9020290dd-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "e2ae0cd7-7d55-4dc1-8ea2-1eb9020290dd" (UID: "e2ae0cd7-7d55-4dc1-8ea2-1eb9020290dd"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 02:46:31.024664 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:46:31.024577    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e2ae0cd7-7d55-4dc1-8ea2-1eb9020290dd-istio-data" (OuterVolumeSpecName: "istio-data") pod "e2ae0cd7-7d55-4dc1-8ea2-1eb9020290dd" (UID: "e2ae0cd7-7d55-4dc1-8ea2-1eb9020290dd"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 02:46:31.024664 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:46:31.024639    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e2ae0cd7-7d55-4dc1-8ea2-1eb9020290dd-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "e2ae0cd7-7d55-4dc1-8ea2-1eb9020290dd" (UID: "e2ae0cd7-7d55-4dc1-8ea2-1eb9020290dd"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 02:46:31.024878 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:46:31.024676    2583 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e2ae0cd7-7d55-4dc1-8ea2-1eb9020290dd-credential-socket\") on node \"ip-10-0-128-111.ec2.internal\" DevicePath \"\""
Sep 16 02:46:31.024878 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:46:31.024694    2583 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e2ae0cd7-7d55-4dc1-8ea2-1eb9020290dd-workload-certs\") on node \"ip-10-0-128-111.ec2.internal\" DevicePath \"\""
Sep 16 02:46:31.024878 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:46:31.024710    2583 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e2ae0cd7-7d55-4dc1-8ea2-1eb9020290dd-istiod-ca-cert\") on node \"ip-10-0-128-111.ec2.internal\" DevicePath \"\""
Sep 16 02:46:31.026233 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:46:31.026205    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e2ae0cd7-7d55-4dc1-8ea2-1eb9020290dd-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "e2ae0cd7-7d55-4dc1-8ea2-1eb9020290dd" (UID: "e2ae0cd7-7d55-4dc1-8ea2-1eb9020290dd"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 16 02:46:31.026233 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:46:31.026218    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e2ae0cd7-7d55-4dc1-8ea2-1eb9020290dd-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "e2ae0cd7-7d55-4dc1-8ea2-1eb9020290dd" (UID: "e2ae0cd7-7d55-4dc1-8ea2-1eb9020290dd"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 02:46:31.026419 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:46:31.026334    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e2ae0cd7-7d55-4dc1-8ea2-1eb9020290dd-istio-token" (OuterVolumeSpecName: "istio-token") pod "e2ae0cd7-7d55-4dc1-8ea2-1eb9020290dd" (UID: "e2ae0cd7-7d55-4dc1-8ea2-1eb9020290dd"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 02:46:31.026463 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:46:31.026422    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e2ae0cd7-7d55-4dc1-8ea2-1eb9020290dd-kube-api-access-9mqdg" (OuterVolumeSpecName: "kube-api-access-9mqdg") pod "e2ae0cd7-7d55-4dc1-8ea2-1eb9020290dd" (UID: "e2ae0cd7-7d55-4dc1-8ea2-1eb9020290dd"). InnerVolumeSpecName "kube-api-access-9mqdg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 02:46:31.125412 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:46:31.125380    2583 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e2ae0cd7-7d55-4dc1-8ea2-1eb9020290dd-istio-podinfo\") on node \"ip-10-0-128-111.ec2.internal\" DevicePath \"\""
Sep 16 02:46:31.125412 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:46:31.125404    2583 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e2ae0cd7-7d55-4dc1-8ea2-1eb9020290dd-istio-envoy\") on node \"ip-10-0-128-111.ec2.internal\" DevicePath \"\""
Sep 16 02:46:31.125412 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:46:31.125413    2583 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e2ae0cd7-7d55-4dc1-8ea2-1eb9020290dd-istio-token\") on node \"ip-10-0-128-111.ec2.internal\" DevicePath \"\""
Sep 16 02:46:31.125412 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:46:31.125421    2583 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e2ae0cd7-7d55-4dc1-8ea2-1eb9020290dd-workload-socket\") on node \"ip-10-0-128-111.ec2.internal\" DevicePath \"\""
Sep 16 02:46:31.125710 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:46:31.125429    2583 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e2ae0cd7-7d55-4dc1-8ea2-1eb9020290dd-istio-data\") on node \"ip-10-0-128-111.ec2.internal\" DevicePath \"\""
Sep 16 02:46:31.125710 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:46:31.125440    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9mqdg\" (UniqueName: \"kubernetes.io/projected/e2ae0cd7-7d55-4dc1-8ea2-1eb9020290dd-kube-api-access-9mqdg\") on node \"ip-10-0-128-111.ec2.internal\" DevicePath \"\""
Sep 16 02:46:31.432360 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:46:31.432324    2583 generic.go:356] "Generic (PLEG): container finished" podID="e2ae0cd7-7d55-4dc1-8ea2-1eb9020290dd" containerID="bd27330f5e2003d3a5f2e9c6d8fef57a47955b553312d958746e27f4688be485" exitCode=0
Sep 16 02:46:31.432861 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:46:31.432373    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-cleanup-342933-openshift-default-66f6f7b544-c5hzx"
Sep 16 02:46:31.432861 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:46:31.432406    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-342933-openshift-default-66f6f7b544-c5hzx" event={"ID":"e2ae0cd7-7d55-4dc1-8ea2-1eb9020290dd","Type":"ContainerDied","Data":"bd27330f5e2003d3a5f2e9c6d8fef57a47955b553312d958746e27f4688be485"}
Sep 16 02:46:31.432861 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:46:31.432460    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-342933-openshift-default-66f6f7b544-c5hzx" event={"ID":"e2ae0cd7-7d55-4dc1-8ea2-1eb9020290dd","Type":"ContainerDied","Data":"16faa3a3f7417e728c53f5d3cc1ce1a55aa1f26f56a61390fd6c7c7f6362da41"}
Sep 16 02:46:31.432861 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:46:31.432492    2583 scope.go:117] "RemoveContainer" containerID="bd27330f5e2003d3a5f2e9c6d8fef57a47955b553312d958746e27f4688be485"
Sep 16 02:46:31.434234 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:46:31.434210    2583 generic.go:356] "Generic (PLEG): container finished" podID="97866e0b-aab7-4115-9e3f-e6895c50a644" containerID="e7e22381d4f095d7b12d928dfaafb799e70c87f4dd5009e6c367cc2c900b9254" exitCode=0
Sep 16 02:46:31.434351 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:46:31.434302    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-10a6b9-hc9fg" event={"ID":"97866e0b-aab7-4115-9e3f-e6895c50a644","Type":"ContainerDied","Data":"e7e22381d4f095d7b12d928dfaafb799e70c87f4dd5009e6c367cc2c900b9254"}
Sep 16 02:46:31.445293 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:46:31.445278    2583 scope.go:117] "RemoveContainer" containerID="14de09c0f6ff121463cb92a82976c4fb156a33dc8d66672fd3627e71f0ad77d8"
Sep 16 02:46:31.454592 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:46:31.454570    2583 scope.go:117] "RemoveContainer" containerID="bd27330f5e2003d3a5f2e9c6d8fef57a47955b553312d958746e27f4688be485"
Sep 16 02:46:31.454871 ip-10-0-128-111 kubenswrapper[2583]: E0916 02:46:31.454843    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bd27330f5e2003d3a5f2e9c6d8fef57a47955b553312d958746e27f4688be485\": container with ID starting with bd27330f5e2003d3a5f2e9c6d8fef57a47955b553312d958746e27f4688be485 not found: ID does not exist" containerID="bd27330f5e2003d3a5f2e9c6d8fef57a47955b553312d958746e27f4688be485"
Sep 16 02:46:31.454937 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:46:31.454883    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bd27330f5e2003d3a5f2e9c6d8fef57a47955b553312d958746e27f4688be485"} err="failed to get container status \"bd27330f5e2003d3a5f2e9c6d8fef57a47955b553312d958746e27f4688be485\": rpc error: code = NotFound desc = could not find container \"bd27330f5e2003d3a5f2e9c6d8fef57a47955b553312d958746e27f4688be485\": container with ID starting with bd27330f5e2003d3a5f2e9c6d8fef57a47955b553312d958746e27f4688be485 not found: ID does not exist"
Sep 16 02:46:31.454937 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:46:31.454899    2583 scope.go:117] "RemoveContainer" containerID="14de09c0f6ff121463cb92a82976c4fb156a33dc8d66672fd3627e71f0ad77d8"
Sep 16 02:46:31.455231 ip-10-0-128-111 kubenswrapper[2583]: E0916 02:46:31.455194    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"14de09c0f6ff121463cb92a82976c4fb156a33dc8d66672fd3627e71f0ad77d8\": container with ID starting with 14de09c0f6ff121463cb92a82976c4fb156a33dc8d66672fd3627e71f0ad77d8 not found: ID does not exist" containerID="14de09c0f6ff121463cb92a82976c4fb156a33dc8d66672fd3627e71f0ad77d8"
Sep 16 02:46:31.455296 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:46:31.455236    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"14de09c0f6ff121463cb92a82976c4fb156a33dc8d66672fd3627e71f0ad77d8"} err="failed to get container status \"14de09c0f6ff121463cb92a82976c4fb156a33dc8d66672fd3627e71f0ad77d8\": rpc error: code = NotFound desc = could not find container \"14de09c0f6ff121463cb92a82976c4fb156a33dc8d66672fd3627e71f0ad77d8\": container with ID starting with 14de09c0f6ff121463cb92a82976c4fb156a33dc8d66672fd3627e71f0ad77d8 not found: ID does not exist"
Sep 16 02:46:31.461991 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:46:31.461970    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-342933-openshift-default-66f6f7b544-c5hzx"]
Sep 16 02:46:31.464981 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:46:31.464959    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-342933-openshift-default-66f6f7b544-c5hzx"]
Sep 16 02:46:32.045438 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:46:32.045395    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e2ae0cd7-7d55-4dc1-8ea2-1eb9020290dd" path="/var/lib/kubelet/pods/e2ae0cd7-7d55-4dc1-8ea2-1eb9020290dd/volumes"
Sep 16 02:46:32.576294 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:46:32.576273    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-10a6b9-hc9fg"
Sep 16 02:46:32.640162 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:46:32.640134    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/97866e0b-aab7-4115-9e3f-e6895c50a644-maas-api-service-ca\") pod \"97866e0b-aab7-4115-9e3f-e6895c50a644\" (UID: \"97866e0b-aab7-4115-9e3f-e6895c50a644\") "
Sep 16 02:46:32.640493 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:46:32.640467    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/97866e0b-aab7-4115-9e3f-e6895c50a644-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "97866e0b-aab7-4115-9e3f-e6895c50a644" (UID: "97866e0b-aab7-4115-9e3f-e6895c50a644"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 02:46:32.741881 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:46:32.741802    2583 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/97866e0b-aab7-4115-9e3f-e6895c50a644-maas-api-service-ca\") on node \"ip-10-0-128-111.ec2.internal\" DevicePath \"\""
Sep 16 02:46:33.447343 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:46:33.447308    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-10a6b9-hc9fg" event={"ID":"97866e0b-aab7-4115-9e3f-e6895c50a644","Type":"ContainerDied","Data":"1eb0f986c8544552d0352da422e1c278c8258cf5e0373c51026a2182411e5663"}
Sep 16 02:46:33.447343 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:46:33.447347    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-10a6b9-hc9fg"
Sep 16 02:46:33.447569 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:46:33.447357    2583 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1eb0f986c8544552d0352da422e1c278c8258cf5e0373c51026a2182411e5663"
Sep 16 02:46:41.253948 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:46:41.253917    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-fe9d17-29825445-qjxl4"]
Sep 16 02:46:41.259745 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:46:41.259718    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-fe9d17-29825445-qjxl4"]
Sep 16 02:46:41.275611 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:46:41.275580    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w1-fe9d17-6b5dbf5695-rxrld"]
Sep 16 02:46:41.275993 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:46:41.275817    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-fe9d17-6b5dbf5695-rxrld" podUID="1828a8ad-c4ec-4f54-b6e7-f34bc6735457" containerName="payload-processing" containerID="cri-o://3586f74a0c5982b88161f5009a889a2eab9401871ccc1e37fe0b446a09418011" gracePeriod=30
Sep 16 02:46:41.492485 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:46:41.492442    2583 generic.go:356] "Generic (PLEG): container finished" podID="1828a8ad-c4ec-4f54-b6e7-f34bc6735457" containerID="3586f74a0c5982b88161f5009a889a2eab9401871ccc1e37fe0b446a09418011" exitCode=0
Sep 16 02:46:41.492683 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:46:41.492501    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-fe9d17-6b5dbf5695-rxrld" event={"ID":"1828a8ad-c4ec-4f54-b6e7-f34bc6735457","Type":"ContainerDied","Data":"3586f74a0c5982b88161f5009a889a2eab9401871ccc1e37fe0b446a09418011"}
Sep 16 02:46:41.527926 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:46:41.527897    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-fe9d17-6b5dbf5695-rxrld"
Sep 16 02:46:41.624686 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:46:41.624650    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1828a8ad-c4ec-4f54-b6e7-f34bc6735457-plugins-config-volume\") pod \"1828a8ad-c4ec-4f54-b6e7-f34bc6735457\" (UID: \"1828a8ad-c4ec-4f54-b6e7-f34bc6735457\") "
Sep 16 02:46:41.624880 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:46:41.624718    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mszgq\" (UniqueName: \"kubernetes.io/projected/1828a8ad-c4ec-4f54-b6e7-f34bc6735457-kube-api-access-mszgq\") pod \"1828a8ad-c4ec-4f54-b6e7-f34bc6735457\" (UID: \"1828a8ad-c4ec-4f54-b6e7-f34bc6735457\") "
Sep 16 02:46:41.625437 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:46:41.625382    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1828a8ad-c4ec-4f54-b6e7-f34bc6735457-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "1828a8ad-c4ec-4f54-b6e7-f34bc6735457" (UID: "1828a8ad-c4ec-4f54-b6e7-f34bc6735457"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 02:46:41.627324 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:46:41.627297    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1828a8ad-c4ec-4f54-b6e7-f34bc6735457-kube-api-access-mszgq" (OuterVolumeSpecName: "kube-api-access-mszgq") pod "1828a8ad-c4ec-4f54-b6e7-f34bc6735457" (UID: "1828a8ad-c4ec-4f54-b6e7-f34bc6735457"). InnerVolumeSpecName "kube-api-access-mszgq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 02:46:41.726195 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:46:41.726156    2583 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1828a8ad-c4ec-4f54-b6e7-f34bc6735457-plugins-config-volume\") on node \"ip-10-0-128-111.ec2.internal\" DevicePath \"\""
Sep 16 02:46:41.726195 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:46:41.726184    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mszgq\" (UniqueName: \"kubernetes.io/projected/1828a8ad-c4ec-4f54-b6e7-f34bc6735457-kube-api-access-mszgq\") on node \"ip-10-0-128-111.ec2.internal\" DevicePath \"\""
Sep 16 02:46:42.047298 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:46:42.047266    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5f700448-b33c-43ec-ba1d-b850e3e7618a" path="/var/lib/kubelet/pods/5f700448-b33c-43ec-ba1d-b850e3e7618a/volumes"
Sep 16 02:46:42.262863 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:46:42.262830    2583 scope.go:117] "RemoveContainer" containerID="3586f74a0c5982b88161f5009a889a2eab9401871ccc1e37fe0b446a09418011"
Sep 16 02:46:42.497703 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:46:42.497659    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-fe9d17-6b5dbf5695-rxrld" event={"ID":"1828a8ad-c4ec-4f54-b6e7-f34bc6735457","Type":"ContainerDied","Data":"5d29658a5d0ca8821ba3c474dcd2e9912885f64b3a8a87fca1a046c6976249ba"}
Sep 16 02:46:42.497898 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:46:42.497723    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-fe9d17-6b5dbf5695-rxrld"
Sep 16 02:46:42.523682 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:46:42.523654    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w1-fe9d17-6b5dbf5695-rxrld"]
Sep 16 02:46:42.525361 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:46:42.525341    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w1-fe9d17-6b5dbf5695-rxrld"]
Sep 16 02:46:43.352310 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:46:43.352279    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-10a6b9-29825445-5n5s4"]
Sep 16 02:46:43.364681 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:46:43.364657    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-10a6b9-29825445-5n5s4"]
Sep 16 02:46:43.583690 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:46:43.583656    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-10a6b9-hc9fg"]
Sep 16 02:46:43.587738 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:46:43.587710    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-10a6b9-hc9fg"]
Sep 16 02:46:44.045612 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:46:44.045576    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1828a8ad-c4ec-4f54-b6e7-f34bc6735457" path="/var/lib/kubelet/pods/1828a8ad-c4ec-4f54-b6e7-f34bc6735457/volumes"
Sep 16 02:46:44.046011 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:46:44.045991    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="97866e0b-aab7-4115-9e3f-e6895c50a644" path="/var/lib/kubelet/pods/97866e0b-aab7-4115-9e3f-e6895c50a644/volumes"
Sep 16 02:46:44.046314 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:46:44.046297    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b8b8b76a-5f01-4a45-9750-4b5814c0eedc" path="/var/lib/kubelet/pods/b8b8b76a-5f01-4a45-9750-4b5814c0eedc/volumes"
Sep 16 02:46:44.851471 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:46:44.851438    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-fzl8t"]
Sep 16 02:46:48.049279 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:46:48.049246    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-e86d9e-9q5qj"]
Sep 16 02:46:48.049745 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:46:48.049635    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="97866e0b-aab7-4115-9e3f-e6895c50a644" containerName="revoke-keys"
Sep 16 02:46:48.049745 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:46:48.049647    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="97866e0b-aab7-4115-9e3f-e6895c50a644" containerName="revoke-keys"
Sep 16 02:46:48.049745 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:46:48.049658    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1828a8ad-c4ec-4f54-b6e7-f34bc6735457" containerName="payload-processing"
Sep 16 02:46:48.049745 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:46:48.049663    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="1828a8ad-c4ec-4f54-b6e7-f34bc6735457" containerName="payload-processing"
Sep 16 02:46:48.049745 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:46:48.049682    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e2ae0cd7-7d55-4dc1-8ea2-1eb9020290dd" containerName="istio-proxy"
Sep 16 02:46:48.049745 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:46:48.049688    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="e2ae0cd7-7d55-4dc1-8ea2-1eb9020290dd" containerName="istio-proxy"
Sep 16 02:46:48.049938 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:46:48.049750    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="a7e3b05a-b2e6-4a33-b102-c7c98550b985" containerName="istio-proxy"
Sep 16 02:46:48.049938 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:46:48.049757    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="97866e0b-aab7-4115-9e3f-e6895c50a644" containerName="revoke-keys"
Sep 16 02:46:48.049938 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:46:48.049765    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="1828a8ad-c4ec-4f54-b6e7-f34bc6735457" containerName="payload-processing"
Sep 16 02:46:48.049938 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:46:48.049772    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="e2ae0cd7-7d55-4dc1-8ea2-1eb9020290dd" containerName="istio-proxy"
Sep 16 02:46:48.049938 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:46:48.049777    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="e2ae0cd7-7d55-4dc1-8ea2-1eb9020290dd" containerName="istio-proxy"
Sep 16 02:46:48.054125 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:46:48.054108    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-e86d9e-9q5qj"
Sep 16 02:46:48.056925 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:46:48.056899    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 16 02:46:48.057073 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:46:48.056899    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-w8hnp\""
Sep 16 02:46:48.059137 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:46:48.059114    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-e86d9e-9q5qj"]
Sep 16 02:46:48.185996 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:46:48.185958    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/130bfdb3-450f-4be3-a49c-c851c1235afa-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w1-e86d9e-9q5qj\" (UID: \"130bfdb3-450f-4be3-a49c-c851c1235afa\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-e86d9e-9q5qj"
Sep 16 02:46:48.287599 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:46:48.287546    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/130bfdb3-450f-4be3-a49c-c851c1235afa-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w1-e86d9e-9q5qj\" (UID: \"130bfdb3-450f-4be3-a49c-c851c1235afa\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-e86d9e-9q5qj"
Sep 16 02:46:48.288151 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:46:48.288126    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/130bfdb3-450f-4be3-a49c-c851c1235afa-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w1-e86d9e-9q5qj\" (UID: \"130bfdb3-450f-4be3-a49c-c851c1235afa\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-e86d9e-9q5qj"
Sep 16 02:46:48.366256 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:46:48.366228    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-e86d9e-9q5qj"
Sep 16 02:46:48.637109 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:46:48.637031    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-e86d9e-9q5qj"]
Sep 16 02:46:48.640323 ip-10-0-128-111 kubenswrapper[2583]: W0916 02:46:48.640294    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod130bfdb3_450f_4be3_a49c_c851c1235afa.slice/crio-9cc2930dd8499b94a943d4ccdb9d467d1289696defcfc22c40aafeabb067ee79 WatchSource:0}: Error finding container 9cc2930dd8499b94a943d4ccdb9d467d1289696defcfc22c40aafeabb067ee79: Status 404 returned error can't find the container with id 9cc2930dd8499b94a943d4ccdb9d467d1289696defcfc22c40aafeabb067ee79
Sep 16 02:46:49.536279 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:46:49.536193    2583 generic.go:356] "Generic (PLEG): container finished" podID="130bfdb3-450f-4be3-a49c-c851c1235afa" containerID="c3dbd299e127eeab3c09f13544d622018c00d589e93603eec9ce35b6a2039d3b" exitCode=0
Sep 16 02:46:49.536279 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:46:49.536262    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-e86d9e-9q5qj" event={"ID":"130bfdb3-450f-4be3-a49c-c851c1235afa","Type":"ContainerDied","Data":"c3dbd299e127eeab3c09f13544d622018c00d589e93603eec9ce35b6a2039d3b"}
Sep 16 02:46:49.536720 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:46:49.536315    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-e86d9e-9q5qj" event={"ID":"130bfdb3-450f-4be3-a49c-c851c1235afa","Type":"ContainerStarted","Data":"9cc2930dd8499b94a943d4ccdb9d467d1289696defcfc22c40aafeabb067ee79"}
Sep 16 02:46:50.672801 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:46:50.672779    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-e86d9e-9q5qj"
Sep 16 02:46:50.810679 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:46:50.810595    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/130bfdb3-450f-4be3-a49c-c851c1235afa-maas-api-service-ca\") pod \"130bfdb3-450f-4be3-a49c-c851c1235afa\" (UID: \"130bfdb3-450f-4be3-a49c-c851c1235afa\") "
Sep 16 02:46:50.810958 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:46:50.810938    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/130bfdb3-450f-4be3-a49c-c851c1235afa-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "130bfdb3-450f-4be3-a49c-c851c1235afa" (UID: "130bfdb3-450f-4be3-a49c-c851c1235afa"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 02:46:50.912066 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:46:50.912031    2583 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/130bfdb3-450f-4be3-a49c-c851c1235afa-maas-api-service-ca\") on node \"ip-10-0-128-111.ec2.internal\" DevicePath \"\""
Sep 16 02:46:51.548058 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:46:51.548028    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-e86d9e-9q5qj"
Sep 16 02:46:51.548058 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:46:51.548055    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-e86d9e-9q5qj" event={"ID":"130bfdb3-450f-4be3-a49c-c851c1235afa","Type":"ContainerDied","Data":"9cc2930dd8499b94a943d4ccdb9d467d1289696defcfc22c40aafeabb067ee79"}
Sep 16 02:46:51.548278 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:46:51.548096    2583 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9cc2930dd8499b94a943d4ccdb9d467d1289696defcfc22c40aafeabb067ee79"
Sep 16 02:46:58.765016 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:46:58.764978    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-e86d9e-9q5qj"]
Sep 16 02:46:58.767631 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:46:58.767606    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-e86d9e-9q5qj"]
Sep 16 02:46:58.952050 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:46:58.952015    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-fzl8t"]
Sep 16 02:47:00.045502 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:47:00.045467    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="130bfdb3-450f-4be3-a49c-c851c1235afa" path="/var/lib/kubelet/pods/130bfdb3-450f-4be3-a49c-c851c1235afa/volumes"
Sep 16 02:47:05.394758 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:47:05.394675    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-e86d9e-openshift-default-65bf4ddbdd-jnrbq"]
Sep 16 02:47:05.395170 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:47:05.394925    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w1-e86d9e-openshift-default-65bf4ddbdd-jnrbq" podUID="1490b32b-6d61-404a-ac86-f902766a0cfb" containerName="istio-proxy" containerID="cri-o://377984d084144614c81b3a83f3ef8f32c780f408f84ccb7490fc6338a658b88a" gracePeriod=30
Sep 16 02:47:05.434812 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:47:05.434775    2583 patch_prober.go:28] interesting pod/e2e-shared-b-w1-e86d9e-openshift-default-65bf4ddbdd-jnrbq container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Sep 16 02:47:05.434971 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:47:05.434841    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-w1-e86d9e-openshift-default-65bf4ddbdd-jnrbq" podUID="1490b32b-6d61-404a-ac86-f902766a0cfb" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 16 02:47:08.251603 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:47:08.251562    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-fzl8t"]
Sep 16 02:47:10.641589 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:47:10.641562    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-e86d9e-openshift-default-65bf4ddbdd-jnrbq"
Sep 16 02:47:10.647459 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:47:10.647434    2583 generic.go:356] "Generic (PLEG): container finished" podID="1490b32b-6d61-404a-ac86-f902766a0cfb" containerID="377984d084144614c81b3a83f3ef8f32c780f408f84ccb7490fc6338a658b88a" exitCode=0
Sep 16 02:47:10.647584 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:47:10.647473    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-e86d9e-openshift-default-65bf4ddbdd-jnrbq"
Sep 16 02:47:10.647584 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:47:10.647514    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-e86d9e-openshift-default-65bf4ddbdd-jnrbq" event={"ID":"1490b32b-6d61-404a-ac86-f902766a0cfb","Type":"ContainerDied","Data":"377984d084144614c81b3a83f3ef8f32c780f408f84ccb7490fc6338a658b88a"}
Sep 16 02:47:10.647584 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:47:10.647570    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-e86d9e-openshift-default-65bf4ddbdd-jnrbq" event={"ID":"1490b32b-6d61-404a-ac86-f902766a0cfb","Type":"ContainerDied","Data":"ddae25ba735eb894babf6a228ac6a23b5ffdda3ffd318fd136a8b579663ee572"}
Sep 16 02:47:10.647698 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:47:10.647593    2583 scope.go:117] "RemoveContainer" containerID="377984d084144614c81b3a83f3ef8f32c780f408f84ccb7490fc6338a658b88a"
Sep 16 02:47:10.655987 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:47:10.655965    2583 scope.go:117] "RemoveContainer" containerID="caf2a731e53b6280980decc1fa9f3283e0a18c56dbb979f680229687a00edad3"
Sep 16 02:47:10.665316 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:47:10.665289    2583 scope.go:117] "RemoveContainer" containerID="377984d084144614c81b3a83f3ef8f32c780f408f84ccb7490fc6338a658b88a"
Sep 16 02:47:10.665709 ip-10-0-128-111 kubenswrapper[2583]: E0916 02:47:10.665685    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"377984d084144614c81b3a83f3ef8f32c780f408f84ccb7490fc6338a658b88a\": container with ID starting with 377984d084144614c81b3a83f3ef8f32c780f408f84ccb7490fc6338a658b88a not found: ID does not exist" containerID="377984d084144614c81b3a83f3ef8f32c780f408f84ccb7490fc6338a658b88a"
Sep 16 02:47:10.665825 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:47:10.665716    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"377984d084144614c81b3a83f3ef8f32c780f408f84ccb7490fc6338a658b88a"} err="failed to get container status \"377984d084144614c81b3a83f3ef8f32c780f408f84ccb7490fc6338a658b88a\": rpc error: code = NotFound desc = could not find container \"377984d084144614c81b3a83f3ef8f32c780f408f84ccb7490fc6338a658b88a\": container with ID starting with 377984d084144614c81b3a83f3ef8f32c780f408f84ccb7490fc6338a658b88a not found: ID does not exist"
Sep 16 02:47:10.665825 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:47:10.665735    2583 scope.go:117] "RemoveContainer" containerID="caf2a731e53b6280980decc1fa9f3283e0a18c56dbb979f680229687a00edad3"
Sep 16 02:47:10.666086 ip-10-0-128-111 kubenswrapper[2583]: E0916 02:47:10.666063    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"caf2a731e53b6280980decc1fa9f3283e0a18c56dbb979f680229687a00edad3\": container with ID starting with caf2a731e53b6280980decc1fa9f3283e0a18c56dbb979f680229687a00edad3 not found: ID does not exist" containerID="caf2a731e53b6280980decc1fa9f3283e0a18c56dbb979f680229687a00edad3"
Sep 16 02:47:10.666161 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:47:10.666090    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"caf2a731e53b6280980decc1fa9f3283e0a18c56dbb979f680229687a00edad3"} err="failed to get container status \"caf2a731e53b6280980decc1fa9f3283e0a18c56dbb979f680229687a00edad3\": rpc error: code = NotFound desc = could not find container \"caf2a731e53b6280980decc1fa9f3283e0a18c56dbb979f680229687a00edad3\": container with ID starting with caf2a731e53b6280980decc1fa9f3283e0a18c56dbb979f680229687a00edad3 not found: ID does not exist"
Sep 16 02:47:10.702266 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:47:10.702237    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1490b32b-6d61-404a-ac86-f902766a0cfb-istio-token\") pod \"1490b32b-6d61-404a-ac86-f902766a0cfb\" (UID: \"1490b32b-6d61-404a-ac86-f902766a0cfb\") "
Sep 16 02:47:10.702415 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:47:10.702285    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1490b32b-6d61-404a-ac86-f902766a0cfb-credential-socket\") pod \"1490b32b-6d61-404a-ac86-f902766a0cfb\" (UID: \"1490b32b-6d61-404a-ac86-f902766a0cfb\") "
Sep 16 02:47:10.702415 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:47:10.702329    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1490b32b-6d61-404a-ac86-f902766a0cfb-istio-podinfo\") pod \"1490b32b-6d61-404a-ac86-f902766a0cfb\" (UID: \"1490b32b-6d61-404a-ac86-f902766a0cfb\") "
Sep 16 02:47:10.702415 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:47:10.702347    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1490b32b-6d61-404a-ac86-f902766a0cfb-istio-envoy\") pod \"1490b32b-6d61-404a-ac86-f902766a0cfb\" (UID: \"1490b32b-6d61-404a-ac86-f902766a0cfb\") "
Sep 16 02:47:10.702415 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:47:10.702364    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-p654j\" (UniqueName: \"kubernetes.io/projected/1490b32b-6d61-404a-ac86-f902766a0cfb-kube-api-access-p654j\") pod \"1490b32b-6d61-404a-ac86-f902766a0cfb\" (UID: \"1490b32b-6d61-404a-ac86-f902766a0cfb\") "
Sep 16 02:47:10.702415 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:47:10.702394    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1490b32b-6d61-404a-ac86-f902766a0cfb-istiod-ca-cert\") pod \"1490b32b-6d61-404a-ac86-f902766a0cfb\" (UID: \"1490b32b-6d61-404a-ac86-f902766a0cfb\") "
Sep 16 02:47:10.702758 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:47:10.702428    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1490b32b-6d61-404a-ac86-f902766a0cfb-workload-socket\") pod \"1490b32b-6d61-404a-ac86-f902766a0cfb\" (UID: \"1490b32b-6d61-404a-ac86-f902766a0cfb\") "
Sep 16 02:47:10.702758 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:47:10.702477    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1490b32b-6d61-404a-ac86-f902766a0cfb-workload-certs\") pod \"1490b32b-6d61-404a-ac86-f902766a0cfb\" (UID: \"1490b32b-6d61-404a-ac86-f902766a0cfb\") "
Sep 16 02:47:10.702758 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:47:10.702519    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1490b32b-6d61-404a-ac86-f902766a0cfb-istio-data\") pod \"1490b32b-6d61-404a-ac86-f902766a0cfb\" (UID: \"1490b32b-6d61-404a-ac86-f902766a0cfb\") "
Sep 16 02:47:10.702758 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:47:10.702646    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1490b32b-6d61-404a-ac86-f902766a0cfb-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "1490b32b-6d61-404a-ac86-f902766a0cfb" (UID: "1490b32b-6d61-404a-ac86-f902766a0cfb"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 02:47:10.702969 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:47:10.702859    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1490b32b-6d61-404a-ac86-f902766a0cfb-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "1490b32b-6d61-404a-ac86-f902766a0cfb" (UID: "1490b32b-6d61-404a-ac86-f902766a0cfb"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 02:47:10.702969 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:47:10.702897    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1490b32b-6d61-404a-ac86-f902766a0cfb-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "1490b32b-6d61-404a-ac86-f902766a0cfb" (UID: "1490b32b-6d61-404a-ac86-f902766a0cfb"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 02:47:10.702969 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:47:10.702921    2583 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1490b32b-6d61-404a-ac86-f902766a0cfb-credential-socket\") on node \"ip-10-0-128-111.ec2.internal\" DevicePath \"\""
Sep 16 02:47:10.703131 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:47:10.703054    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1490b32b-6d61-404a-ac86-f902766a0cfb-istio-data" (OuterVolumeSpecName: "istio-data") pod "1490b32b-6d61-404a-ac86-f902766a0cfb" (UID: "1490b32b-6d61-404a-ac86-f902766a0cfb"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 02:47:10.703131 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:47:10.703094    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1490b32b-6d61-404a-ac86-f902766a0cfb-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "1490b32b-6d61-404a-ac86-f902766a0cfb" (UID: "1490b32b-6d61-404a-ac86-f902766a0cfb"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 02:47:10.704613 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:47:10.704587    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1490b32b-6d61-404a-ac86-f902766a0cfb-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "1490b32b-6d61-404a-ac86-f902766a0cfb" (UID: "1490b32b-6d61-404a-ac86-f902766a0cfb"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 02:47:10.704718 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:47:10.704650    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1490b32b-6d61-404a-ac86-f902766a0cfb-kube-api-access-p654j" (OuterVolumeSpecName: "kube-api-access-p654j") pod "1490b32b-6d61-404a-ac86-f902766a0cfb" (UID: "1490b32b-6d61-404a-ac86-f902766a0cfb"). InnerVolumeSpecName "kube-api-access-p654j". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 02:47:10.704903 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:47:10.704880    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1490b32b-6d61-404a-ac86-f902766a0cfb-istio-token" (OuterVolumeSpecName: "istio-token") pod "1490b32b-6d61-404a-ac86-f902766a0cfb" (UID: "1490b32b-6d61-404a-ac86-f902766a0cfb"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 02:47:10.704960 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:47:10.704949    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/1490b32b-6d61-404a-ac86-f902766a0cfb-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "1490b32b-6d61-404a-ac86-f902766a0cfb" (UID: "1490b32b-6d61-404a-ac86-f902766a0cfb"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 16 02:47:10.804392 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:47:10.804349    2583 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1490b32b-6d61-404a-ac86-f902766a0cfb-istio-token\") on node \"ip-10-0-128-111.ec2.internal\" DevicePath \"\""
Sep 16 02:47:10.804392 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:47:10.804383    2583 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1490b32b-6d61-404a-ac86-f902766a0cfb-istio-podinfo\") on node \"ip-10-0-128-111.ec2.internal\" DevicePath \"\""
Sep 16 02:47:10.804392 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:47:10.804396    2583 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1490b32b-6d61-404a-ac86-f902766a0cfb-istio-envoy\") on node \"ip-10-0-128-111.ec2.internal\" DevicePath \"\""
Sep 16 02:47:10.804680 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:47:10.804409    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p654j\" (UniqueName: \"kubernetes.io/projected/1490b32b-6d61-404a-ac86-f902766a0cfb-kube-api-access-p654j\") on node \"ip-10-0-128-111.ec2.internal\" DevicePath \"\""
Sep 16 02:47:10.804680 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:47:10.804421    2583 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1490b32b-6d61-404a-ac86-f902766a0cfb-istiod-ca-cert\") on node \"ip-10-0-128-111.ec2.internal\" DevicePath \"\""
Sep 16 02:47:10.804680 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:47:10.804432    2583 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1490b32b-6d61-404a-ac86-f902766a0cfb-workload-socket\") on node \"ip-10-0-128-111.ec2.internal\" DevicePath \"\""
Sep 16 02:47:10.804680 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:47:10.804443    2583 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1490b32b-6d61-404a-ac86-f902766a0cfb-workload-certs\") on node \"ip-10-0-128-111.ec2.internal\" DevicePath \"\""
Sep 16 02:47:10.804680 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:47:10.804453    2583 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1490b32b-6d61-404a-ac86-f902766a0cfb-istio-data\") on node \"ip-10-0-128-111.ec2.internal\" DevicePath \"\""
Sep 16 02:47:10.968721 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:47:10.968693    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-e86d9e-openshift-default-65bf4ddbdd-jnrbq"]
Sep 16 02:47:10.971789 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:47:10.971762    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-e86d9e-openshift-default-65bf4ddbdd-jnrbq"]
Sep 16 02:47:12.045739 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:47:12.045704    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1490b32b-6d61-404a-ac86-f902766a0cfb" path="/var/lib/kubelet/pods/1490b32b-6d61-404a-ac86-f902766a0cfb/volumes"
Sep 16 02:47:42.511146 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:47:42.511108    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-4fe974-5c5d79b89c-jfqrs"]
Sep 16 02:47:42.511661 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:47:42.511361    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-4fe974-5c5d79b89c-jfqrs" podUID="a18ebddf-5578-4752-852a-3dcfe8eb881b" containerName="payload-processing" containerID="cri-o://01e59d1d2026dc539ddbd2a39a32f8f42b51a44bfb204d4faaa55022110c692a" gracePeriod=30
Sep 16 02:47:42.519116 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:47:42.518697    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-4fe974-659977f8c4-926tc"]
Sep 16 02:47:42.519116 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:47:42.518944    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-4fe974-659977f8c4-926tc" podUID="7d492f3f-cf1b-4902-b111-4c0f248d6fe1" containerName="payload-pre-processing" containerID="cri-o://54d8eb8b89d8f12cae91787ed97d35b6e1ffb1c0cabbacc9e3c061b57f92e2d0" gracePeriod=30
Sep 16 02:47:42.564584 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:47:42.564466    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-ipp-b-4fe974-5c5d79b89c-jfqrs" podUID="a18ebddf-5578-4752-852a-3dcfe8eb881b" containerName="payload-processing" probeResult="failure" output="dial tcp 10.134.0.69:9004: connect: connection refused"
Sep 16 02:47:42.564584 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:47:42.564477    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-4fe974-659977f8c4-926tc" podUID="7d492f3f-cf1b-4902-b111-4c0f248d6fe1" containerName="payload-pre-processing" probeResult="failure" output="dial tcp 10.134.0.68:9004: connect: connection refused"
Sep 16 02:47:42.804986 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:47:42.804956    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-4fe974-5c5d79b89c-jfqrs"
Sep 16 02:47:42.808193 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:47:42.808173    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-4fe974-659977f8c4-926tc"
Sep 16 02:47:42.808712 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:47:42.808689    2583 generic.go:356] "Generic (PLEG): container finished" podID="a18ebddf-5578-4752-852a-3dcfe8eb881b" containerID="01e59d1d2026dc539ddbd2a39a32f8f42b51a44bfb204d4faaa55022110c692a" exitCode=0
Sep 16 02:47:42.808821 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:47:42.808718    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-4fe974-5c5d79b89c-jfqrs"
Sep 16 02:47:42.808821 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:47:42.808771    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-4fe974-5c5d79b89c-jfqrs" event={"ID":"a18ebddf-5578-4752-852a-3dcfe8eb881b","Type":"ContainerDied","Data":"01e59d1d2026dc539ddbd2a39a32f8f42b51a44bfb204d4faaa55022110c692a"}
Sep 16 02:47:42.808911 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:47:42.808824    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-4fe974-5c5d79b89c-jfqrs" event={"ID":"a18ebddf-5578-4752-852a-3dcfe8eb881b","Type":"ContainerDied","Data":"55528f2447a525d7cae1c313034c488a91fcf171b20f3fb63a0aab4f7bf93fee"}
Sep 16 02:47:42.808911 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:47:42.808846    2583 scope.go:117] "RemoveContainer" containerID="01e59d1d2026dc539ddbd2a39a32f8f42b51a44bfb204d4faaa55022110c692a"
Sep 16 02:47:42.810244 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:47:42.810221    2583 generic.go:356] "Generic (PLEG): container finished" podID="7d492f3f-cf1b-4902-b111-4c0f248d6fe1" containerID="54d8eb8b89d8f12cae91787ed97d35b6e1ffb1c0cabbacc9e3c061b57f92e2d0" exitCode=0
Sep 16 02:47:42.810330 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:47:42.810253    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-4fe974-659977f8c4-926tc"
Sep 16 02:47:42.810330 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:47:42.810297    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-4fe974-659977f8c4-926tc" event={"ID":"7d492f3f-cf1b-4902-b111-4c0f248d6fe1","Type":"ContainerDied","Data":"54d8eb8b89d8f12cae91787ed97d35b6e1ffb1c0cabbacc9e3c061b57f92e2d0"}
Sep 16 02:47:42.810408 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:47:42.810335    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-4fe974-659977f8c4-926tc" event={"ID":"7d492f3f-cf1b-4902-b111-4c0f248d6fe1","Type":"ContainerDied","Data":"21bd20969d1770036b6376964db22c19c8b60069900c4e299d99a4f2310d7fd2"}
Sep 16 02:47:42.817069 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:47:42.817054    2583 scope.go:117] "RemoveContainer" containerID="01e59d1d2026dc539ddbd2a39a32f8f42b51a44bfb204d4faaa55022110c692a"
Sep 16 02:47:42.817339 ip-10-0-128-111 kubenswrapper[2583]: E0916 02:47:42.817319    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"01e59d1d2026dc539ddbd2a39a32f8f42b51a44bfb204d4faaa55022110c692a\": container with ID starting with 01e59d1d2026dc539ddbd2a39a32f8f42b51a44bfb204d4faaa55022110c692a not found: ID does not exist" containerID="01e59d1d2026dc539ddbd2a39a32f8f42b51a44bfb204d4faaa55022110c692a"
Sep 16 02:47:42.817390 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:47:42.817346    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"01e59d1d2026dc539ddbd2a39a32f8f42b51a44bfb204d4faaa55022110c692a"} err="failed to get container status \"01e59d1d2026dc539ddbd2a39a32f8f42b51a44bfb204d4faaa55022110c692a\": rpc error: code = NotFound desc = could not find container \"01e59d1d2026dc539ddbd2a39a32f8f42b51a44bfb204d4faaa55022110c692a\": container with ID starting with 01e59d1d2026dc539ddbd2a39a32f8f42b51a44bfb204d4faaa55022110c692a not found: ID does not exist"
Sep 16 02:47:42.817390 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:47:42.817363    2583 scope.go:117] "RemoveContainer" containerID="54d8eb8b89d8f12cae91787ed97d35b6e1ffb1c0cabbacc9e3c061b57f92e2d0"
Sep 16 02:47:42.826727 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:47:42.826710    2583 scope.go:117] "RemoveContainer" containerID="54d8eb8b89d8f12cae91787ed97d35b6e1ffb1c0cabbacc9e3c061b57f92e2d0"
Sep 16 02:47:42.827020 ip-10-0-128-111 kubenswrapper[2583]: E0916 02:47:42.826997    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"54d8eb8b89d8f12cae91787ed97d35b6e1ffb1c0cabbacc9e3c061b57f92e2d0\": container with ID starting with 54d8eb8b89d8f12cae91787ed97d35b6e1ffb1c0cabbacc9e3c061b57f92e2d0 not found: ID does not exist" containerID="54d8eb8b89d8f12cae91787ed97d35b6e1ffb1c0cabbacc9e3c061b57f92e2d0"
Sep 16 02:47:42.827088 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:47:42.827029    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"54d8eb8b89d8f12cae91787ed97d35b6e1ffb1c0cabbacc9e3c061b57f92e2d0"} err="failed to get container status \"54d8eb8b89d8f12cae91787ed97d35b6e1ffb1c0cabbacc9e3c061b57f92e2d0\": rpc error: code = NotFound desc = could not find container \"54d8eb8b89d8f12cae91787ed97d35b6e1ffb1c0cabbacc9e3c061b57f92e2d0\": container with ID starting with 54d8eb8b89d8f12cae91787ed97d35b6e1ffb1c0cabbacc9e3c061b57f92e2d0 not found: ID does not exist"
Sep 16 02:47:42.900855 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:47:42.900825    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a18ebddf-5578-4752-852a-3dcfe8eb881b-plugins-config-volume\") pod \"a18ebddf-5578-4752-852a-3dcfe8eb881b\" (UID: \"a18ebddf-5578-4752-852a-3dcfe8eb881b\") "
Sep 16 02:47:42.901016 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:47:42.900882    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7j8pp\" (UniqueName: \"kubernetes.io/projected/7d492f3f-cf1b-4902-b111-4c0f248d6fe1-kube-api-access-7j8pp\") pod \"7d492f3f-cf1b-4902-b111-4c0f248d6fe1\" (UID: \"7d492f3f-cf1b-4902-b111-4c0f248d6fe1\") "
Sep 16 02:47:42.901016 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:47:42.900966    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7d492f3f-cf1b-4902-b111-4c0f248d6fe1-plugins-config-volume\") pod \"7d492f3f-cf1b-4902-b111-4c0f248d6fe1\" (UID: \"7d492f3f-cf1b-4902-b111-4c0f248d6fe1\") "
Sep 16 02:47:42.901016 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:47:42.900993    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qdxq5\" (UniqueName: \"kubernetes.io/projected/a18ebddf-5578-4752-852a-3dcfe8eb881b-kube-api-access-qdxq5\") pod \"a18ebddf-5578-4752-852a-3dcfe8eb881b\" (UID: \"a18ebddf-5578-4752-852a-3dcfe8eb881b\") "
Sep 16 02:47:42.901281 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:47:42.901252    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a18ebddf-5578-4752-852a-3dcfe8eb881b-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "a18ebddf-5578-4752-852a-3dcfe8eb881b" (UID: "a18ebddf-5578-4752-852a-3dcfe8eb881b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 02:47:42.901362 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:47:42.901343    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7d492f3f-cf1b-4902-b111-4c0f248d6fe1-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "7d492f3f-cf1b-4902-b111-4c0f248d6fe1" (UID: "7d492f3f-cf1b-4902-b111-4c0f248d6fe1"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 02:47:42.903078 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:47:42.903055    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7d492f3f-cf1b-4902-b111-4c0f248d6fe1-kube-api-access-7j8pp" (OuterVolumeSpecName: "kube-api-access-7j8pp") pod "7d492f3f-cf1b-4902-b111-4c0f248d6fe1" (UID: "7d492f3f-cf1b-4902-b111-4c0f248d6fe1"). InnerVolumeSpecName "kube-api-access-7j8pp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 02:47:42.903078 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:47:42.903072    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a18ebddf-5578-4752-852a-3dcfe8eb881b-kube-api-access-qdxq5" (OuterVolumeSpecName: "kube-api-access-qdxq5") pod "a18ebddf-5578-4752-852a-3dcfe8eb881b" (UID: "a18ebddf-5578-4752-852a-3dcfe8eb881b"). InnerVolumeSpecName "kube-api-access-qdxq5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 02:47:43.002694 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:47:43.002660    2583 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a18ebddf-5578-4752-852a-3dcfe8eb881b-plugins-config-volume\") on node \"ip-10-0-128-111.ec2.internal\" DevicePath \"\""
Sep 16 02:47:43.002694 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:47:43.002688    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7j8pp\" (UniqueName: \"kubernetes.io/projected/7d492f3f-cf1b-4902-b111-4c0f248d6fe1-kube-api-access-7j8pp\") on node \"ip-10-0-128-111.ec2.internal\" DevicePath \"\""
Sep 16 02:47:43.002694 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:47:43.002697    2583 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7d492f3f-cf1b-4902-b111-4c0f248d6fe1-plugins-config-volume\") on node \"ip-10-0-128-111.ec2.internal\" DevicePath \"\""
Sep 16 02:47:43.002923 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:47:43.002710    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qdxq5\" (UniqueName: \"kubernetes.io/projected/a18ebddf-5578-4752-852a-3dcfe8eb881b-kube-api-access-qdxq5\") on node \"ip-10-0-128-111.ec2.internal\" DevicePath \"\""
Sep 16 02:47:43.135630 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:47:43.135601    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-4fe974-5c5d79b89c-jfqrs"]
Sep 16 02:47:43.151601 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:47:43.151577    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-4fe974-5c5d79b89c-jfqrs"]
Sep 16 02:47:43.181770 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:47:43.181735    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-4fe974-659977f8c4-926tc"]
Sep 16 02:47:43.187207 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:47:43.187185    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-4fe974-659977f8c4-926tc"]
Sep 16 02:47:44.045134 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:47:44.045099    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7d492f3f-cf1b-4902-b111-4c0f248d6fe1" path="/var/lib/kubelet/pods/7d492f3f-cf1b-4902-b111-4c0f248d6fe1/volumes"
Sep 16 02:47:44.045511 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:47:44.045458    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a18ebddf-5578-4752-852a-3dcfe8eb881b" path="/var/lib/kubelet/pods/a18ebddf-5578-4752-852a-3dcfe8eb881b/volumes"
Sep 16 02:47:45.649757 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:47:45.649724    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-fzl8t"]
Sep 16 02:47:48.925899 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:47:48.925869    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-4fe974-openshift-default-59b7dbcd46-cpkvz"]
Sep 16 02:47:48.926314 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:47:48.926088    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-b-4fe974-openshift-default-59b7dbcd46-cpkvz" podUID="4b2c7d51-9fcb-4bae-a99c-66ad0a8e62eb" containerName="istio-proxy" containerID="cri-o://8002db96b2b0dc6ea48f3e73037be4a28fa3d3fcf1ece34705154668d7f21e72" gracePeriod=30
Sep 16 02:47:53.943610 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:47:53.943571    2583 patch_prober.go:28] interesting pod/e2e-ipp-b-4fe974-openshift-default-59b7dbcd46-cpkvz container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" start-of-body=
Sep 16 02:47:53.944027 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:47:53.943635    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-b-4fe974-openshift-default-59b7dbcd46-cpkvz" podUID="4b2c7d51-9fcb-4bae-a99c-66ad0a8e62eb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused"
Sep 16 02:47:54.176023 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:47:54.175998    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-4fe974-openshift-default-59b7dbcd46-cpkvz"
Sep 16 02:47:54.304229 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:47:54.304145    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4b2c7d51-9fcb-4bae-a99c-66ad0a8e62eb-workload-certs\") pod \"4b2c7d51-9fcb-4bae-a99c-66ad0a8e62eb\" (UID: \"4b2c7d51-9fcb-4bae-a99c-66ad0a8e62eb\") "
Sep 16 02:47:54.304229 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:47:54.304186    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4b2c7d51-9fcb-4bae-a99c-66ad0a8e62eb-istio-data\") pod \"4b2c7d51-9fcb-4bae-a99c-66ad0a8e62eb\" (UID: \"4b2c7d51-9fcb-4bae-a99c-66ad0a8e62eb\") "
Sep 16 02:47:54.304435 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:47:54.304239    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4b2c7d51-9fcb-4bae-a99c-66ad0a8e62eb-istio-podinfo\") pod \"4b2c7d51-9fcb-4bae-a99c-66ad0a8e62eb\" (UID: \"4b2c7d51-9fcb-4bae-a99c-66ad0a8e62eb\") "
Sep 16 02:47:54.304435 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:47:54.304264    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4b2c7d51-9fcb-4bae-a99c-66ad0a8e62eb-istiod-ca-cert\") pod \"4b2c7d51-9fcb-4bae-a99c-66ad0a8e62eb\" (UID: \"4b2c7d51-9fcb-4bae-a99c-66ad0a8e62eb\") "
Sep 16 02:47:54.304435 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:47:54.304279    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4b2c7d51-9fcb-4bae-a99c-66ad0a8e62eb-istio-envoy\") pod \"4b2c7d51-9fcb-4bae-a99c-66ad0a8e62eb\" (UID: \"4b2c7d51-9fcb-4bae-a99c-66ad0a8e62eb\") "
Sep 16 02:47:54.304435 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:47:54.304323    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4b2c7d51-9fcb-4bae-a99c-66ad0a8e62eb-credential-socket\") pod \"4b2c7d51-9fcb-4bae-a99c-66ad0a8e62eb\" (UID: \"4b2c7d51-9fcb-4bae-a99c-66ad0a8e62eb\") "
Sep 16 02:47:54.304696 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:47:54.304448    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4b2c7d51-9fcb-4bae-a99c-66ad0a8e62eb-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "4b2c7d51-9fcb-4bae-a99c-66ad0a8e62eb" (UID: "4b2c7d51-9fcb-4bae-a99c-66ad0a8e62eb"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 02:47:54.304696 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:47:54.304454    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4b2c7d51-9fcb-4bae-a99c-66ad0a8e62eb-istio-token\") pod \"4b2c7d51-9fcb-4bae-a99c-66ad0a8e62eb\" (UID: \"4b2c7d51-9fcb-4bae-a99c-66ad0a8e62eb\") "
Sep 16 02:47:54.304696 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:47:54.304605    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-b97v2\" (UniqueName: \"kubernetes.io/projected/4b2c7d51-9fcb-4bae-a99c-66ad0a8e62eb-kube-api-access-b97v2\") pod \"4b2c7d51-9fcb-4bae-a99c-66ad0a8e62eb\" (UID: \"4b2c7d51-9fcb-4bae-a99c-66ad0a8e62eb\") "
Sep 16 02:47:54.304696 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:47:54.304625    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4b2c7d51-9fcb-4bae-a99c-66ad0a8e62eb-istio-data" (OuterVolumeSpecName: "istio-data") pod "4b2c7d51-9fcb-4bae-a99c-66ad0a8e62eb" (UID: "4b2c7d51-9fcb-4bae-a99c-66ad0a8e62eb"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 02:47:54.304696 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:47:54.304637    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4b2c7d51-9fcb-4bae-a99c-66ad0a8e62eb-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "4b2c7d51-9fcb-4bae-a99c-66ad0a8e62eb" (UID: "4b2c7d51-9fcb-4bae-a99c-66ad0a8e62eb"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 02:47:54.304696 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:47:54.304643    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4b2c7d51-9fcb-4bae-a99c-66ad0a8e62eb-workload-socket\") pod \"4b2c7d51-9fcb-4bae-a99c-66ad0a8e62eb\" (UID: \"4b2c7d51-9fcb-4bae-a99c-66ad0a8e62eb\") "
Sep 16 02:47:54.305003 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:47:54.304721    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4b2c7d51-9fcb-4bae-a99c-66ad0a8e62eb-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "4b2c7d51-9fcb-4bae-a99c-66ad0a8e62eb" (UID: "4b2c7d51-9fcb-4bae-a99c-66ad0a8e62eb"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 02:47:54.305003 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:47:54.304867    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4b2c7d51-9fcb-4bae-a99c-66ad0a8e62eb-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "4b2c7d51-9fcb-4bae-a99c-66ad0a8e62eb" (UID: "4b2c7d51-9fcb-4bae-a99c-66ad0a8e62eb"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 02:47:54.305116 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:47:54.305003    2583 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4b2c7d51-9fcb-4bae-a99c-66ad0a8e62eb-credential-socket\") on node \"ip-10-0-128-111.ec2.internal\" DevicePath \"\""
Sep 16 02:47:54.305116 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:47:54.305024    2583 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4b2c7d51-9fcb-4bae-a99c-66ad0a8e62eb-workload-socket\") on node \"ip-10-0-128-111.ec2.internal\" DevicePath \"\""
Sep 16 02:47:54.305116 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:47:54.305040    2583 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4b2c7d51-9fcb-4bae-a99c-66ad0a8e62eb-workload-certs\") on node \"ip-10-0-128-111.ec2.internal\" DevicePath \"\""
Sep 16 02:47:54.305116 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:47:54.305053    2583 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4b2c7d51-9fcb-4bae-a99c-66ad0a8e62eb-istio-data\") on node \"ip-10-0-128-111.ec2.internal\" DevicePath \"\""
Sep 16 02:47:54.305116 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:47:54.305065    2583 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4b2c7d51-9fcb-4bae-a99c-66ad0a8e62eb-istiod-ca-cert\") on node \"ip-10-0-128-111.ec2.internal\" DevicePath \"\""
Sep 16 02:47:54.306497 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:47:54.306475    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4b2c7d51-9fcb-4bae-a99c-66ad0a8e62eb-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "4b2c7d51-9fcb-4bae-a99c-66ad0a8e62eb" (UID: "4b2c7d51-9fcb-4bae-a99c-66ad0a8e62eb"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 02:47:54.306626 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:47:54.306505    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/4b2c7d51-9fcb-4bae-a99c-66ad0a8e62eb-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "4b2c7d51-9fcb-4bae-a99c-66ad0a8e62eb" (UID: "4b2c7d51-9fcb-4bae-a99c-66ad0a8e62eb"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 16 02:47:54.306626 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:47:54.306599    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4b2c7d51-9fcb-4bae-a99c-66ad0a8e62eb-istio-token" (OuterVolumeSpecName: "istio-token") pod "4b2c7d51-9fcb-4bae-a99c-66ad0a8e62eb" (UID: "4b2c7d51-9fcb-4bae-a99c-66ad0a8e62eb"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 02:47:54.306701 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:47:54.306631    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4b2c7d51-9fcb-4bae-a99c-66ad0a8e62eb-kube-api-access-b97v2" (OuterVolumeSpecName: "kube-api-access-b97v2") pod "4b2c7d51-9fcb-4bae-a99c-66ad0a8e62eb" (UID: "4b2c7d51-9fcb-4bae-a99c-66ad0a8e62eb"). InnerVolumeSpecName "kube-api-access-b97v2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 02:47:54.405984 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:47:54.405954    2583 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4b2c7d51-9fcb-4bae-a99c-66ad0a8e62eb-istio-podinfo\") on node \"ip-10-0-128-111.ec2.internal\" DevicePath \"\""
Sep 16 02:47:54.405984 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:47:54.405979    2583 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4b2c7d51-9fcb-4bae-a99c-66ad0a8e62eb-istio-envoy\") on node \"ip-10-0-128-111.ec2.internal\" DevicePath \"\""
Sep 16 02:47:54.405984 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:47:54.405991    2583 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4b2c7d51-9fcb-4bae-a99c-66ad0a8e62eb-istio-token\") on node \"ip-10-0-128-111.ec2.internal\" DevicePath \"\""
Sep 16 02:47:54.406208 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:47:54.406004    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b97v2\" (UniqueName: \"kubernetes.io/projected/4b2c7d51-9fcb-4bae-a99c-66ad0a8e62eb-kube-api-access-b97v2\") on node \"ip-10-0-128-111.ec2.internal\" DevicePath \"\""
Sep 16 02:47:54.882887 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:47:54.882846    2583 generic.go:356] "Generic (PLEG): container finished" podID="4b2c7d51-9fcb-4bae-a99c-66ad0a8e62eb" containerID="8002db96b2b0dc6ea48f3e73037be4a28fa3d3fcf1ece34705154668d7f21e72" exitCode=0
Sep 16 02:47:54.883089 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:47:54.882897    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-4fe974-openshift-default-59b7dbcd46-cpkvz"
Sep 16 02:47:54.883089 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:47:54.882931    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-4fe974-openshift-default-59b7dbcd46-cpkvz" event={"ID":"4b2c7d51-9fcb-4bae-a99c-66ad0a8e62eb","Type":"ContainerDied","Data":"8002db96b2b0dc6ea48f3e73037be4a28fa3d3fcf1ece34705154668d7f21e72"}
Sep 16 02:47:54.883089 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:47:54.882987    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-4fe974-openshift-default-59b7dbcd46-cpkvz" event={"ID":"4b2c7d51-9fcb-4bae-a99c-66ad0a8e62eb","Type":"ContainerDied","Data":"8f1cc1b52728c0440cabb5c2632e0a0e4d059a2a27a583ea1f7e799632a2fd6f"}
Sep 16 02:47:54.883089 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:47:54.883010    2583 scope.go:117] "RemoveContainer" containerID="8002db96b2b0dc6ea48f3e73037be4a28fa3d3fcf1ece34705154668d7f21e72"
Sep 16 02:47:54.892021 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:47:54.892007    2583 scope.go:117] "RemoveContainer" containerID="3c8c75e419f914054215a63cf3ddba4923d913959f3d76a9c8aa9af0d19e8adb"
Sep 16 02:47:54.899575 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:47:54.899537    2583 scope.go:117] "RemoveContainer" containerID="8002db96b2b0dc6ea48f3e73037be4a28fa3d3fcf1ece34705154668d7f21e72"
Sep 16 02:47:54.899837 ip-10-0-128-111 kubenswrapper[2583]: E0916 02:47:54.899817    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8002db96b2b0dc6ea48f3e73037be4a28fa3d3fcf1ece34705154668d7f21e72\": container with ID starting with 8002db96b2b0dc6ea48f3e73037be4a28fa3d3fcf1ece34705154668d7f21e72 not found: ID does not exist" containerID="8002db96b2b0dc6ea48f3e73037be4a28fa3d3fcf1ece34705154668d7f21e72"
Sep 16 02:47:54.899899 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:47:54.899850    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8002db96b2b0dc6ea48f3e73037be4a28fa3d3fcf1ece34705154668d7f21e72"} err="failed to get container status \"8002db96b2b0dc6ea48f3e73037be4a28fa3d3fcf1ece34705154668d7f21e72\": rpc error: code = NotFound desc = could not find container \"8002db96b2b0dc6ea48f3e73037be4a28fa3d3fcf1ece34705154668d7f21e72\": container with ID starting with 8002db96b2b0dc6ea48f3e73037be4a28fa3d3fcf1ece34705154668d7f21e72 not found: ID does not exist"
Sep 16 02:47:54.899899 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:47:54.899867    2583 scope.go:117] "RemoveContainer" containerID="3c8c75e419f914054215a63cf3ddba4923d913959f3d76a9c8aa9af0d19e8adb"
Sep 16 02:47:54.900114 ip-10-0-128-111 kubenswrapper[2583]: E0916 02:47:54.900096    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3c8c75e419f914054215a63cf3ddba4923d913959f3d76a9c8aa9af0d19e8adb\": container with ID starting with 3c8c75e419f914054215a63cf3ddba4923d913959f3d76a9c8aa9af0d19e8adb not found: ID does not exist" containerID="3c8c75e419f914054215a63cf3ddba4923d913959f3d76a9c8aa9af0d19e8adb"
Sep 16 02:47:54.900155 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:47:54.900118    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3c8c75e419f914054215a63cf3ddba4923d913959f3d76a9c8aa9af0d19e8adb"} err="failed to get container status \"3c8c75e419f914054215a63cf3ddba4923d913959f3d76a9c8aa9af0d19e8adb\": rpc error: code = NotFound desc = could not find container \"3c8c75e419f914054215a63cf3ddba4923d913959f3d76a9c8aa9af0d19e8adb\": container with ID starting with 3c8c75e419f914054215a63cf3ddba4923d913959f3d76a9c8aa9af0d19e8adb not found: ID does not exist"
Sep 16 02:47:54.903506 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:47:54.903484    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-4fe974-openshift-default-59b7dbcd46-cpkvz"]
Sep 16 02:47:54.906934 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:47:54.906914    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-b-4fe974-openshift-default-59b7dbcd46-cpkvz"]
Sep 16 02:47:56.045391 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:47:56.045354    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4b2c7d51-9fcb-4bae-a99c-66ad0a8e62eb" path="/var/lib/kubelet/pods/4b2c7d51-9fcb-4bae-a99c-66ad0a8e62eb/volumes"
Sep 16 02:47:59.799207 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:47:59.799176    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-db1ed7-29825445-tnkp6"]
Sep 16 02:47:59.802278 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:47:59.802252    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-db1ed7-29825445-tnkp6"]
Sep 16 02:47:59.829755 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:47:59.829728    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-db1ed7-57b4fc7f46-6jjzx"]
Sep 16 02:47:59.829964 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:47:59.829935    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-db1ed7-57b4fc7f46-6jjzx" podUID="a9250574-d9e5-4188-b8db-0fb7a9cf1f09" containerName="payload-pre-processing" containerID="cri-o://a2a42f5a989940fdd33a897fecba063c1d32e61d20711e40a477ff3061724454" gracePeriod=30
Sep 16 02:47:59.907899 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:47:59.907854    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-db1ed7-57b4fc7f46-6jjzx" podUID="a9250574-d9e5-4188-b8db-0fb7a9cf1f09" containerName="payload-pre-processing" probeResult="failure" output="dial tcp 10.134.0.46:9004: connect: connection refused"
Sep 16 02:48:00.060127 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:48:00.060043    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="12bfa699-c47a-495e-8c11-5c0ff7927bf3" path="/var/lib/kubelet/pods/12bfa699-c47a-495e-8c11-5c0ff7927bf3/volumes"
Sep 16 02:48:00.092163 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:48:00.092141    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-db1ed7-57b4fc7f46-6jjzx"
Sep 16 02:48:00.161697 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:48:00.161670    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a9250574-d9e5-4188-b8db-0fb7a9cf1f09-plugins-config-volume\") pod \"a9250574-d9e5-4188-b8db-0fb7a9cf1f09\" (UID: \"a9250574-d9e5-4188-b8db-0fb7a9cf1f09\") "
Sep 16 02:48:00.161866 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:48:00.161711    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lkbfq\" (UniqueName: \"kubernetes.io/projected/a9250574-d9e5-4188-b8db-0fb7a9cf1f09-kube-api-access-lkbfq\") pod \"a9250574-d9e5-4188-b8db-0fb7a9cf1f09\" (UID: \"a9250574-d9e5-4188-b8db-0fb7a9cf1f09\") "
Sep 16 02:48:00.162022 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:48:00.162001    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a9250574-d9e5-4188-b8db-0fb7a9cf1f09-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "a9250574-d9e5-4188-b8db-0fb7a9cf1f09" (UID: "a9250574-d9e5-4188-b8db-0fb7a9cf1f09"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 02:48:00.163826 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:48:00.163808    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a9250574-d9e5-4188-b8db-0fb7a9cf1f09-kube-api-access-lkbfq" (OuterVolumeSpecName: "kube-api-access-lkbfq") pod "a9250574-d9e5-4188-b8db-0fb7a9cf1f09" (UID: "a9250574-d9e5-4188-b8db-0fb7a9cf1f09"). InnerVolumeSpecName "kube-api-access-lkbfq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 02:48:00.263184 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:48:00.263154    2583 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a9250574-d9e5-4188-b8db-0fb7a9cf1f09-plugins-config-volume\") on node \"ip-10-0-128-111.ec2.internal\" DevicePath \"\""
Sep 16 02:48:00.263184 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:48:00.263180    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lkbfq\" (UniqueName: \"kubernetes.io/projected/a9250574-d9e5-4188-b8db-0fb7a9cf1f09-kube-api-access-lkbfq\") on node \"ip-10-0-128-111.ec2.internal\" DevicePath \"\""
Sep 16 02:48:00.914825 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:48:00.914788    2583 generic.go:356] "Generic (PLEG): container finished" podID="a9250574-d9e5-4188-b8db-0fb7a9cf1f09" containerID="a2a42f5a989940fdd33a897fecba063c1d32e61d20711e40a477ff3061724454" exitCode=0
Sep 16 02:48:00.915249 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:48:00.914833    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-db1ed7-57b4fc7f46-6jjzx"
Sep 16 02:48:00.915249 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:48:00.914870    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-db1ed7-57b4fc7f46-6jjzx" event={"ID":"a9250574-d9e5-4188-b8db-0fb7a9cf1f09","Type":"ContainerDied","Data":"a2a42f5a989940fdd33a897fecba063c1d32e61d20711e40a477ff3061724454"}
Sep 16 02:48:00.915249 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:48:00.914920    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-db1ed7-57b4fc7f46-6jjzx" event={"ID":"a9250574-d9e5-4188-b8db-0fb7a9cf1f09","Type":"ContainerDied","Data":"ee8cfdc1023748743df27e5e264809f78f45ab85b3cc7ce5864f0c0f9d1b56b6"}
Sep 16 02:48:00.915249 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:48:00.914942    2583 scope.go:117] "RemoveContainer" containerID="a2a42f5a989940fdd33a897fecba063c1d32e61d20711e40a477ff3061724454"
Sep 16 02:48:00.924208 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:48:00.924189    2583 scope.go:117] "RemoveContainer" containerID="a2a42f5a989940fdd33a897fecba063c1d32e61d20711e40a477ff3061724454"
Sep 16 02:48:00.924472 ip-10-0-128-111 kubenswrapper[2583]: E0916 02:48:00.924454    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a2a42f5a989940fdd33a897fecba063c1d32e61d20711e40a477ff3061724454\": container with ID starting with a2a42f5a989940fdd33a897fecba063c1d32e61d20711e40a477ff3061724454 not found: ID does not exist" containerID="a2a42f5a989940fdd33a897fecba063c1d32e61d20711e40a477ff3061724454"
Sep 16 02:48:00.924535 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:48:00.924479    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a2a42f5a989940fdd33a897fecba063c1d32e61d20711e40a477ff3061724454"} err="failed to get container status \"a2a42f5a989940fdd33a897fecba063c1d32e61d20711e40a477ff3061724454\": rpc error: code = NotFound desc = could not find container \"a2a42f5a989940fdd33a897fecba063c1d32e61d20711e40a477ff3061724454\": container with ID starting with a2a42f5a989940fdd33a897fecba063c1d32e61d20711e40a477ff3061724454 not found: ID does not exist"
Sep 16 02:48:00.935013 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:48:00.934994    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-db1ed7-57b4fc7f46-6jjzx"]
Sep 16 02:48:00.938419 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:48:00.938401    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-db1ed7-57b4fc7f46-6jjzx"]
Sep 16 02:48:02.045509 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:48:02.045466    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a9250574-d9e5-4188-b8db-0fb7a9cf1f09" path="/var/lib/kubelet/pods/a9250574-d9e5-4188-b8db-0fb7a9cf1f09/volumes"
Sep 16 02:48:02.943291 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:48:02.943249    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-fzl8t"]
Sep 16 02:48:17.256401 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:48:17.256362    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-eee708-78849d699f-lm2ks"]
Sep 16 02:48:17.256930 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:48:17.256703    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-eee708-78849d699f-lm2ks" podUID="be3b30cb-484b-4f99-a52c-0567667b31c3" containerName="payload-processing" containerID="cri-o://5b81837183304191ee7ebd4cd247a66c05bb565772c3a97a81e753858a390afe" gracePeriod=30
Sep 16 02:48:17.516275 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:48:17.516210    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-eee708-78849d699f-lm2ks"
Sep 16 02:48:17.525700 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:48:17.525679    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/be3b30cb-484b-4f99-a52c-0567667b31c3-plugins-config-volume\") pod \"be3b30cb-484b-4f99-a52c-0567667b31c3\" (UID: \"be3b30cb-484b-4f99-a52c-0567667b31c3\") "
Sep 16 02:48:17.525825 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:48:17.525719    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hjn9s\" (UniqueName: \"kubernetes.io/projected/be3b30cb-484b-4f99-a52c-0567667b31c3-kube-api-access-hjn9s\") pod \"be3b30cb-484b-4f99-a52c-0567667b31c3\" (UID: \"be3b30cb-484b-4f99-a52c-0567667b31c3\") "
Sep 16 02:48:17.526071 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:48:17.526052    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/be3b30cb-484b-4f99-a52c-0567667b31c3-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "be3b30cb-484b-4f99-a52c-0567667b31c3" (UID: "be3b30cb-484b-4f99-a52c-0567667b31c3"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 02:48:17.527882 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:48:17.527858    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/be3b30cb-484b-4f99-a52c-0567667b31c3-kube-api-access-hjn9s" (OuterVolumeSpecName: "kube-api-access-hjn9s") pod "be3b30cb-484b-4f99-a52c-0567667b31c3" (UID: "be3b30cb-484b-4f99-a52c-0567667b31c3"). InnerVolumeSpecName "kube-api-access-hjn9s". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 02:48:17.627257 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:48:17.627224    2583 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/be3b30cb-484b-4f99-a52c-0567667b31c3-plugins-config-volume\") on node \"ip-10-0-128-111.ec2.internal\" DevicePath \"\""
Sep 16 02:48:17.627257 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:48:17.627251    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hjn9s\" (UniqueName: \"kubernetes.io/projected/be3b30cb-484b-4f99-a52c-0567667b31c3-kube-api-access-hjn9s\") on node \"ip-10-0-128-111.ec2.internal\" DevicePath \"\""
Sep 16 02:48:18.003873 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:48:18.003837    2583 generic.go:356] "Generic (PLEG): container finished" podID="be3b30cb-484b-4f99-a52c-0567667b31c3" containerID="5b81837183304191ee7ebd4cd247a66c05bb565772c3a97a81e753858a390afe" exitCode=0
Sep 16 02:48:18.004078 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:48:18.003887    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-eee708-78849d699f-lm2ks"
Sep 16 02:48:18.004078 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:48:18.003926    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-eee708-78849d699f-lm2ks" event={"ID":"be3b30cb-484b-4f99-a52c-0567667b31c3","Type":"ContainerDied","Data":"5b81837183304191ee7ebd4cd247a66c05bb565772c3a97a81e753858a390afe"}
Sep 16 02:48:18.004078 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:48:18.004016    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-eee708-78849d699f-lm2ks" event={"ID":"be3b30cb-484b-4f99-a52c-0567667b31c3","Type":"ContainerDied","Data":"c5859a4606964bda20d6f16fe9793c321716f7cf1ea7a1f87c0c33c500284f08"}
Sep 16 02:48:18.004078 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:48:18.004048    2583 scope.go:117] "RemoveContainer" containerID="5b81837183304191ee7ebd4cd247a66c05bb565772c3a97a81e753858a390afe"
Sep 16 02:48:18.014001 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:48:18.013984    2583 scope.go:117] "RemoveContainer" containerID="5b81837183304191ee7ebd4cd247a66c05bb565772c3a97a81e753858a390afe"
Sep 16 02:48:18.014294 ip-10-0-128-111 kubenswrapper[2583]: E0916 02:48:18.014270    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5b81837183304191ee7ebd4cd247a66c05bb565772c3a97a81e753858a390afe\": container with ID starting with 5b81837183304191ee7ebd4cd247a66c05bb565772c3a97a81e753858a390afe not found: ID does not exist" containerID="5b81837183304191ee7ebd4cd247a66c05bb565772c3a97a81e753858a390afe"
Sep 16 02:48:18.014342 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:48:18.014293    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5b81837183304191ee7ebd4cd247a66c05bb565772c3a97a81e753858a390afe"} err="failed to get container status \"5b81837183304191ee7ebd4cd247a66c05bb565772c3a97a81e753858a390afe\": rpc error: code = NotFound desc = could not find container \"5b81837183304191ee7ebd4cd247a66c05bb565772c3a97a81e753858a390afe\": container with ID starting with 5b81837183304191ee7ebd4cd247a66c05bb565772c3a97a81e753858a390afe not found: ID does not exist"
Sep 16 02:48:18.024925 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:48:18.024900    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-eee708-78849d699f-lm2ks"]
Sep 16 02:48:18.030777 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:48:18.030757    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-eee708-78849d699f-lm2ks"]
Sep 16 02:48:18.044895 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:48:18.044870    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="be3b30cb-484b-4f99-a52c-0567667b31c3" path="/var/lib/kubelet/pods/be3b30cb-484b-4f99-a52c-0567667b31c3/volumes"
Sep 16 02:48:19.945824 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:48:19.945788    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-fzl8t"]
Sep 16 02:48:29.152093 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:48:29.152058    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-fzl8t"]
Sep 16 02:49:50.352634 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:49:50.352603    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-fzl8t"]
Sep 16 02:50:17.546618 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:50:17.546587    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-fzl8t"]
Sep 16 02:50:25.146358 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:50:25.146326    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-fzl8t"]
Sep 16 02:50:34.352802 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:50:34.352769    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-fzl8t"]
Sep 16 02:50:51.946619 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:50:51.946577    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-fzl8t"]
Sep 16 02:50:58.547771 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:50:58.547730    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-fzl8t"]
Sep 16 02:51:42.358156 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:51:42.358068    2583 scope.go:117] "RemoveContainer" containerID="ee71db41c3a3cc305eca954c8cff5227a4f531e66c279f742c95f65bf0890aff"
Sep 16 02:51:42.370253 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:51:42.370233    2583 scope.go:117] "RemoveContainer" containerID="7a4714cc303cbbd7a3384553ddbc126ee3a6feffb10755d0a621c602abe1c72a"
Sep 16 02:51:42.377385 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:51:42.377367    2583 scope.go:117] "RemoveContainer" containerID="844ad7487aa5c0d5843e3bb2bbaf827b591e71344095b9518012ce5d7c2ce8d0"
Sep 16 02:51:42.384515 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:51:42.384499    2583 scope.go:117] "RemoveContainer" containerID="33c0e7601aab0514fd6551cc5a457193b28cff2971bb7daad7d7bc6624eac362"
Sep 16 02:51:42.391402 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:51:42.391383    2583 scope.go:117] "RemoveContainer" containerID="9cf60f4f256bbb7679ad9a591101fbfe00beb63f6d794d05254bc39b7ad516b2"
Sep 16 02:52:32.351898 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:52:32.351861    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-fzl8t"]
Sep 16 02:52:38.056074 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:52:38.056040    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-fzl8t"]
Sep 16 02:52:42.420790 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:52:42.420753    2583 scope.go:117] "RemoveContainer" containerID="e7e22381d4f095d7b12d928dfaafb799e70c87f4dd5009e6c367cc2c900b9254"
Sep 16 02:52:42.428961 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:52:42.428935    2583 scope.go:117] "RemoveContainer" containerID="3798671a60d65e0e531261233bd251c6c3f4b8ffd722f391239cec7e98f39570"
Sep 16 02:52:53.236715 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:52:53.236680    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-same-id-a-c0541850-kserve-66f4d48cdc-zqdls"]
Sep 16 02:52:53.237105 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:52:53.237002    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="be3b30cb-484b-4f99-a52c-0567667b31c3" containerName="payload-processing"
Sep 16 02:52:53.237105 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:52:53.237012    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="be3b30cb-484b-4f99-a52c-0567667b31c3" containerName="payload-processing"
Sep 16 02:52:53.237105 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:52:53.237029    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4b2c7d51-9fcb-4bae-a99c-66ad0a8e62eb" containerName="istio-proxy"
Sep 16 02:52:53.237105 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:52:53.237034    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="4b2c7d51-9fcb-4bae-a99c-66ad0a8e62eb" containerName="istio-proxy"
Sep 16 02:52:53.237105 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:52:53.237041    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a9250574-d9e5-4188-b8db-0fb7a9cf1f09" containerName="payload-pre-processing"
Sep 16 02:52:53.237105 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:52:53.237046    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="a9250574-d9e5-4188-b8db-0fb7a9cf1f09" containerName="payload-pre-processing"
Sep 16 02:52:53.237105 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:52:53.237053    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4b2c7d51-9fcb-4bae-a99c-66ad0a8e62eb" containerName="istio-proxy"
Sep 16 02:52:53.237105 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:52:53.237059    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="4b2c7d51-9fcb-4bae-a99c-66ad0a8e62eb" containerName="istio-proxy"
Sep 16 02:52:53.237105 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:52:53.237066    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a18ebddf-5578-4752-852a-3dcfe8eb881b" containerName="payload-processing"
Sep 16 02:52:53.237105 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:52:53.237071    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="a18ebddf-5578-4752-852a-3dcfe8eb881b" containerName="payload-processing"
Sep 16 02:52:53.237105 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:52:53.237080    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1490b32b-6d61-404a-ac86-f902766a0cfb" containerName="istio-proxy"
Sep 16 02:52:53.237105 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:52:53.237086    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="1490b32b-6d61-404a-ac86-f902766a0cfb" containerName="istio-proxy"
Sep 16 02:52:53.237105 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:52:53.237095    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7d492f3f-cf1b-4902-b111-4c0f248d6fe1" containerName="payload-pre-processing"
Sep 16 02:52:53.237105 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:52:53.237100    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="7d492f3f-cf1b-4902-b111-4c0f248d6fe1" containerName="payload-pre-processing"
Sep 16 02:52:53.237105 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:52:53.237108    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e2ae0cd7-7d55-4dc1-8ea2-1eb9020290dd" containerName="istio-proxy"
Sep 16 02:52:53.237105 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:52:53.237115    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="e2ae0cd7-7d55-4dc1-8ea2-1eb9020290dd" containerName="istio-proxy"
Sep 16 02:52:53.237635 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:52:53.237125    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="130bfdb3-450f-4be3-a49c-c851c1235afa" containerName="revoke-keys"
Sep 16 02:52:53.237635 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:52:53.237130    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="130bfdb3-450f-4be3-a49c-c851c1235afa" containerName="revoke-keys"
Sep 16 02:52:53.237635 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:52:53.237180    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="be3b30cb-484b-4f99-a52c-0567667b31c3" containerName="payload-processing"
Sep 16 02:52:53.237635 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:52:53.237188    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="130bfdb3-450f-4be3-a49c-c851c1235afa" containerName="revoke-keys"
Sep 16 02:52:53.237635 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:52:53.237194    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="1490b32b-6d61-404a-ac86-f902766a0cfb" containerName="istio-proxy"
Sep 16 02:52:53.237635 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:52:53.237202    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="a9250574-d9e5-4188-b8db-0fb7a9cf1f09" containerName="payload-pre-processing"
Sep 16 02:52:53.237635 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:52:53.237207    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="1490b32b-6d61-404a-ac86-f902766a0cfb" containerName="istio-proxy"
Sep 16 02:52:53.237635 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:52:53.237213    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="7d492f3f-cf1b-4902-b111-4c0f248d6fe1" containerName="payload-pre-processing"
Sep 16 02:52:53.237635 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:52:53.237217    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="a18ebddf-5578-4752-852a-3dcfe8eb881b" containerName="payload-processing"
Sep 16 02:52:53.237635 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:52:53.237224    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="4b2c7d51-9fcb-4bae-a99c-66ad0a8e62eb" containerName="istio-proxy"
Sep 16 02:52:53.240237 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:52:53.240217    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-c0541850-kserve-66f4d48cdc-zqdls"
Sep 16 02:52:53.242443 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:52:53.242424    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-same-id-a-c0541850-kserve-self-signed-certs\""
Sep 16 02:52:53.247256 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:52:53.247234    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-a-c0541850-kserve-66f4d48cdc-zqdls"]
Sep 16 02:52:53.384939 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:52:53.384902    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/344c1b59-2645-48e2-85e1-649d6fe7c65d-home\") pod \"e2e-same-id-a-c0541850-kserve-66f4d48cdc-zqdls\" (UID: \"344c1b59-2645-48e2-85e1-649d6fe7c65d\") " pod="llm/e2e-same-id-a-c0541850-kserve-66f4d48cdc-zqdls"
Sep 16 02:52:53.384939 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:52:53.384939    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/344c1b59-2645-48e2-85e1-649d6fe7c65d-model-cache\") pod \"e2e-same-id-a-c0541850-kserve-66f4d48cdc-zqdls\" (UID: \"344c1b59-2645-48e2-85e1-649d6fe7c65d\") " pod="llm/e2e-same-id-a-c0541850-kserve-66f4d48cdc-zqdls"
Sep 16 02:52:53.385181 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:52:53.384971    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/344c1b59-2645-48e2-85e1-649d6fe7c65d-dshm\") pod \"e2e-same-id-a-c0541850-kserve-66f4d48cdc-zqdls\" (UID: \"344c1b59-2645-48e2-85e1-649d6fe7c65d\") " pod="llm/e2e-same-id-a-c0541850-kserve-66f4d48cdc-zqdls"
Sep 16 02:52:53.385181 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:52:53.385014    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/344c1b59-2645-48e2-85e1-649d6fe7c65d-tls-certs\") pod \"e2e-same-id-a-c0541850-kserve-66f4d48cdc-zqdls\" (UID: \"344c1b59-2645-48e2-85e1-649d6fe7c65d\") " pod="llm/e2e-same-id-a-c0541850-kserve-66f4d48cdc-zqdls"
Sep 16 02:52:53.385181 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:52:53.385039    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-srnkq\" (UniqueName: \"kubernetes.io/projected/344c1b59-2645-48e2-85e1-649d6fe7c65d-kube-api-access-srnkq\") pod \"e2e-same-id-a-c0541850-kserve-66f4d48cdc-zqdls\" (UID: \"344c1b59-2645-48e2-85e1-649d6fe7c65d\") " pod="llm/e2e-same-id-a-c0541850-kserve-66f4d48cdc-zqdls"
Sep 16 02:52:53.385181 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:52:53.385054    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/344c1b59-2645-48e2-85e1-649d6fe7c65d-tmp-dir\") pod \"e2e-same-id-a-c0541850-kserve-66f4d48cdc-zqdls\" (UID: \"344c1b59-2645-48e2-85e1-649d6fe7c65d\") " pod="llm/e2e-same-id-a-c0541850-kserve-66f4d48cdc-zqdls"
Sep 16 02:52:53.485894 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:52:53.485853    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/344c1b59-2645-48e2-85e1-649d6fe7c65d-home\") pod \"e2e-same-id-a-c0541850-kserve-66f4d48cdc-zqdls\" (UID: \"344c1b59-2645-48e2-85e1-649d6fe7c65d\") " pod="llm/e2e-same-id-a-c0541850-kserve-66f4d48cdc-zqdls"
Sep 16 02:52:53.485894 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:52:53.485896    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/344c1b59-2645-48e2-85e1-649d6fe7c65d-model-cache\") pod \"e2e-same-id-a-c0541850-kserve-66f4d48cdc-zqdls\" (UID: \"344c1b59-2645-48e2-85e1-649d6fe7c65d\") " pod="llm/e2e-same-id-a-c0541850-kserve-66f4d48cdc-zqdls"
Sep 16 02:52:53.486157 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:52:53.485942    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/344c1b59-2645-48e2-85e1-649d6fe7c65d-dshm\") pod \"e2e-same-id-a-c0541850-kserve-66f4d48cdc-zqdls\" (UID: \"344c1b59-2645-48e2-85e1-649d6fe7c65d\") " pod="llm/e2e-same-id-a-c0541850-kserve-66f4d48cdc-zqdls"
Sep 16 02:52:53.486157 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:52:53.485988    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/344c1b59-2645-48e2-85e1-649d6fe7c65d-tls-certs\") pod \"e2e-same-id-a-c0541850-kserve-66f4d48cdc-zqdls\" (UID: \"344c1b59-2645-48e2-85e1-649d6fe7c65d\") " pod="llm/e2e-same-id-a-c0541850-kserve-66f4d48cdc-zqdls"
Sep 16 02:52:53.486157 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:52:53.486029    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-srnkq\" (UniqueName: \"kubernetes.io/projected/344c1b59-2645-48e2-85e1-649d6fe7c65d-kube-api-access-srnkq\") pod \"e2e-same-id-a-c0541850-kserve-66f4d48cdc-zqdls\" (UID: \"344c1b59-2645-48e2-85e1-649d6fe7c65d\") " pod="llm/e2e-same-id-a-c0541850-kserve-66f4d48cdc-zqdls"
Sep 16 02:52:53.486330 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:52:53.486158    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/344c1b59-2645-48e2-85e1-649d6fe7c65d-tmp-dir\") pod \"e2e-same-id-a-c0541850-kserve-66f4d48cdc-zqdls\" (UID: \"344c1b59-2645-48e2-85e1-649d6fe7c65d\") " pod="llm/e2e-same-id-a-c0541850-kserve-66f4d48cdc-zqdls"
Sep 16 02:52:53.486330 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:52:53.486321    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/344c1b59-2645-48e2-85e1-649d6fe7c65d-home\") pod \"e2e-same-id-a-c0541850-kserve-66f4d48cdc-zqdls\" (UID: \"344c1b59-2645-48e2-85e1-649d6fe7c65d\") " pod="llm/e2e-same-id-a-c0541850-kserve-66f4d48cdc-zqdls"
Sep 16 02:52:53.486441 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:52:53.486403    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/344c1b59-2645-48e2-85e1-649d6fe7c65d-model-cache\") pod \"e2e-same-id-a-c0541850-kserve-66f4d48cdc-zqdls\" (UID: \"344c1b59-2645-48e2-85e1-649d6fe7c65d\") " pod="llm/e2e-same-id-a-c0541850-kserve-66f4d48cdc-zqdls"
Sep 16 02:52:53.486488 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:52:53.486473    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/344c1b59-2645-48e2-85e1-649d6fe7c65d-tmp-dir\") pod \"e2e-same-id-a-c0541850-kserve-66f4d48cdc-zqdls\" (UID: \"344c1b59-2645-48e2-85e1-649d6fe7c65d\") " pod="llm/e2e-same-id-a-c0541850-kserve-66f4d48cdc-zqdls"
Sep 16 02:52:53.488360 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:52:53.488307    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/344c1b59-2645-48e2-85e1-649d6fe7c65d-dshm\") pod \"e2e-same-id-a-c0541850-kserve-66f4d48cdc-zqdls\" (UID: \"344c1b59-2645-48e2-85e1-649d6fe7c65d\") " pod="llm/e2e-same-id-a-c0541850-kserve-66f4d48cdc-zqdls"
Sep 16 02:52:53.488465 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:52:53.488430    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/344c1b59-2645-48e2-85e1-649d6fe7c65d-tls-certs\") pod \"e2e-same-id-a-c0541850-kserve-66f4d48cdc-zqdls\" (UID: \"344c1b59-2645-48e2-85e1-649d6fe7c65d\") " pod="llm/e2e-same-id-a-c0541850-kserve-66f4d48cdc-zqdls"
Sep 16 02:52:53.492993 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:52:53.492972    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-srnkq\" (UniqueName: \"kubernetes.io/projected/344c1b59-2645-48e2-85e1-649d6fe7c65d-kube-api-access-srnkq\") pod \"e2e-same-id-a-c0541850-kserve-66f4d48cdc-zqdls\" (UID: \"344c1b59-2645-48e2-85e1-649d6fe7c65d\") " pod="llm/e2e-same-id-a-c0541850-kserve-66f4d48cdc-zqdls"
Sep 16 02:52:53.550466 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:52:53.550430    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-c0541850-kserve-66f4d48cdc-zqdls"
Sep 16 02:52:53.818469 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:52:53.818390    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-a-c0541850-kserve-66f4d48cdc-zqdls"]
Sep 16 02:52:53.821327 ip-10-0-128-111 kubenswrapper[2583]: W0916 02:52:53.821295    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod344c1b59_2645_48e2_85e1_649d6fe7c65d.slice/crio-5298687d79591d0555023823c12adaf0e3380e8a2b7e30a47b4db83cbb7abb73 WatchSource:0}: Error finding container 5298687d79591d0555023823c12adaf0e3380e8a2b7e30a47b4db83cbb7abb73: Status 404 returned error can't find the container with id 5298687d79591d0555023823c12adaf0e3380e8a2b7e30a47b4db83cbb7abb73
Sep 16 02:52:54.327798 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:52:54.327753    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-c0541850-kserve-66f4d48cdc-zqdls" event={"ID":"344c1b59-2645-48e2-85e1-649d6fe7c65d","Type":"ContainerStarted","Data":"6611445ad87ac4b125258f2ad81a6ee0f4740244917cbbc33ac3b5440d849141"}
Sep 16 02:52:54.327798 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:52:54.327794    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-c0541850-kserve-66f4d48cdc-zqdls" event={"ID":"344c1b59-2645-48e2-85e1-649d6fe7c65d","Type":"ContainerStarted","Data":"5298687d79591d0555023823c12adaf0e3380e8a2b7e30a47b4db83cbb7abb73"}
Sep 16 02:52:54.342303 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:52:54.342235    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-same-id-a-c0541850-kserve-66f4d48cdc-zqdls" podStartSLOduration=1.34221722 podStartE2EDuration="1.34221722s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 02:52:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 02:52:54.341394993 +0000 UTC m=+2052.809183774" watchObservedRunningTime="2026-09-16 02:52:54.34221722 +0000 UTC m=+2052.810006002"
Sep 16 02:52:55.659194 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:52:55.659164    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-fzl8t"]
Sep 16 02:53:03.058936 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:53:03.058899    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-fzl8t"]
Sep 16 02:53:03.551428 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:53:03.551391    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-same-id-a-c0541850-kserve-66f4d48cdc-zqdls"
Sep 16 02:53:03.551753 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:53:03.551467    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-same-id-a-c0541850-kserve-66f4d48cdc-zqdls"
Sep 16 02:53:03.563537 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:53:03.563510    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-same-id-a-c0541850-kserve-66f4d48cdc-zqdls"
Sep 16 02:53:04.387852 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:53:04.387773    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-same-id-a-c0541850-kserve-66f4d48cdc-zqdls"
Sep 16 02:53:39.954655 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:53:39.954619    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-a-c0541850-kserve-66f4d48cdc-zqdls"]
Sep 16 02:53:42.250590 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:53:42.250538    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-fzl8t"]
Sep 16 02:53:42.451574 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:53:42.451531    2583 scope.go:117] "RemoveContainer" containerID="c3dbd299e127eeab3c09f13544d622018c00d589e93603eec9ce35b6a2039d3b"
Sep 16 02:53:51.046224 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:53:51.046185    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-fzl8t"]
Sep 16 02:53:54.974798 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:53:54.974742    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="llm/e2e-same-id-a-c0541850-kserve-66f4d48cdc-zqdls" podUID="344c1b59-2645-48e2-85e1-649d6fe7c65d" containerName="main" containerID="cri-o://6611445ad87ac4b125258f2ad81a6ee0f4740244917cbbc33ac3b5440d849141" gracePeriod=45
Sep 16 02:53:55.211412 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:53:55.211386    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-c0541850-kserve-66f4d48cdc-zqdls"
Sep 16 02:53:55.325681 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:53:55.325584    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/344c1b59-2645-48e2-85e1-649d6fe7c65d-tmp-dir\") pod \"344c1b59-2645-48e2-85e1-649d6fe7c65d\" (UID: \"344c1b59-2645-48e2-85e1-649d6fe7c65d\") "
Sep 16 02:53:55.325681 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:53:55.325641    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/344c1b59-2645-48e2-85e1-649d6fe7c65d-tls-certs\") pod \"344c1b59-2645-48e2-85e1-649d6fe7c65d\" (UID: \"344c1b59-2645-48e2-85e1-649d6fe7c65d\") "
Sep 16 02:53:55.325681 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:53:55.325671    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/344c1b59-2645-48e2-85e1-649d6fe7c65d-model-cache\") pod \"344c1b59-2645-48e2-85e1-649d6fe7c65d\" (UID: \"344c1b59-2645-48e2-85e1-649d6fe7c65d\") "
Sep 16 02:53:55.325976 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:53:55.325711    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-srnkq\" (UniqueName: \"kubernetes.io/projected/344c1b59-2645-48e2-85e1-649d6fe7c65d-kube-api-access-srnkq\") pod \"344c1b59-2645-48e2-85e1-649d6fe7c65d\" (UID: \"344c1b59-2645-48e2-85e1-649d6fe7c65d\") "
Sep 16 02:53:55.325976 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:53:55.325730    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/344c1b59-2645-48e2-85e1-649d6fe7c65d-home\") pod \"344c1b59-2645-48e2-85e1-649d6fe7c65d\" (UID: \"344c1b59-2645-48e2-85e1-649d6fe7c65d\") "
Sep 16 02:53:55.325976 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:53:55.325744    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/344c1b59-2645-48e2-85e1-649d6fe7c65d-dshm\") pod \"344c1b59-2645-48e2-85e1-649d6fe7c65d\" (UID: \"344c1b59-2645-48e2-85e1-649d6fe7c65d\") "
Sep 16 02:53:55.325976 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:53:55.325885    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/344c1b59-2645-48e2-85e1-649d6fe7c65d-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "344c1b59-2645-48e2-85e1-649d6fe7c65d" (UID: "344c1b59-2645-48e2-85e1-649d6fe7c65d"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 02:53:55.326172 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:53:55.325967    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/344c1b59-2645-48e2-85e1-649d6fe7c65d-model-cache" (OuterVolumeSpecName: "model-cache") pod "344c1b59-2645-48e2-85e1-649d6fe7c65d" (UID: "344c1b59-2645-48e2-85e1-649d6fe7c65d"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 02:53:55.326172 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:53:55.325976    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/344c1b59-2645-48e2-85e1-649d6fe7c65d-home" (OuterVolumeSpecName: "home") pod "344c1b59-2645-48e2-85e1-649d6fe7c65d" (UID: "344c1b59-2645-48e2-85e1-649d6fe7c65d"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 02:53:55.326172 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:53:55.326063    2583 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/344c1b59-2645-48e2-85e1-649d6fe7c65d-tmp-dir\") on node \"ip-10-0-128-111.ec2.internal\" DevicePath \"\""
Sep 16 02:53:55.328031 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:53:55.327999    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/344c1b59-2645-48e2-85e1-649d6fe7c65d-dshm" (OuterVolumeSpecName: "dshm") pod "344c1b59-2645-48e2-85e1-649d6fe7c65d" (UID: "344c1b59-2645-48e2-85e1-649d6fe7c65d"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 02:53:55.328147 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:53:55.328066    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/344c1b59-2645-48e2-85e1-649d6fe7c65d-kube-api-access-srnkq" (OuterVolumeSpecName: "kube-api-access-srnkq") pod "344c1b59-2645-48e2-85e1-649d6fe7c65d" (UID: "344c1b59-2645-48e2-85e1-649d6fe7c65d"). InnerVolumeSpecName "kube-api-access-srnkq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 02:53:55.328147 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:53:55.328074    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/344c1b59-2645-48e2-85e1-649d6fe7c65d-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "344c1b59-2645-48e2-85e1-649d6fe7c65d" (UID: "344c1b59-2645-48e2-85e1-649d6fe7c65d"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 02:53:55.426905 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:53:55.426868    2583 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/344c1b59-2645-48e2-85e1-649d6fe7c65d-model-cache\") on node \"ip-10-0-128-111.ec2.internal\" DevicePath \"\""
Sep 16 02:53:55.426905 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:53:55.426897    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-srnkq\" (UniqueName: \"kubernetes.io/projected/344c1b59-2645-48e2-85e1-649d6fe7c65d-kube-api-access-srnkq\") on node \"ip-10-0-128-111.ec2.internal\" DevicePath \"\""
Sep 16 02:53:55.426905 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:53:55.426908    2583 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/344c1b59-2645-48e2-85e1-649d6fe7c65d-home\") on node \"ip-10-0-128-111.ec2.internal\" DevicePath \"\""
Sep 16 02:53:55.426905 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:53:55.426916    2583 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/344c1b59-2645-48e2-85e1-649d6fe7c65d-dshm\") on node \"ip-10-0-128-111.ec2.internal\" DevicePath \"\""
Sep 16 02:53:55.427174 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:53:55.426925    2583 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/344c1b59-2645-48e2-85e1-649d6fe7c65d-tls-certs\") on node \"ip-10-0-128-111.ec2.internal\" DevicePath \"\""
Sep 16 02:53:55.602910 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:53:55.602874    2583 generic.go:356] "Generic (PLEG): container finished" podID="344c1b59-2645-48e2-85e1-649d6fe7c65d" containerID="6611445ad87ac4b125258f2ad81a6ee0f4740244917cbbc33ac3b5440d849141" exitCode=0
Sep 16 02:53:55.603114 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:53:55.602928    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-c0541850-kserve-66f4d48cdc-zqdls"
Sep 16 02:53:55.603114 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:53:55.602963    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-c0541850-kserve-66f4d48cdc-zqdls" event={"ID":"344c1b59-2645-48e2-85e1-649d6fe7c65d","Type":"ContainerDied","Data":"6611445ad87ac4b125258f2ad81a6ee0f4740244917cbbc33ac3b5440d849141"}
Sep 16 02:53:55.603114 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:53:55.603020    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-c0541850-kserve-66f4d48cdc-zqdls" event={"ID":"344c1b59-2645-48e2-85e1-649d6fe7c65d","Type":"ContainerDied","Data":"5298687d79591d0555023823c12adaf0e3380e8a2b7e30a47b4db83cbb7abb73"}
Sep 16 02:53:55.603114 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:53:55.603043    2583 scope.go:117] "RemoveContainer" containerID="6611445ad87ac4b125258f2ad81a6ee0f4740244917cbbc33ac3b5440d849141"
Sep 16 02:53:55.612731 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:53:55.612713    2583 scope.go:117] "RemoveContainer" containerID="6611445ad87ac4b125258f2ad81a6ee0f4740244917cbbc33ac3b5440d849141"
Sep 16 02:53:55.613040 ip-10-0-128-111 kubenswrapper[2583]: E0916 02:53:55.613023    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6611445ad87ac4b125258f2ad81a6ee0f4740244917cbbc33ac3b5440d849141\": container with ID starting with 6611445ad87ac4b125258f2ad81a6ee0f4740244917cbbc33ac3b5440d849141 not found: ID does not exist" containerID="6611445ad87ac4b125258f2ad81a6ee0f4740244917cbbc33ac3b5440d849141"
Sep 16 02:53:55.613089 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:53:55.613049    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6611445ad87ac4b125258f2ad81a6ee0f4740244917cbbc33ac3b5440d849141"} err="failed to get container status \"6611445ad87ac4b125258f2ad81a6ee0f4740244917cbbc33ac3b5440d849141\": rpc error: code = NotFound desc = could not find container \"6611445ad87ac4b125258f2ad81a6ee0f4740244917cbbc33ac3b5440d849141\": container with ID starting with 6611445ad87ac4b125258f2ad81a6ee0f4740244917cbbc33ac3b5440d849141 not found: ID does not exist"
Sep 16 02:53:55.623194 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:53:55.623172    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-a-c0541850-kserve-66f4d48cdc-zqdls"]
Sep 16 02:53:55.626708 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:53:55.626686    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["llm/e2e-same-id-a-c0541850-kserve-66f4d48cdc-zqdls"]
Sep 16 02:53:56.045255 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:53:56.045156    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="344c1b59-2645-48e2-85e1-649d6fe7c65d" path="/var/lib/kubelet/pods/344c1b59-2645-48e2-85e1-649d6fe7c65d/volumes"
Sep 16 02:54:00.848087 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:54:00.848052    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-fzl8t"]
Sep 16 02:57:06.947231 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:57:06.947186    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-fzl8t"]
Sep 16 02:57:13.753077 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:57:13.753042    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-fzl8t"]
Sep 16 02:57:19.447239 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:57:19.447207    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-fzl8t"]
Sep 16 02:57:26.549594 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:57:26.549559    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-fzl8t"]
Sep 16 02:57:30.146153 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:57:30.146117    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-fv4vf/must-gather-d7sl8"]
Sep 16 02:57:30.146648 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:57:30.146624    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="344c1b59-2645-48e2-85e1-649d6fe7c65d" containerName="main"
Sep 16 02:57:30.146648 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:57:30.146642    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="344c1b59-2645-48e2-85e1-649d6fe7c65d" containerName="main"
Sep 16 02:57:30.146790 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:57:30.146662    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1490b32b-6d61-404a-ac86-f902766a0cfb" containerName="istio-proxy"
Sep 16 02:57:30.146790 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:57:30.146671    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="1490b32b-6d61-404a-ac86-f902766a0cfb" containerName="istio-proxy"
Sep 16 02:57:30.146790 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:57:30.146769    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="344c1b59-2645-48e2-85e1-649d6fe7c65d" containerName="main"
Sep 16 02:57:30.146790 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:57:30.146783    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="4b2c7d51-9fcb-4bae-a99c-66ad0a8e62eb" containerName="istio-proxy"
Sep 16 02:57:30.150164 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:57:30.150144    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-fv4vf/must-gather-d7sl8"
Sep 16 02:57:30.152565 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:57:30.152534    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-fv4vf\"/\"openshift-service-ca.crt\""
Sep 16 02:57:30.152670 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:57:30.152542    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-fv4vf\"/\"default-dockercfg-nxld2\""
Sep 16 02:57:30.153681 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:57:30.153663    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-fv4vf\"/\"kube-root-ca.crt\""
Sep 16 02:57:30.165829 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:57:30.165803    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-fv4vf/must-gather-d7sl8"]
Sep 16 02:57:30.216093 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:57:30.216044    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/9dca4f21-77ec-4123-bb31-257fc936fce7-must-gather-output\") pod \"must-gather-d7sl8\" (UID: \"9dca4f21-77ec-4123-bb31-257fc936fce7\") " pod="openshift-must-gather-fv4vf/must-gather-d7sl8"
Sep 16 02:57:30.216093 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:57:30.216101    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f56kv\" (UniqueName: \"kubernetes.io/projected/9dca4f21-77ec-4123-bb31-257fc936fce7-kube-api-access-f56kv\") pod \"must-gather-d7sl8\" (UID: \"9dca4f21-77ec-4123-bb31-257fc936fce7\") " pod="openshift-must-gather-fv4vf/must-gather-d7sl8"
Sep 16 02:57:30.316913 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:57:30.316876    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f56kv\" (UniqueName: \"kubernetes.io/projected/9dca4f21-77ec-4123-bb31-257fc936fce7-kube-api-access-f56kv\") pod \"must-gather-d7sl8\" (UID: \"9dca4f21-77ec-4123-bb31-257fc936fce7\") " pod="openshift-must-gather-fv4vf/must-gather-d7sl8"
Sep 16 02:57:30.317077 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:57:30.316987    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/9dca4f21-77ec-4123-bb31-257fc936fce7-must-gather-output\") pod \"must-gather-d7sl8\" (UID: \"9dca4f21-77ec-4123-bb31-257fc936fce7\") " pod="openshift-must-gather-fv4vf/must-gather-d7sl8"
Sep 16 02:57:30.317269 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:57:30.317255    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/9dca4f21-77ec-4123-bb31-257fc936fce7-must-gather-output\") pod \"must-gather-d7sl8\" (UID: \"9dca4f21-77ec-4123-bb31-257fc936fce7\") " pod="openshift-must-gather-fv4vf/must-gather-d7sl8"
Sep 16 02:57:30.324416 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:57:30.324391    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-f56kv\" (UniqueName: \"kubernetes.io/projected/9dca4f21-77ec-4123-bb31-257fc936fce7-kube-api-access-f56kv\") pod \"must-gather-d7sl8\" (UID: \"9dca4f21-77ec-4123-bb31-257fc936fce7\") " pod="openshift-must-gather-fv4vf/must-gather-d7sl8"
Sep 16 02:57:30.459443 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:57:30.459353    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-fv4vf/must-gather-d7sl8"
Sep 16 02:57:30.730690 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:57:30.730663    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-fv4vf/must-gather-d7sl8"]
Sep 16 02:57:30.733358 ip-10-0-128-111 kubenswrapper[2583]: W0916 02:57:30.733331    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod9dca4f21_77ec_4123_bb31_257fc936fce7.slice/crio-5985834a97146e622111ccb361f6ab150fec295bdef35fd70fa93bd94098c501 WatchSource:0}: Error finding container 5985834a97146e622111ccb361f6ab150fec295bdef35fd70fa93bd94098c501: Status 404 returned error can't find the container with id 5985834a97146e622111ccb361f6ab150fec295bdef35fd70fa93bd94098c501
Sep 16 02:57:30.735426 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:57:30.735409    2583 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 16 02:57:31.564469 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:57:31.564434    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-fv4vf/must-gather-d7sl8" event={"ID":"9dca4f21-77ec-4123-bb31-257fc936fce7","Type":"ContainerStarted","Data":"5985834a97146e622111ccb361f6ab150fec295bdef35fd70fa93bd94098c501"}
Sep 16 02:57:33.577137 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:57:33.577079    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-fv4vf/must-gather-d7sl8" event={"ID":"9dca4f21-77ec-4123-bb31-257fc936fce7","Type":"ContainerStarted","Data":"124624fe3de08764e15248eecb52f6fd244fe1f70547be3187e82bfc09872ad7"}
Sep 16 02:57:33.577137 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:57:33.577138    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-fv4vf/must-gather-d7sl8" event={"ID":"9dca4f21-77ec-4123-bb31-257fc936fce7","Type":"ContainerStarted","Data":"eeafa6e4d99ae4c63c44780ae4da3d928e0f0a28df57b9e7b6c6ef3751100e2c"}
Sep 16 02:57:33.591462 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:57:33.591395    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-fv4vf/must-gather-d7sl8" podStartSLOduration=1.566832741 podStartE2EDuration="3.591374817s" totalImagesPullingTime="2.024542076s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 02:57:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 02:57:33.589976598 +0000 UTC m=+2332.057765382" watchObservedRunningTime="2026-09-16 02:57:33.591374817 +0000 UTC m=+2332.059163601"
Sep 16 02:57:34.340602 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:57:34.340507    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-mbxmd_2fbf440b-4c09-428e-b6bf-1236d95f97cc/global-pull-secret-syncer/0.log"
Sep 16 02:57:34.474478 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:57:34.474447    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-psqcm_bdf274ad-3983-42ba-b712-8ce11da63210/konnectivity-agent/0.log"
Sep 16 02:57:34.534911 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:57:34.534879    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-128-111.ec2.internal_b7568fa8cea4626b154e4750bd44461f/haproxy/0.log"
Sep 16 02:57:39.569616 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:57:39.569510    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-7698c7d5bb-mds65_60642fc2-3eab-4e46-85a9-30bc0e930717/authorino/0.log"
Sep 16 02:57:39.733503 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:57:39.733473    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-84994f7ffc-fzl8t_f795b100-3c16-48da-ac11-3899946ca508/limitador/0.log"
Sep 16 02:57:41.368495 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:57:41.368458    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-59fbdd77d-4c8ff_f7765761-5911-4ee7-9587-9454b2e15ddc/kube-state-metrics/0.log"
Sep 16 02:57:41.387912 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:57:41.387863    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-59fbdd77d-4c8ff_f7765761-5911-4ee7-9587-9454b2e15ddc/kube-rbac-proxy-main/0.log"
Sep 16 02:57:41.409421 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:57:41.409391    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-59fbdd77d-4c8ff_f7765761-5911-4ee7-9587-9454b2e15ddc/kube-rbac-proxy-self/0.log"
Sep 16 02:57:41.434202 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:57:41.434116    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_metrics-server-f586c465f-6kvt2_41130ea6-39bb-4c30-956c-b94b1b1cc68a/metrics-server/0.log"
Sep 16 02:57:41.599259 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:57:41.599229    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-hv9kk_80cd7034-5b29-4116-aa53-74168a3a96ed/node-exporter/0.log"
Sep 16 02:57:41.624078 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:57:41.624045    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-hv9kk_80cd7034-5b29-4116-aa53-74168a3a96ed/kube-rbac-proxy/0.log"
Sep 16 02:57:41.658097 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:57:41.658069    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-hv9kk_80cd7034-5b29-4116-aa53-74168a3a96ed/init-textfile/0.log"
Sep 16 02:57:41.851914 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:57:41.851879    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_c29a4a9d-f6a2-446e-b2a9-90292218ae94/prometheus/0.log"
Sep 16 02:57:41.872896 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:57:41.872864    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_c29a4a9d-f6a2-446e-b2a9-90292218ae94/config-reloader/0.log"
Sep 16 02:57:41.896396 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:57:41.896350    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_c29a4a9d-f6a2-446e-b2a9-90292218ae94/thanos-sidecar/0.log"
Sep 16 02:57:41.917611 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:57:41.917581    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_c29a4a9d-f6a2-446e-b2a9-90292218ae94/kube-rbac-proxy-web/0.log"
Sep 16 02:57:41.946194 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:57:41.946104    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_c29a4a9d-f6a2-446e-b2a9-90292218ae94/kube-rbac-proxy/0.log"
Sep 16 02:57:41.970170 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:57:41.970142    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_c29a4a9d-f6a2-446e-b2a9-90292218ae94/kube-rbac-proxy-thanos/0.log"
Sep 16 02:57:41.992843 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:57:41.992754    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_c29a4a9d-f6a2-446e-b2a9-90292218ae94/init-config-reloader/0.log"
Sep 16 02:57:42.020656 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:57:42.020599    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-68bcd7f987-fk5jv_529830e9-72ae-4032-899d-6af9f83105ea/prometheus-operator/0.log"
Sep 16 02:57:42.046072 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:57:42.045508    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-68bcd7f987-fk5jv_529830e9-72ae-4032-899d-6af9f83105ea/kube-rbac-proxy/0.log"
Sep 16 02:57:42.074900 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:57:42.074818    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-admission-webhook-5864cd99d7-z28pr_a97c117e-0ae7-48e7-ba8e-b100f553ab2b/prometheus-operator-admission-webhook/0.log"
Sep 16 02:57:42.184930 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:57:42.184852    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-79d4d88bf4-4nc9x_bd366e5d-6d4a-45b9-b259-9d5488a3b7a6/thanos-query/0.log"
Sep 16 02:57:42.205590 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:57:42.205538    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-79d4d88bf4-4nc9x_bd366e5d-6d4a-45b9-b259-9d5488a3b7a6/kube-rbac-proxy-web/0.log"
Sep 16 02:57:42.225983 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:57:42.225952    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-79d4d88bf4-4nc9x_bd366e5d-6d4a-45b9-b259-9d5488a3b7a6/kube-rbac-proxy/0.log"
Sep 16 02:57:42.252484 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:57:42.252401    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-79d4d88bf4-4nc9x_bd366e5d-6d4a-45b9-b259-9d5488a3b7a6/prom-label-proxy/0.log"
Sep 16 02:57:42.278443 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:57:42.278407    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-79d4d88bf4-4nc9x_bd366e5d-6d4a-45b9-b259-9d5488a3b7a6/kube-rbac-proxy-rules/0.log"
Sep 16 02:57:42.299407 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:57:42.299374    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-79d4d88bf4-4nc9x_bd366e5d-6d4a-45b9-b259-9d5488a3b7a6/kube-rbac-proxy-metrics/0.log"
Sep 16 02:57:45.056078 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:57:45.056040    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-fv4vf/perf-node-gather-daemonset-xdtw8"]
Sep 16 02:57:45.063732 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:57:45.063696    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-fv4vf/perf-node-gather-daemonset-xdtw8"
Sep 16 02:57:45.079173 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:57:45.079142    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-fv4vf/perf-node-gather-daemonset-xdtw8"]
Sep 16 02:57:45.178953 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:57:45.178917    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/08032da5-de60-41dd-b149-99714f80a182-sys\") pod \"perf-node-gather-daemonset-xdtw8\" (UID: \"08032da5-de60-41dd-b149-99714f80a182\") " pod="openshift-must-gather-fv4vf/perf-node-gather-daemonset-xdtw8"
Sep 16 02:57:45.179285 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:57:45.179260    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/08032da5-de60-41dd-b149-99714f80a182-proc\") pod \"perf-node-gather-daemonset-xdtw8\" (UID: \"08032da5-de60-41dd-b149-99714f80a182\") " pod="openshift-must-gather-fv4vf/perf-node-gather-daemonset-xdtw8"
Sep 16 02:57:45.179450 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:57:45.179424    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/08032da5-de60-41dd-b149-99714f80a182-lib-modules\") pod \"perf-node-gather-daemonset-xdtw8\" (UID: \"08032da5-de60-41dd-b149-99714f80a182\") " pod="openshift-must-gather-fv4vf/perf-node-gather-daemonset-xdtw8"
Sep 16 02:57:45.179541 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:57:45.179513    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/08032da5-de60-41dd-b149-99714f80a182-podres\") pod \"perf-node-gather-daemonset-xdtw8\" (UID: \"08032da5-de60-41dd-b149-99714f80a182\") " pod="openshift-must-gather-fv4vf/perf-node-gather-daemonset-xdtw8"
Sep 16 02:57:45.179622 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:57:45.179586    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kvx9t\" (UniqueName: \"kubernetes.io/projected/08032da5-de60-41dd-b149-99714f80a182-kube-api-access-kvx9t\") pod \"perf-node-gather-daemonset-xdtw8\" (UID: \"08032da5-de60-41dd-b149-99714f80a182\") " pod="openshift-must-gather-fv4vf/perf-node-gather-daemonset-xdtw8"
Sep 16 02:57:45.281286 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:57:45.281235    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/08032da5-de60-41dd-b149-99714f80a182-podres\") pod \"perf-node-gather-daemonset-xdtw8\" (UID: \"08032da5-de60-41dd-b149-99714f80a182\") " pod="openshift-must-gather-fv4vf/perf-node-gather-daemonset-xdtw8"
Sep 16 02:57:45.281490 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:57:45.281313    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kvx9t\" (UniqueName: \"kubernetes.io/projected/08032da5-de60-41dd-b149-99714f80a182-kube-api-access-kvx9t\") pod \"perf-node-gather-daemonset-xdtw8\" (UID: \"08032da5-de60-41dd-b149-99714f80a182\") " pod="openshift-must-gather-fv4vf/perf-node-gather-daemonset-xdtw8"
Sep 16 02:57:45.281490 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:57:45.281383    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/08032da5-de60-41dd-b149-99714f80a182-sys\") pod \"perf-node-gather-daemonset-xdtw8\" (UID: \"08032da5-de60-41dd-b149-99714f80a182\") " pod="openshift-must-gather-fv4vf/perf-node-gather-daemonset-xdtw8"
Sep 16 02:57:45.281490 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:57:45.281428    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/08032da5-de60-41dd-b149-99714f80a182-proc\") pod \"perf-node-gather-daemonset-xdtw8\" (UID: \"08032da5-de60-41dd-b149-99714f80a182\") " pod="openshift-must-gather-fv4vf/perf-node-gather-daemonset-xdtw8"
Sep 16 02:57:45.281490 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:57:45.281469    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/08032da5-de60-41dd-b149-99714f80a182-lib-modules\") pod \"perf-node-gather-daemonset-xdtw8\" (UID: \"08032da5-de60-41dd-b149-99714f80a182\") " pod="openshift-must-gather-fv4vf/perf-node-gather-daemonset-xdtw8"
Sep 16 02:57:45.281754 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:57:45.281711    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/08032da5-de60-41dd-b149-99714f80a182-lib-modules\") pod \"perf-node-gather-daemonset-xdtw8\" (UID: \"08032da5-de60-41dd-b149-99714f80a182\") " pod="openshift-must-gather-fv4vf/perf-node-gather-daemonset-xdtw8"
Sep 16 02:57:45.281852 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:57:45.281827    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/08032da5-de60-41dd-b149-99714f80a182-podres\") pod \"perf-node-gather-daemonset-xdtw8\" (UID: \"08032da5-de60-41dd-b149-99714f80a182\") " pod="openshift-must-gather-fv4vf/perf-node-gather-daemonset-xdtw8"
Sep 16 02:57:45.282279 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:57:45.282245    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/08032da5-de60-41dd-b149-99714f80a182-sys\") pod \"perf-node-gather-daemonset-xdtw8\" (UID: \"08032da5-de60-41dd-b149-99714f80a182\") " pod="openshift-must-gather-fv4vf/perf-node-gather-daemonset-xdtw8"
Sep 16 02:57:45.282279 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:57:45.282263    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/08032da5-de60-41dd-b149-99714f80a182-proc\") pod \"perf-node-gather-daemonset-xdtw8\" (UID: \"08032da5-de60-41dd-b149-99714f80a182\") " pod="openshift-must-gather-fv4vf/perf-node-gather-daemonset-xdtw8"
Sep 16 02:57:45.290167 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:57:45.290131    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kvx9t\" (UniqueName: \"kubernetes.io/projected/08032da5-de60-41dd-b149-99714f80a182-kube-api-access-kvx9t\") pod \"perf-node-gather-daemonset-xdtw8\" (UID: \"08032da5-de60-41dd-b149-99714f80a182\") " pod="openshift-must-gather-fv4vf/perf-node-gather-daemonset-xdtw8"
Sep 16 02:57:45.376800 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:57:45.376511    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-fv4vf/perf-node-gather-daemonset-xdtw8"
Sep 16 02:57:45.681815 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:57:45.680531    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-fv4vf/perf-node-gather-daemonset-xdtw8"]
Sep 16 02:57:45.683948 ip-10-0-128-111 kubenswrapper[2583]: W0916 02:57:45.683910    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod08032da5_de60_41dd_b149_99714f80a182.slice/crio-9ac4f9fd9a21e13eddc288a240e8d82ace39d97e598fa0a13fa8b0f9ba87689f WatchSource:0}: Error finding container 9ac4f9fd9a21e13eddc288a240e8d82ace39d97e598fa0a13fa8b0f9ba87689f: Status 404 returned error can't find the container with id 9ac4f9fd9a21e13eddc288a240e8d82ace39d97e598fa0a13fa8b0f9ba87689f
Sep 16 02:57:46.670515 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:57:46.670481    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-fv4vf/perf-node-gather-daemonset-xdtw8" event={"ID":"08032da5-de60-41dd-b149-99714f80a182","Type":"ContainerStarted","Data":"ab79bfb64bb93cbe6a91ed7a387a978837f9929fd39960a0d129ba7f61ac73ce"}
Sep 16 02:57:46.671199 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:57:46.670532    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-fv4vf/perf-node-gather-daemonset-xdtw8" event={"ID":"08032da5-de60-41dd-b149-99714f80a182","Type":"ContainerStarted","Data":"9ac4f9fd9a21e13eddc288a240e8d82ace39d97e598fa0a13fa8b0f9ba87689f"}
Sep 16 02:57:46.671199 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:57:46.670579    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-fv4vf/perf-node-gather-daemonset-xdtw8"
Sep 16 02:57:46.687020 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:57:46.686954    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-fv4vf/perf-node-gather-daemonset-xdtw8" podStartSLOduration=1.6869298910000001 podStartE2EDuration="1.686929891s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 02:57:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 02:57:46.684420299 +0000 UTC m=+2345.152209082" watchObservedRunningTime="2026-09-16 02:57:46.686929891 +0000 UTC m=+2345.154718674"
Sep 16 02:57:46.851613 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:57:46.851582    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-v2h5n_216004c9-119c-415e-b79e-634e7382c0ea/dns/0.log"
Sep 16 02:57:46.869701 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:57:46.869671    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-v2h5n_216004c9-119c-415e-b79e-634e7382c0ea/kube-rbac-proxy/0.log"
Sep 16 02:57:46.915274 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:57:46.915245    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-kf6w7_2c5d542f-5718-4aeb-8bf5-680a69ddecbc/dns-node-resolver/0.log"
Sep 16 02:57:47.497206 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:57:47.497172    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-5nwmf_82d1b4b2-7910-42f8-b2f3-0d675dc2c530/node-ca/0.log"
Sep 16 02:57:48.821951 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:57:48.821903    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-6f86d6bc5-hqmmj_4d013d76-d01d-4ac8-9baf-7809fa1c9ede/kube-auth-proxy/0.log"
Sep 16 02:57:48.849572 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:57:48.849522    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5ccb968c66-xjmsh_53ca0d14-d5b8-422c-a044-ab3471e584c9/istio-proxy/0.log"
Sep 16 02:57:48.853933 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:57:48.853902    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5ccb968c66-xjmsh_53ca0d14-d5b8-422c-a044-ab3471e584c9/istio-proxy/1.log"
Sep 16 02:57:49.989818 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:57:49.989791    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-l8hmx_c66d34de-0500-4a6a-9fd7-056ecb784bf7/serve-healthcheck-canary/0.log"
Sep 16 02:57:50.568728 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:57:50.568695    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-ntjs4_93090997-c69e-4547-bf00-2e6650ac00eb/kube-rbac-proxy/0.log"
Sep 16 02:57:50.588252 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:57:50.588211    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-ntjs4_93090997-c69e-4547-bf00-2e6650ac00eb/exporter/0.log"
Sep 16 02:57:50.609245 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:57:50.609218    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-ntjs4_93090997-c69e-4547-bf00-2e6650ac00eb/extractor/0.log"
Sep 16 02:57:52.687652 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:57:52.687621    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-fv4vf/perf-node-gather-daemonset-xdtw8"
Sep 16 02:57:52.747362 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:57:52.747322    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-d6f4b65b8-clzwd_fc012f8a-f2bc-4ac6-97ee-1a1604b82c9c/manager/0.log"
Sep 16 02:57:53.343905 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:57:53.343858    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-599b75f66d-g4c5v_7849eb53-ff7d-4c8e-82a2-982f21b80577/manager/0.log"
Sep 16 02:57:53.363658 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:57:53.363631    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-599b75f66d-g4c5v_7849eb53-ff7d-4c8e-82a2-982f21b80577/copy-manifests/0.log"
Sep 16 02:57:54.992932 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:57:54.992902    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_lws-controller-manager-54ff6bb656-f2fpn_28d69534-e2e0-4bda-9eea-f5ff6eb7634a/manager/0.log"
Sep 16 02:57:55.658082 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:57:55.658037    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-6f86d6bc5-hqmmj_4d013d76-d01d-4ac8-9baf-7809fa1c9ede/kube-auth-proxy/0.log"
Sep 16 02:57:55.679166 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:57:55.679136    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5ccb968c66-xjmsh_53ca0d14-d5b8-422c-a044-ab3471e584c9/istio-proxy/0.log"
Sep 16 02:57:55.684334 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:57:55.684307    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5ccb968c66-xjmsh_53ca0d14-d5b8-422c-a044-ab3471e584c9/istio-proxy/1.log"
Sep 16 02:57:57.607633 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:57:57.607600    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-6f86d6bc5-hqmmj_4d013d76-d01d-4ac8-9baf-7809fa1c9ede/kube-auth-proxy/0.log"
Sep 16 02:57:57.628097 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:57:57.628057    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5ccb968c66-xjmsh_53ca0d14-d5b8-422c-a044-ab3471e584c9/istio-proxy/0.log"
Sep 16 02:57:57.633459 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:57:57.633431    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5ccb968c66-xjmsh_53ca0d14-d5b8-422c-a044-ab3471e584c9/istio-proxy/1.log"
Sep 16 02:58:01.754631 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:58:01.754599    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-7nk9m_e34369f4-88ba-4859-ad03-80219c62e272/kube-multus/0.log"
Sep 16 02:58:02.070301 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:58:02.070219    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-ll7tp_3a1ad832-d1ab-4676-8eee-31d2eccce648/kube-multus-additional-cni-plugins/0.log"
Sep 16 02:58:02.089239 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:58:02.089207    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-ll7tp_3a1ad832-d1ab-4676-8eee-31d2eccce648/egress-router-binary-copy/0.log"
Sep 16 02:58:02.108054 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:58:02.108021    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-ll7tp_3a1ad832-d1ab-4676-8eee-31d2eccce648/cni-plugins/0.log"
Sep 16 02:58:02.126888 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:58:02.126856    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-ll7tp_3a1ad832-d1ab-4676-8eee-31d2eccce648/bond-cni-plugin/0.log"
Sep 16 02:58:02.145613 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:58:02.145581    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-ll7tp_3a1ad832-d1ab-4676-8eee-31d2eccce648/routeoverride-cni/0.log"
Sep 16 02:58:02.164243 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:58:02.164212    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-ll7tp_3a1ad832-d1ab-4676-8eee-31d2eccce648/whereabouts-cni-bincopy/0.log"
Sep 16 02:58:02.183957 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:58:02.183920    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-ll7tp_3a1ad832-d1ab-4676-8eee-31d2eccce648/whereabouts-cni/0.log"
Sep 16 02:58:02.268517 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:58:02.268482    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-bdwgr_67a44fbe-95d7-4808-bcff-f7d214c6d82c/network-metrics-daemon/0.log"
Sep 16 02:58:02.285568 ip-10-0-128-111 kubenswrapper[2583]: I0916 02:58:02.285521    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-bdwgr_67a44fbe-95d7-4808-bcff-f7d214c6d82c/kube-rbac-proxy/0.log"